research rigour in qualitative research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Am J Pharm Educ
• v.84(1); 2020 Jan

A Review of the Quality Indicators of Rigor in Qualitative Research

Jessica l. johnson.

a William Carey University School of Pharmacy, Biloxi, Mississippi

Donna Adkins

Sheila chauvin.

b Louisiana State University, School of Medicine, New Orleans, Louisiana

Attributes of rigor and quality and suggested best practices for qualitative research design as they relate to the steps of designing, conducting, and reporting qualitative research in health professions educational scholarship are presented. A research question must be clear and focused and supported by a strong conceptual framework, both of which contribute to the selection of appropriate research methods that enhance trustworthiness and minimize researcher bias inherent in qualitative methodologies. Qualitative data collection and analyses are often modified through an iterative approach to answering the research question. Researcher reflexivity, essentially a researcher’s insight into their own biases and rationale for decision-making as the study progresses, is critical to rigor. This article reviews common standards of rigor, quality scholarship criteria, and best practices for qualitative research from design through dissemination.

INTRODUCTION

Within the past 20 years, qualitative research in health professions education has increased significantly, both in practice and publication. Today, one can pick up most any issue of a wide variety of health professions education journals and find at least one article that includes some type of qualitative research, whether a full study or the inclusion of a qualitative component within a quantitative or mixed methods study. Simultaneously, there have been recurrent calls for enhancing rigor and quality in qualitative research.

As members of the academic community, we share responsibility for ensuring rigor in qualitative research, whether as researchers who design and implement, manuscript reviewers who critique, colleagues who discuss and learn from each other, or scholarly teachers who draw upon results to enhance and innovate education. Therefore, the purpose of this article is to summarize standards of rigor and suggested best practices for designing, conducting, and reporting high-quality qualitative research. To begin, Denzin and Lincoln’s definition of qualitative research, a long-standing cornerstone in the field, provides a useful foundation for summarizing quality standards and best practices:

Qualitative research involves the studied use and collection of a variety of empirical materials – case study; personal experience; introspection; life story; interview; artifacts; cultural texts and productions; observational, historical, interactional, and visual texts – that describe the routine and problematic moments and meanings in individual lives. Accordingly, qualitative researchers deploy a wide range of interconnected interpretative practices, hoping always to get a better understanding of the subject matter at hand. It is understood, however, that each practice makes the world visible in a different way. Hence there is frequently a commitment to using more than one interpretative practice in any study. 1

In recent years, multiple publications have synthesized quality criteria and recommendations for use by researchers and peer reviewers alike, often in the form of checklists. 2-6 Some authors have raised concerns about the use of such checklists and adherence to strict, universal criteria because they do not afford sufficient flexibility to accommodate the diverse approaches and multiple interpretive practices often represented in qualitative studies. 7-11 They argue that a strict focus on using checklists of specific technical criteria may stifle the diversity and multiplicity of practices that are so much a part of achieving quality and rigor within the qualitative paradigm. As an alternative, some of these authors have published best practice guidelines for use by researchers and peer reviewers to achieve and assess methodological rigor and research quality. 12,13

Some journals within the field of health professions education have also established best practice guidance, as opposed to strict criteria or a checklist, for qualitative research. These have been disseminated as guiding questions or evaluation categories. In 2015, Academic Medicine produced an expanded second edition of a researcher/author manual that includes specific criteria with extensive explanations and examples. 14 Still others have disseminated best practice guidelines through a series of methodological articles within journal publications. 2

In this article, attributes of rigor and quality and suggested best practices are presented as they relate to the steps of designing, conducting, and reporting qualitative research in a step-wise approach.

BEST PRACTICES: STEP-WISE APPROACH

Step 1: identifying a research topic.

Identifying and developing a research topic is comprised of two major tasks: formulating a research question, and developing a conceptual framework to support the study. Formulating a research question is often stimulated by real-life observations, experiences, or events in the researcher’s local setting that reflect a perplexing problem begging for systematic inquiry. The research question begins as a problem statement or set of propositions that describe the relationship among certain concepts, behaviors, or experiences. Agee 15 and others 16,17 note that initial questions are usually too broad in focus and too vague regarding the specific context of the study to be answerable and researchable. Creswell reminds us that initial qualitative research questions guide inquiry, but they often change as the author’s understanding of the issue develops throughout the study. 16 Developing and refining a primary research question focused on both the phenomena of interest and the context in which it is situated is essential to research rigor and quality.

Glassick, Huber, and Maeroff identified six criteria applicable to assessing the quality of scholarship. 18,19 Now commonly referred to as the Glassick Criteria ( Table 1 ), these critical attributes outline the essential elements of any scholarly approach and serve as a general research framework for developing research questions and designing studies. The first two criteria, clear purpose and adequate preparation, are directly related to formulating effective research questions and a strong conceptual framework.

Glassick’s Criteria for Assessing the Quality of Scholarship of a Research Study 18

Generating and refining a qualitative research question requires thorough, systematic, and iterative review of the literature, and the use of those results to establish a clear context and foundation for the question and study design. Using an iterative approach, relevant concepts, principles, theories or models, and prior evidence are identified to establish what is known, and more importantly, what is not known. The iterative process contributes to forming a better research question, the criteria for which can be abbreviated by the acronym FINER, ie, f easible, i nteresting, n ovel, e thical, and r elevant, that is answerable and researchable, in terms of research focus, context specificity, and the availability of time, logistics, and resources to carry out the study. Developing a FINER research question is critical to study rigor and quality and should not be rushed, as all other aspects of research design depend on the focus and clarity of the research question(s) guiding the study. 15 Agee provides clear and worthwhile additional guidance for developing qualitative research questions. 15

Reflexivity, the idea that a researcher’s preconceptions and biases can influence decisions and actions throughout qualitative research activities, is a critical aspect of rigor even at the earliest stages of the study. A researcher’s background, beliefs, and experiences may affect any aspect of the research from choosing which specific question to investigate through determining how to present the results. Therefore, even at this early stage, the potential effect of researcher bias and any ethical considerations should be acknowledged and addressed. That is, how will the question’s influence on study design affect participants’ lives, position the researcher in relationship with others, or require specific methods for addressing potential areas of research bias and ethical considerations?

A conceptual framework is then actively constructed to provide a logical and convincing argument for the research. The framework defines and justifies the research question, the methodology selected to answer that question, and the perspectives from which interpretation of results and conclusions will be made. 5,6,20 Developing a well-integrated conceptual framework is essential to establishing a research topic based upon a thorough and integrated review of relevant literature (addressing Glassick criteria #1 and #2: clear purpose and adequate preparation). Key concepts, principles, assumptions, best practices, and theories are identified, defined, and integrated in ways that clearly demonstrate the problem statement and corresponding research question are answerable, researchable, and important to advancing thinking and practice.

Ringsted, Hodges, and Sherpbier describe three essential parts to an effective conceptual framework: theories and/or concepts and principles relevant to the phenomenon of interest; what is known and unknown from prior work, observations, and examples; and the researcher’s observations, ideas, and suppositions regarding the research problem statement and question. 21 Lingard describes four types of unknowns to pursue during literature review: what no one knows; what is not yet well understood; what controversy or conflicting results, understandings, or perspectives exist; and what are unproven assumptions. 22 In qualitative research, these unknowns are critical to achieving a well-developed conceptual framework and a corresponding rigorous study design.

Recent contributions from Ravitch and colleagues present best practices in developing frameworks for conceptual and methodological coherence within a study design, regardless of the research approach. 23,24 Their recommendations and arguments are highly relevant to qualitative research. Figure 1 reflects the primary components of a conceptual framework adapted from Ravitch and Carl 23 and how all components contribute to decisions regarding research design, implementation, and applications of results to future thinking, study, and practice. Notice that each element of the framework interacts with and influences other elements in a dynamic and interactive process from the beginning to the end of a research project. The intersecting bidirectional arrows represent direct relationships between elements as they relate to specific aspects of a qualitative research study.

Adaptation of Ravitch and Carl’s Components of a Conceptual Framework 23

Maxwell also provides useful guidance for developing an effective conceptual framework specific to the qualitative research paradigm. 17 The 2015 second edition of the Review Criteria for Research Manuscripts 14 and work by Ravitch and colleagues 23,24 provide specific guidance for applying the conceptual framework to each stage of the research process to enhance rigor and quality. Quality criteria for assessing a study’s problem statement, conceptual framework, and research question include the following: introduction builds a logical case and provides context for the problem statement; problem statement is clear and well-articulated; conceptual framework is explicit and justified; research purpose and/or question is clearly stated; and constructs being investigated are clearly identified and presented. 14,24,25 As best practice guidelines, these criteria facilitate quality and rigor while providing sufficient flexibility in how each is achieved and demonstrated.

While a conceptual framework is important to rigor in qualitative research, Huberman and Miles caution qualitative researchers about developing and using a framework to the extent that it influences qualitative design deductively because this would violate the very principles of induction that define the qualitative research paradigm. 25 Our profession’s recent emphasis on a holistic admissions process for pharmacy students provides a reasonable example of inductive and deductive reasoning and their respective applications in qualitative and quantitative research studies. Principles of inductive reasoning are applied when a qualitative research study examines a representative group of competent pharmacy professionals to generate a theory about essential cognitive and affective skills for patient-centered care. Deductive reasoning could then be applied to design a hypothesis-driven prospective study that compares the outcomes of two cohorts of students, one group admitted using traditional criteria and one admitted based on a holistic admissions process revised to value the affective skills of applicants. Essentially, the qualitative researcher must carefully generate a conceptual framework that guides the research question and study design without allowing the conceptual framework to become so rigid as to dictate a testable hypothesis, which is the founding principle of deductive reasoning. 26

Step 2: Qualitative Study Design

The development of a strong conceptual framework facilitates selection of appropriate study methods to minimize the bias inherent in qualitative studies and help readers to trust the research and the researcher (see Glassick criteria #3 in Table 1 ). Although researchers can employ great flexibility in the selection of study methods, inclusion of best practice methods for assuring the rigor and trustworthiness of results is critical to study design. Lincoln and Guba outline four criteria for establishing the overall trustworthiness of qualitative research results: credibility, the researcher ensures and imparts to the reader supporting evidence that the results accurately represent what was studied; transferability, the researcher provides detailed contextual information such that readers can determine whether the results are applicable to their or other situations; dependability, the researcher describes the study process in sufficient detail that the work could be repeated; confirmability, the researcher ensures and communicates to the reader that the results are based on and reflective of the information gathered from the participants and not the interpretations or bias of the researcher. 27

Specific best practice methods used in the sampling and data collection processes to increase the rigor and trustworthiness of qualitative research include: clear rationale for sampling design decisions, determination of data saturation, ethics in research design, member checking, prolonged engagement with and persistent observation of study participants, and triangulation of data sources. 28

Qualitative research is focused on making sense of lived, observed phenomenon in a specific context with specifically selected individuals, rather than attempting to generalize from sample to population. Therefore, sampling design in qualitative research is not random but defined purposively to include the most appropriate participants in the most appropriate context for answering the research question. Qualitative researchers recognize that certain participants are more likely to be “rich” with data or insight than others, and therefore, more relevant and useful in achieving the research purpose and answering the question at hand. The conceptual framework contributes directly to determining sample definitions, size, and recruitment of participants. A typical best practice is purposive sampling methods, and when appropriate, convenience sampling may be justified. 29

Purposive sampling reflects intentional selection of research participants to optimize data sources for answering the research question. For example, the research question may be best answered by persons who have particular experience (critical case sampling) or certain expertise (key informant sampling). Similarly, additional participants may be referred for participation by active participants (snowball sampling) or may be selected to represent either similar or opposing viewpoints (confirming or disconfirming samples). Again, the process of developing and using a strong conceptual framework to guide and justify methodological decisions, in this case defining and establishing the study sample, is critical to rigor and quality. 30 Convenience sampling, using the most accessible research participants, is the least rigorous approach to defining a study sample and may result in low accuracy, poor representativeness, low credibility, and lack of transferability of study results.

Qualitative studies typically reflect designs in which data collection and analysis are done concurrently, with results of ongoing analysis informing continuing data collection. Determination of a final sample size is largely based on having sufficient opportunity to collect relevant data until new information is no longer emerging from data collection, new coding is not feasible, and/or no new themes are emerging; that is, reaching data saturation , a common standard of rigor for data collection in qualitative studies . Thus, accurately predicting a sample size during the planning phases of qualitative research can be challenging. 30 Care should be taken that sufficient quantity (think thick description) and quality (think rich description) of data have been collected prior to concluding that data saturation has been achieved. A poor decision regarding sample size is a direct consequence of sampling strategy and quality of data generated, which leaves the researcher unable to fully answer the research question in sufficient depth. 30

Though data saturation is probably the most common terminology used to describe the achievement of sufficient sample size, it does not apply to all study designs. For example, one could argue that in some approaches to qualitative research, data collection could continue infinitely if the event continues infinitely. In education, we often anecdotally observe variations in the personality and structure of a class of students, and as generations of students continue to evolve with time, so too would the data generated from observing each successive class. In such situations, data saturation might never be achieved. Conversely, the number of participants available for inclusion in a sample may be small and some risk of not reaching data saturation may be unavoidable. Thus, the idea of fully achieving data saturation may be unrealistic when applied to some populations or research questions. In other instances, attrition and factors related to time and resources may contribute to not reaching data saturation within the limits of the study. By being transparent in the process and reporting of results when saturation may not have been possible, the resulting data may still contribute to the field and to further inquiry. Replication of the study using other samples and conducting additional types of follow-up studies are other options for better understanding the research phenomenon at hand. 31

In addition to defining the sample and selecting participants, other considerations related to sampling bias may impact the quantity and quality of data generated and therefore the quality of the study result. These include: methods of recruiting, procedures for informed consent, timing of the interviews in relation to experience or emotion, procedures for ensuring participant anonymity/confidentiality, interview setting, and methods of recording/transcribing the data. Any of these factors could potentially change the nature of the relationship between the researcher and the study participants and influence the trustworthiness of data collected or the study result. Thus, ongoing application of previously mentioned researcher reflexivity is critical to the rigor of the study and quality of sampling. 29,30

Common qualitative data collection methods used in health professions education include interview, direct observation methods, and textual/document analysis. Given the unique and often highly sensitive nature of data being collected by the researcher, trustworthiness is an essential component of the researcher-participant relationship. Ethical conduct refers to how moral principles and values are part of the research process. Participants’ perceptions of ethical conduct are fundamental to a relationship likely to generate high quality data. During each step of the research process, care must be taken to protect the confidentiality of participants and shield them from harm relating to issues of respect and dignity. Researchers must be respectful of the participants’ contributions and quotes, and results must be reported truthfully and honestly. 8

Interview methods range from highly structured to increase dependability or completely open-ended to allow for interviewers to clarify a participant’s response for increased credibility and confirmability. Regardless, interview protocols and structure are often modified or refined, based on concurrent data collection and analysis processes to support or refute preliminary interpretations and refine focus and continuing inquiry. Researcher reflexivity, or acknowledgement of researcher bias, is absolutely critical to the credibility and trustworthiness of data collection and analysis in such study designs. 32

Interviews should be recorded and transcribed verbatim prior to coding and analysis. 28 Member checking, a common standard of rigor, is a practice to increase study credibility and confirmability that involves asking a research subject to verify the transcription of an interview. 1,16,28 The research subject is asked to verify the completeness and accuracy of an interview transcript to ensure the transcript truthfully reflects the meaning and intent of the subject’s contribution.

Prolonged engagement involves the researcher gaining familiarity and understanding of the culture and context surrounding the persons or situations being studied. This strategy supports reflexivity, allowing the researcher to determine how they themselves may be a source of bias during the data collection process by altering the nature of how individuals behave or interact with others in the presence of the researcher. Facial expressions, spoken language, body language, style of dress, age, race, gender, social status, culture, and the researcher’s relationship with the participants may potentially influence either participants’ responses or how the researcher interprets those responses. 33 “Fitting in” by demonstrating an appreciation and understanding of the cultural norms of the population being studied potentially allows the researcher to obtain more open and honest responses from participants. However, if the research participants or topic are too familiar or personal, this may also influence data collection or analysis and interpretation of the results. 33 The possible applications of this section to faculty research with student participants in the context of pharmacy education are obvious, and researcher reflexivity is critical to rigor.

Some researchers using observational methods adopt a strategy of direct field observation, while others play partial or full participant roles in the activity being observed. In both observation scenarios, it is impossible to separate the researcher from the environment, and researcher reflexivity is essential. The pros and cons of observation approach, relative to the research question and study purpose, should be evaluated by the researcher, and the justification for the observational strategy selected should be made clear. 34 Regardless of the researcher’s degree of visibility to the study participants, persistent observation of the targeted sample is critical to the confirmability standard and to achieving data saturation. That is, study conclusions must be clearly grounded in persistent phenomena witnessed during the study, rather than on a fluke event. 28

Researchers acknowledge that observational methodologies are limited by the reality that the researcher carries a bias in determining what is observed, what is recorded, how it is recorded, and how it is transcribed for analysis. A study’s conceptual framework is critical to achieving rigor and quality and provides guidance in developing predetermined notions or plans for what to observe, how to record, and how to minimize the influence of potential bias. 34 Researcher notes should be recorded as soon as possible after the observation event to optimize accuracy. The more detailed and complete the notes, the more accurate and useful they can be in data analysis or in auditing processes for enhancing rigor in the interpretation phase of the study. 34

Triangulation is among the common standards of rigor applied within the qualitative research paradigm. Data triangulation is used to identify convergence of data obtained through multiple data sources and methods (eg, observation field notes and interview transcripts) to avoid or minimize error or bias and optimize accuracy in data collection and analysis processes. 33,35,36

Again, researcher practice in reflexivity throughout research processes is integral to rigor in study design and implementation. Researchers must demonstrate attention to appropriate methods and reflective critique, which are represented in both core elements of the conceptual framework ( Figure 1 ) and Glassick criteria ( Table 1 ). In so doing, the researcher will be well-prepared to justify sampling design and data collection decisions to manuscript reviewers and, ultimately, readers.

Step 3: Data Analysis

In many qualitative studies, data collection runs concurrently with data analysis. Specific standards of rigor are commonly used to ensure trustworthiness and integrity within the data analysis process, including use of computer software, peer review, audit trail, triangulation, and negative case analysis.

Management and analyses of qualitative data from written text, observational field notes, and interview transcriptions may be accomplished using manual methods or the assistance of computer software applications for coding and analysis. When managing very large data sets or complex study designs, computer software can be very helpful to assist researchers in coding, sorting, organizing, and weighting data elements. Software applications can facilitate ease in calculating semi-quantitative descriptive statistics, such as counts of specific events, that can be used as evidence that the researcher’s analysis is based on a representative majority of data collected ( inclusivism ) rather than focusing on selected rarities ( anecdotalism ). Using software to code data can also make it easier to identify deviant cases, detect coding errors, and estimate interrater reliability among multiple coders. 37 While such software helps to manage data, the actual analyses and interpretation still reside with the researcher.

Peer review, another common standard of rigor, is a process by which researchers invite an independent third-party researcher to analyze a detailed audit trail maintained by the study author. The audit trail methodically describes the step-by-step processes and decision-making throughout the study. Review of this audit trail occurs prior to manuscript development and enhances study confirmability. 1,16 The peer reviewer offers a critique of the study methods and validation of the conclusions drawn by the author as a thorough check on researcher bias.

Triangulation also plays a role in data analysis, as the term can also be used to describe how multiple sources of data can be used to confirm or refute interpretation, assertions, themes, and study conclusions. If a theme or theory can be arrived at and validated using multiple sources of data, the result of the study has greater credibility and confirmability. 16,33,36 Should any competing or controversial theories emerge during data collection or analysis, it is vital to the credibility and trustworthiness of the study that the author disclose and explore those negative cases. Negative case analysis refers to actively seeking out and scrutinizing data that do not fit or support the researcher’s interpretation of the data. 16

The use of best practices applying to data collection and data analysis facilitates the full examination of data relative to the study purpose and research question and helps to prevent premature closure of the study. Rather than stopping at the initial identification of literal, first-level assertion statements and themes, authors must progress to interpreting how results relate to, revise, or expand the conceptual framework, or offer an improved theory or model for explaining the study phenomenon of interest. Closing the loop on data collection is critical and is achieved when thorough and valid analysis can be linked back to the conceptual framework, as addressed in the next section.

Step 4: Drawing Valid Conclusions

Lingard and Kennedy 38 succinctly state that the purpose of qualitative research is to deepen one’s understanding of specific perspectives, observations, experiences, or events evidenced through the behaviors or products of individuals and groups as they are situated in specific contexts or circumstances. Conclusions generated from study results should enhance the conceptual framework, or contribute to a new theory or model development, and are most often situated within the discussion and conclusion sections of a manuscript.

The discussion section should include interpretation of the results and recommendations for practice. Interpretations should go beyond first-level results or literal description of observed behaviors, patterns, and themes from analysis. The author’s challenge is to provide a complete and thorough examination and explanation of how specific results relate to each other, contribute to answering the research question, and achieve the primary purpose of the research endeavor. The discussion should “close the loop” by integrating study results and analysis with the original conceptual framework. The discussion section should also provide a parsimonious narrative or graphical explanation and interpretation of study results that enhances understanding of the targeted phenomena.

The conclusion section should provide an overall picture or synopsis of the study, including its important and unique contributions to the field from the perspective of both conceptual and practical significance. The conclusion should also include personal and theoretical perspectives and future directions for research. Together, the discussion and conclusion should include responses to the larger questions of the study’s contributions, such as: So what? Why do these results matter? What next?

The strength of conclusions is dependent upon the extent to which standards of rigor and best practices were demonstrated in design, data collection, data analysis, and interpretation, as described in previous sections of this article. 4,12,17,23,24 Quality and rigor expectations for drawing valid conclusions and generating new theories are reflected in the following essential features of rigor and quality, which include: “Close the loop” to clearly link research questions, study design, data collection and analysis, and interpretation of results. Reflect effective integration of the study results with the conceptual framework and explain results in ways that relate, support, elaborate, and/or challenge conclusions of prior scholarship. Descriptions of new or enhanced frameworks or models are clear and effectively grounded in the study results and conclusions. Practical or theoretical implications are effectively discussed, including guidance for future studies. Limitations and issues of reflexivity and ethics are clearly and explicitly described, including references to actions taken to address these areas. 3,4,12,14

Step 5: Reporting Research Results

Key to quality reporting of qualitative research results are clarity, organization, completeness, accuracy, and conciseness in communicating the results to the reader of the research manuscript. O’Brien and others 4 proposed a standardized framework specifically for reporting qualitative studies known as the Standards for Reporting Qualitative Research (SRQR, Table 2 ). This framework provides detailed explanations of what should be reported in each of 21 sections of a qualitative research manuscript. While the SRQR does not explicitly mention a conceptual framework, the descriptions and table footnote clarification for the introduction and problem statement reflect the essential elements and focus of a conceptual framework. Ultimately, readers of published work determine levels of credibility, trustworthiness, and the like. A manuscript reviewer, the first reader of a study report, has the responsibility and privilege of providing critique and guidance to authors regarding achievement of quality criteria, execution and reporting of standards of rigor, and the extent to which meaningful contributions to thinking and practice in the field are presented. 13,39

An Adaptation of the 21 Elements of O’Brien and Colleagues’ Standards for Reporting Qualitative Research (SRQR) 4

Authors must avoid language heavy with connotations or adjectives that insert the researcher’s opinion into the database or manuscript. 14,40 The researcher should be as neutral and objective as possible in interpreting data and in presenting results. Thick and rich descriptions, where robust descriptive language is used to provide sufficient contextual information, enable the reader to determine credibility, transferability, dependability, and confirmability .

The process of demonstrating the credibility of research is rooted in honest and transparent reporting of how biases and other possible confounders were identified and addressed throughout study processes. Such reporting, first described within the study’s conceptual framework, should be revisited in reporting the work. Confounders may include the researcher’s training and previous experiences, personal connections to the background theory, access to the study population, and funding sources. These elements and processes are best represented in Glassick’s criteria for effective presentation and reflective critique ( Table 1 , criteria 5 and 6). Transferability is communicated, in part, through description of sampling factors such as: geographical location of the study, number and characteristics of participants, and the timeframe of data collection and analysis. 40 Such descriptions also contribute to the credibility of the results and readers’ determination of transfer to their and other contexts. To ensure dependability, the research method must be reported in detail such that the reader can determine proper research practices have been followed and that future researchers can repeat the study. 40 The confirmability of the results is influenced by reducing or at a minimum explaining any researcher influence on the result by applying and meeting standards of rigor such as member checking, triangulation, and peer review. 29,33

In qualitative studies, the researcher is often the primary instrument for data collection. Any researcher biases not adequately addressed or errors in judgement can affect the quality of data and subsequent research results. 33 Thus, due to the creative interpretative and contextually bound nature of qualitative studies, the application of standards of rigor and adherence to systematic processes well-documented in an audit trail are essential. The application of rigor and quality criteria extend beyond the researcher and are also important to effective peer review processes within a study and for scholarly dissemination. The goal of rigor in qualitative research can be described as ensuring that the research design, method, and conclusions are explicit, public, replicable, open to critique, and free of bias. 41 Rigor in the research process and results are achieved when each element of study methodology is systematic and transparent through complete, methodical, and accurate reporting. 33 Beginning the study with a well-developed conceptual framework and active use of both researcher reflexivity and rigorous peer review during study implementation can drive both study rigor and quality.

As the number of published qualitative studies in health professions educational research increases, it is important for our community of health care educators to keep in mind the unique aspects of rigor in qualitative studies presented here. Qualitative researchers should select and apply any of the above referenced study methods and research practices, as appropriate to the research question, to achieve rigor and quality. As in any research paradigm, the goal of quality and rigor in qualitative research is to minimize the risk of bias and maximize the accuracy and credibility of research results. Rigor is best achieved through thoughtful and deliberate planning, diligent and ongoing application of researcher reflexivity, and honest communication between the researcher and the audience regarding the study and its results.

MS in Nursing (MSN)

A Guide To Qualitative Rigor In Research

Advances in technology have made quantitative data more accessible than ever before; but in the human-centric discipline of nursing, qualitative research still brings vital learnings to the health care industry. It is sometimes difficult to derive viable insights from qualitative research; but in the article below, the authors identify three criteria for developing acceptable qualitative studies.

Qualitative rigor in research explained

Qualitative rigor. It’s one of those terms you either understand or you don’t. And it seems that many of us fall into the latter of those two categories. From novices to experienced qualitative researchers, qualitative rigor is a concept that can be challenging. However, it also happens to be one of the most critical aspects of qualitative research, so it’s important that we all start getting to grips with what it means.

Rigor, in qualitative terms, is a way to establish trust or confidence in the findings of a research study. It allows the researcher to establish consistency in the methods used over time. It also provides an accurate representation of the population studied. As a nurse, you want to build your practice on the best evidence you can and to do so you need to have confidence in those research findings.

This article will look in more detail at the unique components of qualitative research in relation to qualitative rigor. These are: truth-value (credibility); applicability (transferability); consistency (dependability); and neutrality (confirmability).

Credibility

Credibility allows others to recognize the experiences contained within the study through the interpretation of participants’ experiences. In order to establish credibility, a researcher must review the individual transcripts, looking for similarities within and across all participants.

A study is considered credible when it presents an interpretation of an experience in such a way that people sharing that experience immediately recognize it. Examples of strategies used to establish credibility include:

• Reflexivity
• Member checking (aka informant feedback)
• Peer examination
• Peer debriefing
• Prolonged time spent with participants
• Using the participants’ words in the final report

Transferability

The ability to transfer research findings or methods from one group to another is called transferability in qualitative language, equivalent to external validity. One way of establishing transferability is to provide a dense description of the population studied by describing the demographics and geographic boundaries of the study.

Ways in which transferability can be applied by researchers include:

• Using the same data collection methods with different demographic groups or geographical locations
• Giving a range of experiences on which the reader can build interventions and understanding to decide whether the research is applicable to practice

Dependability

Related to reliability in quantitative terms, dependability occurs when another researcher can follow the decision trail used by the researcher. This trail is achieved by:

• Describing the specific purpose of the study
• Discussing how and why the participants were selected for the study
• Describing how the data was collected and how long collection lasted
• Explaining how the data was reduced or transformed for analysis
• Discussing the interpretation and presentation of the findings
• Explaining the techniques used to determine the credibility of the data

Strategies used to establish dependability include:

• Having peers participate in the analysis process
• Providing a detailed description of the research methods
• Conducting a step-by-step repetition of the study to identify similarities in results or to enhance findings

Confirmability

Confirmability occurs once credibility, transferability and dependability have been established. Qualitative research must be reflective, maintaining a sense of awareness and openness to the study and results. The researcher needs a self-critical attitude, taking into account how his or her preconceptions affect the research.

Techniques researchers use to achieve confirmability include:

• Taking notes regarding personal feelings, biases and insights immediately after an interview
• Following, rather than leading, the direction of interviews by asking for clarifications when needed

Reflective research produces new insights, which lead the reader to trust the credibility of the findings and applicability of the study

Become a Champion of Qualitative Rigor

Clinical Nurse Leaders, or CNLs, work with interdisciplinary teams to improve care for populations of patients. CNLs can impact quality and safety by assessing risks and utilizing research findings to develop quality improvement strategies and evidence-based solutions.

As a student in Queens University of Charlotte’s online Master of Science in Nursing program , you will solidify your skills in research and analysis allowing you to make informed, strategic decisions to drive measurable results for your patients.

Request more information to learn more about how this degree can improve your nursing practice, or call 866-313-2356.

Adapted from: Thomas, E. and Magilvy, J. K. (2011), Qualitative Rigor or Research Validity in Qualitative Research. Journal for Specialists in Pediatric Nursing, 16: 151–155. [WWW document]. URL  http://onlinelibrary.wiley.com/doi/10.1111/j.1744-6155.2011.00283.x  [accessed 2 July 2014]

Recommended Articles

Tips for nurse leaders to maintain moral courage amid ethical dilemmas, relationship between nursing leadership & patient outcomes, day in the life of a clinical nurse leader®, 8 leadership skills nurses need to be successful, the rise of medical errors in hospitals, get started.

• Search Menu
• Advance Articles
• Editor's Choice
• Author Guidelines
• Submission Site
• Open Access
• About Journal of Public Administration Research and Theory
• About the Public Management Research Association
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, where qualitative methods shine, forget qualitative versus quantitative, rigor: the point of departure, deductive, mixed, and hybrid qualitative methods.

• < Previous

A Reviewer’s Guide to Qualitative Rigor

• Article contents
• Figures & tables
• Supplementary Data

Branda Nowell, Kate Albrecht, A Reviewer’s Guide to Qualitative Rigor, Journal of Public Administration Research and Theory , Volume 29, Issue 2, April 2019, Pages 348–363, https://doi.org/10.1093/jopart/muy052

• Permissions Icon Permissions

Institutions are useful for advancing methodologies within disciplines. Through required coursework, doctoral students are indoctrinated into basic guidelines and frameworks that provide a common foundation for scholars to interact with one another. Lacking such forums in many of our doctoral granting institutions ( Stout 2013 ), the field of public management continues to struggle with an ambivalence toward qualitative approaches. Lack of shared understanding concerning basic tenets of qualitative methodology abounds. This article is intended for qualitative consumers, those not formally trained in qualitative methods but who serve as peer reviewers, content experts, and advisors in arenas where qualitative methods are encountered. Adopting a postpositivistic stance dominant in the field, we seek to offer a pragmatic perspective on qualitative methods with regards to some basic tenets of rigor appropriate (and inappropriate) for assessing the contribution of qualitative research. We argue that the first step in this effort is to stop conflating data type (qualitative versus quantitative) with inductive versus deductive modes of inquiry. Using deductive modes as the basis for comparison, we discuss both common, as well as, diverging criteria of quality and rigor for inductive modes of inquiry. We conclude with a discussion of rigor in emerging methods which utilize qualitative data but from within a deductive, mixed, or hybrid mode of inquiry.

The field of public management continues to have a rocky relationship with qualitative methods. Like most methods, qualitative research has both its champions and its critics in the field. However, it is our sense that the majority of the field sits somewhere a bit right of center, open to a discussion but still suspect of what to do with findings from any study consisting of a small unrepresentative sample and no standard error. Much of this stems from fundamental misunderstandings about what qualitative inquiry is, and is not, designed to do. The cost of this to our discipline is significant. In a recent review, Ospina and colleagues (2017) reported only 7.5% of the articles published in top PA journals over the past 5 years relied solely on qualitative methods. This is not particularly surprising as our doctoral training institutions allow graduates to remain largely uninformed about qualitative approaches ( Stout 2013 ). However, there are many questions germane to our discipline that are best suited to qualitative inquiry (for discussion, see Brower, Abolafia, and Carr 2000 ; Milward forthcoming , Ospina et al. 2017 ). In order to advance the contribution qualitative methods can make to the field, some foundational understanding about qualitative rigor is needed.

In embarking on this effort, we join an esteemed cadre of scholars who have grappled with the issue of qualitative rigor in public management (e.g., Brower et al. 2000 ; Dodge et al. 2005 ; Lowery and Evans 2004 ; Ospina et al. 2017 ). However, we seek a very specific audience. This is not an article written for the initiated qualitative scholar; we are not seeking to offer advancements in qualitative techniques or further the discourse on the precepts of qualitative inquiry. Nor is this an article particularly aimed at the edification of the novice qualitative scholar looking to embark upon qualitative inquiry for the first time; there are many excellent texts out there that deal with the issues contained in this article in a much more thorough manner. Rather, this article was conceptualized and written primarily for the qualitative consumer who, at present, represents the over-whelming majority in the field of public management.

As we are envisioning our intended audience, three general categories of consumers come to mind. First, this article is for the quantitatively trained peer reviewer who finds themselves asked to assess the quality and contribution of a qualitative study brought to them for review. These folks serve as the gatekeepers and a quality assurance mechanism critical to the advancement of the discipline. Second, this article is for the scholar reviewing the literature within a content domain populated by both qualitative and quantitative studies. If we want qualitative research to have a greater substantive impact on the discipline, we need to give non-qualitatively trained scholars the tools to assess the contribution of qualitative research within their own research paradigm. Otherwise, citations will inevitably trend into methodological silos. Finally, this article is written for the quantitatively trained professor who finds themselves on a committee trying to support a student pursuing a qualitative or mixed method dissertation. We have a beloved colleague who routinely asks students whether their dissertations are going to be empirical or qualitative. Her intent is not to be pejorative; she simply has no frame of reference for how to think about quotations as data.

A Brief Note on Epistemology

We recognize that the writing of this article requires the adoption of some normative stances linked to the philosophy of science; namely, an epistemological stance that is primarily postpositivist in nature. We have intentionally deviated from normative practice in qualitative scholarship in minimizing our discussion of epistemology (for further discussions, see Creswell 2018 ; Creswell and Miller 2000 ; Raadschelders 2011 ; Riccucci 2010 ). This is not because we do not appreciate the value and relevance of alternative epistemological stances for the field of public management. However, many methods associated with qualitative rigor can be applied across different epistemological stances, varying in intention and orientation rather than practical execution. 1 For example, Lincoln and Guba’s (1986) criteria of trustworthiness are useful. This is true regardless of whether you are utilizing those practices because you believe in the postpositivistic limitations of humans to fully comprehend social processes present in natural settings, or because you believe these social processes are co-constructed in an inseparable relationship between the researcher and the participant. In a similar way, reflexivity 2 is relevant to both the postpositivist as well as the interpretivists regardless of whether you embrace the inseparability between the knower and knowledge (constructivism) or just view humans as fallible in part because they cannot fully and objectively separate who they are from the questions they ask and the answers they find (postpositivism; Guba 1990 ).

In this paper, we seek to offer a pragmatic perspective on qualitative inquiry with a focus on how to conceptualize and assess quality and rigor within a postpositivistic framework; the dominant philosophical stance of most qualitative consumers within public management. We do this with the aim of widening the pathways through which qualitative studies might influence scholarship in public management. We recognize that such an endeavor may be highly controversial within some branches of the qualitative community which maintain a strong allegiance to advancing a constructivist philosophy of science (e.g., Carter and Little 2007 ; Rolfe 2006 ). However, we argue it is neither reasonable nor necessary for qualitative consumers to suspend their fundamental view of reality in order to appreciate and assess the contribution of qualitative work to the broader field. There is a rich history of the integration of qualitative research within postpositivism (e.g., Clark 1998 ; Glaser and Strauss 2017 ; Prasad 2015 ; Yin 2017 ), particularly in the organizational sciences ( Eisenhardt and Graebner 2007 ).

We do not foresee a reconciliation between constructivism and postpositivist philosophies occurring any time soon. However, we do see sizable opportunity for naturalistic, inductive qualitative inquiry to have a broader impact in the field of public management if we start from the perspective that both qualitative and quantitative methods are compatible and complementary buckets of tools within social science. Different tools are best suited for different jobs and there is almost as much variation within each bucket as there is between them. Regardless, the world is an increasingly complex place. As a discipline that routinely trudges off into some really messy domains of inquiry and holds itself accountable to informing practice as well as advancing theory ( Brooks 2002 ; Denhardt 2001 ; Gill and Meier 2000 ; Head 2010 ; Weber and Khademian 2008 ), we need every tool we can get.

To address building this toolbox for qualitative consumers, we present first an overview of critical domains of inquiry in the field of public management where we see qualitative methods as being particularly well suited to advancing scholarship. This review highlights some of the most cited and celebrated theories of our field that have been initially shaped or meaningfully re-imagined from qualitative approaches. Next, we argue for a reframing of the question of qualitative rigor, asserting the more productive distinction lies in differentiating inductive versus deductive modes of inquiry. Leveraging this perspective, we discuss both commonalities and points of departure in appropriate criteria of quality and rigor between deductive versus inductive models. Finally, we discuss issues of rigor in three emerging methods in public management that use qualitative data in deductive, mixed and hybrid models of inquiry.

If qualitative methods are viewed as a category of tools, it is relevant to next consider some of the functionality one maximizes through the use of such tools. Although this list is not exhaustive, it is intended to provide a general grounding into the types of situations where qualitative approaches are particularly well equipped to make a contribution to the field of public management.

Advancing New Theory and Discovering Nuance in Existing Theory

Quantitative hypothesis testing requires a priori theory. Arbitrarily searching for significant correlations between variables in a dataset without a theoretically grounded hypothesis to direct the analysis is infamously problematic for well-documented reasons ( Kuhn 1996 ; Steiner 1988 ). Theory is a combination of a premise as well as a well-explicated mechanism that explains the why behind the premise or proposition.

Cross-sectional quantitative designs can test the strength and nature of association between two or more constructs. Longitudinal quantitative designs can examine the patterning of association over time, and experimental designs can even narrow in on causality. These are powerful tools, but none are well equipped to discover the mechanisms by which these observed patterns are operating or identifying intervening factors that explain inconsistencies across cases. We use existing theory to infer the mechanism associated with an observed pattern but this is generally not an empirical exercise, it is a conceptual one. Further, it often requires the extrapolation of theoretical mechanisms conceptualized in one organizational context (e.g., private firms) to be applied in a completely different organizational context (e.g., public organizations). When the hypothesized association holds, we generally conclude that the same mechanisms are in operation in the same manner. How critically do we look at this assumption? What else might be going on? Qualitative methods offer tools specifically designed to empirically shed light on these questions.

Qualitative methods are particularly useful in the theory development process because they are able to provide detailed description of a phenomenon as it occurs in context. These methods do not require the scholar to guess in advance the most important factors and their relationship to each other. Mechanisms associated with the co-occurrence of two phenomena can be observed in real time or described by first hand informants who experienced it. For example, Feldman’s (e.g. Feldman 2000 ; Feldman and Pentland 2003 ) seminal work on the role of routines as sources of change and innovation in organizations was based on organizational ethnography. Some other classic examples of theory development in public management that began as qualitative research can be found in organizational culture and sense making case studies ( Schein 2003 ; Weick 1993 ). Toepler’s (2005) case study of a CEO in crisis, the phenomena of iron triangles ( Freeman 1965 ), and the social construction of target populations ( Schneider and Ingram 1993 ) are also illustrations of theoretical advances through qualitative inquiry. Additionally, a major contribution to theory of both formal and informal accountability in the public sector and multi-sector collaboration was a direct result of a grounded theory qualitative approach ( Romzek and Dubnick 1987 ; Romzek, LeRoux, and Blackmar 2012 ; Romzek et al. 2014 ). All of these examples leverage a qualitative researcher’s ability to harness an inductive approach that allows for the emergence of our understanding of the nature of phenomena from those organizations and people who experienced it.

Beyond advancing new theories, qualitative methods have a strong tradition of clarifying and expanding upon existing theory. Underpinning many public management research areas is the ever-present politics-administration dichotomy. Maynard-Moody and Kelly’s (1993) foundational piece used a phenomenological approach to present the views of public workers who must navigate their administrative and political responsibilities every day. Agency and stewardship theories have also been examined and further delineated using qualitative methods ( Schillemans 2013 ). Theories of goal-directed networks and managerial tensions around unity and diversity have been expanded through qualitative studies ( Saz-Carranza and Ospina 2010 ). Finally, the nature of public participation has been theorized and statistically tested, but along the way the notion of authentic engagement—described as “deep and continuous involvement…with the potential for all involved to have an effect on the situation” (p. 320) was introduced to clarify theories, in part as a result of King et al.’s (1998) qualitative study.

Developing New Constructs, Frameworks, and Typologies

Quantitative hypothesis testing and construct validation requires the conceptualization and suggested operationalization of a construct. The development or usage of a new measure is aptly treated with skepticism if it is not empirically and theoretically grounded. In this way, many variables that we quantitatively leverage could not exist without prior development through qualitative research. For example, a foundational idea, and the basis for subsequent quantitative considerations of the differences between managers and front-line workers, is rooted in Lipsky’s (1971) examination and discussion of the street-level bureaucrat. Drawing from case studies and interviews, Lipsky highlights the nature of front-line worker discretion and challenges public management scholars to include this important context in future research.

Public Service Motivation (PSM), public management’s very own and home-grown construct, was born from Perry and Wise’s (1990) discussion citing both cases and quotes from public servants. Their argument for PSM to be more fully operationalized and then measured is rooted in their content analysis. Although they do not explicitly state the qualitative nature of their article, their argument for, and legacy of PSM scale measures, is drawn directly from the words and actions of public servants themselves.

Defining Mechanisms Underlying Statistical Associations

Although some quantitative articles do include mechanisms in their discussion sections, many simply rehash results and what hypotheses were or were not supported. Indeed, quantitative research in public management gives considerable weight to well-documented statistical association, even when the causal mechanism is ambiguous. In this world, how then do mechanisms get clarified when an association is found? This is an area where qualitative researchers have been working with less recognition of the importance of their research striving to answer “how” and “why” questions. The literature mentioned here again is not an exhaustive list, but emblematic of some prime examples of how our field’s understanding of a statistical result has been given more texture and a much richer application to both theory and practice through qualitative methods.

In the area of government contracting, Dias and Maynard-Moody (2007) further examine past quantitative findings that turn on Transaction Cost Economics (TCE) ( Williamson 1981 ) by explicating how and why implementing competing contracting philosophies of agencies and service providers underpins the nature of the transaction itself. Another qualitative piece examining the deeper mechanisms behind TCE is Van Slyke’s (2003) discussion of the “mythology” of contracting. In his research, data from semi-structured interviews suggests competition is not a simple construct in testing TCE interactions between governments and service providers because of the nature of environmental constraints, actions by nonprofit organizations, networked relationships, and government-enacted barriers have important dynamics. Honig (2018) offers another apt example in a mixed method study in which he demonstrates how comparative case study designs can reveal insights about the role of the environment in moderating the relationship between managerial control and success that were not possible to capture through quantitative modeling.

We have observed many scholars get conceptually hung up on the numbers versus text dichotomy associated with qualitative versus quantitative traditions. Although it is true that qualitative methods generally involve the analysis of some form of text and quantitative methods always involve the analysis of numbers, this focus on data type is largely a distraction from the more important distinction of inductive versus deductive forms of inquiry ( Eisenhardt and Graebner 2007 ). Deductive approaches to inquiry start with a general premise or proposition and then investigate whether this premise holds within a specific sample intended to represent a broader population. Inductive approaches start with a specific case or set of cases of theoretical importance and seek to describe a phenomenon of interest within that case in such a manner as to draw rich insight into that phenomenon (for discussion, see Eisenhardt and Graebner 2007 ; McNabb 2014 ). Although there are a handful of qualitative and/or hybrid qualitative/quantitative methods intended for deductive inquiry (more on this below), the bulk of tools in the qualitative bucket are intended for inductive inquiry.

Overarching Principles of Quality Between Inductive and Deductive Inquiry

Before we get into differences, it is important to first consider similarities. Although inductive and deductive traditions of scholarship differ in many important respects, they also share some commonalities that form the mutual basis of considerations of quality in terms of assessing their contribution to the literature. In our exuberance to elaborate their differences, we can forget what these forms of inquiry can hold in common. We argue that inductive and deductive approaches share in common three core values that are foundational to the notion of quality scholarship in public management: 1) the importance of scholarship that advances theory, 2) the principle of inquiry-driven design, and 3) the criticality of gap-driven inquiry.

Relevance of Scholarship for Advancing Theory

In public management, our focus is to inform practice as well as advance theory ( Kettl 2000 ). As a result, we give the greatest currency to knowledge that has relevance beyond the boundaries of the specific case, individual, or instance. Thus, within our field, the degree to which findings can have relevance beyond the study case or sample is foundational to conceptualizations of quality regardless of inductive or deductive approach ( Dubnick 1999 ). Inductive scholarship, different from most deductive studies, allows for a plurality of truths and an equifinality of pathways to the same outcome ( Eisenhardt, Graebner, and Sonenshein 2016 ), but the same standards of quality still apply. In other words, in inductive approaches, one need not argue an observed finding is the only explanation for a given outcome observed in another space or time, but it must be a plausible explanation for a similar outcome given a similar set of circumstances ( Holland 1986 ; Lewis 1973 ).

As such, both inductive and deductive studies are in the same boat of trying to figure out the extent to which and ways in which their limited study has broader implications for the field. The criteria and processes used to establish this element of quality certainly differs, but the precept that findings must have relevance beyond the scope of the data analyzed is common to both qualitative and quantitative scholarship in the field of public management ( McNabb 2015 ).

Inquiry-Driven Design

Both inductive and deductive traditions are inquiry driven. This means that evaluating the quality of any design—qualitative or quantitative—is inseparable from understanding the research question the study is designed to address. It is possible to hammer a nail with a screwdriver, but it is not considered good practice as you are likely to just make a mess of it. In the same way, different research questions are more or less appropriate to different designs. Thus, while it is possible to attempt to describe the different ways in which people experience transformational leadership with an exploratory survey or use a series of focus groups to examine the relative prevalence of public service motivation among different groups, it is not a good practice as you are likely to just make a mess of it.

A common misconception is that inductive qualitative methods seek to ask and answer the same questions as quantitative methods, just using different types of data and standards of rigor. This is not the case. Inductive approaches are designed to pose and address fundamentally different kinds of questions that necessitate different types of data and criteria of rigor. However, methodological appropriateness ( Haverland and Yanow 2012 ), or using the right tool for the job, is a value common to both inductive and deductive traditions and a key element of quality for all public management scholarship.

Gap-Driven Inquiry

Both inductive and deductive traditions recognize that knowledge does not advance in isolation—it takes a community of scholars to build a body of knowledge ( Kuhn 1996 ; Gill and Meier 2000 ). The practice of positioning a research question in terms of its relevance within broader conversations that are taking place within the literature is mainstream to both traditions ( McNabb 2015 ). In the field of public management—as elsewhere—the greatest currency is given to studies that clearly identify and address a significant gap within the literature; we seek to investigate something overlooked, under-appreciated, or potentially misunderstood in our current understanding of a given phenomenon. The extent to which a study accomplishes such a contribution is a shared element of quality for both deductive and inductive traditions.

In the previous section, we have argued that inductive and deductive approaches in public management share a common foundation in conceptualizing the quality of inquiry. Specifically, we suggest quality can be conceptualized as inquiry that addresses a significant gap in the literature in a manner that advances our general understanding of a broader phenomenon through the use of a method appropriate to the nature of the research question. Rigor, then, can be conceptualized as the appropriate execution of that method. Put simply, if quality is the what, rigor for our purposes becomes the how. It is here that inductive and deductive traditions diverge in a significant way.

It is useful to start with the negative case. Two criteria appropriate for deductive research but NOT appropriate for inductive inquiry include:

1) Is there evidence that the causal factors, processes, nature, meaning, and/or significance of the phenomenon generalize to the broader population?

2) Are the findings able to be replicated in the sense that two researchers asking the same question would come to the same interpretation of the data?

These two criteria, held sacred as cornerstones of rigor in deductive inquiry, seem to cause the greatest amount of heartburn within the field of public management and its relationship to inductive qualitative inquiry. If it is not generalizable and it does not replicate, how is that possibly science? This results in on-going frustration among qualitative scholars as they attempt to respond to criticisms of their design by reviewers, colleagues, and advisors in terms of the lack of representative sampling and/or inter-rater reliability measures. This is rooted in some fundamental misunderstandings about what inductive inquiry is and what it seeks to accomplish.

Generalizability

In deductive methods, when there are more cases that conform to an a priori hypothesis than do not, relative to the standard error and controlling for all other factors in the model, we reject the null hypothesis that this pattern could have been observed merely by random chance. However, in every deductive sample, there can be numerous observations which do not conform to our models. These we vaguely disregard as “error.” When cases deviate substantially, we call them “outliers” and may remove them from consideration entirely. This is reasonable because the aim of deductive inquiry is to test the presence of an a priori relationship in the population based on a limited, representative sample ( Neuman and Robson 2014 ). Associations deal with probabilities and likelihoods; not all cases must conform to a pattern to conclude that an association exists as long as the sample is reasonably representative and sufficient to detect differences ( Wasserman 2013 ).

Inductive research is attempting to do something quite different. The sample of an inductive study is never purely random nor convenient. Instead, each case or participant should be purposively selected because they represent a theoretically interesting exemplar of, or key informant about, a phenomenon of interest ( Patton 2014 ). In other words, by nature of being selected for inclusion in an inductive study, the scholar is making the argument that we should care about understanding the experience of this person(s) or the events of this case. Whether a pattern discerned in an inductive study is common in the general population is not the question an inductive scholar is seeking to answer. In fact, the case may have been selected specifically because it represents something rare or unusual. Rather, they are seeking to use a systematic method to interpret and represent, in rich detail, what is true for a particular set of individual(s) and/or cases, identifying themes and patterns across cases that add insight into the phenomenon of interest. Cases with divergent patterns or informants with contradictory experiences are not ignored or discounted as measurement error or outliers. Rather, the inductive scholar seeks to understand the factors and mechanisms that explain these points of divergence ( Eisenhardt et al. 2016 ).

Although the inductive scholar does not empirically test the extent to which an association or experience is common in the general population, this does not mean that inductive findings are not intended to have relevance for advancing general theory and practice. If done well, an inductive study should provide a detailed, contextualized, and empirically grounded interpretation of what was true in one or more cases of interest. Just as one experience in one setting should never be assumed to dictate what one might experience in another setting, it would likewise be absurd to assume prior experience is totally irrelevant if a similar set of conditions are present. In this way, qualitative inductive scholarship seeks to systematically describe and interpret what is occurring in a finite set of cases in sufficient detail as to lend insight into what might be going on in cases like these . Discerning the quantitative prevalence of these key patterns or experiences within populations is where deductive methods can pick up where inductive leave off. However, it is only through also gaining a grounded and detailed understanding of phenomenon of theoretical interest do we gain new insights and have hope of developing understanding and theory that has relevance to field of practice.

Replication

As mentioned previously, inductive methods are seeking to develop a deep understanding of causal factors, processes, nature, meaning, and/or significance of a particular phenomenon ( Creswell and Poth 2018 ; Denzin and Lincoln 2012 ; Patton 2014 ). This understanding generally comes from asking a lot of questions, observing settings and behavior, and collecting stories, images, and other artifacts that aid the scholar in also gaining insight into their phenomenon of interest. Different approaches have been created to narrow in on specific types of phenomenon. For example, phenomenology looks at how individuals experience and ascribe meaning to a given phenomenon ( Giorgi 1997 ; Moran 2002 ; Waugh and Waugh 2003 ). Grounded theory seeks to identify the causal relationships that give rise to, and result from, a given phenomenon ( Glaser and Strauss 2017 ; Morse et al. 2016 ). Ethnography seeks to uncover the cultural elements within human systems ( Agar 1996 ; Hammersley 1983 ; Preissle and Le Compte 1984 ).

Each tradition has its own systematic process of data collection and analysis. However, regardless of the tradition, it is always the analyst who must draw inference and interpretation from the vast array of qualitative information in front of them . Just as there are some doctors who can observe the same patient information to diagnose root causes while others focus on first order symptomology, multiple analysts working independently on the same data sources may also come to different interpretations of what is going on ( Langley 1999 ). One doctor is not necessarily right and the others wrong; rather the same thing can be many things at once (e.g., structural, psychological, cultural). Therefore, the appropriate criteria of rigor is not whether the same interpretation would be independently arrived upon by different analysts. Rather, in inductive analysis, the criteria is: based on the evidence provided, is a given interpretation credible ( Patton 1999 )? In other words, if an independent analyst were informed of another analyst’s interpretation and then given all the same source information, would the interpretation stand up to scrutiny as being a justified, empirically grounded, exposition of the phenomenon?

Elements of Rigor

If we cannot assess inductive studies in terms of generalizability and replication, what are valid criteria upon which they might be evaluated? In very global terms, rigorous inductive research in public management can be judged on two core criteria:

1) Does the research design and its execution generate new insight into the causal factors, processes, nature, meaning, and/or significance of a phenomenon of interest to the field? (reviewed in Table 1 ) and

2) Is the account of these causal factors, processes, nature, meaning, and/or significance within these cases trustworthy? (reviewed in Table 3 )

Relevant and Inappropriate Criteria of Rigor for Inductive Research

The trustworthiness and depth of insight of an inductive study is manifest in its research design, execution, reporting.

Research Design

Because the contribution of inductive qualitative research fundamentally hinges on the theoretical relevance of the units (e.g., individuals, cases, texts) selected for study, sampling is of paramount importance. Different approaches of qualitative analysis have specific guidance on sampling consistent with that approach. For example, grounded theory uses a protocol of proposition-driven sampling in which the investigator strategically chooses cases iteratively in conjunction with data analysis in an effort to examine variation in patterns observed in the previous cases (for discussion, see Corbin and Strauss 1990 ; Glaser 2002 ). However, regardless of which analysis tradition an inductive scholar is using, the inductive qualitative sample must always be justified in terms of why the informants, texts, and/or cases selected should be considered of theoretical interest to the field. This description should be situated in terms of who these informants are in the broader population of possible informants relevant to the research question. Inductive scholarship should include a clear explication of why these individuals were chosen specifically and what they represent. What qualifies them as key informants of this phenomenon? Why would we expect them to have insight into this question that is particularly information rich and/or relevant to the field? How might their position likely influence the perspective they offer about the phenomenon (for discussion, see Marshall 1996 ; for exemplar, see Saz-Carranza and Ospina 2010 and Romzek et al. 2012 justification of both case and informant selection)?

As outlined in most introductory texts in qualitative analysis (e.g., Denzin and Lincoln 2012 ; Miles, Huberman, and Saldana 2013 ; Patton 2014 ), there are numerous sampling strategies that may guide participant or case selection in an inductive study. Common approaches include efforts to capture the “typical case,” the “extreme case,” the disconfirming case, or the “unusual case.” Sampling is also often purposefully stratified to represent informants from theoretically important sub-populations. In studies of individual level phenomenon, this may include stratifying samples to include men and women, young/middle age/old, more or less experience, or different ethnicities/racial groups. In studies of higher order phenomenon such as at the organizational, coalition, group, or network level, the scholar may choose to stratify cases across geographic region or based on some developmental phase (e.g., new versus old organizations). Although there are numerous potential sampling strategies for an inductive study, they all share in common the criteria that whatever or whomever is chosen for inclusion or exclusion of an inductive study, sampling decisions must be inquiry driven, theoretically justified, and information rich.

How Many is Enough?

The question of sample size in inductive qualitative research is less straight forward than it is in deductive research. In the deductive world, the sample size criteria turns primarily on the power to detect differences given the model applied to the data ( Wasserman 2013 ). In inductive research, the sample size question focuses on the sources of variability of the phenomenon of interest that are of theoretical importance to the field given the research question. However, inductive studies complicate the sample size question because numerous and varied sources of data can be, and often are, integrated. For example, in several qualitative approaches, triangulation of findings among multiple data sources is one of elements of the rigor (e.g., for review see Jonsen and Jehn 2009 ).

Just as with deductive research, no one inductive study can address every dimension or perspective that might be relevant to understanding a phenomenon of interest. Therefore, in addition to clearly articulating the criteria upon which individuals or other data sources were sampled for inclusion into the study, there is need to explicate the boundary criteria that sets the limits for who or what is not considered within the scope of the inquiry. Following this, the authors must clearly articulate the unit or units of analysis that define the phenomenon of inquiry. Is it case-based such as an inquiry into the factors that hindered international NGO community from being effective contributors to the response phase of Hurricane Katrina (e.g., Eikenberry, Arroyave, and Cooper 2007 )? Is it organizational such as a study of service providers’ usage of monitoring tools based on agency theory (e.g., Lambright 2008 ). Is it focused on the individual, such as examining public service motivation and transformation leadership (e.g., Andersen et al. 2016 )? Or is it episodic such as a study of the process through which routines can lead to a source of innovation within an organization (e.g., Feldman 2003 )? Higher order phenomenon (i.e., case-level, coalition-level, organizational-level, etc.) often require multiple data sources or informants associated with that case, group, or organization to gain sufficient depth of understanding of the dynamics present. This will necessarily place limits on the number of cases that can be studied comparatively. Alternatively, a single informant may be able to reflect on multiple episodic units based on varied experiences over time.

Qualitative Saturation

Qualitative saturation is a technique commonly referenced in inductive research to demonstrate that the dataset is robust in terms of capturing the important variability that exists around the phenomenon of interest ( O’Reilly and Parker 2013 ). However, we advise caution in the use of saturation in defending the sample characteristics of a qualitative sample. Qualitative saturation refers to a point at which the analyst has obtained a sort of information redundancy such that continued analysis has revealed no new insight not already captured by previous cases ( Morse 1995 ). Generally, during analysis, scholars do reach a point at which no new themes or propositions emerge and analysis of new transcripts leads only to additional instances of existing themes or relationships. However, this standard is problematic as a criterion for rigor in public management for two reasons.

First, in order to be used as a condition of sampling rigor, it requires that the scholar analyze their data as it is being collected so as to recognize the point at which no additional data collection is needed. Although this design feature is integral to grounded theory, it is uncommon in other qualitative traditions which often mimic deductive models having a distinct data collection phase preceding a data analysis phase ( Miles, Huberman, and Saldana 2013 ). Second, the methods by which a scholar determines saturation are generally methodologically difficult to standardize or demonstrate as a criteria of rigor ( Morse 1995 ; O’Reilly and Parker 2013 ). Therefore, while saturation is an important heuristic in guiding data analysis—for example, for informing the analyst when they should transition from open coding to axial coding, we do not find it is a particularly useful concept for qualitative consumers to evaluate the suitability of a dataset in terms of whether it should be considered theoretically robust.

Consequently, qualitative consumers generally must rely on qualitative as opposed to quantitative benchmarks for determining the suitability of a given dataset for addressing an inductive research question. The questions qualitative consumers need to answer are these: 1) is the dataset analyzed information rich and 2) does it have a reasonable chance of representing variability of the phenomena of interest that are of theoretical importance given the research question ( Brower, Abolafia, and Carr 2000 )? In efforts to orient new inductive scholars into the general ballpark of sample expectations, some scholars have cautiously made heavily caveated recommendations (for review, see Onwuegbuzie and Leech 2007 ). Qualitative studies of 100 or more units are unusual and generally unnecessary for most inductive analytic traditions unless some type of quantification is desired (see below discussion on hybrid designs; Gentles et al. 2015 ). Studies of ten or less units would require a unique justification in terms of how such a data set provides a theoretically robust perspective on the phenomenon of interest. Within that sizeable range, qualitative consumers will have to make a subjective call about the theoretical robustness of a given dataset in relation to the research question asked, the phenomenon of interest, the analytic tradition used, and the interpretive claims made. Benchmarking sampling approaches against existing literature utilizing the same analytic approach is helpful for creating consistency within the field. Additionally, qualitative consumers may find the following questions a useful rubric in determining how theoretically robust a given dataset might be considered to be:

1) Is the phenomenon rare or infrequently encountered?

2) Are the data rare or particularly difficult to obtain?

3) Is the phenomenon simple or complex?

4) Is the phenomenon new or well investigated in the literature?

5) How information rich is each unit in relation to the phenomenon of interest?

6) Is the same unit being engaged at multiple points in time?

Data Collection Protocols and Procedures

In deductive research, constructs and relationships are articulated prior to analysis, and what one can discover is therefore necessarily constrained to what one looks to find. In inductive research, constructs and relationships are articulated through analysis, and the scholar seeks to minimize constraint on what can be discovered ( Lincoln and Guba 1986 ). However, because in most inductive studies, the data must still be collected from individuals, the actions of the investigator will inevitably constrain and shape what the data looks like. This is done a priori through the creation of protocols which guide the types of questions that the investigator asks informants or the elements the investigator observes and records their observations. While these protocols can, and often should evolve over the course of the study, it is the execution of these protocols that create the data used in analysis. Consequently, the quality of these protocols and their execution is an important consideration in determining the rigor of an inductive study ( Miles, Huberman, and Saldana 2013 ).

In demonstrating the rigor of an inductive research design, the investigator should be able to clearly describe what data was considered relevant for a given research question and how this data was obtained. Data collection protocol design should be consistent with the specific methodological tradition embraced by the study (see Table 2 ). Vague descriptors such as “data were obtained through open ended interviews” is not sufficient description to determine rigor. Just as in deductive research the same construct can be operationalized in multiple ways, two inductive investigators may be interested in the same research question but ask very different types of interview questions of their informants. Researchers should be able to describe the types of questions the investigator asked informants related to the phenomenon of interest. These questions should have a clear conceptual linkage to the research question of concern, the analytic tradition embraced, and be a key consideration in the analysis and interpretation of the findings (for exemplar, see Rerup and Feldman’s (2011) , description of the interview protocol used to illicit espoused schemas of staff in a tech start up). It is also important for the qualitative consumer to recognize that data looks different depending on the different analytic tradition one uses. Table 2 outlines some of the more prevalent qualitative traditions.

Qualitative Data Collection and Analysis Traditions

Data Analysis and Interpretation

Like deductive approaches, inductive qualitative data analysis come in many forms linked to different analytic traditions and are more or less appropriate to different types of research questions. These traditions carry with them specific guidance on design, sampling, and analysis. Methodological deviations or qualitative “mixology” ( Kahlke 2014 ) in which design element from multiple traditions are combined or certain design elements omitted should be well-justified and evaluated carefully by the qualitative consumer to ensure the resulting design remains robust. Just as with deductive designs, robust inductive designs should have a clear logical flow from the research question, to the data collection protocol, to the description of the analysis procedure, to the explication of the findings. There should be no black curtain behind which hundreds of pages of transcripts are magically transformed into seven key findings. Rather, the scholar should be able to provide a clear and concise description of their analysis process and its relationship to the reported findings (for exemplar, see Rivera’s (2017) case analysis description in her study of gender discrimination in academic hiring committees).

As discussed, the overarching criteria of rigor associated with an inductive study is not reliability or replication. Rather, rigorous analysis is based on 1) whether the interpretation is credible in light of the data, 2) whether it was the result of a robust and systematic analytical process designed to move beyond superficial findings and minimize and/or account for investigator bias, and 3) whether it is reported with sufficient attention to context so as to facilitate the potential relevance of insights to similar contexts. These features were first described by Lincoln and Guba (1986) as the criteria of qualitative trustworthiness. They developed an initial set of practices designed to achieve these elements of rigor that have since been expanded upon by various qualitative scholars. Although these elements remain under development and debate, especially in public management (for discussion see, Lowery and Evans 2004 ), Table 3 offers a broad overview of some of the more commonly advocated strategies and associated aims that qualitative consumers might consider when evaluating the rigor of an inductive study. However, it is important to note that these elements represent strategies. They are not a checklist and certain strategies may be more or less appropriate in certain study designs. As such, we argue rigor is best conceptualized in terms of its functionality. Was the design logically coherent in relation to the research question? Was the analysis systematically designed to move beyond superficial findings and minimize and/or account for investigator bias? Did the design result in both credible and insightful findings? Were the findings reported with sufficient attention to context so as to facilitate empirically grounded theory building?

Elements of Qualitative Rigor (Adapted From Creswell and Poth 2018 ; Denzin and Lincoln 2003 ; Lincoln and Guba 1986 ; Morse 2015 )

We have argued that the distinction between inductive versus deductive approaches is a most relevant delineation for identifying appropriate criteria of rigor. Up to this point, we have focused primarily on inductive applications of qualitative data. However, as noted previously, not all qualitative data analysis is inductive. In this final section, we give special consideration to qualitative approaches in the field of public management that that are either deductive, mixed, and hybrid methods.

Narrative Policy Framework

In policy process research, the Narrative Policy Framework (NPF) has more recently emerged as an approach for quantifying qualitative data that has been coded from policy documents and various mediums of public comment ( Shanahan, Jones, and McBeth 2013 ). The NPF was designed to address postpositivist challenges to policy process theories by taking into account the critical role that narratives play in generating and facilitating meaning for people and how those narratives then relate to the politics of constructing reality ( Shanahan, Jones, and McBeth 2013 ). Within the NPF, narratives are considered to be constructed of important elements that include many traditional parts of stories like a hero, a villain, a plot, and a moral. These narrative elements are applied as codes in a more directly deductive approach and then often used for hypothesis testing at micro , meso , and macro levels ( McBeth, Jones, and Shanahan 2014 ).

Despite being derived from qualitative data, much of the work on NPF embraces a deductive model of hypothesis testing ( Shanahan, Jones, and McBeth 2017 ). In deductive applications, the standards of rigor as it relates to representative sampling, construct validity, reliability, statistical power, and generalizability apply. These methods require the development of a stable coding framework that can be applied by multiple coders with a high degree of reliability. As such, metrics such as inter-rater reliability are appropriate tools for demonstrating that the coding framework is being applied in a consistent manner. Another design challenge with NPF is the fact that its core propositions are associated with discrete “narratives” as the unit of analysis, which can be difficult to isolate in a standardized way across different types of policy documents which may contain multiple narratives (for discussion, see Shanahan, Jones, and McBeth 2018 ). Further, the representative sampling of policy documents relative to a defined population can be difficult to conceptualize ( Shanahan, Jones, and McBeth 2018 ). Despite these challenges, NPF is valuable in its ability to examine whether specific narrative patterns have a stable and generalizable influence on different outcomes of the policy process ( McBeth, Jones, and Shanahan 2014 ); a question ill-suited to an inductive narrative analysis approach.

Mixed Methods

Another development that has gained popularity in public management and applied social sciences more generally is the mixed methods study (see Honig, this issue). A mixed methods study is often characterized as one that uses a combination of both qualitative and quantitative data ( Creswell and Clark 2018 ; for alternative definitions see Johnson et al. 2007 ). It is generally assumed that mixed methods studies will also utilize a combination of inductive and deductive approaches. The ordering of the inductive/deductive mixture can vary. For example, the scholar may use an inductive qualitative phase aimed at gaining a greater insight about a poorly understood phenomenon. Constructs, dimensions, and propositions resulting in the findings from this first inductive phase of analysis can then be translated into a second confirmatory phase in the form of survey measure development, psychometrics, and hypothesis testing. In a second variation, a scholar may use existing literature and theory to deductively create measures and propose and test hypotheses. The scholar may then design an inductive phase in which the mechanisms and contextual factor underlying these hypotheses are explored in great depth through qualitative methods (for discussion of various design options, see Mele & Belardinelli, this issue; Creswell, Clark, Gutmann, and Hanson 2003 ).

Considerations of rigor in a mixed methods study are two pronged. First, mixed methods studies have the dual burden of adhering to all the requirements of rigorous design associated with both inductive and deductive models. For example, the sample for the inductive phase must meet the criteria of offering an information rich, inquiry-driven sample while the sample for the deductive phase must have still sufficient power to detect differences and be a reasonably representative sample of the population. This generally makes such studies relatively large and ambitious. Second, a rigorous mixed methods study should ideally reflect some degree of complementarity between the approaches, maximizing the different advantages in inductive versus deductive designs. Each design element should reflect thoughtful attention to the points at which the findings from the different phases of analysis co-inform one another ( Johnson, Burke, and Onwuegbuzie 2004 ).

Qualitative Comparative Analysis

Qualitative Comparative Analysis (QCA; Ragin 1998 ; Ragin and Rihoux 2004 ) represents a hybrid approach, being neither fully inductive or deductive. QCA has an established presence in public management ( Cristofoli and Markovic 2016 ; Hudson and Kuhner 2013 ; Malatesta and Carboni 2015 ; Pattyn, Molevald, and Befani 2017 ; Raab, Mannak, and Cabre 2015 ; Sanger 2013 ; Thomann 2015 ). Like NPF, QCA involves the quantification of qualitative data and the application of mathematical models. However, different from NPF, which is principally deductive in its approach, QCA can use inductive qualitative methods to identify outcomes of interest and factors of relevance to explaining that outcome. These interpretations of the data are then quantified and entered into mathematical models designed to examine pathways of necessary and sufficient conditions that are derived from a researcher creating a numeric data table, often using binary codes.

QCA, first introduced by Ragin (1987) , is intended to unify aspects of qualitative, case-based research, and quantitative, variable-based, approaches ( Fischer 2011 ). QCA is rooted in the assumption of equifinality; that different causal conditions can lead to the same outcome, and that the effect of each condition is dependent on how it is combined with other conditions ( Fischer 2011 ; Ragin 1987 ). Accordingly, QCA is not hindered by the assumptions of homogeneous effects that encumber many quantitative approaches. Rather, it enables the researcher to consider multiple pathways and combinations that may lead to the same outcome. Also unique to QCA is the focus on combinatorial logic that assumes that cases should be viewed holistically within the context of all conditions combined. As such, QCA can reveal patterns across cases that might be difficult to discern through purely qualitative approaches (for discussion see Rihoux and Ragin 2008 ).

One of the challenges to assessing QCA from a rigor perspective stems from its inherently hybrid nature. The samples in QCA are generally small and presumably inductively selected ( Hug 2013 ). As such, an inductive criteria of rigor could apply. However, the results of a QCA have a distinctive deductive flavor in both the style of analysis and interpretation. For example, the process by which the specific constructs are identified for inclusion is often not well explicated and may contain a mixture of a priori theory and inductively derived theory. Some authors embrace a fully deductive hypothesis driven approaches based on theory and using predetermined codebooks (e.g., Raab, Mannak, and Cambre 2015 ; Thomman 2015 ). Cases, which do not fit into one of the identified pathways are excluded from the output due to criteria like relevancy and consistency that enable the Boolean algebra of QCA to more readily converge on causal pathways. 3 Publications of QCA findings generally focus primarily on the pathways identified with little or no attention to the cases that deviated from these patterns.

It is our belief that as QCA applications evolve, scholars will need to, metaphorically, pick a horse to ride in their utilization of this technique in order for a study to be associated with the appropriate standards of rigor. In other words, QCA is a descriptive tool that can be used either inductively or deductively. Is a study a deductive effort to examine possible configurations of pathways toward a predefined outcome using a priori factors examined within a representative sample of a population? If so, deductive criteria of rigor would apply to a QCA as it relates to construct validity, inter-rater reliability, and representative sampling. On the other hand, QCA could also be a powerful tool used within an inductive model of research with associated inductive criteria of rigor. In this model, cases would be purposively justified as theoretically important to understanding a given phenomenon. The QCA would represent a tool within a broader process of inquiry for examining complex patterning across cases that may be difficult to otherwise discern. The inductive process by which coding categories were generated and qualitative variability that exists within coding delineations would be central concerns of the analysis. The analysis would include an empirically grounded contextualization and interpretation of the cases that conform to, as well as deviate from, the identified patterns so as to inform the mechanisms by which one pattern versus another may emerge. Either application of QCA, whether deductive or inductive, holds promise as a technique but murky applications which do not fully commit to either standard of rigor seem problematic (for additional discussion, see Hug 2013 ).

We began this article with the assertion that qualitative methods are poised to make a greater contribution in shaping our understanding of public management. We view this as a good thing; having the potential to inject new insight and depth of understanding into the questions that define the field. We perceive a general openness in the discipline to advancing bodies of literature through the integration of contributions from both inductive and deductive styles of inquiry. However, much of the discipline lacks even basic training in inductive approaches to research (see Stout 2013 ) which serves as a barrier. Deductive models—by virtue of the more structured task they are designed to accomplish coupled with the greater duration of time this approach has had to institutionalize—are simply more straightforward in their precepts of rigor. However, advancing the contribution of qualitative methods in public management will not happen without some shared construction of rigor that is compatible with a postpositivistic stance on science. We argue that the first step in advancing this agenda is to stop conflating data type (qualitative versus quantitative) with methodological approach (inductive versus deductive).

Beyond this, this article is positioned as a conversation-starter and as a resource for breaking down barriers for meaningful interactions that have put qualitative and quantitative methods at odds. We argue here that these past misunderstandings have less to do with the analysis of text versus number-based data, and more to do with murky or altogether misunderstood differences between the requirements of quality and rigor for inductive versus deductive methods. In clearing some of the air on quality and rigor of both kinds of methods in this space, we put forth a postpositivist stance with the understanding that not all scholars will agree, but that this perspective offers a productive pathway for broadly engaging the most common public management researcher today.

Agar , Michael H . 1996 . The professional stranger: An informal introduction to ethnography . 2nd ed. UK : Emerald Publishing Group .

Google Scholar

Google Preview

Andersen , Lotte Bøgh , Bente Bjørnholt , Louise Ladegaard Bro , and Christina Holm-Petersen . 2016 . Leadership and motivation: A qualitative study of transformational leadership and public service motivation . International Review of Administrative Sciences : 1 – 17 .

Brooks , Arthur C . 2002 . Can nonprofit management help answer public management’s “big questions” ? Public Administration Review , 62 ( 3 ): 259 – 66 .

Brower , Ralph S. , Mitchel Y. Abolafia , and Jered B. Carr . 2000 . On improving qualitative methods in public administration research . Administration & Society 32 ( 4 ): 363 – 97 . doi: 10.1177/00953990022019470 .

Carter , Stacy M. , and Miles Little . 2007 . Justifying knowledge, justifying method, taking action: Epistemologies, methodologies, and methods in qualitative research . Qualitative Health Research 17 ( 10 ): 1316 – 28 .

Caelli , Kate , Lynne Ray , and Judy Mill . 2003 . ‘Clear as mud’: Toward greater clarity in generic qualitative research . International Journal of Qualitative Methods 2 ( 2 ): 1 – 13 .

Clark , Alexander M . 1998 . The qualitative-quantitative debate: Moving from positivism and confrontation to post-positivism and reconciliation . Journal of Advanced Nursing 27 ( 6 ): 1242 – 9 .

Corbin , Juliet M. , and Anselm Strauss . 1990 . Grounded theory research: Procedures, canons, and evaluative criteria . Qualitative sociology 13 ( 1 ): 3 – 21 .

Corbin , Juliet , and Anselm L. Strauss . 2014 . Basics of qualitative research . Thousand Oaks, CA : Sage .

Creswell , John W. , and Dana L. Miller . 2000 . “ Determining validity in qualitative inquiry .” Theory into Practice . 39 ( 3 ): 124 – 130 .

Creswell , John W. , and Vicki L. Plano Clark . 2018 . Designing and conducting mixed methods research , 3rd ed. Thousand Oaks, CA : Sage .

Creswell , John W. and Cheryl Poth . 2018 . Qualitative inquiry and research design: Choosing among five approaches . 4th ed. Thousand Oaks, CA : Sage .

Creswell , John W. , Vicki L. Plano Clark , Michelle L. Gutmann , and William E. Hanson . 2003 . “ Advanced mixed methods research designs .” Handbook of Mixed Methods in Social and Behavioral Research 209 : 240 .

Cristofoli , Daniela and Josip Markovic . 2016 . How to make public networks really work: A qualitative comparative analysis . Public Administration 94 : 89 – 110 . doi: 10.1111/padm.12192

Dias , Janice Johnson , and Steven Maynard-Moody . 2007 . For-Profit Welfare: Contracts, Conflicts, and the Performance Paradox . Journal of Public Administration Research and Theory 17 : 189 – 211 .

Denhardt , Robert B . 2001 . The big questions of public administration education . Public Administration Review . 61 ( 5 ): 526 – 34 .

Denzin , Norman K. , and Yvonna S. Lincoln . 2003 . The landscape of qualitative research: Theories and issues . 2nd ed. Thousand Oaks, CA : Sage .

Denzin , Norman K. , and Yvonna S. Lincoln . 2012 . Strategies of qualitative inquiry . Vol. 4 . Thousand Oaks, CA : Sage .

Dodge , J. , S. M. Ospina , and E. G. Foldy . 2005 . Integrating rigor and relevance in public administration scholarship: The contribution of narrative inquiry . Public Administration Review 65 ( 3 ): 286 – 300 .

Dubnick , M. J . 1999 . Demons, spirits, and elephants: Reflections on the failure of public administration theory . Paper presented at the annual meeting of the American Political Science Association , Atlanta, GA .

Eikenberry , Angela M. , Verónica Arroyave , and Tracy Cooper . 2007 . Administrative failure and the international NGO response to Hurricane Katrina . Public Administration Review 67 ( 1 ): 160 – 70 .

Eisenhardt , Kathleen M. , and Melissa E. Graebner . 2007 . Theory building from cases: Opportunities and challenges . Academy of Management Journal . 50 ( 1 ): 25 – 32 .

Eisenhardt , Kathleen M. , Melissa E. Graebner , and Scott Sonenshein . 2016 . “ Grand challenges and inductive methods: Rigor without rigor mortis .” Academy Of Management Journal 59 ( 4 ): 1113 – 1123 .

Feldman , Martha S . 2000 . “ Organizational routines as a source of continuous change .” Organization Science 11 ( 6 ): 611 – 629 .

Feldman , Martha S. , and B. T. Pentland . 2003 . Reconceptualizing organizational routines as a source of flexibility and change . Administrative Science Quarterly 48 ( 1 ): 94 – 118 .

Feldman , Martha S. , Kaj Sköldberg , Ruth Nicole Brown , and Debra Horner . 2004 . Making sense of stories: A rhetorical approach to narrative analysis . Journal of Public Administration Research and Theory 14 ( 2 ): 147 – 70 .

Fischer , Manuel . 2011 . Social Network Analysis and Qualitative Comparative Analysis: Their mutual benefit for the explanation of policy network structures . Methodological Innovations Online 6 ( 2 ): 27 – 51 .

Freeman , John Leiper . 1965 . The Political Process: Executive Bureau-Legislative, Committee Relations . Vol. 13 . New York : Random House .

Glaser , Barney G . 2002 . Conceptualization: On theory and theorizing using grounded theory . International Journal of Qualitative Methods 1 ( 2 ): 23 – 38 .

Glaser , Barney G. , and Anselm L. Strauss . 2017 . Discovery of grounded theory: Strategies for qualitative research . London : Routledge .

Gentles , Stephen J. , Cathy Charles , Jenny Ploeg , and K. Ann McKibbon . 2015 . Sampling in qualitative research: Insights from an overview of the methods literature . The Qualitative Report 20 ( 11 ): 1772 .

Gill , J. , and K. J. Meier . 2000 . Public administration research and practice: A methodological manifesto . Journal of Public Administration Research and Theory 10 ( 1 ): 157 – 99 .

Giorgi , Amedeo . 1997 . The theory, practice, and evaluation of the phenomenological method as a qualitative research procedure . Journal of Phenomenological Psychology 28 ( 2 ): 235 – 60 .

Guba Egon G ., ed. 1990 . The paradigm dialog . Newbury Park, CA : Sage .

Hammersley , Martyn . 1983 . Ethnography . San Francisco, CA : John Wiley & Sons .

Haverland , Markus , and Dvora Yanow . 2012 . A hitchhiker ‘ s guide to the public administration research universe: Surviving conversations on methodologies and methods . Public Administration Review 72 ( 3 ): 401 – 8 .

Head , Brian William . 2010 . Public management research: Towards relevance . Public Management Review 12 ( 5 ): 571 – 85 .

Holland , P. W . 1986 . Statistics and causal inference . Journal of the American Statistical Association . 81 ( 396 ): 945 – 960 .

Honig , Dan . 2018 . Case study design and analysis as a complementary empirical strategy to econometric analysis in the study of public agencies: deploying mutually supportive mixed methods . Current issue.

Hudson , John , and Stefan Kühner . 2013 . Qualitative comparative analysis and applied public policy analysis: New applications of innovative methods . Policy and Society 32 ( 4 ): 279 – 87 .

Hug , Simon . 2013 . Qualitative comparative analysis: How inductive use and measurement error lead to problematic inference . Political Analysis 21 ( 2 ): 252 – 65 .

Johnson , R. Burke , and Anthony J. Onwuegbuzie . 2004 . Mixed methods research: A research paradigm whose time has come . Educational Researcher 33 ( 7 ): 14 – 26 .

Johnson , R. Burke , Anthony J. Onwuegbuzie , and Lisa A. Turner . 2007 . Toward a definition of mixed methods research . Journal of Mixed Methods Research 1 ( 2 ): 112 – 33 .

Jonsen , K. , and K. A. Jehn . 2009 . Using triangulation to validate themes in qualitative studies . Qualitative Research in Organizations and Management: An International Journal 4 ( 2 ): 123 – 50 .

Kahlke , Renate M . 2014 . Generic qualitative approaches: Pitfalls and benefits of methodological mixology . International Journal of Qualitative Methods 13 ( 1 ): 37 – 52 .

Kettl , Donald F . 2000 . Public administration at the millennium: The state of the field . Journal of Public Administration Research and Theory 10 ( 1 ): 7 – 34 .

King , Cheryl Simrell , Kathryn M. Feltey , and Bridget O’Neill Susel . 1998 . The Question of Participation: Toward Authentic Public Participation in Public Administration . Public Administration Review 58 ( 4 ): 317 . doi: 10.2307/977561 .

Kuhn , Thomas S . 1996 . The nature and necessity of scientific revolutions . In The structure of scientific revolutions , Kihn T. S . ed. 3rd ed. Chicago : University of Chicago Press .

Lambright , Kristina T . 2008 . Agency theory and beyond: Contracted providers’ motivations to properly use service monitoring tools . Journal of Public Administration Research and Theory 19 ( 2 ): 207 – 27 .

Langley , Ann . 1999 . Strategies for theorizing from process data . Academy of Management review 24 ( 4 ): 691 – 710 .

Lewis , D . 1973 . Causation . The Journal of Philosophy 70 ( 17 ): 556 – 67 .

Lincoln , Yvonna S. , and Egon G. Guba . 1986 . But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation . New Directions for Evaluation 30 : 73 – 84 .

Lipsky , Michael . 1971 . Street-level bureaucracy and the analysis of urban reform . Urban Affairs Quarterly 6 ( 4 ): 391 – 409 . doi: 10.1177/107808747100600401

Lowery , Daniel , and Karen G. Evans . 2004 . The iron cage of methodology: The vicious circle of means limiting ends limiting means ... Administration & Society 36 ( 3 ): 306 – 27 .

Malatesta , Deanna , and Julia L. Carboni . 2015 . The public–private distinction: Insights for public administration from the state action doctrine . Public Administration Review 75 ( 1 ): 63 – 74 .

Marshall , M. N . 1996 . Sampling for qualitative research . Family Practice 13 ( 6 ): 522 – 6 .

Maynard-Moody , Steven , and Marisa Kelly . 1993 . Stories managers tell about elected officials: Making sense of the politics-administration dichotomy . In Public management: The state of the art , ed. B. Bozeman, 71 – 90 . San Francisco : Jossey-Bass .

McNabb , David E . 2014 . Case research in public management . London : Routledge .

McNabb , David E . 2015 . Research methods in public administration and nonprofit management . London : Routledge .

McBeth , Mark K. , Michael D. Jones , and Elizabeth A. Shanahan . 2014 . “ The narrative policy framework .” In Theories of the policy process , edited by Sabatier , Paul A. , and Weible Christopher M , 225 – 266 . Boulder : Westview Press .

Mertens , Donna M. , and Amy T. Wilson . 2012 . Program evaluation theory and practice: A comprehensive guide . New York : Guilford Press .

Milward , H. Brinton. Forthcoming. Toward a theory of organizational networks: Structure, process, and people . Perspectives on Public Management and Governance .

Miles , Matthew B. , A. Michael Huberman , and Johnny Saldana . 2013 . Qualitative data analysis . Thousand Oaks, CA : Sage .

Moran , Dermot . 2002 . Introduction to phenomenology . London : Routledge .

Morse , Janice M . 1995 . The significance of saturation . Qualitative Health Research 5 ( 2 ): 147 – 9 .

Morse , Janice M . 2015 . Critical analysis of strategies for determining rigor in qualitative inquiry . Qualitative Health Research 25 ( 9 ): 1212 – 22 .

Morse , Janice M. , Phyllis Noerager Stern , Juliet Corbin , Barbara Bowers , Kathy Charmaz , and Adele E. Clarke . 2016 . Developing grounded theory: The second generation . London : Routledge .

Neuman , William Lawrence , and Karen Robson . 2014 . Basics of social research . Canada : Pearson .

Onwuegbuzie , A. J. , and N. L. Leech . 2007 . A call for qualitative power analyses . Quality & Quantity 41 ( 1 ): 105 – 21 .

O’Reilly , Michelle , and Nicola Parker . 2013 . ‘Unsatisfactory Saturation’: A critical exploration of the notion of saturated sample sizes in qualitative research . Qualitative Research 13 ( 2 ): 190 – 7 .

Ospina , Sonia M. , and Jennifer Dodge . 2005 . It’s about time: Catching method up to meaning—the usefulness of narrative inquiry in public administration research . Public Administration Review 65 ( 2 ): 143 – 57 .

Ospina , Sonia M. , Marc Esteve , and Seulki Lee . 2017 . Assessing Qualitative Studies in Public Administration Research . Public Administration Review 78 ( 4 ): 593 – 605 . doi: 10.1111/puar.12837

Patton , Michael Quinn . 1999 . Enhancing the quality and credibility of qualitative analysis . Health Services Research 34 ( 5 ): 1189 – 208 .

Patton , Michael Quinn . 2014 . Qualitative Research & Evaluation Methods: Integrating Theory and Practice . Thousand Oaks, CA : Sage .

Pattyn , Valérie , Astrid Molenveld , and Barbara Befani . 2017 . Qualitative comparative analysis as an evaluation tool: Lessons from an application in development cooperation . American Journal of Evaluation .

Perry , James L. , and Lois Recascino Wise . 1990 . The motivational bases of public service . Public Administration Review 50 ( 3 ): 367 . doi: 10.2307/976618

Pillow , W . 2003 . Confession, catharsis, or cure? Rethinking the uses of reflexivity as methodological power in qualitative research . International Journal of Qualitative Studies in Education 16 ( 2 ): 175 – 96 .

Prasad , Pushkala . 2015 . Crafting qualitative research: Working in the postpositivist traditions . London : Routledge .

Preissle , Judith , and Margaret D. Le Compte . 1984 . Ethnography and qualitative design in educational research . New York : Academic Press .

Raab , Jörg. , R. S. Mannak , and B. Cambre . 2015 . Combining structure, governance, and context: A configurational approach to network effectiveness . Journal of Public Administration Research and Theory 25 ( 2 ): 479 – 511 . doi: 10.1093/jopart/mut039

Raadschelders , J. C . 2011 . The future of the study of public administration: Embedding research object and methodology in epistemology and ontology . Public Administration Review 71 ( 6 ): 916 – 24 .

Ragin , Charles . 1987 . The comparative method: Moving beyond qualitative and quantitative methods . Berkeley : University of California Press .

Ragin , Charles C . 1998 . The logic of qualitative comparative analysis . International Review of Social History 43 ( S6 ): 105 – 24 . doi: 10.1017/S0020859000115111

Ragin , Charles C. , and Benoit Rihoux . 2004 . Qualitative Comparative Analysis (QCA): State of the Art and Prospects . Qualitative Methods 2 ( 2 ): 3 – 13 .

Rerup , C. , and M. S. Feldman . 2011 . Routines as a source of change in organizational schemata: The role of trial-and-error learning . Academy of Management Journal 54 ( 3 ): 577 – 610 .

Riccucci , Norma M . 2010 . Public administration: Traditions of inquiry and philosophies of knowledge . Washington, DC : Georgetown University Press .

Riessman , Catherine Kohler . 1993 . Narrative analysis . Vol. 30 . Thousand Oaks, CA : Sage .

Rihoux , Benoît , and Charles C. Ragin . 2008 . Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques . Vol. 51 . Thousand Oaks, CA : Sage .

Rivera , Lauren A . 2017 . “ When two bodies are (not) a problem: Gender and relationship status discrimination in academic hiring .” American Sociological Review 82 ( 6 ): 1111 – 1138 .

Rolfe , Gary . 2006 . Validity, trustworthiness and rigour: Quality and the idea of qualitative research . Journal of Advanced Nursing 53 ( 3 ): 304 – 10 .

Romzek , B. , and Melvin J. Dubnick . 1987 . Accountability in the Public Sector: Lessons from the Challenger Disaster . Public Administration Review 47 ( 3 ): 227 – 38 .

Romzek , B. , Kelly LeRoux , and Jeannette M. Blackmar . 2012 . A preliminary theory of informal accountability among network organizational actors . Public Administration Review 72 ( 3 ): 442 – 53 .

Romzek , B. , K. LeRoux , J. Johnston , R. J. Kempf , and J. S. Piatak . 2014 . Informal accountability in multisector service delivery collaborations . Journal of Public Administration Research and Theory 24 ( 4 ): 813 – 42 . doi: 10.1093/jopart/mut027

Sanger , Mary Bryna . 2013 . Does measuring performance lead to better performance ?. Journal of Policy Analysis and Management 32 ( 1 ): 185 – 203 .

Saz-Carranza , Angel , and Sonia M. Ospina . 2010 . The behavioral dimension of governing interorganizational goal-directed networks—Managing the unity-diversity tension . Journal of Public Administration Research and Theory . 21 ( 2 ): 327 – 65 .

Schein , Edgar H . 2003 . On dialogue, culture, and organizational learning . Reflections: The SoL Journal 4 ( 4 ): 27 – 38 . doi: 10.1162/152417303322004184

Schillemans , Thomas . 2013 . Moving beyond the clash of interests: On stewardship theory and the relationships between central government departments and public agencies . Public Management Review 15 ( 4 ): 541 – 62 .

Shanahan , Elizabeth A. , Michael D. Jones , and Mark K. McBeth . 2018 . How to conduct a Narrative Policy Framework study . The Social Science Journal . 55 ( 3 ): 332 – 345 .

Shanahan , Elizabeth A. , Michael D. Jones , Mark K. McBeth , and Ross R. Lane . 2013 . An angel on the wind: How heroic policy narratives shape policy realities: Narrative policy framework . Policy Studies Journal 41 ( 3 ): 453 – 83 . doi: 10.1111/psj.12025

Schneider , Anne , and Helen Ingram . 1993 . Social construction of target populations: Implications for politics and policy . American Political Science Review 87 ( 2 ): 334 – 47 .

Steiner , Elizabeth . 1988 . Methodology of theory building . Sydney : Educology Research Associates .

Stout , Margaret . 2013 . Preparing public administration scholars for qualitative inquiry: A status report . Public Administration Research 2 ( 1 ): 11 – 28 .

Thomann , Eva . 2015 . Is output performance all about the resources? A fuzzy ‐ set qualitative comparative analysis of street ‐ level bureaucrats in Switzerland . Public Administration 93 ( 1 ): 177 – 94 .

Toepler , Stefan . 2005 . Called to order: A board president in trouble . Nonprofit Management and Leadership 15 ( 4 ): 469 – 76 .

Van Slyke , David M . 2003 . The mythology of privatization in contracting for social services . Public Administration Review 63 ( 3 ): 296 – 315 . doi:10.1111/1540–6210.00291

Wasserman , Larry . 2013 . All of statistics: A concise course in statistical inference . New York : Springer Science & Business Media .

Waugh , William L. Jr , and Wesley W. Waugh . 2003 . Phenomenology and public administration . International Journal of Organization Theory & Behavior . 7 ( 3 ): 405 – 31 .

Weber , E. P. , and A. M. Khademian . 2008 . Wicked problems, knowledge challenges, and collaborative capacity builders in network settings . Public administration review 68 ( 2 ): 334 – 49 .

Weick , Karl E . 1993 . The collapse of sensemaking in organizations: The mann gulch disaster . Administrative Science Quarterly 38 ( 4 ): 628 . doi: 10.2307/2393339

Williamson , O. E . 1981 . The economics of organization: The transaction cost approach . American Journal of Sociology 87 ( 3 ): 548 – 77 .

Yin , Robert K . 2017 . Case study research and applications: Design and methods . Thousand Oaks, CA : Sage .

The method, methodology, epistemology coupling is a topic of considerable debate and concern in the field of qualitative methods ( Corbin and Strauss 2014 ; Haverland and Yanow 2012 ; Ospina et al 2017 ). Whether certain methods can or should be implemented by scholars embracing diverging epistemological stances is a topic warranting further discourse in the field of public management.

Reflexivity refers to the practice of being intentionally reflective about who you are both as a person situated within society and as a scholar professionally socialized within a cultural and institutional milieu. Specifically, reflexive practice calls upon scholars to consider how the totality of who they are as individuals influences the manner in which they approach scholarship, the questions they ask, the way the subjects of one’s inquiry may react/respond, and how one interprets what they observe. This is done with an eye toward critically examining how these factors may shape and constrain what one “finds” (for discussion, see Pillow 2003 ).

Within the QCA lexicon, results are referred to as causal pathways, although researchers are cautioned against the use of terms like causation as QCA uses a combinatorial logic/conjunctural causation instead of main effect/parameter estimate logic.

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1477-9803
• Print ISSN 1053-1858
• Copyright © 2024 Public Management Research Association
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Qualitative Research:...

Qualitative Research: Rigour and qualitative research

• Related content
• Peer review
• Nicholas Mays , director of health services research a ,
• Catherine Pope , lecturer in social and behavioural medicine b
• a King's Fund Institute, London W2 4HT
• b Department of Epidemiology and Public Health, University of Leicester, Leicester LE1 6TP
• Correspondence to: Mr Mays.

Various strategies are available within qualitative research to protect against bias and enhance the reliability of findings. This paper gives examples of the principal approaches and summarises them into a methodological checklist to help readers of reports of qualitative projects to assess the quality of the research.

Criticisms of qualitative research

In the health field--with its strong tradition of biomedical research using conventional, quantitative, and often experimental methods--qualitative research is often criticised for lacking scientific rigour. To label an approach “unscientific” is peculiarly damning in an era when scientific knowledge is generally regarded as the highest form of knowing. The most commonly heard criticisms are, firstly, that qualitative research is merely an assembly of anecdote and personal impressions, strongly subject to researcher bias; secondly, it is argued that qualiative research lacks reproducibility--the research is so personal to the researcher that there is no guarantee that a different researcher would not come to radically different conclusions; and, finally, qualitative research is criticised for lacking generalisability. It is said that qualitative methods tend to generate large amounts of detailed information about a small number of settings.

Is qualitative research different?

The pervasive assumption underlying all these criticisms is that quantitative and qualitative approaches are fundamentally different in their ability to ensure the validity and reliability of their findings. This distinction, however, is more one of degree than of type. The problem of the relation of a piece of research to some presumed underlying “truth” applies to the conduct of any form of social research. “One of the greatest methodological fallacies of the last half century in social research is the belief that science is a particular set of techniques; it is, rather, a state of mind, or attitude, and the organisational conditions which allow that attitude to be expressed.” 1 In quantitative data analysis it is possible to generate statistical representations of phenomena which may or may not be fully justified since, just as in qualitative work, they will depend on the judgment and skill of the researcher and the appropriateness to the question answered of the data collected. All research is selective--there is no way that the researcher can in any sense capture the literal truth of events. All research depends on collecting particular sorts of evidence through the prism of particular methods, each of which has its strengths and weaknesses. For example, in a sample survey it is difficult for the researcher to ensure that the questions, categories, and language used in the questionnaire are shared uniformly by respondents and that the replies returned have the same meanings for all respondents. Similarly, research that relies exclusively on observation by a single researcher is limited by definition to the perceptions and introspection of the investigator and by the possibility that the presence of the observer may, in some way that is hard to characterise, have influenced the behaviour and speech that was witnessed. Britten and Fisher summarise the position neatly by pointing out that “there is some truth in the quip that quantitative methods are reliable but not valid and that qualitative methods are valid but not reliable.” 2

Strategies to ensure rigour in qualitative research

As in quantitative research, the basic strategy to ensure rigour in qualitative research is systematic and self conscious research design, data collection, interpretation, and communication. Beyond this, there are two goals that qualitative researchers should seek to achieve: to create an account of method and data which can stand independently so that another trained researcher could analyse the same data in the same way and come to essentially the same conclusions; and to produce a plausible and coherent explanation of the phenomenon under scrutiny. Unfortunately, many qualitative researchers have neglected to give adequate descriptions in their research reports of their assumptions and methods, particularly with regard to data analysis. This has contributed to some of the criticisms of bias from quantitative researchers.

Yet the integrity of qualitative projects can be protected throughout the research process. The remainder of this paper discusses how qualitative researchers attend to issues of validity, reliability, and generalisability.

Much social science is concerned with classifying different “types” of behaviour and distinguishing the “typical” from the “atypical.” In quantitative research this concern with similarity and difference leads to the use of statistical sampling so as to maximise external validity or generalisability. Although statistical sampling methods such as random sampling are relatively uncommon in qualitative investigations, there is no reason in principle why they cannot be used to provide the raw material for a comparative analysis, particularly when the researcher has no compelling a priori reason for a purposive approach. For example, a random sample of practices could be studied in an investigation of how and why teamwork in primary health care is more and less successful in different practices. However, since qualitative data collection is generally more time consuming and expensive than, for example, a quantitative survey, it is not usually practicable to use a probability sample. Furthermore, statistical representativeness is not a prime requirement when the objective is to understand social processes.

An alternative approach, often found in qualitative research and often misunderstood in medical circles, is to use systematic, non-probabilistic sampling. The purpose is not to establish a random or representative sample drawn from a population but rather to identify specific groups of people who either possess characteristics or live in circumstances relevant to the social phenomenon being studied. Informants are identified because they will enable exploration of a particular aspect of behaviour relevant to the research. This approach to sampling allows the researcher deliberately to include a wide range of types of informants and also to select key informants with access to important sources of knowledge.

“Theoretical” sampling is a specific type of non-probability sampling in which the objective of developing theory or explanation guides the process of sampling and data collection. 3 Thus, the analyst makes an initial selection of informants; collects, codes, and analyses the data; and produces a preliminary theoretical explanation before deciding which further data to collect and from whom. Once these data are analysed, refinements are made to the theory, which may in turn guide further sampling and data collection. The relation between sampling and explanation is iterative and theoretically led.

To return to the example of the study of primary care team working, some of the theoretically relevant characteristics of general practices affecting variations in team working might be the range of professions represented in the team, the frequency of opportunities for communication among team members, the local organisation of services, and whether the practice is in an urban, city, or rural area. These factors could be identified from other similar research and within existing social science theories of effective and ineffective team working and would then be used explicitly as sampling categories. Though not statistically representative of general practices, such a sample is theoretically informed and relevant to the research questions. It also minimises the possible bias arising from selecting a sample on the basis of convenience.

ENSURING THE RELIABILITY OF AN ANALYSIS

In many forms of qualitative research the raw data are collected in a relatively unstructured form such as tape recordings or transcripts of conversations. The main ways in which qualitative researchers ensure the retest reliability of their analyses is in maintaining meticulous records of interviews and observations and by documenting the process of analysis in detail. While it is possible to analyse such data singlehandedly and use ways of classifying and categorising the data which emerge from the analysis and remain implicit, more explicit group approaches, which perhaps have more in common with the quantitative social sciences, are increasingly used. The interpretative procedures are often decided on before the analysis. Thus, for example, computer software is available to facilitate the analysis of the content of interview transcripts. 4 A coding frame can be developed to characterise each utterance (for example, in relation to the age, sex, and role of the speaker; the topic; and so on), and transcripts can then be coded by more than one researcher. 5 One of the advantages of audiotaping or videotaping is the opportunity the tapes offer for subsequent analysis by independent observers.

The reliability of the analysis of qualitative data can be enhanced by organising an independent assessment of transcripts by additional skilled qualitative researchers and comparing agreement between the raters. For example, in a study of clinical encounters between cardiologists and their patients which looked at the differential value each derived from the information provided by echocardiography, transcripts of the clinic interviews were analysed for content and structure by the principal researcher and by an independent panel, and the level of agreement was assessed. 6

SAFEGUARDING VALIDITY

Alongside issues of reliability, qualitative researchers give attention to the validity of their findings. “Triangulation” refers to an approach to data collection in which evidence is deliberately sought from a wide range of different, independent sources and often by different means (for instance, comparing oral testimony with written records). This approach was used to good effect in a qualitative study of the effects of the introduction of general management into the NHS. The accounts of doctors, managers, and patient advocates were explored in order to identify patterns of convergence between data sources to see whether power relations had shifted appreciably in favour of professional managers and against the medical profession. 7

The differences in GPs' interviews with parents of handicapped and non-handicapped children have been shown by qualitative methods

• Download figure
• Open in new tab
• Download powerpoint

Validation strategies sometimes used in qualitative research are to feed the findings back to the participants to see if they regard the findings as a reasonable account of their experience 8 and to use interviews or focus groups with the same people so that their reactions to the evolving analysis become part of the emerging research data. 9 If used in isolation these techniques assume that fidelity to the participants' commonsense perceptions is the touchstone of validity. In practice, this sort of validation has to be set alongside other evidence of the plausibility of the research account since different groups are likely to have different perspectives on what is happening. 10

A related analytical and presentational issue is concerned with the thoroughness with which the researcher examines “negative” or “deviant” cases--those in which the researcher's explanatory scheme appears weak or is contradicted by the evidence. The researcher should give a fair account of these occasions and try to explain why the data vary. 11 In the same way, if the findings of a single case study diverge from those predicted by a previously stated theory, they can be useful in revising the existing theory in order to increase its reliability and validity.

VALIDITY AND EXPLANATION

It is apparent in qualitative research, particularly in observational studies (see the next paper in this series for more on observational methods), that the researcher can be regarded as a research instrument. 12 Allowing for the inescapable fact that purely objective observation is not possible in social science, how can the reader judge the credibility of the observer's account? One solution is to ask a set of questions: how well does this analysis explain why people behave in the way they do; how comprehensible would this explanation be to a thoughtful participant in the setting; and how well does the explanation it advances cohere with what we already know?

This is a challenging enough test, but the ideal test of a qualitative analysis, particularly one based on observation, is that the account it generates should allow another person to learn the “rules” and language sufficiently well to be able to function in the research setting. In other words, the report should carry sufficient conviction to enable someone else to have the same experience as the original observer and appreciate the truth of the account. 13 Few readers have the time or inclination to go to such lengths, but this provides an ideal against which the quality of a piece of qualitative work can be judged.

The development of “grounded theory” 3 offers another response to this problem of objectivity. Under the strictures of grounded theory, the findings must be rendered through a systematic account of a setting that would be clearly recognisable to the people in the setting (by, for example, recording their words, ideas, and actions) while at the same time being more structured and self consciously explanatory than anything that the participants themselves would produce.

Attending to the context

Some pieces of qualitative research consist of a case study carried out in considerable detail in order to produce a naturalistic account of everyday life. For example, a researcher wishing to observe care in an acute hospital around the clock may not be able to study more than one hospital. Again the issue of generalisability, or what can be learnt from a single case, arises. Here, it is essential to take care to describe the context and particulars of the case study and to flag up for the reader the similarities and differences between the case study and other settings of the same type. A related way of making the best use of case studies is to show how the case study contributes to and fits with a body of social theory and other empirical work. 12 The final paper in this series discusses qualitative case studies in more detail.

COLLECTING DATA DIRECTLY

Another defence against the charge that qualitative research is merely impressionistic is that of separating the evidence from secondhand sources and hearsay from the evidence derived from direct observation of behaviour in situ. It is important to ensure that the observer has had adequate time to become thoroughly familiar with the milieu under scrutiny and that the participants have had the time to become accustomed to having the researcher around. It is also worth asking whether the observer has witnessed a wide enough range of activities in the study site to be able to draw conclusions about typical and atypical forms of behaviour--for example, were observations undertaken at different times? The extent to which the observer has succeeded in establishing an intimate understanding of the research setting is often shown in the way in which the subsequent account shows sensitivity to the specifics of language and its meanings in the setting.

MINIMISING RESEARCHER BIAS IN THE PRESENTATION OF RESULTS

Although it is not normally appropriate to write up qualitative research in the conventional format of the scientific paper, with a rigid distinction between the results and discussion sections of the account, it is important that the presentation of the research allows the reader as far as possible to distinguish the data, the analytic framework used, and the interpretation. 1 In quantitative research these distinctions are conventionally and neatly presented in the methods section, numerical tables, and the accompanying commentary. Qualitative research depends in much larger part on producing a convincing account. 14 In trying to do this it is all too easy to construct a narrative that relies on the reader's trust in the integrity and fairness of the researcher. The equivalent in quantitative research is to present tables of data setting out the statistical relations between operational definitions of variables without giving any idea of how the phenomena they represent present themselves in naturally occurring settings. 1 The need to quantify can lead to imposing arbitrary categories on complex phenomena, just as data extraction in qualitative research can be used selectively to tell a story that is rhetorically convincing but scientifically incomplete.

The problem with presenting qualitative analyses objectively is the sheer volume of data customarily available and the relatively greater difficulty faced by the researcher in summarising qualitative data. It has been suggested that a full transcript of the raw data should be made available to the reader on microfilm or computer disk, 11 although this would be cumbersome. Another partial solution is to present extensive sequences from the original data (say, of conversations), followed by a detailed commentary.

Another option is to combine a qualitative analysis with some quantitative summary of the results. The quantification is used merely to condense the results to make them easily intelligible; the approach to the analysis remains qualitative since naturally occurring events identified on theoretical grounds are being counted. The table shows how Silverman compared the format of the doctor's initial questions to parents in a paediatric cardiology clinic when the child was not handicapped with a smaller number of cases when the child had Down's syndrome. A minimum of interpretation was needed to contrast the two sorts of interview. 15 16

Assessing a piece of qualitative research

This short paper has shown some of the ways in which researchers working in the qualitative tradition have endeavoured to ensure the rigour of their work. It is hoped that this summary will help the prospective reader of reports of qualitative research to identify some of the key questions to ask when trying to assess its quality. A range of helpful checklists has been published to assist readers of quantitative research assess the design 17 and statistical 18 and economic 19 aspects of individual published papers and review articles. 20 Likewise, the contents of this paper have been condensed into a checklist for readers of qualitative studies, covering design, data collection, analysis, and reporting (box). We hope that the checklist will give readers of studies in health and health care research that use qualitative methods the confidence to subject them to critical scrutiny.

Questions to ask of a qualitative study

Overall, did the researcher make explicit in the account the theoretical framework and methods used at every stage of the research?

Was the context clearly described?

Was the sampling strategy clearly described and justified?

Was the sampling strategy theoretically com-prehensive to ensure the generalisability of the conceptual analyses (diverse range of individuals and settings, for example)?

How was the fieldwork undertaken? Was it described in detail?

Could the evidence (fieldwork notes, inter-view transcripts, recordings, documentary analysis, etc) be inspected independently by others; if relevant, could the process of transcription be independently inspected?

Were the procedures for data analysis clearly described and theoretically justified? Did they relate to the original research questions? How were themes and concepts identified from the data?

Was the analysis repeated by more than one researcher to ensure reliability?

Did the investigator make use of quantitative evidence to test qualitative conclusions where appropriate?

Did the investigator give evidence of seeking out observations that might have contradicted or modified the analysis?

Was sufficient of the original evidence pre-sented systematically in the written account to satisfy the sceptical reader of the relation between the interpretation and the evidence (for example, were quotations numbered and sources given)?

Form of doctor's questions to parents at a paediatric cardiology clinic 15

• View inline

Further reading

Hammersley M. Reading ethnographic research. London: Longman, 1990.

• MacDonald I ,
• Britten N ,
• Glaser BG ,
• Krippendorff K
• Pollitt C ,
• Harrison S ,
• Hunter DJ ,
• McKeganey NP ,
• Glassner B ,
• Silverman D
• Fowkes FGR ,
• Gardner MJ ,
• Campbell MJ
• Department of Clinical Epidemiology and Biostatistics

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 26: Rigour

Darshini Ayton

Learning outcomes

Upon completion of this chapter, you should be able to:

• Understand the concepts of rigour and trustworthiness in qualitative research.
• Describe strategies for dependability, credibility, confirmability and transferability in qualitative research.
• Define reflexivity and describe types of reflexivity

What is rigour?

In qualitative research, rigour, or trustworthiness, refers to how researchers demonstrate the quality of their research. 1, 2 Rigour is an umbrella term for several strategies and approaches that recognise the influence on qualitative research by multiple realities; for example, of the researcher during data collection and analysis, and of the participant. The research process is shaped by multiple elements, including research skills, the social and research environment and the community setting. 2

Research is considered rigorous or trustworthy when members of the research community are confident in the study’s methods, the data and its interpretation. 3 As mentioned in Chapters 1 and 2, quantitative and qualitative research are founded on different research paradigms and, hence, quality in research cannot be addressed in the same way for both types of research studies. Table 26.1 provides a comparison overview of the approaches of quantitative and qualitative research in ensuring quality in research.

Table 26.1: Comparison of quantitative and qualitative approaches to ensuring quality in research

Below is an overview of the main approaches to rigour in qualitative research. For each of the approaches, examples of how rigour was demonstrated are provided from the author’s PhD thesis.

Approaches to dependability

Dependability requires the researcher to provide an account of changes to the research process and setting. 3 The main approach to dependability is an audit trail.

• Audit trail – the researcher records or takes notes on the conduct of the research and the process of reaching conclusions from the data. The audit trail includes information on the data collection and data analysis, including decision-making and interpretations of the data that influence the study’s results. 8 , 9

The interview questions for this study evolved as the study progressed, and accordingly, the process was iterative. I spent 12 months collecting data, and as my understanding and responsiveness to my participants and to the culture and ethos of the various churches developed, so did my line of questioning. For example, in the early interviews for phase 2, I included questions regarding the qualifications a church leader might look for in hiring someone to undertake health promotion activities. This question was dropped after the first couple of interviews, as it was clear that church leaders did not necessarily view their activities as health promoting and therefore did not perceive the relevance of this question. By ‘being church’, they were health promoting, and therefore activities that were health promoting were not easily separated from other activities that were part of the core mission of the church 10 ( pp93–4)

Approaches to credibility

Credibility requires the researcher to demonstrate the truth or confidence in the findings. The main approaches to credibility include triangulation, prolonged engagement, persistent observation, negative case analysis and member checking. 3

• Triangulation – the assembly of data and interpretations from multiple methods (methods triangulation), researchers (research triangulation), theory (theory triangulation) and data sources (different participant groups). 9 Refer to Chapter 28 for a detailed discussion of this process.
• Prolonged engagement – the requirement for researchers to spend sufficient time with participants and/or within the research context to familiarise them with the research setting, to build trust and rapport with participants and to recognise and correct any misinformation. 9

Prolonged engagement with churches was also achieved through the case study phase as the ten case study churches were involved in more than one phase of data collection. These ten churches were the case studies in which significant time was spent conducting interviews and focus groups, and attending activities and programs. Subsequently, there were many instances where I interacted with the same people on more than one occasion, thereby facilitating the development of interactive and deeper relationships with participants 10 (pp.94–5)

• Persistent observation – the identification of characteristics and elements that are most relevant to the problem or issue under study, and upon which the research will focus in detail. 9

In the following chapters, I present my analysis of the world of churches in which I was immersed as I conducted fieldwork. I describe the processes of church practice and action, and explore how this can be conceptualised into health promotion action 10 (p97)

• Negative case analysis – the process of finding and discussing data that contradicts the study’s main findings. Negative case analysis demonstrates that nuance and granularity in perspectives of both shared and divergent opinions have been examined, enhancing the quality of the interpretation of the data.

Although I did not use negative case selection, the Catholic churches in this study acted as examples of the ‘low engagement’ 10 (p97 )

• Member checking – the presentation of data analysis, interpretations and conclusions of the research to members of the participant groups. This enables participants or people with shared identity with the participants to provide their perspectives on the research. 9

Throughout my candidature – during data collection and analysis, and in the construction of my results chapters – I engaged with a number of Christians, both paid church staff members and volunteers, to test my thoughts and concepts. These people were not participants in the study, but they were embedded in the cultural and social context of churches in Victoria. They were able to challenge and also affirm my thinking and so contributed to a process of member checking 10 (p96)

Approaches to confirmability

Confirmability is demonstrated by grounding the results in the data from participants. 3 This can be achieved through the use of quotes, specifying the number of participants and data sources and providing details of the data collection.

• Quotes from participants are used to demonstrate that the themes are generated from the data. The results section of the thesis chapters commences with a story based on the field notes or recordings, with extensive quotes from participants presented throughout. 10
• The number of participants in the study provides the context for where the data is ‘sourced’ from for the results and interpretation. Table 26.2 is reproduced with permission from the Author’s thesis and details the data sources for the project. This also contributes to establishing how triangulation across data sources and methods was achieved.
• Details of data collection – Table 26.2 provides detailed information about the processes of data collection, including dates and locations but the duration of each research encounter was not specified.

Table 26.2 Data sources for the PhD research project of the Author.

Approaches to transferability.

To enable the transferability of qualitative research, researchers need to provide information about the context and the setting. A key approach for transferability is thick description. 6

• Thick description – detailed explanations and descriptions of the research questions are provided, including about the research setting, contextual factors and changes to the research setting. 9

I chose to include the Catholic Church because it is the largest Christian group in Australia and is an example of a traditional church. The Protestant group were represented through the Uniting, Anglican Baptist and Church of Christ denominations. The Uniting Church denomination is unique to Australia and was formed in 1977 through the merging of the Methodist, Presbyterian and Congregationalist denominations. The Church of Christ denomination was chosen to represent a contemporary less hierarchical denomination in comparison to the other protestant denominations. The last group, the Salvation Army, was chosen because of its high profile in social justice and social welfare, therefore offering different perspectives on the role and activities of the church in health promotion 10 (pp82–3)

What is reflexivity?

Reflexivity is the process in which researchers engage to explore and explain how their subjectivity (or bias) has influenced the research. 12 Researchers engage in reflexive practices to ensure and demonstrate rigour, quality and, ultimately, trustworthiness in their research. 13 The researcher is the instrument of data collection and data analysis, and hence awareness of what has influenced their approach and conduct of the research – and being able to articulate them – is vital in the creation of knowledge. One important element is researcher positionality (see Chapter 27), which acknowledges the characteristics, interests, beliefs and personal experiences of the researcher and how this influences the research process. Table 26.3 outlines different types of reflexivity, with examples from the author’s thesis.

Table 26.3: Types of reflexivity

The quality of qualitative research is measured through the rigour or trustworthiness of the research, demonstrated through a range of strategies in the processes of data collection, analysis, reporting and reflexivity.

• Chowdhury IA. Issue of quality in qualitative research: an overview. Innovative Issues and Approaches in Social Sciences . 2015;8(1):142-162. doi:10.12959/issn.1855-0541.IIASS-2015-no1-art09
• Cypress BS. Rigor or reliability and validity in qualitative research: perspectives, strategies, reconceptualization, and recommendations. Dimens Crit Care Nurs . 2017;36(4):253-263. doi:10.1097/DCC.0000000000000253
• Connelly LM. Trustworthiness in qualitative research. Medsurg Nurs . 2016;25(6):435-6.
• Golafshani N. Understanding reliability and validity in qualitative research. Qual Rep . 2003;8(4):597-607. Accessed September 18, 2023. https://nsuworks.nova.edu/tqr/vol8/iss4/6/
• Yilmaz K. Comparison of quantitative and qualitative research traditions: epistemological, theoretical, and methodological differences. Eur J  Educ . 2013;48(2):311-325. doi:10.1111/ejed.12014
• Shenton AK. Strategies for ensuring trustworthiness in qualitative research projects. Education for Information 2004;22:63-75. Accessed September 18, 2023. https://content.iospress.com/articles/education-for-information/efi00778
• Varpio L, O’Brien B, Rees CE, Monrouxe L, Ajjawi R, Paradis E. The applicability of generalisability and bias to health professions education’s research. Med Educ . Feb 2021;55(2):167-173. doi:10.1111/medu.14348
• Carcary M. The Research Audit Trail: Methodological guidance for application in practice. Electronic Journal of Business Research Methods . 2020;18(2):166-177. doi:10.34190/JBRM.18.2.008
• Korstjens I, Moser A. Series: Practical guidance to qualitative research. Part 4: Trustworthiness and publishing. Eur J Gen Pract . Dec 2018;24(1):120-124. doi:10.1080/13814788.2017.1375092
• Ayton D. ‘From places of despair to spaces of hope’ – the local church and health promotion in Victoria . PhD. Monash University; 2013. https://figshare.com/articles/thesis/_From_places_of_despair_to_spaces_of_hope_-_the_local_church_and_health_promotion_in_Victoria/4628308/1
• Hanson A. Negative case analysis. In: Matthes J, ed. The International Encyclopedia of Communication Research Methods . John Wiley & Sons, Inc.; 2017. doi: 10.1002/9781118901731.iecrm0165
• Olmos-Vega FM. A practical guide to reflexivity in qualitative research: AMEE Guide No. 149. Med Teach . 2023;45(3):241-251. doi: 10.1080/0142159X.2022.2057287
• Dodgson JE. Reflexivity in qualitative research. J Hum Lact . 2019;35(2):220-222. doi:10.1177/08903344198309

Qualitative Research – a practical guide for health and social care researchers and practitioners Copyright © 2023 by Darshini Ayton is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

Share This Book

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 18, Issue 2
• Issues of validity and reliability in qualitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Helen Noble 1 ,
• Joanna Smith 2
• 1 School of Nursing and Midwifery, Queens's University Belfast , Belfast , UK
• 2 School of Human and Health Sciences, University of Huddersfield , Huddersfield , UK
• Correspondence to Dr Helen Noble School of Nursing and Midwifery, Queens's University Belfast, Medical Biology Centre, 97 Lisburn Rd, Belfast BT9 7BL, UK; helen.noble{at}qub.ac.uk

https://doi.org/10.1136/eb-2015-102054

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Evaluating the quality of research is essential if findings are to be utilised in practice and incorporated into care delivery. In a previous article we explored ‘bias’ across research designs and outlined strategies to minimise bias. 1 The aim of this article is to further outline rigour, or the integrity in which a study is conducted, and ensure the credibility of findings in relation to qualitative research. Concepts such as reliability, validity and generalisability typically associated with quantitative research and alternative terminology will be compared in relation to their application to qualitative research. In addition, some of the strategies adopted by qualitative researchers to enhance the credibility of their research are outlined.

Are the terms reliability and validity relevant to ensuring credibility in qualitative research?

Although the tests and measures used to establish the validity and reliability of quantitative research cannot be applied to qualitative research, there are ongoing debates about whether terms such as validity, reliability and generalisability are appropriate to evaluate qualitative research. 2–4 In the broadest context these terms are applicable, with validity referring to the integrity and application of the methods undertaken and the precision in which the findings accurately reflect the data, while reliability describes consistency within the employed analytical procedures. 4 However, if qualitative methods are inherently different from quantitative methods in terms of philosophical positions and purpose, then alterative frameworks for establishing rigour are appropriate. 3 Lincoln and Guba 5 offer alternative criteria for demonstrating rigour within qualitative research namely truth value, consistency and neutrality and applicability. Table 1 outlines the differences in terminology and criteria used to evaluate qualitative research.

• View inline

Terminology and criteria used to evaluate the credibility of research findings

What strategies can qualitative researchers adopt to ensure the credibility of the study findings?

Unlike quantitative researchers, who apply statistical methods for establishing validity and reliability of research findings, qualitative researchers aim to design and incorporate methodological strategies to ensure the ‘trustworthiness’ of the findings. Such strategies include:

Accounting for personal biases which may have influenced findings; 6

Acknowledging biases in sampling and ongoing critical reflection of methods to ensure sufficient depth and relevance of data collection and analysis; 3

Meticulous record keeping, demonstrating a clear decision trail and ensuring interpretations of data are consistent and transparent; 3 , 4

Establishing a comparison case/seeking out similarities and differences across accounts to ensure different perspectives are represented; 6 , 7

Including rich and thick verbatim descriptions of participants’ accounts to support findings; 7

Demonstrating clarity in terms of thought processes during data analysis and subsequent interpretations 3 ;

Engaging with other researchers to reduce research bias; 3

Respondent validation: includes inviting participants to comment on the interview transcript and whether the final themes and concepts created adequately reflect the phenomena being investigated; 4

Data triangulation, 3 , 4 whereby different methods and perspectives help produce a more comprehensive set of findings. 8 , 9

Table 2 provides some specific examples of how some of these strategies were utilised to ensure rigour in a study that explored the impact of being a family carer to patients with stage 5 chronic kidney disease managed without dialysis. 10

Strategies for enhancing the credibility of qualitative research

In summary, it is imperative that all qualitative researchers incorporate strategies to enhance the credibility of a study during research design and implementation. Although there is no universally accepted terminology and criteria used to evaluate qualitative research, we have briefly outlined some of the strategies that can enhance the credibility of study findings.

• Sandelowski M
• Lincoln YS ,
• Barrett M ,
• Mayan M , et al
• Greenhalgh T
• Lingard L ,

Twitter Follow Joanna Smith at @josmith175 and Helen Noble at @helnoble

Competing interests None.

Read the full text or download the PDF:

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

4.7 Qualitative Rigour

Qualitative research is sometimes crticised for being biased, small scale, anecdotal, and/or lacking rigour; but, when done properly, it is impartial, in-depth, valid, dependable, believable, and rigorous. Similar to quantitive research, where validity and reliability are assessed, qualitative research may be evaluated for trustworthiness. 81 Criteria used to establish trustworthiness include credibility, transferability, confirmability, and dependability. 81

• Credibility: refers to the degree to which the findings of a study are believable, trustworthy, and accurate. 81,82 The “fit” between respondents’ opinions and the researcher’s depiction of them is addressed by credibility. There are a variety of approaches to address credibility, including prolonged engagement, persistent observation, data collection triangulation, and researcher triangulation. 82 Peer debriefing to give an external check on the study process has also been identified to increase credibility. Another way is via member checking, which involves testing the results and interpretations with the participants. 82
• Transferability:  refers to the extent to which the findings of the study can be applied to other settings and contexts. 81 In other words, transferability refers to the generalizability of the study. While the researcher cannot know which sites may desire to transfer the findings, the researcher must provide detailed descriptions so that people who wish to transfer the results to their own location may assess transferability. 82
• Confirmability: refers to the degree to which the research findings are objective and not influenced by the researcher’s biases or beliefs. 81 It is concerned with demonstrating that the researcher’s interpretations and findings are drawn from the data, which necessitates the researcher demonstrating how conclusions and interpretations were reached. 83 For people to comprehend how and why decisions were taken, researchers need to incorporate markers like the justifications for theoretical, methodological, and analytical choices throughout the study. 84
• Dependability: refers to findings that are consistent and sustainable over time. 81 Researchers need to ensure the study process is rational, traceable, and thoroughly recorded. Readers are better equipped to assess the reliability of the study when they can see how the research was conducted. 82,83

In the Padlet below, use a mind map to articulate your thoughts on a particular research of interest to you in your discipline/field of study.

An Introduction to Research Methods for Undergraduate Health Profession Students Copyright © 2023 by Faith Alele and Bunmi Malau-Aduli is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

Login to your account

If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password

If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password

• AACP Member Login      Submit

Download started.

• PDF [516 KB] PDF [516 KB]
• Figure Viewer
• Download Figures (PPT)
• Add To Online Library Powered By Mendeley
• Add To My Reading List
• Export Citation
• Create Citation Alert

A Review of the Quality Indicators of Rigor in Qualitative Research

• Jessica L. Johnson, PharmD Jessica L. Johnson Correspondence Corresponding Author: Jessica L. Johnson, William Carey University School of Pharmacy, 19640 Hwy 67, Biloxi, MS 39574. Tel: 228-702-1897. Contact Affiliations William Carey University School of Pharmacy, Biloxi, Mississippi Search for articles by this author
• Donna Adkins, PharmD Donna Adkins Affiliations William Carey University School of Pharmacy, Biloxi, Mississippi Search for articles by this author
• Sheila Chauvin, PhD Sheila Chauvin Affiliations Louisiana State University, School of Medicine, New Orleans, Louisiana Search for articles by this author
• qualitative research design
• standards of rigor
• best practices

INTRODUCTION

• Denzin Norman
• Lincoln Y.S.
• Google Scholar
• Anderson C.
• Full Text PDF
• Scopus (561)
• Santiago-Delefosse M.
• Stephen S.L.
• Scopus (83)
• Scopus (32)
• Levinson W.
• Scopus (504)
• Dixon-Woods M.
• Scopus (439)
• Malterud K.
• Midtgarden T.
• Scopus (203)
• Wasserman J.A.
• Wilson K.L.
• Scopus (68)
• Barbour R.S.
• Sale J.E.M.
• Scopus (12)
• Fraser M.W.
• Scopus (35)
• Sandelowski M.
• Scopus (1554)

BEST PRACTICES: STEP-WISE APPROACH

Step 1: identifying a research topic.

• Scopus (279)
• Creswell J.
• Maxwell J.A.
• Glassick C.E.
• Maeroff G.I.
• Scopus (264)

• Scopus (271)
• Ringsted C.
• Scherpbier A.
• Scopus (132)
• Ravitch S.M.

• View Large Image
• Download Hi-res image
• Download (PPT)
• Huberman M.

Step 2: Qualitative Study Design

• Whittemore R.
• Mandle C.L.
• Scopus (969)
• Marshall M.N.
• Scopus (2208)
• Horsfall J.
• Scopus (182)
• O’Reilly M.
• Scopus (1036)
• Burkard A.W.
• Scopus (164)
• Patton M.Q.
• Scopus (353)
• Scopus (4166)
• Johnson R.B.

Step 3: Data Analysis

Step 4: drawing valid conclusions.

• Swanwick T.
• Swanwick T.O.
• O’Brien B.C.
• Harris I.B.
• Beckman T.J.
• Scopus (4705)

Step 5: Reporting Research Results

• Shenton A.K.
• Scopus (4115)

Article info

Publication history, identification.

DOI: https://doi.org/10.5688/ajpe7120

ScienceDirect

• Download .PPT

Related Articles

• Access for Developing Countries
• Articles & Issues
• Articles In Press
• Current Issue
• Past Issues
• Journal Information
• About Open Access
• Aims & Scope
• Editorial Board
• Editorial Team
• History of AJPE
• Contact Information
• For Authors
• Guide for Authors
• Researcher Academy
• Rights & Permissions
• Submission Process
• Submit Article
• For Reviewers
• Reviewer Instructions
• Reviewer Frequently Asked Questions

The content on this site is intended for healthcare professionals.

• Privacy Policy   
• Terms and Conditions   
• Accessibility   
• Help & Contact

A Review of the Quality Indicators of Rigor in Qualitative Research

Affiliations.

• 1 William Carey University School of Pharmacy, Biloxi, Mississippi.
• 2 Louisiana State University, School of Medicine, New Orleans, Louisiana.
• PMID: 32292186
• PMCID: PMC7055404
• DOI: 10.5688/ajpe7120

Attributes of rigor and quality and suggested best practices for qualitative research design as they relate to the steps of designing, conducting, and reporting qualitative research in health professions educational scholarship are presented. A research question must be clear and focused and supported by a strong conceptual framework, both of which contribute to the selection of appropriate research methods that enhance trustworthiness and minimize researcher bias inherent in qualitative methodologies. Qualitative data collection and analyses are often modified through an iterative approach to answering the research question. Researcher reflexivity, essentially a researcher's insight into their own biases and rationale for decision-making as the study progresses, is critical to rigor. This article reviews common standards of rigor, quality scholarship criteria, and best practices for qualitative research from design through dissemination.

Keywords: best practices; qualitative research design; quality; standards of rigor.

© 2020 American Association of Colleges of Pharmacy.

Publication types

• Data Collection / methods
• Education, Pharmacy / methods*
• Fellowships and Scholarships / methods
• Qualitative Research
• Quality Indicators, Health Care
• Research Design
• Research Personnel

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Part II. Rigour in qualitative research: complexities and solutions

2005, Nurse Researcher

Related Papers

Jurnal Akuntansi dan Keuangan

helianti utami

Prior research has explored qualitative studies in relation to the paradigms used. This paper enriches the literature by investigating the quality of qualitative studies in relation to the data collection method and participants&#39; selection. In this study, we collected SNA qualitative paper proceedings from 2007 to 2017. Guided by the minimum criteria of the data collection method described in the literature review sections, we analyze those proceedings. We found the three most common methods used in the studies: interview, observation, and documentation. The majority of the paper clearly stated their data collection method. However, only a minority of them provides a clear description of how the data were collected and how to obtain participants/data used in their studies and why invite dthem in the research. Thus, it is suggested that researchers provide a detail explanation of their methods to show the rigour of the study that they conducted

Julia Crook

Nick Schuermans

Electronic Journal of Business Research Methods

Assessing the quality of qualitative research to ensure rigour in the findings is critical, especially if findings are to contribute to theory and be utilised in practice. However, teaching students concepts of rigour and how to apply them to their research is challenging. This article presents a generic framework of rigour with four critical dimensions—Trustworthiness, Auditability, Credibility and Transferability (TACT) intended to teach issues of rigour to postgraduate students and those new to qualitative research methodology. The framework enables them to explore the key dimensions necessary for assessing the rigour of qualitative research studies and checklist questions against each of the dimensions. TACT was offered through 10 workshops, attended by 64 participants. Participants positively evaluated the workshops and reported that the workshops enable them to learn the principles of qualitative research and better understanding issues of rigour. Work presented in the article...

American Journal of Pharmaceutical Education

Sheila Chauvin

Zimitri Erasmus , Jacques de Wet

Indian Journal of Public Health

Sanjay Zodpey

Traditionally, qualitative studies are founded on interpretative and constructive epistemology. The process of data collection in these studies is longer and intensive. This helps to build a strong rapport with the community, hence enabling to capture the field as naturally as possible. These characteristics provide an ample scope to take care of quality and validity of data. However, in applied situations, data collection is often a truncated activity. This robs away a number of taken-for-granted strengths of traditional qualitative research methods: No time is spent on rapport building; holism is left behind, instead we engage in selection; we focus narrowly on specific phenomenon of concern, divorced from its context; analysis does not evolve out of an iterative process. In this paper, we aim to discuss some of the issues related to rigor and quality of such studies and strategies available to address them.

Catherine Pope

Wawan Yulianto

Despite a growing interest in quaLitative research in occupationaL therapy, littLe attention has been pLaced on establishing its rigor. This article presents one modeL that can be used for the assessment of trustworthi-ness or merit of qualitative inquiry. Cuba's (1981) modeL describes four generaL criteria for evaLuation of research and then defines each from both a quantitative and a qualitative perspective. SeveraL strategies for the achievement of rigor in qualitative research usefuL for both researchers and consumers of research are described.

Journal of Advanced Nursing

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024