Cognitive Interview Technique

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, Ph.D., is a qualified psychology teacher with over 18 years experience of working in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

On This Page:

Findings concerning the unreliability of eyewitness  accounts have led researchers to attempt to devise methods for improving retrieval.  One of these methods is the cognitive interview (Fisher & Geiselman, 1992).

The cognitive interview (CI) is a questioning technique used by the police to enhance the retrieval of information about a crime scene from the eyewitness’s and victim’s memory.

Geiselman et al. (1985) developed the Cognitive Interview (CI) as an alternative to the Standard Interview.It takes into account psychological findings about cue-dependent forgetting and has four stages designed to stimulate as many cues as possible in order to maximize different retrieval routes.

  • Stage 1 : Reinstate the context
  • Stage 2 : Recall events in reverse order
  • Stage 3 : Report everything they can remember
  • Stage 4 : Describe events from someone else’s point of view

Because our memories are made up of a network of associations rather than discrete and unconnected events, there are several ways that these memories can be accessed. The cognitive interview exploits this by using multiple retrieval strategies.

The cognitive interview involves a number of techniques/mnemonics:

Mental Reinstatement of Environmental and Personal Contexts

The interviewer tries to mentally reinstate the environmental and personal context of the crime for the witnesses, perhaps by asking them about their general activities and feelings on the day.  This could include sights, sounds, feelings and emotions, the weather etc.

In the interview, witnesses are often asked to use all of their five senses in their recollection of the event. This can help in recreating the event clearly in their mind and may trigger the recall of context-dependent memories.

Reporting the Event from Different Perspectives

Witnesses are asked to report the incident differently, describing what they think other witnesses (or even the criminals themselves) might have seen.

Describing the Event in Several Orders

Recounting the incident in a different narrative order.  Geiselman and Fisher proposed that due to the recency effect, people tend to recall more recent events more clearly than others. Witnesses should be encouraged to work backwards from the end to the beginning.

When events are recalled in forward order, witnesses reconstruct based on their schemas, this might lead to distortion. If the order is changed, they are more accurate as they are less likely to use their schemas.

In-depth Reporting

Witnesses are asked to report every detail, even if they think that detail is trivial. In this way, apparently unimportant detail might act as a trigger for key information about the event.

It is believed that the change of narrative order and change of perceptive techniques aid recall because they reduce witness’ use of prior knowledge, expectations or schema.

A psychology laboratory experiment conducted by Geiselman, Fisher, MacKinnon, and Holland (1985) compared the cognitive interview with a standard police interview and hypnosis.

Geiselman (1985)

Aim : Geiselman et al. (1985) set out to investigate the effectiveness of the cognitive interview.

Method : Participants viewed a film of a violent crime and, after 48 hours, were interviewed by a policeman using one of three methods: the cognitive interview; a standard interview used by the Los Angeles Police; or an interview using hypnosis. The number of facts accurately recalled and the number of errors made were recorded.

Results : The average number of correctly recalled facts for the cognitive interview was 41.2, for hypnosis it was 38.0 and for the standard interview it was 29.4.  There was no significant difference in the number of errors in each condition.

Conclusion : The cognitive interview leads to better memory of events, with witnesses able to recall more relevant information compared with a traditional interview method.

Cognitive Interview Evaluation

One limitation is the cognitive interview is that it’s time-consuming to conduct and takes much longer than a standard police interview. A rapport must be established, and it requires specialist training.

Kebbell and Wagstaff (1999) found many police officers did not use the CI technique in less serious crimes as they did not have the time. The CI may produce a vast amount of information but it may not always be practical or helpful in terms of allocating the police to efficiently investigate incidents.

It is also time-consuming to train police officers to use this method. This means that it is unlikely that the “proper” version of the cognitive interview is used.

Another limitation is that some elements of the cognitive interview may be more valuable than others. For example, research has shown that using a combination of “report everything” and “context reinstatement” produced better recall than any of the conditions individually.

Geiselman (1985) set out to investigate the effectiveness of the cognitive interview. Participants viewed a film of a violent crime and, after 48 hours, were interviewed by a policeman using one of three methods: the cognitive interview; a standard interview used by the Los Angeles Police; or an interview using hypnosis.

The number of facts accurately recalled and the number of errors made were recorded. The average number of correctly recalled facts for the cognitive interview was 41.2, for hypnosis it was 38.0 and for the standard interview it was 29.4.

In a real-life test, Fisher et al. (1990) trained detectives from the Miami Police Department to use the cognitive interview. Police interviews with eyewitnesses and victims were videotaped and the total number of statements was scored. A second eyewitness was then asked to confirm whether these were true or false.

Compared to the standard procedure used, the cognitive interview produced 46% increase in recall and 90% accuracy. The findings suggested that the cognitive interview is more effective than the standard interview, producing higher recall and reducing errors.

The cognitive interview is useful when interviewing older witnesses. Wright and Holliday (2007) found that the older the participant, the less complete and accurate the recall but when they used the CI technique, the older participants recalled significantly greater detail without giving any false information.

Therefore, the CI can be used to ensure that all eyewitness testimony is as accurate as possible to avoid a possible age bias on recall.

Fisher, R. P., Chin, D. M., & McCauley, M. R. (1990). Enhancing eyewitness recollection with the cognitive interview. National Police Research Unit Review, 6 (3), 11.

Fisher, R. P., & Geiselman, R. E. (1992). Memory enhancing techniques for investigative interviewing: The cognitive interview . Springfield, IL: Charles C. Thomas.

Geiselman, R. E., Fisher, R. P., MacKinnon, D. P., & Holland, H. L. (1985). Eyewitness memory enhancement in the police interview: Cognitive retrieval mnemonics versus hypnos is. Journal of Applied Psychology , 70[2], 401-412.

Kebbell, M. R., Milne, R., & Wagstaff, G. F. (1999). The cognitive interview: A survey of its forensic effectiveness . Psychology, crime and law, 5(1-2) , 101-115.

Wright, A. M., & Holliday, R. E. (2007). Interviewing cognitively impaired older adults: How useful is a Cognitive Interview ? Memory, 15 (1), 17-33.

Further Reading

Memon, A., & Higham, P. A. (1999). A review of the cognitive interview. Psychology, Crime and Law, 5(1-2), 177-196.

Memon, A., & Bull, R. (1991). The cognitive interview: Its origins, empirical support, evaluation and practical implications. Journal of Community & Applied Social Psychology, 1(4), 291-307.

Köhnken, G., Milne, R., Memon, A., & Bull, R. (1999). The cognitive interview: A meta-analysis. Psychology, Crime and Law, 5(1-2), 3-27.

Print Friendly, PDF & Email

Police tape outside a house in Carnarvon, WA

Cleo Smith case: how ‘cognitive interviewing’ can help police compile the most reliable evidence

cognitive interview case study

Associate professor in Criminology and Justice Studies, RMIT University

Disclosure statement

Georgina Heydon does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

RMIT University provides funding as a strategic partner of The Conversation AU.

View all partners

With Cleo Smith safely returned to her family, and charges laid against her alleged abductor, much of the media’s focus has now turned to the question of how police can best piece together the details of what exactly happened to her during the 18 days she was missing.

Child psychologists have described how, with appropriate questioning techniques, a four-year-old can indeed recall and describe detailed memories of their recent experiences. But what about the questioning of the relevant adults, whose recall might also not be perfect, or who might be unwilling to volunteer the necessary information?

Movies and television shows often show police using tough, aggressive questioning techniques on suspects. But the evidence suggests a calm, measured and open approach can help interviewers collect the best information with which to build a prosecution.

Perhaps surprisingly, that means the techniques used to compile evidence from Cleo herself may not be all that different from an interview with an adult witness, or even with Terence Darrell Kelly, the 36-year-old man now charged with her abduction.

Building the case

Child interviewing techniques are based on research about memory and cognition. Creating a safe and comfortable space for the child will help them recall crucial details, and every care is taken to avoid “contaminating” the evidence by asking leading questions. Asking if a child saw “some dolls” when the child has previously only mentioned “some toys” would introduce new information into the child’s narrative, making it hard for them to be sure what they have remembered rather than what was suggested to them.

Young children such as Cleo are also still learning language skills, so the interview questions must also take account of their level of linguistic development. In a question that uses the passive voice, such as “was the dog attacked by the man”, children younger than about seven years will struggle to figure out who is doing the attacking and who is being attacked. Their answers to these types of question can be very unreliable. Multi-part questions (“who was there and what were they wearing”) and forced-choice questions (“was the car white or grey?”) are also problematic.

Cleo Smith pictured during her rescue by police

Of course, some of these considerations apply equally to adults.

The best-practice models of police investigative interviewing are those developed in the United Kingdom during the 1990s, called “ cognitive interviewing ”. This approach involves five phases, known by the mnemonic “PEACE”:

Preparation and Planning - assembling all the available information and preparing an appropriate strategy and questions. This happens before the interview begins.

Engage and Explain - the first part of the interview, which involves building rapport and explaining the purpose and format of the interview.

Account - the main body of the interview, in which the interviewee has an opportunity to give their narrative, and interviewers ask questions for further detail.

Closure - confirming the information collected with the interviewee, and providing further information and support.

Evaluation - after the interview is complete, the interviewers assess the information gathered, the quality of the interview, and the next steps.

This approach emphasises the use of open questions, such as “tell me everything that happened on the night Cleo went missing – even little things you think aren’t important”.

The second part of this question is designed to counter people’s natural tendency not to include lengthy and detailed descriptions in response to questions in ordinary conversation.

Police interviewing puts a strong emphasis on the elicitation of a free narrative at the start of the interview, and at any point where new information is sought. All possible effort must be made to encourage the suspect to tell their own story in their own words, before probing questions are used to clarify the details and prepare for the strategic presentation of evidence.

Closed questions with a limited range of possible answers, such as “who were you with?” or “what was he wearing on his head?”, and yes/no questions such as “did you go to Cleo’s tent?” are regarded as last-resort options .

Adults vary enormously in their language abilities, and a skilled interviewer will adapt their vocabulary, grammar and questioning style as appropriate. This might mean asking simpler questions, but also using a similar level of informality, and the same terminology, that the interviewee uses. This can build rapport and promote clear communication, which is especially important in relation to taboo topics such as body parts, or other idiosyncratic words like names.

Some interviewees might also have special communication needs, which might require a language interpreter or support person to be present.

Read more: When missing children return: how can we avoid adding to Cleo Smith's trauma?

It’s not like the movies

Television and movie convention tells us that police interviewing is forceful and even coercive, particularly in strongly emotional cases such as an alleged child abduction.

This trope is based on the real-life Reid technique , widely used in the United States since the 1960s. Not dissimilar to the proverbial “good cop, bad cop” routine, this approach aims to put extreme pressure on any subject deemed uncooperative or untruthful, before offering sympathy and understanding if a confession seems to be forthcoming.

Unsurprisingly, researchers have raised concerns that this technique generates high rates of false confessions.

According to a growing weight of evidence, it’s now clear the best way to interview an adult, even someone accused of child abduction, is to draw on the same principles that underpin child interviewing.

To achieve the best results – that is, a judicial verdict based on fair and just processes and high-quality evidence – investigators will use interviewing techniques that avoid police “contamination” of the evidence, and focus instead on reliable, narrative-based contributions from the suspect.

When the stakes are high, the principles of investigative interviewing established by the UK Home Office offer the best guidance. The focus is on respecting the interviewee’s rights, gathering high-quality, unbiased and reliable information, and following due process in the hopes of maximising the chances of securing a conviction if appropriate.

  • Eyewitness testimony

cognitive interview case study

Research Assistant in Immunology and Virology

cognitive interview case study

General Manager | La Trobe University, Sydney Campus

cognitive interview case study

Lecturer / Senior Lecturer - Business Law & Taxation

cognitive interview case study

Industrial Officer (Senior)

cognitive interview case study

Supply Chain Management – Open Rank (Tenure-Track)

  • Search Menu
  • Advance articles
  • Author Guidelines
  • Submission Site
  • Open Access
  • Why Submit?
  • About Public Opinion Quarterly
  • About the American Association for Public Opinion Research
  • Editorial Board
  • Advertising and Corporate Services
  • Journals Career Network
  • Self-Archiving Policy
  • Dispatch Dates
  • Journals on Oxford Academic
  • Books on Oxford Academic

Article Contents

Alternative paradigms of cognitive interviewing: thinking-aloud and probing, specific parameters in design and implementation: who to interview, and how to probe, evaluating evidence from cognitive interviews.

  • < Previous

Research Synthesis: The Practice of Cognitive Interviewing

paul c. beatty National Center for Health Statistics, Centers for Disease Control and Prevention, 3311 Toledo Road, Room 3218, Hyattsville, MD 20782.

gordon b. willis National Cancer Institute, National Institutes of Health, 6130 Executive Blvd., MSC 7344, EPN 4005 Bethesda, MD 20892-7344.

  • Article contents
  • Figures & tables
  • Supplementary Data

Paul C. Beatty, Gordon B. Willis, Research Synthesis: The Practice of Cognitive Interviewing, Public Opinion Quarterly , Volume 71, Issue 2, Summer 2007, Pages 287–311, https://doi.org/10.1093/poq/nfm006

  • Permissions Icon Permissions

Cognitive interviewing has emerged as one of the more prominent methods for identifying and correcting problems with survey questions. We define cognitive interviewing as the administration of draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response or to help determine whether the question is generating the information that its author intends. But beyond this general categorization, cognitive interviewing potentially includes a variety of activities that may be based on different assumptions about the type of data that are being collected and the role of the interviewer in that process. This synthesis reviews the range of current cognitive interviewing practices, focusing on three considerations: (1) what are the dominant paradigms of cognitive interviewing—what is produced under each, and what are their apparent advantages; (2) what key decisions about cognitive interview study design need to be made once the general approach is selected (e.g., who should be interviewed, how many interviews should be conducted, and how should probes be selected), and what bases exist for making these decisions; and (3) how cognitive interviewing data should be evaluated, and what standards of evidence exist for making questionnaire design decisions based on study findings. In considering these issues, we highlight where standards for best practices are not clearly defined, and suggest broad areas worthy of additional methodological research.

Developing and evaluating questions has always been one of the key challenges for survey researchers. Beginning in the 1980's, cognitive interviewing has emerged as one of the more prominent methods for identifying and correcting problems with survey questions. Numerous academic survey centers, government agencies, and commercial research firms have incorporated cognitive interviews into their usual procedures for questionnaire development, and some organizations (e.g., the Census Bureau, National Center for Health Statistics, Bureau of Labor Statistics, Westat, RTI International, and Abt Associates) have created permanent questionnaire design laboratories to facilitate this practice. Cognitive interviewing was a prominent topic at the 2002 conference on Questionnaire Design, Evaluation and Testing, and the volume resulting from the conference describes several facets of this activity (Presser et al. 2004 ). In addition, a book by Willis ( 2005 ) contains an extensive review of the methodology and serves as a practical guide for carrying out cognitive interviewing projects.

In spite of these developments, there does not appear to be a succinct and commonly accepted definition of what cognitive interviewing 1 consists of, or consensus regarding its best practices (Presser et al. 2004 ). A basic definition proposed by Beatty ( 2003 ) that seems to reflect its most common application is that cognitive interviewing entails administering draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response or to help determine whether the question is generating the information that its author intends. But beyond this general categorization, cognitive interviewing potentially includes a variety of activities that may be based on different assumptions about the type of data that are being collected and the role of the interviewer in that process. For example, the verbal material generated by such interviews could consist of (1) respondent elaborations regarding how they constructed their answers, (2) explanations of what they interpret the questions to mean, (3) reports of any difficulties they had answering, or (4) anything else that sheds light on the broader circumstances that their answers were based upon. This material could be based on explicit follow-up questions (probes) from an interviewer, or based on general instructions to “think out loud” as much as possible. The interviewer herself could range from a relatively unskilled data collector to an expert investigator; the interview could be based on a scripted protocol, be semi scripted, or largely improvised based on the issues that emerge from discussion. Analysis may be based on systematic review of interview transcripts, or entirely from notes taken during the interview [more detailed discussion of the potential varieties of cognitive interviewing practice are provided by Willis 2005 and Conrad and Blair 2004 ].

Given such variety, it may be difficult to understand what someone means when claiming to have conducted cognitive interviews. This is certainly a problem for consumers of cognitive interview findings, but a lack of consensus on objectives, procedures, or even general terminology can also inhibit methodological development. Providing a “foundation for optimal cognitive interviewing,” as called for by Presser et al. ( 2004 , p. 115) seems to require both an understanding of current cognitive interviewing practices, and an analysis of areas where best practices are unclear. This synthesis has been written to serve as a key element of that foundation, focusing on three key considerations: (1) what are the dominant paradigms of cognitive interviewing—what is produced under each, and what are their apparent advantages; (2) what key decisions about cognitive interview study design need to be made once the general approach is selected (e.g., who should be interviewed, how many interviews should be conducted, and how should probes be selected), and what bases exist for making these decisions; and (3) how cognitive interviewing data should be evaluated, and what standards of evidence exist for making questionnaire design decisions based on study findings.

All forms of cognitive interviewing entail administering survey questions to a participant 2 while collecting additional verbal information relevant to survey responses. Beyond that, practices seem to be based on two primary paradigms. One involves a cognitive interviewer whose role is to facilitate participants’ verbalization of their thought processes, but to intervene as little as possible in generating this verbal information. The other involves an interviewer who guides the interaction more proactively, generally asking additional, direct questions about the basis for responses. The former paradigm is rooted in the think-aloud procedure, in which interviewers encourage participants to verbalize thoughts while answering questions (e.g., “tell me what you are thinking … how are you coming up with your answer to this?”). The latter paradigm is rooted in the practice of intensive interviewing with follow-up probes (e.g., “Can you tell me in your own words what that question was asking?”)

THE “PURE” THINK-ALOUD AND NON-INTERVENING COGNITIVE INTERVIEWER

Although the definition of cognitive interviewing provided earlier is quite broad, its original paradigm was both more specific and more explicitly psychological. The initial impetus for cognitive interviewing came from an interdisciplinary seminar on the Cognitive Aspects of Survey Methodology, often referred to as the first “CASM” meeting (and summarized in Jabine et al. 1984 ). Loftus ( 1984 ), elaborating upon ideas presented at this meeting, proposed that a psychological research technique known as protocol analysis could be adapted as a pretesting methodology for survey questions. The blueprint for this technique was developed by Ericsson and Simon ( 1980 , expanded in 1993) and relies upon concurrent think-aloud reports. Think-aloud reports were used to yield insights into the thought processes involved in participants’ completion of certain tasks in a laboratory setting. The validity of the procedure assumes that these reports are available, can be accurately reported, and that reporting them does not fundamentally change the activities that participants reported about. Ericsson and Simon argued that these assumptions can often be met, and Loftus suggested that think-alouds yielded information about how participants tended to retrieve memories (e.g., of medical visits). Such data could be used to develop questions that reflected these retrieval strategies. For example, she suggested defining the reference period of recall questions from a past date up to the present, rather than from the present backwards.

Early papers on cognitive laboratory methods (e.g., Royston et al. 1986 ) suggest that initial cognitive interviews were based heavily, if not exclusively, upon instructions to participants to think-aloud as they thought about and answered survey questions. In practice, this meant that cognitive laboratory participants were asked to report what they were thinking while answering, and interviewers simply reminded respondents to continue providing such information as necessary. Think-aloud responses were the dominant data produced in the interviews, and interviewer behavior was constrained accordingly. It is probably not surprising that this was the original paradigm of cognitive interviewing, since it closely followed the methodology of protocol analysis that served as its basis.

AN ALTERNATIVE PARADIGM: INTERVIEWERS ASKING DIRECT QUESTIONS TO EXPLORE THE SURVEY RESPONSE PROCESS

At some point, an alternative paradigm of cognitive interviewing emerged that expanded upon the use of “pure” think-alouds—in particular, allowing for the addition of direct probing by the interviewer. Several precedents involving “intensive interviewing” had emerged independent of the first CASM meeting—for example Streett ( 1983 ) had described “frame of reference” questions about how respondents interpreted terms, which could be administered within pretests. In addition, Converse and Presser ( 1986 ) proposed “participating pretests,” in which respondents would be specifically told that the interview is evaluative and asked to explain their answers; scripted and unscripted probes might be used. Both works cite Belson's ( 1981 ) work as a precedent, in which he probed about respondent interpretations of questions and the circumstances surrounding their responses in an effort to identify reporting errors. Apparently, the distinction between true think-aloud interviews and “intensive interviews” became blurred at some point, with both eventually falling under the header of “cognitive interviewing.” It is easy to imagine how this could have occurred, especially given that intensive probes were often “cognitive”—addressing how terms were interpreted, how participants remembered certain facts, whether answers fit into available response categories, and so on.

This probing-based paradigm appears to have evolved gradually. Although some early descriptions of cognitive interviewing (Bercini 1992 ; Forsyth and Lessler 1991 ; Royston 1989 ) focus on think-alouds as the dominant component of cognitive interviewing, others (Royston and Bercini 1987 ; Willis, Royston, and Bercini 1991 ) suggested that both think-alouds and probing could be viable alternatives. Later, Willis ( 1994 ) proposed putting a greater emphasis on probing, primarily based on the observation that thinking-aloud seems awkward and burdensome for many participants. DeMaio and Rothgeb ( 1996 ) then proposed that cognitive interviews could include interviewer-guided activities such as probes about comprehension, confidence ratings, and requests to paraphrase questions in the absence of thinking-aloud. Several other articles suggest that the trend toward acceptance of such activities continued; Gerber and Wellens ( 1997 ) noted that cognitive interviewing had seemed to evolve from its original form to include “more probes and probes about meaning than was originally intended” (p. 35). O’Muircheartaigh ( 1999 ), along the same lines, suggests that cognitive interview practice had “diverged substantially” from the original paradigm derived from Ericsson and Simon. Willis, DeMaio, and Harris-Kojetin ( 1999 ), noting that cognitive interviews are often called “think-aloud interviews,” recommended that the latter term should be used more sparingly because think-aloud protocols were not necessarily the dominant component of cognitive interviewing as it was then practiced.

In short, many practitioners of cognitive interviewing began to collect verbal material other than “pure” think-alouds and, at least in some cases, empowered the interviewer to guide interviews based on individual content. This is not to say that the use of think-alouds was abandoned, as virtually all descriptions of cognitive interviewing mention think-alouds as one possible component (see DeMaio and Landreth 2004 ; Willis 2005 ), and some researchers (e.g., Conrad, Blair, and Tracy 2000 ) have continued to favor that approach. However, this new paradigm owed relatively little allegiance to the procedures for verbal protocol analysis proposed by Ericsson and Simon. Rather, it appears to have emerged for pragmatic reasons—some researchers gravitated toward this paradigm simply because it gave them additional useful information.

AN ASSESSMENT OF THE TWO PARADIGMS

The goal under both probing and think-aloud paradigms is to generate verbal information that is usually unseen in a survey interview in order to evaluate how well the questions are meeting their objectives. This puts interviews from both paradigms on an important common ground. Yet they are carried out differently and are based on different assumptions, which may have important implications regarding the nature of the data that they generate. Advocates of the think-aloud paradigm propose that it has several advantages. One is that the procedures are relatively standardized, reducing the chances that interviewers could introduce bias into the data collection process; another is that interviewers do not need to be knowledgeable about questionnaire design or the objectives of specific questions (Bolton and Bronkhorst 1996 ). Conrad, Blair, and Tracy ( 2000 ) note that interviewer probing can create artificiality (e.g., changing content and flow), whereas these problems are presumably avoided in think-aloud interviews. This artificiality is the major reason why Oksenberg, Cannell, and Kalton ( 1991 ) proposed that probing, when used, should follow only a few questions per interview. It is possible that apparent problems with a survey question could be products of a unique interaction between cognitive interviewers and participants, rather than “real” questionnaire problems (Beatty, Willis, and Schechter 1997 ).

Forsyth and Lessler ( 1991 ) and van der Veer, Hak, and Jansen ( 2002 ) propose an additional advantage: because think-aloud data are collected during the response process, they have a certain purity that probe responses (provided after responding) do not. However, a considerable body of research beginning with Nisbett and Wilson ( 1977 ) calls into question whether think-alouds are literal reflections of thought processes. More likely, they are re-constructions, although they are likely to reflect actual processes to some degree (Wilson, LaFleur, and Anderson 1996 ). Furthermore, as Willis ( 2004 ) notes, Ericsson and Simon ( 1980 ) did not insist upon exclusive use of thinking-aloud: their crucial point was that self-reported information should be in short-term memory . From that perspective, reports based on probes immediately following questions are probably not much different than think-aloud reports. Also, the practical value of think-aloud reports for questionnaire designers may not depend upon literal accuracy. For example, DeMaio, Ciochetto, and Davis ( 1993 ) determined through verbal protocols that some dietary recall questions forced participants to think chronologically, which was difficult for many. Revised questions focused on what was eaten rather than when it was eaten, freeing participants to use whatever recall strategy worked best for them. Whether or not the participants reported literal thought processes, the material they provided highlighted why one version of the question was difficult, and suggested a viable alternative.

The think-aloud procedure generates information that can be useful to questionnaire designers, can be administered with only modest training, and appears to avoid problems of artificiality that could be created through probing. However, Willis ( 2005 ) also reviews evidence that some cognitive interview participants perform think-alouds poorly. Some psychological researchers acknowledge variation in ability to perform this task, but suggest that it does not seem to be correlated with any other observable variables (van Someren, Barnard, and Sandberg 1994 ). The extent to which this varying ability limits the effectiveness of think-alouds for evaluating questionnaires is not completely clear.

In addition to overcoming this potential weakness, advocates of the more probing-centered paradigm claim that probing offers several advantages. For example, Willis ( 1994 , 2005 ) suggests that probing provides focus to the participant's behavior. Given that participants may diverge onto irrelevant tangents when relying entirely upon general instructions to articulate thought-processes, he suggests that carefully selected probes help to focus attention on pertinent issues. Of course, such probing requires interviewer judgment regarding both what the most pertinent issues are, and what probes are most appropriate to return attention to those points. This is important, because it is not the use of probes per se that regains control of the interview, but an interviewer skilled at using the “right” probes. The implication of Willis’ suggestion is that interviewers should retain discretion over interview content; however, they may also require special expertise to wield that discretion effectively.

Another potential advantage of probing is that theoretically it should not interfere with the actual process of responding, whereas thinking-aloud might. Although probe responses are likely to be quite similar to think-aloud reports (see above), procedures for obtaining them are different; in the think-aloud case, participants at least attempt to provide some verbal information during the response process. Although Ericsson and Simon ( 1980 ) claim that thinking-aloud probably does not interfere with the response process, Russo, Johnson, and Stephens ( 1989 ) found that it had an impact on the accuracy of various mental computations; furthermore, Willis ( 1994 ) argues that thinking aloud is likely to increase the effort spent on creating a response, which has an unknown impact on the response process. Probing may create less interference than thinking aloud, while still capturing information stored in short-term memory. (However, as noted previously, probing may interfere with the usual flow of the interview, potentially compromising the realism of questionnaire administration in a different manner.)

Perhaps the strongest justification for the probe-based paradigm is that it generates verbal material that questionnaire designers find useful, but that may not emerge unless a cognitive interviewer specifically asks for it. As Willis ( 2004 ) observes, think-aloud procedures were originally proposed to shed light specifically on retrieval processes. It is unclear whether think-aloud responses are useful for assessing comprehension problems, inadequate response options, or other questionnaire issues, whereas it is generally straightforward to probe directly about such matters. Similarly, Conrad, Blair, and Tracy ( 2000 ) note that think-alouds alone sometimes suggest a problem with a question but do not provide enough information to diagnose what the problem is. Probe responses might help to fill in this gap.

Although it is useful to present these paradigms as distinct, the boundaries between practices are probably not as precise as they once were, with both allowing for some degree of probing, albeit with some reservations. Advocates of the original paradigm (Conrad, Blair, and Tracy 2000 ) have conceded that probing makes important contributions, and advocates of the alternative (Beatty 2004 ) have acknowledged that probing can shape interview content in some undesirable ways. However, an important distinction remains regarding the expected role of the interviewer. The original paradigm employs an unobtrusive cognitive interviewer, who relies on standardized think-aloud protocols and possibly scripted probes. The alternative paradigm employs an active cognitive interviewer who is given more latitude to explore topics as they emerge within interviews. Thus, the practical decision has moved from whether or not to allow probes, to how much probing is appropriate, and whether this probing should be standardized or determined by interviewer judgment (or to what extent).

At this point it would be relatively easy to conclude, as does Willis ( 2005 ), that “In practice, think-aloud and verbal probing actually fit together very naturally” (p. 57), and that it is appropriate to call a truce such that practitioners adopt both methods. However, the appropriate relative weight to be put on either procedure may depend on specific factors relevant to testing. Optimal cognitive testing procedures may vary depending upon the age and cognitive ability of participants, topic of the questionnaire, or intended mode of administration. For example, Redline et al. ( 1998 ) evaluated a self-administered paper instrument with both think-aloud and purely retrospective approaches. They found that the methods produced similar results, except that participants with low educational levels tended to miss skip patterns under the think-aloud approach. This suggests that think-alouds interfered with normal navigation among some participants, making the method less desirable; however, this problem might not exist with computer-assisted instruments that handle skip logic. Furthermore, the increased prevalence of web surveys puts increased emphasis on self-administered visual modes. While a probing paradigm seems more common for interviewer-administered surveys, several researchers have considered whether different approaches might be best for self-administered questionnaires or web surveys (Bates and DeMaio 1989 ; Dillman and Redline 2004 ; Redline et al. 1998 ; Schechter, Blair, and Vande Hey 1996 ). It may be that navigational and other creative decisions involved in completing web surveys are more appropriately evaluated through think-alouds, and the need for such assessments could spur a resurgence of that technique (Willis 2005 ).

Although Forsyth and Lessler ( 1991 ), Willis ( 1994 ), and DeMaio and Rothgeb ( 1996 ), among others, have contributed significantly to establishing general descriptions of cognitive interviewing, the literature is generally not very detailed concerning many specifics regarding design and implementation of studies based upon this method. There does not appear to be general consensus regarding issues such as sample sizes needed for adequate testing, participant selection, the ideal background and training of interviewers, and choice of cognitive probes [although Willis ( 2005 ) does provide recommendations in these areas]. This section reviews some considerations along these lines, and suggests considerations that are useful in making decisions about specific approaches.

WHO TO INTERVIEW, AND HOW MUCH INTERVIEWING TO DO

Cognitive interviewing literature has paid surprisingly little attention to issues of appropriate composition and size of cognitive interviewing samples. Practitioners generally acknowledge that participants are chosen by convenience and that such samples are “not designed to be representative [of any larger population], but to reflect the detailed thoughts and problems of the few respondents who participate in [cognitive interviews]” (DeMaio et al. 1993 ). One clear consequence of such sampling is that cognitive interview practitioners cannot directly determine the extent of questionnaire problems in the population—they can only identify question characteristics that are believed to pose problems with some unspecified frequency. Other than that, the specific guidance that is available advocates demographic variety of respondents, and that participants should include people relevant to the topic of the questionnaire being tested (Willis 1994 , 2005 ).

Still, some sampling considerations could help to strengthen claims that a reasonably thorough effort has been made to identify the most pressing problems with a questionnaire. For example, participants can be selected to cover as much of a questionnaire's conceptual terrain as possible. If questionnaires include skip patterns that lead to various branches, the sample should be sufficiently diverse to explore as many of these different paths as possible. Whatever topic the questions focus on (e.g., health insurance), the sample should cover a variety of circumstances relevant to that topic (e.g., people with a variety of health insurance situations, including some with no insurance at all). Within those parameters, it also seems desirable to select participants representing some demographic variety. This does not ensure “representativeness,” but casting as wide a net as possible over varying circumstances maximizes the chances that discovery will be effective. Similarly, interviewing in multiple locations could improve the variety of circumstances that are captured in testing (Miller 2002 ).

It is also not clear whether cognitive interviewers select samples of adequate size. Current practices seem based on the assumption that the most critical questionnaire problems will be revealed by a small sample of relevant participants. However, Blair et al. ( 2006 ) found that in one study, significant questionnaire problems were uncovered even after 50 or more cognitive interviews, and that some of the most serious problems (as judged by external reviewers) were not identified until relatively late in the process. General guidance calls for cognitive interviews to be conducted in “rounds” that mostly commonly range between 5 and 15 interviews, which are ideally repeated following efforts to revise questions and eliminate problems (Willis 1994 , 2005 ; McColl 2001 ). This iterative approach seems useful, but it is not clear how many rounds are usually conducted given time and resource constraints. Even under ideal circumstances, this approach still leaves open the question of how researchers can determine that they have conducted enough rounds of interviews to warrant stopping the process. Some qualitative researchers make such decisions based on the idea of category saturation (Strauss and Corbin 1990 ). Put simply, this means that the researcher identifies groups of people most relevant to the study and conducts interviews with members of each until they yield relatively few new insights. While the total sample sizes generally used in cognitive interviewing might fall short of those required to reach the point where insights actually stop emerging, the general principle of operating based on diminishing returns may be useful. On the whole, it seems unlikely that typical sample sizes currently used for cognitive interviewing are sufficient to provide truly comprehensive insight into the performance of a questionnaire, and additional standards are needed to determine optimal sizes.

COGNITIVE INTERVIEWERS AS DATA COLLECTORS OR INVESTIGATORS

Tucker ( 1997 )—in a position largely consistent with the original paradigm discussed earlier—calls for much greater standardization of cognitive interview procedures. Without this, he argues that “effective manipulation [of variables] will be impossible … the notion of falsifiability has no meaning … [and] the conditions necessary for generalizability will be absent” (p. 72). Conrad and Blair ( 1996 ) similarly argue that “rigorous experimental methods that rely on objective quantifiable data” are preferable to “largely impressionistic methods” that they suggest are generally used in cognitive laboratories (p. 8). The methodological research agenda proposed by Conrad and Blair ( 2004 ) to evaluate the quality of verbal report data is based primarily on experimentation across alternative, clearly defined techniques; this would require relatively high standardization as well. Under these perspectives, creative contributions from interviewers that lead to nonstandardized behavior are undesirable and should be minimized. Interviewers would primarily serve as data collectors, and researchers would ideally determine issues they wished to probe about in advance. This puts a relatively high investigative burden on the front-end of the process (i.e., before data collection).

An alternative perspective is that interviewers themselves would serve as investigators, potentially making decisions about content and scope during data collection. Such interviews might have an exploratory character, being more attuned toward generating ideas about potential problems than determining their extent in the population. For example, Willis ( 1994 ) compares cognitive interviewers to “detectives” who rely at least partially upon improvisation in looking for clues about questionnaire problems. In subsequent work, he draws an analogy between cognitive and clinical interviews, which may be guided by intuition, experience, and flexibility (Willis 2004 , 2005 ). This perspective forgoes consistency across interviews in favor of freedom to explore issues that emerge in discussions with participants. Its advantage is that it allows interviewers to explore issues that might have been missed through more tightly scripted interviews.

Beatty ( 2004 ), for example, reported results from a test of the following question: “In the past 12 months, how many times have you seen or talked on the telephone about your physical, emotional, or mental health with a family doctor or general practitioner?” One participant answered “zero,” but answers to other questions suggested that he had received medical care recently. In response to a series of follow-up probes, the participant eventually explained that his answer was accurate because the question referred to talking on the phone —the word “seen” had been lost within the verbiage of the question. In subsequent interviews, interviewers found that this mistake was commonly made. The identification of this problem emerged specifically from interviewer improvisation based on careful attention to responses. Furthermore, this probing provided insight into the particular causes of the problem.

However, the decision to rely upon this sort of emergent probing places considerable trust in the interviewer's ability to notice potential problems and choose appropriate follow-up probes. There does not appear to be sufficient guidance about what sort of training or background is most appropriate to prepare such interviewers for this role, though Willis ( 2005 ) describes a developmental training process that emphasizes study of questionnaire design, observation of cognitive interviews, and “on-the-job” practice that is critiqued by experienced interviewers. Clearly, it is important for such interviewers to understand potential types of cognitive or communicative errors that could affect the accuracy of survey responses, and to have familiarity with various options for eliciting useful verbal material from participants. A thorough grounding in survey methodology would probably be useful. Interviewers would also need to understand the measurement objectives of the questions being tested.

In contrast, cognitive interviewers functioning largely as data collectors would not require this level of expertise. Like traditional survey interviewers (e.g., see Fowler and Mangione 1990 ), their role would be primarily to read pre-determined questions and follow instructions accurately. Detailed knowledge of survey errors and measurement objectives might be useful, but would not be necessary. Interviewers would not need to understand why think-alouds or probes were being administered, although they would need to exercise judgment in recognizing when participants had provided adequate think-aloud or probe responses.

Clearly, the use of interviewer-investigators requires a highly specialized work force that is likely to be more expensive and difficult to assemble than the alternative. Studies based on this paradigm are also likely to involve higher levels of interviewer variation and improvisation. In order to fully assess whether insights gleaned from such efforts outweigh the drawbacks of expense and lack of generalizability, additional research could explore what interviewers actually do under various paradigms of practice and how conclusions are reached. The results would help to make better choices about the costs and benefits of various approaches to cognitive interviewing.

In particular, cognitive interviewers increasingly face the need to conduct cross-cultural and multi-lingual testing, where monolingual staff simply cannot conduct the interviews. It will be necessary to establish means for either quickly training new cognitive interviewers to be proficient, or else to develop standardized testing protocols that require lower levels of proficiency. Kudela et al. ( 2004 ) describe such a cross-cultural cognitive testing project. Researchers were able to coordinate cognitive interviewing of a tobacco use questionnaire across several Asian languages as well as English, relying on relatively quickly trained interviewers who applied a standardized protocol (that is, where probes were completely pre-scripted). The overall results were somewhat reassuring; at least in part, a common set of problems emerged across cultures, languages, and sets of interviewers, indicating that some problems with survey questions appear to be universal, and that the separate cognitive interviewing teams independently identified these defects.

WHAT TO ASK: THE SELECTION OF PROBES

As discussed earlier, most recent conceptualizations of cognitive interviewing involve probing to some degree. If the interviewer is also an investigator, then she may select some of these probes herself; if a data collector, then the probes may be selected for her. But either way, someone must choose what probes are asked. Although the cognitive interviewing literature of the past 20 years has provided many examples of possible probes (e.g., Bercini 1992 ; DeMaio and Rothgeb 1996 ; Forsyth and Lessler 1991 ; Willis 1994 ), it is not clear whether particular probes are likely to be most effective for various purposes. Willis ( 1994 ) suggested that probes should not suggest a “correct” answer, a principle that also applies to survey questions. Foddy ( 1998 ) concluded that specific probes such as “what does [term] mean to you” are more successful than general ones such as “what were you thinking when you first answered the question? 3 Beatty ( 2002 ) found that participants answered probes about the meaning of terms differently when they were administered alone than they did within the context of a particular survey question. In general, however, these sorts of recommendations appear to be uncommon.

Cognitive interviewers may be able to obtain some guidance about how to choose “good” probes from literature on qualitative interviewing, which may include lessons on what to ask, how to ask it, and how to make sense of narrative data. For example, Weiss ( 1994 ) suggests that interviewers generate narrative by asking about specific events rather than generalized experience. Holstein and Gubrium ( 1995 ) encourage interviewers to be on the lookout for “confusion, contradictions, ambiguity and reluctance” as signs that “meanings are being examined, reconstituted, or resisted” (p. 79). In the case of cognitive interviews, such instances might call for additional probing. Variants of qualitative interviewing are also employed by anthropologists, and some guidance may be obtained from that field as well. For example, Gerber ( 1999 ) notes that anthropologists can explore whether terms are “culturally inappropriate” for a particular population. But rather than simply asking a participant what a term such as self-reliance means, an anthropologist might explore its meaning in different contexts, e.g., with regard to child rearing, older family members, or welfare recipients. This might suggest that general cognitive interview probes such as “what does this term mean to you” might be less effective than specific ones exploring how a term is used in a participant's life. Figure 1 suggests a probing classification scheme that attempts to systematically organize the various major categories of probes, and to indicate the circumstances under which each is potentially most beneficial.

Model of verbal probing in the cognitive interview (from Willis 2005 ).

This model of probing distinguishes two major dimensions: (1) whether probes are searching in nature (proactive) as opposed to responsive (reactive), and (2) whether they are fashioned ahead of the interview (standardized) or during its course (nonstandardized). The 2 × 2 combination of these two major dimensions produces four key probing variants:

Anticipated probes are those that are scripted, or at least roughly configured, based on the anticipation of a problem with the question.

Spontaneous probes are flexible in that they are not scripted in advance. However, these probes are not based on any particular response from participants—they derive from interviewers who decide to search for potential problems on their own initiative.

Conditional probes have been introduced by Conrad and Blair ( 2004 ); these are pre-determined probes that are applied only if triggered by particular participant behaviors (e.g., if a participant hesitates, the protocol calls for the interviewer to say: “You took a little while to answer that question—what were you thinking about?”).

Emergent probes are flexible, unscripted, and reactive; the interviewer selects such probes in response to something that a participant says (e.g., something that indicates an apparent problem).

Willis ( 2005 ) discusses the various situations that might give rise to the use of each probe category, the relative benefits and drawbacks of each, and the relative benefits of structured versus flexible approaches to probing. In particular, he proposes that proactive varieties of probing are most useful when problems have been anticipated, yet the participant gives no indication of having the problem until probing is done (i.e., to detect cases of “Silent Misinterpretation”, as phrased by DeMaio and Rothgeb 1996 ). Reactive probing is presumably most appropriate in the opposite case: where problems were unanticipated, yet the participant does indicate some difficulty (e.g., a long silence). It is likely that multiple varieties of probing are appropriate within the same interview, depending on the mix of problems either expected or encountered.

To lend more specificity to these arguments, researchers might put this notion to the test. This could be done by paying careful attention to the nature of probing that is either proactive or reactive—or structured versus unstructured—within cognitive interviewing studies, mainly through careful review of cognitive protocols and interview recordings. Given the sets of (anticipated) probes that were both fashioned and administered, which resulted in the detection of apparent problems, and how often? Conversely, how often did subjects indicate the presence of problems on their own, how often were these followed up by reactive forms of probing, and what types of problems were then identified? A compilation of such basic data would be extremely useful in establishing best practices.

Whether cognitive interviews are conducted based on a fairly standardized protocol or with greater interviewer flexibility, the major product is still verbal text that needs to be evaluated to determine whether or not a question poses a problem for respondents. One advantage of more standardized protocols is that they generate data more amenable to systematic coding and analysis. For example, Conrad and Blair ( 1996 ) propose that verbal protocols be coded in a table with “types of problems” on one axis (lexical, temporal, logical, etc.), and “response stage” (understanding, task performance, and response formatting) on the other. Once criteria are established to indicate the presence of a problem, cognitive interview data can be objectively coded to determine if the criteria are met.

As noted earlier, some practitioners propose that additional, unscripted probing from a skilled investigator brings enough additional material to the surface to justify the lack of standardization. However, the resulting variation in data across questions and interviewers can complicate analysis. Interview content can vary considerably, with some questions or issues given relatively little attention and others pursued in depth. The presence or absence of verbal reports can be attributable to either questionnaire problems or interviewer discretion, making it difficult to establish objective criteria of problems. Nevertheless, we suggest that analysis can be based on whether apparent problems can be logically attributed to question characteristics. For example, consider the tested survey question “Thinking about your physical health, which includes physical illness and injury, for how many days during the past 30 days was your physical health not good?” Schechter, Beatty and Willis ( 1998 ) concluded that respondents in general have a difficult time answering this by the following process of reasoning:

Observing a response problem: several participants could not provide codeable responses (a number between zero and thirty), even when probed.

Considering the specifics: some participants indicated that there was no way to answer the question (“a day is part good and part bad—you can't characterize it as one or the other”); others complained about response task difficulty (“I don't do bookkeeping on this”) especially given complicated health pictures.

Identifying question characteristic that create the problem: the question assumes that a “day” is a reasonable metric, but it may not be for people with varying-quality days.

Evaluating generalizability: it seems reasonable that this problem could recur for people with multiple health problems difficult to keep track of, and for respondents with health that varies throughout the day.

A claim that this process found “proof” of the problem would overstate the evidence. However, a reasonable case could be made that respondents in similar circumstances would have similar difficulties responding, and that the difficulties are caused by a faulty assumption about the way individuals can characterize their health. Note also that the evidence is not linked to the number of participants who report a particular problem. Whether it takes many or a few participants to construct such an argument, it needs to be evaluated based on logical merits. It is conceivable that a solid argument about a questionnaire problem could be constructed around a single case, or that such an argument might fail to materialize around several cases.

ERROR IN COGNITIVE INTERVIEW ANALYSIS

As useful as cognitive interviewing may be, it can still lead to conclusions that are incomplete, misleading, or incorrect. There are several possibilities for error: cognitive interviews could identify problems that would not turn out to be “real” in surveys; cognitive interviews could fail to identify problems that exist in actual survey administration; and cognitive interview findings might be inconsistent when conducted by independent groups of researchers. The first two could be considered problems with the validity of the method and might be respectively classified as errors of commission and errors of omission . The third could be considered to be a problem with the reliability of the method.

In defending themselves against errors of commission, practitioners assume that cognitive interviewing finds problems that will carry over to actual surveys . Unfortunately, there is often no obvious way to verify that hypothesized problems are “real.” Logical arguments may have to do, and researchers will have to determine for themselves whether they find such arguments to be meritorious. However, there have been at lease two attempts to verify that cognitive interview findings were borne out by field data (Beatty, Fowler, and Cosenza 2006 ; Willis and Schechter 1997 ). Both studies administered several questions and revisions based on cognitive interview findings in a split ballot. Arguably, some revisions produced more plausible statistics (e.g., regarding hours spent doing “strenuous physical activity” per day). Such validation is potentially useful, but expensive.

As for errors of omission , there is no reasonable way that cognitive interview practitioners could claim to have found all problems with a questionnaire. As noted earlier, the method can make no claims that it has represented the population as a whole. Its usefulness is based on the assumption that the most egregious problems will become evident in most groups of participants who are reasonably appropriate to the topic of the survey, and interviewing often concludes based on a subjective judgment that interviews are yielding diminishing returns. There is always the possibility that one additional interview could yield a significant new insight, or that an additional interviewer would be more likely to notice additional problems. By the same token, claims that a questionnaire has “no problems” are impossible—the strongest claim that could be made is that no problems have (yet) been discovered.

Finally, there is always the possibility that independent groups of cognitive interviewers might not reach the same conclusions regarding a particular questionnaire (see DeMaio and Landreth 2004 ; Forsyth, Rothgeb, and Willis 2004 ). However, it would probably not be unusual for different groups to discover different insights, especially if interviewers were operating under an “investigator” paradigm. Under both the investigator and data-collector interviewer paradigms, there is an element of chance regarding who is interviewed, meaning that different insights might emerge from interviews, especially in earlier stages of a project. Differences could also be a function of varying backgrounds and sensitivities to various sorts of problems. It seems unlikely that cognitive interviewing would generate reliable findings in the sense that survey researchers might use the term (i.e., each set of interviews identifies the same set of problems with questions). When findings are different, yet not necessarily contradictory, this may indicate that no one set of findings is complete, and they need to be examined more closely. Findings that are difficult to reconcile might indicate either faulty reasoning by analysts, or that interviewing has not yet yielded an adequate understanding of responses associated with a question. The former case calls for a closer look at the data, while the latter indicates a need for continued data collection.

CONCEPTUALIZING THE PRODUCT OF COGNITIVE INTERVIEWING

Central to method evaluation is the choice of the particular outcome measure we choose to evaluate. Implicit in many discussions regarding cognitive interviewing is the assumption that it should help researchers develop measurably better survey questions—that is, it should be able to identify and eliminate problems until researchers have homed in on an “ideal” question wording. If held to such a standard, it is not clear that cognitive interviewing is successful. However, an alternative view is that cognitive interviewing should simply provide questionnaire designers with insights about the consequences of various questionnaire design decisions. These findings may not always point to a clearly superior version of a question. Rather than attempting to find the “right” way to ask a survey question, cognitive interviewing may be more suited to helping researchers assess tradeoffs —the advantages and disadvantages of asking questions in a certain manner.

For example, consider the question: “Are you currently being treated by a doctor for arthritis?” Cognitive interviewing has suggested that this question is simple for participants with either serious arthritis or no arthritis at all. However, it can be complicated for participants whose circumstances may not qualify as “current treatment” (e.g., those who had seen a doctor for arthritis pain over a year ago). Does this qualify as a “problem” with the question? If so, it might make sense to add more specific language than “current treatment.” However, doing so would make the question longer, more burdensome, and potentially confusing to respondents who did not have difficulty with the original version. Cognitive interviewing may be useful simply because it provides information to make such design decisions as logically as possible—indeed, it may be the most efficient method available for illuminating such issues. In that light, cognitive interviewing may be less suited to finding the “best” questions than to guiding “best informed” design decisions.

Maximizing the quality of insights derived by cognitive interviewing calls for more clearly established best-practices, which calls for additional research. Clearly, experimentation on variations of cognitive interview practice (or comparisons with other forms of pretesting) is one possibility. In a straightforward experiment, researchers decide upon a particular variant of cognitive interviewing, manipulate a characteristic of interest, and determine whether this manipulation affected some dependent measure. For example, the researchers may provide identical instructions to two sets of interviewers with different levels of experience, and evaluate whether the number or type of problems that they identified varies. Conrad and Blair ( 2004 ) document a case study of this type, finding (among other things) that a more “conventional” version of cognitive testing identified more problems than a restricted “conditional probe” version applied by less experienced interviewers; however, in the conventional probe version, there was less overall agreement with coders about the presence of problems.

Such experiments might be informative about the effect of manipulating particular variables, which could be especially useful for evaluations of relatively structured forms of cognitive interviewing. However, it is possible that the manipulations may have little resemblance to actual practice (a problem noted by Conrad and Blair 2004 ). This could be a particular problem in efforts to evaluate more qualitative forms of cognitive interviewing—i.e., those in which interviewer freedom to explore in an unpredicted (and possibly unpredictable) manner is a fundamental attribute of the method. It is difficult to use experiments to compare the yield of alternative qualitative methods, because the level of control required for such an evaluation makes the methods non-qualitative by definition. To the extent that cognitive interviewing is qualitative in nature, the usefulness of experimental manipulations of methodology may be limited by artificiality.

Another option for research is to study activities already conducted by cognitive interviewers. Such studies relinquish most or all experimental controls in order to maximize realism. For example, Presser and Blair ( 1994 ) compared results of cognitive interviewing with results from other forms of pretesting the same instrument; DeMaio and Landreth ( 2004 ) compared cognitive interview results produced by three separate research teams; Beatty ( 2004 ) studied what cognitive interviewers on one particular project actually did. All studies identified differences in results across methods or interviewers; however, the lack of experimental control makes it difficult to determine what specific characteristics of the study implementations were driving these differences. In fact, inherent differences in cognitive interviewing practice make it difficult for studies such as these to make firm conclusions about what the method does or does not accomplish in general. The same limitation applies to split-ballot studies (Beatty, Fowler, and Cosenza 2006 ; Willis and Schechter 1997 ) comparing response distributions from original and re-worked survey questions. The studies provide evidence that cognitive interviewing as practiced yielded useful results, but not all cognitive interviewing studies necessarily have equal utility. They also do not identify which attributes of cognitive interview practice were responsible for the usefulness of the results. Such studies can contribute an understanding of what one particular variant of cognitive interviewing produced—but only if the study makes clear exactly what was done in the cognitive interviews.

NEXT STEPS TOWARD DEVELOPING BEST PRACTICES

To close, we consider where the practice of cognitive interviewing seems the least well-specified, and suggest four areas where additional work should be particularly fruitful.

Determining optimal sample sizes for cognitive interviews . Recent work by Blair et al. ( 2006 ) should be replicated using different instruments, types of questions, groups of interviewers, and variants of cognitive interviewing practice. One potentially useful variation would be to employ an iterative testing approach, based on rounds of testing with questionnaire revisions between rounds. This approach is arguably accepted as an ideal practice, and it would be useful to see whether revised questionnaires are in fact “better,” and how rates of problem identification decline across revisions. Along the same lines, it would be useful to see how many interviews are required for independent groups of interviewers to reach consensus that the most significant questionnaire design issues had been identified.

Stronger guidance regarding data collection decisions . As mentioned previously, the prevalence of web and mixed-mode surveys call for revisiting decisions about the proper balance between thinking-aloud versus probing. Further studies involving self-administration that directly compare think-alouds with various probing approaches should follow Redline et al. ( 1998 ) in assessing a range of relevant behaviors, such as usefulness of the verbal reports in identifying various conceptual and navigational problems on the same instrument. Similarly, additional research should evaluate the efficacy of various probing choices given various types of questions, modes, and participants. Although some guidance for selecting probes is available (see Willis 2005 ), little empirical evidence is available to demonstrate which design decisions yield better data for questionnaire design decisions. For example, analyses described by Foddy ( 1998 ) and Beatty ( 2004 ) examined the relationship between interviewing probing and participant reports. Insights into the actual workings of cognitive interviewing projects could suggest which approaches seem to be most effective.

Studying in greater depth what actually happens in cognitive interviews . Although experimentation can be useful, we believe that greater methodological strides can be made through detailed analysis of processes in particular cognitive interviewing projects. That is, analysis could focus on what was done in a particular project, and how certain design decisions or interviewer actions led to certain conclusions, without necessarily attempting to control how studies were carried out. What such studies would lack in control they would make up for in realism. Additional research on previously conducted cognitive interviewing studies may be increasingly feasible with the forthcoming launch of the Q-Bank database, which will document which questions were tested, methods used in the testing, and findings from cognitive interviewing projects conducted in various federal agencies (Miller 2005 ).

Encouraging enhanced documentation of procedures . Perhaps the most significant impediment to the development of best practices is a lack of shared understanding of what cognitive interviewing projects have actually entailed. Given the large variety of practices covered under “cognitive interviewing,” better documentation is essential. Cognitive interview practitioners should agree upon a common set of key parameters that are specified within any cognitive testing report—such as the number of interviews, interviewers, and rounds of interviewing; the relative mix of think-aloud versus probing, the nature of probing (concurrent versus retrospective, whether proactive, reactive, or both); the recruitment methods, and so on.

Although Sirken and Schechter ( 1999 ) have argued that cognitive laboratory testing has “forever changed the attitude that questionnaire design is simply an art rather than a science” (p. 4), the truth may be that there is as much art as science in cognitive interviewing. That does not necessarily diminish its value. Nor does it necessarily matter whether the method is true to its explicitly psychological roots—as Presser ( 1989 , p. 35) concluded, if its application “does nothing more than increase the resources devoted to pretesting, it will have been all to the good.” The practice of cognitive interviewing has resulted in a considerable expansion in the time and energy devoted to developing large-scale survey questionnaires over the past two decades. However it is implemented, cognitive interviewing can put useful information into the hands of researchers who need to make difficult choices. Hopefully this review provides some perspective about the varieties of activities that are encompassed under the term, which may help researchers to understand—and to further explore—what is actually being produced in cognitive interview studies. More importantly, it will hopefully foster further discussions about best practices among practitioners. With such continued discussion, researchers should be better equipped to create questions that are clear, that pose memory and recall tasks that respondents can reasonably be expected to accomplish, and that allow respondents to express their answers accurately.

Earlier versions of this paper have benefited from the comments and encouragement of many people, including Duane Alwin, Bob Groves, Mick Couper, Roger Tourangeau, Norbert Schwarz, Jim House, Howard Schuman, and the anonymous reviewers. We also thank our many colleagues who have discussed various points raised here with us over the course of several years, especially Kristen Miller. Views expressed in this article are those of the authors, and do not necessarily represent the views of the National Center for Health Statistics, Centers for Disease Control and Prevention, or of the National Cancer Institute, National Institutes of Health.

This usage is to be distinguished from the “cognitive interview” described by Fisher and Gieselman ( 1992 ) used to extract information from event eyewitnesses.

We use the term “participant” in place of the word “respondent” to distinguish between those answering questions in a cognitive interview and an actual survey (although we note that the term “subject” is also sometimes used to refer to the former (e.g., Willis 2005 ).

It should also be noted that in this study, general probes were administered before specific probes for most questions. It is possible that the performance of specific probes was enhanced by “priming” from the general probes. It is also possible that results would vary depending on the criteria used to determine that probes were successful.

Google Scholar

Google Preview

Email alerts

Citing articles via.

  • Recommend to your Library

Affiliations

  • Online ISSN 1537-5331
  • Copyright © 2024 American Association for Public Opinion Research
  • About Oxford Academic
  • Publish journals with us
  • University press partners
  • What we publish
  • New features  
  • Open access
  • Institutional account management
  • Rights and permissions
  • Get help with access
  • Accessibility
  • Advertising
  • Media enquiries
  • Oxford University Press
  • Oxford Languages
  • University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

  • Copyright © 2024 Oxford University Press
  • Cookie settings
  • Cookie policy
  • Privacy policy
  • Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

  • Short report
  • Open access
  • Published: 14 February 2024

What do you think it means? Using cognitive interviewing to improve measurement in implementation science: description and case example

  • Zabin Patel-Syed   ORCID: orcid.org/0000-0002-0250-0848 1 ,
  • Sara Becker 1 ,
  • Miranda Olson 1 ,
  • Hailey Rinella 1 &
  • Kelli Scott 1  

Implementation Science Communications volume  5 , Article number:  14 ( 2024 ) Cite this article

325 Accesses

5 Altmetric

Metrics details

Pragmatic measures are essential to evaluate the implementation of evidence-based interventions. Cognitive interviewing, a qualitative method that collects partner feedback throughout measure development, is particularly useful for developing pragmatic implementation measures. Measure developers can use cognitive interviewing to increase a measure’s fit within a particular implementation context. However, cognitive interviewing is underused in implementation research, where most measures remain “homegrown” and used for single studies. We provide a rationale for using cognitive interviewing in implementation science studies and illustrate its use through a case example employing cognitive interviewing to inform development of a measurement-based care protocol for implementation in opioid treatment programs. Applications of cognitive interviewing, including developing a common language with partners and collecting multi-level feedback on assessment procedures, to improve measurement in implementation science are discussed.

Peer Review reports

Contributions to the literature

Measurement concerns in implementation science are among the most significant barriers to advancing the field.

Previous studies suggest that implementation measures are often used in single studies, high burden, and developed without partner input.

There remains limited guidance on methods to develop pragmatic measures.

To address this gap, we provide a brief overview of cognitive interviewing, a qualitative method that uses partner feedback throughout the measure development process.

Measurement issues in implementation science are among the most critical barriers to advancing the field [ 7 , 9 , 21 , 22 , 23 , 30 ]. Measures developed and tested in efficacy trials may not be feasible in service systems, and the widespread use of “homegrown” implementation measures limits generalizability of study findings [ 12 , 25 ]. Implementation science is especially vulnerable to measurement issues given the rapid growth of the field and the need for multi-level measurement in diverse health contexts (e.g., community mental health treatment, medicine, etc.) [ 31 ].

Measure development involves conceptualization (identifying measurement gaps and relevant constructs for a target population); development (generating measure content and administration procedures); and testing (assessing psychometric properties) [ 5 ]. Psychometric testing has received the most attention in the implementation science literature [ 20 , 26 ]. However, implementation partners—treatment developers, implementation researchers, community leaders—are unlikely to select measures based on psychometric evidence alone [ 13 , 14 , 29 ]. Emphasis must also be placed on a measure’s pragmatic qualities, goals for use, and translatability to clinical practice [ 34 ].

Glasgow and colleagues [ 13 ] recommended guidelines for pragmatic implementation measures. Based on a review of the literature, the authors noted that pragmatic measures have four key characteristics: importance to partners; low burden for respondents; actionable; and sensitivity to change. Extending this work, Stanick and colleagues [ 34 ] interviewed implementation science experts and identified the following three characteristics as priorities: integration with an electronic/health record, facilitation of guided action (e.g., selection of an intervention), and low-cost. This work contributed to the development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) for evaluating implementation measures [ 21 , 22 ]. However, there remains limited guidance on methods for developing pragmatic implementation measures to be used across different contexts.

Implementation measures must balance both psychometric and pragmatic quality. To attain this balance, we advocate that implementation scientists routinely use cognitive interviewing, a qualitative method that collects partner feedback throughout measure development [ 40 ]. Cognitive interviewing is uniquely suited to address measurement concerns in implementation science for four key reasons. First, implementation measures often evaluate efforts that engage diverse partners across multiple levels (patient, provider, organization) [ 1 , 35 ]. Cognitive interviewing can reveal whether measure content is relevant across partner groups and inform tailoring as needed. Second, cognitive interviews can help assess psychometric and pragmatic characteristics, including a measure’s construct validity, training burden, relevance, and usefulness across different contexts. Third, unique to implementation research, in which context is paramount [ 4 , 11 , 28 ], cognitive interviews can be used to collect partner feedback on measure administration procedures. Cognitive interviews can assess partner preferences for a measure delivery platform (e.g., electronic or paper), measure format (e.g., time, length, multiple choice versus free response), and strategies to integrate the measure with a clinical setting’s workflow (e.g., when, and how often to administer a measure), all of which can enhance a measure’s utility and scalability. Finally, collaborative research techniques like cognitive interviewing can be used to center partner perspectives, which can promote equitable partnership-building and increase buy-in [ 36 ].

To advance the development of psychometrically and pragmatically valid tools, we advocate for the widespread use of cognitive interviewing in implementation science studies. We first provide a detailed overview of cognitive interviewing theory and the stages of cognitive interviewing. We then provide a case example from an ongoing implementation trial to demonstrate how cognitive interviewing can be used to develop a pragmatic measure and to design a measure administration protocol [ 32 ]. We conclude with reflections on how cognitive interviewing can be used to improve measurement in implementation science.

Cognitive interviewing: overview of theory and techniques for use in implementation science

During a cognitive interview, implementation partners verbalize their thoughts as they evaluate measure questions and provide responses [ 2 , 40 ]. As the partner reads a measure aloud, an interviewer uses intermittent verbal probes to elicit their response process (concurrent interviewing) or has the partner verbalize their thoughts after completion (retrospective interviewing). Interviews may be used to identify constructs that partners value and consider important to assess (concept elicitation) or to revise an existing measure (debriefing). This method is used widely in other areas such as survey methodology and health outcome measurement (e.g., patient-reported outcomes in clinical trials), and by organizations like the United States Census Bureau [ 6 , 16 , 27 ] for measure development.

Cognitive interviews can be tailored to the goals of an implementation study. Given implementation research often includes a broad range of academic and community partners, interviews can be tailored for specific partner groups, to assess specific parts of a measure (e.g., instructions, terms, response options), to examine the relevance of the measure, or to evaluate administration procedures. In addition to its flexibility, cognitive interviewing can produce informative data even with small sample sizes (e.g., 5–10 interviews and a 15–30-min interview period) [ 40 ], which is particularly useful for resource-constrained implementation efforts.

Cognitive interviewing theory

Drawing on cognitive psychology, cognitive interviewing frameworks propose that a partner follows a four-stage mental model: (1) comprehension; (2) memory retrieval; (3) judgement; and (4) response [ 10 , 17 , 37 ]. At the comprehension stage, the goal is for the partner to interpret measure content (e.g., instructions, items, response options) as intended by the developer [ 39 ]. Misunderstandings may result from confusing or complex wording, missing information, inattention, and unfamiliarity with terminology. Measurement error due to comprehension issues [ 40 ] is especially likely in implementation science where it is well documented that users are often unfamiliar with key constructs [ 3 , 8 ]. For example, the question, “Recently, how many days have you participated in a training on evidence-based practice?” presumes the partner comprehends key terms about time reference (“recently”), implementation strategy (“training”), and a construct (“evidence-based practice”). If the partner is unfamiliar with these terms, they may not understand what types of training activities and intervention to include when responding to the question, which contributes to measurement error.

Next, to recall an answer, the partner must draw on information in memory. Several factors influence the memory retrieval process including a partner’s past experiences and the number and quality of memory cues provided, such as the time anchor (e.g., “recently”) and examples (e.g., participation in a workshop versus ongoing training) [ 10 ]. Third, the partner must integrate the information presented and form a judgement [ 40 ]. Previous studies indicate that decreasing item complexity (e.g., length, vocabulary) may facilitate decision-making, leading to more accurate self-reports [ 18 ]. In the example provided, researchers could consider changing the time anchor, replacing the general term “evidence-based practice” with a specific intervention, and simplifying the question (“Over the past month, did you attend a workshop on cognitive behavioral therapy?”).

In the final stage, the partner selects an answer and communicates it to the interviewer [ 17 , 40 ]. It is important to consider how response options are provided, specifically the type of scale used (e.g., Likert scale, rank order, multiple choice, open-ended), the direction of response options (e.g., “Strongly Disagree to Strongly Agree” versus “Strongly Agree to Strongly Disagree”), and whether the partner can meaningfully differentiate among the response options. In sum, cognitive processes involved in recall and recognition are affected by how measure content is presented, and these factors warrant consideration in measure development.

Cognitive interviewing techniques

Several cognitive interviewing techniques, generally categorized as think aloud and verbal probing [ 10 , 40 ], may be used. In think aloud, the interviewer takes an observer role and asks a partner to spontaneously verbalize their thoughts as they respond to questions. In verbal probing, the interviewer takes a more active role by asking a partner pointed follow-up questions after each response. Probes may be general (Does this question makes sense?) or item-specific (What do you think the term “evidence-based practice” means?). Probe selection can be standardized/pre-planned or applied flexibly in response to the partner (You hesitated to answer, can you tell me why?). The goals of the implementation study will guide probe selection. Table 1 presents key goals of cognitive interviewing and probes to elicit implementation relevant feedback.

Cognitive interviewing experts recommend using a structured or semi-structured protocol to guide data collection (see [ 40 ]). The protocol typically includes study-specific interview techniques (e.g., standardized probes) and administration information (e.g., use of technical equipment). For implementation studies, the cognitive interview protocol may also include several key additions: (1) probes to elicit multi-level partner perspectives (e.g., asking a clinical provider: What factors may affect how a patient would answer this question?,asking a clinical supervisor: Do you think clinicians would need additional training to administer this question?); (2) definitions of terms to facilitate shared understanding between partners (e.g., Can you describe what evidence-based practice means in your own words?); and (3) instructions on how to tailor probes for specific partner groups (e.g., clinic supervisors versus front-line providers). Given the multi-level nature of implementation studies, analyzing data at the item- and partner-level may reveal important patterns in terms of conceptual themes, informant discrepancies, targeted revision areas, and measurement feasibility barriers. These patterns can inform subsequent refinements to the measure and measure administration protocol to enhance the usability and scalability in real-world contexts.

Cognitive interviewing case example in ongoing implementation science project

Our team is currently employing cognitive interviewing to develop a pragmatic measurement-based care (MBC) tool. MBC is an evidence-based practice that involves the systematic administration, review, and discussion of patient assessment data to inform treatment decisions [ 19 , 33 ]. Few measures to assess patient progress in opioid use disorder treatment exist [ 24 ]. To address this need, the Director of the National Institute on Drug Abuse (NIDA) put forth a call to develop pragmatic measures of opioid use disorder symptoms and overdose risk. In response to this call, the NIDA-funded Measurement-Based Care to Opioid Treatment Programs (MBC2OTP) Project (K23DA050729) aims to develop a pragmatic overdose risk measure and measure administration protocol [ 32 ]. A preliminary 22-item measure was drafted by members of our study team based on published recommendations from the NIDA Director and colleagues and the DSM-5 diagnostic criteria for opioid use disorder [ 24 ]. Cognitive interviews are being used to collect partner feedback on measure content (symptoms, impairment, frequency of opioid use), format (open-ended questions versus multiple choice, preferred length, scoring), and administration procedures to inform implementation in community opioid treatment programs (OTP).

Multi-level partners are being recruited via email for cognitive interviews in two rounds. In the first round, relevant partners include program leaders who would decide whether to introduce the measure at an opioid treatment program, clinical supervisors who would oversee the training and supervision of counselors in measure administration, and front-line counselors who would deliver the measure to a patient. The second round of interviews focus on patients who would complete the measure in treatment. Eligibility requirements include English fluency and staff employment at the opioid treatment program for at least 3 months. No other exclusion criteria are used. Exclusion criteria are purposefully minimal to capture a range of diverse partner perspectives.

During the interview, three female researchers trained in cognitive interviewing present partners with the measure draft and ask them to answer each question aloud. We then apply the four-stage cognitive model to assess participant comprehension, memory retrieval, judgement, and response. First, in the comprehension phase, we assess whether partners comprehend the question and all the embedded constructs. For instance, our draft tool contains the item, “What typical dose of opioids do you take?” Ensuring comprehension requires us to assess whether a patient understands what opioids are and if they are aware of their average levels of opioid use.

Next, we assess the partner’s ability to recall an answer by drawing on information in memory. For example, we assess whether a patient’s response to the question about typical opioid use may differ based on whether they are experiencing withdrawal symptoms and if they would value examples of opioids in the item wording.

Third, we ask the partner to think aloud and describe how they are answering the question, so that we can assess how they form a judgment [ 40 ]. We also assess whether item complexity (e.g., length, vocabulary) seems appropriate or whether the item can be simplified to facilitate more accurate self-reports [ 18 ]. In the example provided, we ask whether participants might prefer a different time anchor or simpler wording of the question (“Over the past month, did you use more opioids than usual?”).

In the final stage, we ask the partner to communicate their final response to the question to the interviewer [ 17 , 40 ]. In our cognitive interviews, after a partner provides a response to one of the MBC items, we elicit their feedback on how the question is presented using verbal probes, which are outlined in a semi-structured protocol [ 10 , 40 ]. We use both general probes (Does this question makes sense?) and item-specific probes (What do you think the term “dose” means?) that are applied flexibly in response to the partner (You hesitated to answer, can you tell me why?). Importantly, our cognitive interview protocol uses supplemental open-ended questions to collect feedback on the ideal measure administration procedures to facilitate implementation of the protocol into the organizational workflow. Specifically, we elicit feedback on assessment frequency (how often the measure should be administered), administration context (group vs. individual counseling; in-person vs. telehealth sessions), and preferred administration method (electronic health record vs. tablet vs. pen and paper). Additionally, as an extension of typical cognitive interviewing, partners are asked to reflect on the types of implementation supports likely needed. Table 2 presents the four steps of cognitive interviewing currently being applied in the MBC2OTP study. Additional file 1 presents the full cognitive interview script used in the MBC2OTP study.

One-on-one partner interviews are currently being conducted via videoconference, are audio-recorded, and transcribed. Transcripts are being analyzed by three independent coders (ZPS, HR, and KS) to thematically identify areas for revision using NVivo. Using a reflexive team analysis approach [ 15 ], the study team meets weekly to establish consensus and resolve coding discrepancies. Reflexivity in qualitative analysis refers to the process by which the researcher identifies and reflects on the impact they may have (i.e., their own assumptions and biases) on the data being collected and analyzed in a study. The reflexive team analysis approach was selected to enable the coding team to iteratively reflect on their roles as researchers who are unfamiliar with the OTP context, as well as how this outside role may have impacted data collection, analysis, and interpretation.

Suggested revisions are being analyzed by item and partner background. Cognitive interviews will be continued until a representative sample is obtained from each participating OTP, defined as interview completion with all eligible partners who consent at each site. Data from these initial interviews will inform iterative development of the pragmatic MBC measure and measure administration protocol. Discrepancies and conflicting views across different partner groups (e.g., leaders and patients) will be resolved via collaborative co-design meetings with representatives from each OTP and the research team following interview completion. Results from the qualitative data analysis will be presented to OTP representatives, and consensus discussions will be held to make final decisions about conflicting feedback on each measure item.

To date, we have conducted 13 first-round 30 to 60-min cognitive interviews with participants from three opioid treatment programs ( n  = 6 opioid program leaders; n  = 3 clinical supervisors; n  = 4 front-line counselors). Data collection is ongoing and an additional five opioid treatment programs will be recruited to participate in the MBC2OTP study. Table 3 presents illustrative data gathered from the multi-level partners thus far to highlight how cognitive interviewing can be used to elucidate feedback on potential measure refinements as well as workflow administration.

The interviews have identified specific items, instructions, and response options that may require modification to enhance clarity. Specifically, partners have suggested shortening items due to confusing clinical wording to enhance literacy, rephrasing instructions using simpler language, and including a mix of open-ended and multiple-choice response options. Additionally, interviews have identified questions that can likely be removed due to limited perceived utility, conceptual overlap with other items, and fit with counseling procedures at opioid treatment programs. Perhaps most valuably, the interviews conducted thus far have elucidated partner preferences regarding ideal measure administration procedures. Specific administration advice elicited by the interviews has included: administration of the measure prior to individual or group counseling sessions, review of the measure at the start of a clinical encounter to guide service provision, and use of paper and pencil to facilitate administration off-line or in group contexts. The interviews have also provided encouraging preliminary data that the measure is viewed as low burden to be pragmatic within the standard opioid treatment program workflow. Final decisions about which items to eliminate, add, or modify, as well as how to administer the measure in the usual opioid treatment program workflow, will be made once data collection is complete to ensure responsiveness to the elucidated feedback.

Reflections on use of cognitive interviewing

Methods to develop pragmatic measures are critical to advance implementation science [ 23 ]. As the field evolves, ensuring that partners share a common understanding of implementation constructs is essential to further the study of implementation strategies and outcomes [ 38 ]. Although cognitive interviews can be time and labor intensive, involving partners in measure development incorporates the perspectives of the end-users, which can increase measure relevancy, increase the buy-in of front-line staff and administrators, and optimize a measure’s fit within a specific organizational context. Additionally, while interviews elicit discrepant data on measure quality and fit, cognitive interviews allow researchers to qualitatively capture discrepant partner viewpoints. This increased buy-in may result in measures that are more pragmatic, easily implemented, and sustained in community-based settings.

Cognitive interviewing can facilitate a shared understanding between partners and measure developers of implementation constructs, which with time, can reduce the field’s reliance on home grown implementation measures developed for single use. We assert that using cognitive interviewing to engage partners is complementary to psychometric testing because it increases measure utility and, thus, urge implementation researchers to routinely adopt this method. We believe that cognitive interviewing has potential to improve the rigor of implementation measures and facilitate a greater common language for the field.

Measurement concerns in implementation science are among the most significant barriers to advancing the field. There is an immense need for pragmatic and psychometrically sound measures but there remains limited guidance on methods to develop these measures. We hope that the overview of the four-stage approach to cognitive interviewing provided in this manuscript, along with a case example of how these stages are actively being applied in an ongoing implementation study, can help to advance the development of pragmatic measures and address measurement issues in the field.

Availability of data and materials

Not applicable.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23. https://doi.org/10.1007/s10488-010-0327-7 .

Article   PubMed   Google Scholar  

Beatty PC, Willis GB. Research synthesis: the practice of cognitive interviewing. Public Opin Q. 2007;71(2):287–311. https://doi.org/10.1093/poq/nfm006 .

Article   Google Scholar  

Becker SJ, Weeks BJ, Escobar KI, Moreno O, DeMarco CR, Gresko SA. Impressions of “evidence-based practice”: a direct-to-consumer survey of caregivers concerned about adolescent substance use. Evid-Based Pract Child Adolesc Ment Health. 2018;3(2):70–80. https://doi.org/10.1080/23794925.2018.1429228 .

Article   PubMed   PubMed Central   Google Scholar  

Beidas RS, Dorsey S, Lewis CC, Lyon AR, Powell BJ, Purtle J, Saldana L, Shelton RC, Stirman SW, Lane-Fall MB. Promises and pitfalls in implementation science from the perspective of US-based researchers: learning from a pre-mortem. Implement Sci. 2022;17(1):55. https://doi.org/10.1186/s13012-022-01226-3 .

Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149. https://www.frontiersin.org/articles/10.3389/fpubh.2018.00149 .

Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B, Ader D, Fries JF, Bruce B, Rose M, PROMIS Cooperative Group. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007;45(5 Suppl 1):S3–11. https://doi.org/10.1097/01.mlr.0000258615.42478.55 .

Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22. https://doi.org/10.1186/1748-5908-8-22 .

Chorpita BF, Becker KD, Daleiden EL. Understanding the common elements of evidence-based practice: misconceptions and clinical examples. J Am Acad Child Adolesc Psychiatry. 2007;46:647–52. https://doi.org/10.1097/chi.0b013e318033ff71 .

Clinton-McHarg T, Yoong SL, Tzelepis F, Regan T, Fielding A, Skelton E, Kingsland M, Ooi JY, Wolfenden L. Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the Consolidated Framework for Implementation Research: a systematic review. Implement Sci. 2016;11:148. https://doi.org/10.1186/s13012-016-0512-5 .

Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003;12(3):229–38. https://doi.org/10.1023/A:1023254226592 .

Davis M, Beidas RS. Refining contextual inquiry to maximize generalizability and accelerate the implementation process. Implement Res Pract. 2021;2:2633489521994941. https://doi.org/10.1177/2633489521994941 .

Glasgow RE. Critical measurement issues in translational research. Res Soc Work Pract. 2009;19(5):560–8. https://doi.org/10.1177/1049731509335497 .

Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013;45(2):237–43. https://doi.org/10.1016/j.amepre.2013.03.010 .

Halko H, Stanick C, Powell B, Lewis C. Defining the “pragmatic” measures construct: a stakeholder-driven approach. Behav Ther. 2017;40:248–51.

Google Scholar  

Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88. https://doi.org/10.1177/1049732305276687 .

Hughes KA. Comparing pretesting methods: cognitive interviews, respondent debriefing, and behavior coding by. 2004. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.79.1570&rep=rep1&type=pdf .

National Research Council. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press; 1984. https://doi.org/10.17226/930 .

Krosnick J, Presser S, Building A-S. Question and questionnaire design. In: Handbook of survey research. 2009.

Lewis CC, Boyd M, Puspitasari A, Navarro E, Howard J, Kassab H, Hoffman M, Scott K, Lyon A, Douglas S, Simon G, Kroenke K. Implementing measurement-based care in behavioral health: a review. JAMA Psychiatry. 2019;76(3):324–35. https://doi.org/10.1001/jamapsychiatry.2018.3329 .

Lewis CC, Dorsey C. Advancing implementation science measurement. In: Albers B, Shlonsky A, Mildon R, editors. Implementation science 3.0. 2020. pp. 227–251. https://doi.org/10.1007/978-3-030-03874-8_9 .

Lewis CC, Mettert KD, Stanick CF, Halko HM, Nolen EA, Powell BJ, Weiner BJ. The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation. Implement Res Pract. 2021;2:26334895211037390. https://doi.org/10.1177/26334895211037391 .

Lewis CC, Mettert K, Lyon AR. Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: results from a systematic review. Implement Res Pract. 2021;2:2633489521994197. https://doi.org/10.1177/2633489521994197 .

Lewis CC, Weiner BJ, Stanick C, Fischer SM. Advancing implementation science through measure development and evaluation: a study protocol. Implement Sci. 2015;10(1):102. https://doi.org/10.1186/s13012-015-0287-0 .

Marsden J, Tai B, Ali R, Hu L, Rush AJ, Volkow N. Measurement-based care using DSM-5 for opioid use disorder: can we make opioid medication treatment more effective? Addiction (Abingdon, England). 2019;114(8):1346–53. https://doi.org/10.1111/add.14546 .

Martinez RG, Lewis CC, Weiner BJ. Instrumentation issues in implementation science. Implement Sci. 2014;9(1):118. https://doi.org/10.1186/s13012-014-0118-8 .

Mettert K, Lewis C, Dorsey C, Halko H, Weiner B. Measuring implementation outcomes: an updated systematic review of measures’ psychometric properties. Implement Res Pract. 2020;1:2633489520936644. https://doi.org/10.1177/2633489520936644 .

Peterson CH, Peterson NA, Powell KG. Cognitive interviewing for item development: validity evidence based on content and response processes. Meas Eval Couns Dev. 2017;50(4):217–23. https://doi.org/10.1080/07481756.2017.1339564 .

Powell BJ, Beidas R. Advancing implementation research and practice in behavioral health systems. Adm Policy Ment Health. 2016;43(6):825–33. https://doi.org/10.1007/s10488-016-0762-1 .

Powell BJ, Stanick CF, Halko HM, Dorsey CN, Weiner BJ, Barwick MA, Damschroder LJ, Wensing M, Wolfenden L, Lewis CC. Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implement Sci. 2017;12(1):118. https://doi.org/10.1186/s13012-017-0649-x .

Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009;36(1):24–34. https://doi.org/10.1007/s10488-008-0197-4 .

Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN, Brownson RC, Glasgow RE. Measurement resources for dissemination and implementation research in health. Implement Sci. 2016;11(1):42. https://doi.org/10.1186/s13012-016-0401-y .

Scott K, Guigayoma J, Palinkas LA, Beaudoin FL, Clark MA, Becker SJ. The measurement-based care to opioid treatment programs project (MBC2OTP): a study protocol using rapid assessment procedure informed clinical ethnography. Addict Sci Clin Pract. 2022;17(1):44. https://doi.org/10.1186/s13722-022-00327-0 .

Scott K, Lewis CC. Using measurement-based care to enhance any treatment. Cogn Behav Pract. 2015;22(1):49–59. https://doi.org/10.1016/j.cbpra.2014.01.010 .

Stanick CF, Halko HM, Dorsey CN, Weiner BJ, Powell BJ, Palinkas LA, Lewis CC. Operationalizing the ‘pragmatic’ measures construct using a stakeholder feedback and a multi-method approach. BMC Health Serv Res. 2018;18(1):882. https://doi.org/10.1186/s12913-018-3709-2 .

Tabak RG, Khoong EC, Chambers D, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50. https://doi.org/10.1016/j.amepre.2012.05.024 .

Teal R, Enga Z, Diehl SJ, Rohweder CL, Kim M, Dave G, Durr A, Wynn M, Isler MR, Corbie-Smith G, Weiner BJ. Applying cognitive interviewing to inform measurement of partnership readiness: anew approach to strengthening community-academic research. Prog Community Health Partnersh Res Educ Action. 2015;9(4):513–9. https://doi.org/10.1353/cpr.2015.0083 .

Tourangeau R. Cognitive sciences and survey methods. In: Cognitive aspects of survey methodology: building a bridge between disciplines. 1984.

Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, Boynton MH, Halko H. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. https://doi.org/10.1186/s13012-017-0635-3 .

Willis G. The practice of cross-cultural cognitive interviewing. Public Opin Q. 2015;79:359–95. https://doi.org/10.1093/poq/nfu092 .

Willis GB. Cognitive interviewing: A tool for improving questionnaire design. Sage publications; 2004.

Download references

Acknowledgements

This study protocol was supported by a grant award from the National Institute on Drug Abuse (K23DA050729; PI: Scott).

Author information

Authors and affiliations.

Northwestern University Feinberg School of Medicine, Institute for Public Health and Medicine, Center for Dissemination and Implementation Science, Chicago, USA

Zabin Patel-Syed, Sara Becker, Miranda Olson, Hailey Rinella & Kelli Scott

You can also search for this author in PubMed   Google Scholar

Contributions

ZPS conceptualized this manuscript and wrote the first draft of the manuscript. SB also contributed to the conceptualization, writing, and review of the full manuscript. MO, HR, KS contributed to the writing and review of the full manuscript. SB and KS provided mentorship on the development of the manuscript. All authors read and approved the final manuscript.

Authors’ information

Corresponding author.

Correspondence to Zabin Patel-Syed .

Ethics declarations

Ethics approval and consent to participate.

Ethical approval was provided by Northwestern University’s Institutional Review Board. All participants consented to participate.

Consent for publication

Competing interests, additional information, publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Measurement-Based Care Cognitive Interview Script. This file includes the Measurement-Based Care Cognitive Interview Script, Interview Table, and Suggested Follow-up Questions used in the MBC2OTP case example.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Patel-Syed, Z., Becker, S., Olson, M. et al. What do you think it means? Using cognitive interviewing to improve measurement in implementation science: description and case example. Implement Sci Commun 5 , 14 (2024). https://doi.org/10.1186/s43058-024-00549-0

Download citation

Received : 17 January 2023

Accepted : 16 January 2024

Published : 14 February 2024

DOI : https://doi.org/10.1186/s43058-024-00549-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Implementation
  • Pragmatic measures
  • Cognitive interview
  • Case report

Implementation Science Communications

ISSN: 2662-2211

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

cognitive interview case study

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • J Stud Alcohol Drugs

The Case for Cognitive Interviewing in Survey Item Validation: A Useful Approach for Improving the Measurement and Assessment of Substance Use Disorders

Cassandra l. boness.

a Department of Psychological Sciences, University of Missouri, Columbia, Missouri

Kenneth J. Sher

To accurately identify substance use disorders, we must be confident of our ability to define and measure the construct itself. To date, research has demonstrated that the ways in which substance use disorder criteria are operationalized or assessed can significantly affect the information we obtain from these diagnoses. For example, differing operationalizations of the same construct, such as impaired control over substance use, can result in markedly different estimates of prevalence. This points to the need for approaches that aim to improve the validity of diagnostic assessments during the measure development phase.

We performed a scoping review of the cognitive interviewing literature, a technique that aims to provide a systematic way of identifying and reducing measurement error associated with the structure and content of assessment items. Along with this, we apply cognitive interviewing to items assessing alcohol tolerance.

We argue that cognitive interviewing is well suited for reducing measurement error in substance use disorder assessment items.

Conclusions:

Incorporating cognitive interviewing into the item generation stage of measure development for substance use disorder assessments is a worthwhile endeavor for improving validity.

C onstruct validation is central to the research process ( Cronbach & Meehl, 1955 ). In addiction research, our ability to accurately assess and diagnose substance use disorders (SUDs) is largely based on the ability to define and measure the construct of interest (e.g., Flake & Fried, 2019 ). Decades of research have demonstrated that the definition and measurement of SUDs is neither clear nor straightforward. For example, research has demonstrated that the way in which symptoms of psychological disorders, such as SUDs, are operationalized can greatly affect the validity of diagnosis. This was clearly demonstrated by Lane et al. (2016) in a meta-analysis of alcohol use disorder criteria item response theory parameters that demonstrated the relative severity of a given criterion can vary dramatically among commonly employed diagnostic interviews. Further, differing operationalizations of the same construct (e.g., impaired control over alcohol use) can greatly affect overall prevalence rates ( Boness et al., 2019 ) and can influence the estimation of diagnostic structure (e.g., Hoffman et al., 2018 ). In addition, there is evidence that items used to assess certain SUD symptoms, such as alcohol withdrawal, may be misunderstood and result in high false positive rates for that criterion (with associated false positive diagnoses; Boness et al., 2016 ).

Together, findings such as these suggest that it is imperative to consider the way in which a diagnostic instrument operationalizes a given criterion, as we might not be measuring what we think we are measuring. This can result in unknown or poor construct validity of our assessments. Unknown construct validity has far-reaching consequences for addressing the significant impact of SUDs in society. In research, uncertain construct validity for SUD measures means we lack the information necessary to evaluate the validity of a study’s outcomes and conclusions. In clinical practice, the inability to accurately assess SUDs can result in a failure to identify those with the disorder, thus resulting in a lack of treatment or assignment to a treatment that is not effective. Therefore, greater attention to the construct validity of self-report items, particularly during the measure development stage of SUD assessment instruments and surveys, should be an extremely high priority for both research and practice.

One viable approach for improving the validity of diagnostic assessment is through the use of cognitive interviewing (sometimes called cognitive testing). Cognitive interviewing is a technique that aims to provide a systematic way to identify and reduce measurement error associated with the structure and content of interview items and is an important first step in measure development ( Schwarz, 1999 ). It has the goal of understanding how respondents interpret a given item and select a response (i.e., the response process) and can be used to provide feedback on the candidate items and response scales, suggest alternative wording, and ensure relevant constructs are validly represented. Cognitive interviewing is a flexible approach that can be used across varied survey administration modes such as in-person, telephone, and online (see Willis, 2005 ).

Cognitive interviews are typically conducted by trained interviewers who are familiar with the constructs of interest and measurement objectives. Interviewers can use several interviewing techniques including think-aloud and verbal probing procedures. The think-aloud procedure instructs participants to vocalize their thoughts as they select their response to an item. The interviewer pays careful attention to the response to identify misunderstandings or item flaws. With this procedure, open-ended questions with minimal bias in phrasing are used to clarify answers (e.g., “tell me more about that”). In the verbal probing procedure, the interviewer asks the target question and the participant responds. The interviewer then follows up by probing for specific information, such as asking participants to (a) paraphrase the question, (b) discuss the thought processes used to choose an answer (e.g., what kinds of situations were you thinking about in coming up with your answer?), and/or (c) define key terms (e.g., what is a hangover?) ( Fowler, 1995 ). Probing can be concurrent or retrospective and probes can be standardized (i.e., predetermined for every participant) or nonstandardized (i.e., specific to each participant). Thus, there is an immense amount of flexibility with verbal probing (see Willis, 2005 , for a thorough discussion).

Cognitive interviewing has been largely overlooked in psychology and the study of psychopathology. A quick search of the literature revealed only a few publications that used cognitive interviewing for addressing issues related to the assessment of SUDs (e.g., Chung & Martin, 2005 ; Mewton et al., 2014 ; Slade et al., 2013 [the latter two used the same sample]). These publications, focused on adolescents and young adults, highlighted several common areas of misunderstanding related to the assessment of SUDs. One example is impaired control, which is typically conceptualized as the result of addiction-related compulsion. This is often assessed via criteria such as drinking larger amounts or longer than intended. Young adults, although endorsing drinking larger amounts or longer than intended, frequently reported that this behavior was due to social or other noncompulsion-based reasons (e.g., Chung & Martin, 2005 ; Slade et al., 2013 ), suggesting false positives for impaired control among this age group. Tolerance is another criterion that is likely to be misunderstood, particularly among adolescents. Chung and Martin (2005) demonstrated that the extent of tolerance reported by adolescents was more reflective of expected changes in sensitivity to the effects of alcohol rather than a clinically significant increase in consumption that is typically suggestive of physiological dependence. Although important research with significant implications, none of these publications were used in the context of item-refinement for the development of an assessment instrument, but rather to elucidate the problems with existing instruments (e.g., World Mental Health Survey Initiative version of the Composite International Diagnostic Interview; Kessler & Ustün, 2004 ) and, in some cases, make suggestions for revisions (e.g., to DSM-5 diagnostic criteria; Slade et al., 2013 ). Thus, cognitive interviewing has been largely neglected as a part of SUD measure development.

Despite this oversight, we believe that cognitive interviewing should be considered as a part of the standard measure development process. 1 Guidelines for the construction of objective tests (e.g., Clark & Watson, 2019 ) recognize the importance of conceptual and psychometric analysis during the item generation stage of scale construction. However, clinical psychology has leaned heavily toward psychometric analyses using approaches such as factor analysis and item response theory. Although this is crucially important, psychometric analysis alone overlooks the possibility of conceptual problems with items. Thus, we argue that cognitive interviewing is an ideal complement to psychometric approaches in the item generation stage of measure construction and is necessary if we wish to avoid building our diagnostic structures on weak foundations.

Using cognitive interviewing to improve the construct validity of SUD items

Given some of the issues unique to the diagnosis of SUDs (e.g., the difficulty in assessing alcohol withdrawal; Boness et al., 2019 ), cognitive interviewing is especially well suited for dealing with these challenges. This approach allows the researcher to gain a deeper understanding of how respondents are interpreting items, including any areas in need of refinement or additional clarification. It can also aid in the evaluation of measure instructions and response scales, providing feedback that is often treated as less important to scale construction but that, in practice, can have a significant impact on the usefulness of a scale (e.g., Simms et al., 2019 ). Further, we argue that SUD researchers are particularly well suited to carry out these interviews given their substantive expertise in the construct of interest.

To offer an applied example, we conducted cognitive interviews with a small group of participants ( N = 10) to evaluate how tolerance items from the National Epidemiologic Survey on Alcohol and Related Conditions-III (NESARC-III; Grant et al., 2014 ) perform and to recommend revisions. NESARC-III was chosen because it is used to derive population-based estimates of DSM-5 SUD diagnoses. Participants were recruited from the community through fliers posted around a Midwest town and online. They had to be 18 or older and have met heavy drinking criteria 2 within the past 30 days to be eligible. Participants were largely female (90%) and had an average age of 20.7 ( SD = 1.6), and all had graduated high school. Items comprised the following: (a) did you find that your usual number of drinks had much less effect on you than it once did; (b) did you find that you had to drink much more than you once did to get the effect you wanted; (c) did you drink as much as a fifth of liquor in one day, that would be about 20 drinks, or 3 bottles of wine, or as much as 3 six-packs of beer in a single day; and (d) did you increase your drinking because the amount you used to drink didn’t give you the same effect anymore? All questions referred to the past 12 months, and the response scale was “yes” or “no” for each item. Interviews were conducted using a combination of the think-aloud and verbal probe procedures.

The most notable issue that arose across the items was related to the time frame of comparison, which varied markedly across participants. For example, when asked, “In the past 12 months, did you find that your usual number of drinks had much less effect on you than it once did?” some participants reported that they were comparing their usual number of drinks now to when they first started drinking. However, others reported that they were comparing their usual number of drinks now to their freshman year of college (i.e., several years earlier), to the beginning of the current year (i.e., a few months before the interview), to one year ago, and to the start of the academic year (i.e., 4–5 months earlier). This is problematic given that respondents were answering on the basis of different time frames. Misinterpretation of items across respondents is thought to affect the estimation of population-based prevalence rates (e.g., Boness et al., 2016 ) and could contribute, in part, to the so-called treatment gap ( Drummond et al., 2011 ), the difference between estimated prevalence in the population and treatment utilization.

In line with the use of cognitive interviewing for item refinement, we feel most of the tolerance items could be revised to more clearly define the time frame of interest in order to increase consistency in reporting and, thus, improve validity. As such, for the item described above, we recommended the following revision for consideration: “Compared to when you first started drinking regularly, do you find that a given number of drinks has much less effect on you now?” 3 Although cognitive interviewing is useful for identifying potential pitfalls and revising items accordingly, it is only intended to provide information about items rather than to ensure validity ( Willis, 2005 ).

Limitations of cognitive interviewing include being time- and labor-intensive and requiring specific training in interviewing. Some have argued against its use on these grounds alone, which is problematic given that various solutions have been offered to address issues related to the time- and labor-intensity required. For example, the Response Process Evaluation method uses open-ended metasurveys to more rapidly gather information about how participants interpret items, make revisions, and retest the revision across new samples of respondents ( Wolf et al., 2019 ). This is similar to cognitive interviewing but is more efficient for some applications. Other techniques, such as the Questionnaire Appraisal System Checklist ( Willis, 2005 ), have been developed as simpler and more efficient alternatives to full cognitive interviews. These offer many of the benefits, such as the ability to identify items with potential pitfalls (e.g., making inappropriate assumptions about respondents), while decreasing some of the mastery required to conduct cognitive interviews.

Others (e.g., Willis & Artino, 2013 ) have argued that cognitive interviewing, particularly the think-aloud procedure, is an unnatural and a difficult process for respondents to engage in given that it is not a typical way of processing information. It can, therefore, require significant participant training before the interview. Although this criticism may be true for some participants and for some questions, there are few other options for gaining insight into the response process. Even with the Response Process Evaluation method, respondents are asked to report on their response process. Further, some have maintained that cognitive interviewing could cause survey developers to overthink their items. This potential problem is easily addressed by focusing on the identification of clear trends across respondents before making changes to any given item ( Willis, 2005 ). In addition, this criticism requires reflection on the balance between under- and over-consideration of item qualities. We would argue that overthinking items is worth the risk when compared with the alternative of not considering how items are performing at all for fear of “overthinking.”

A final issue to consider is that items may have different meanings and interpretations for different populations. Thus, it is imperative to consider the identification and recruitment of participants from appropriate subpopulations for testing items. In most cases, respondents should have the characteristics of interest for the survey. However, in some cases, such as with SUD diagnosis, the questionnaire may be intended for use in the general population. Thus, recruiting participants with and without the construct of interest (e.g., heavy drinking) is useful for ensuring that the majority of individuals who will be administered the items do, in fact, understand them ( Willis, 2005 ).

Thus, we recommend that cognitive interviewing become more common place in development of SUD assessment instruments. Concretely, we suggest that cognitive interviewing be incorporated in the item generation stage of any new scale development or measure construction undertaking. Further, we believe that cognitive interviewing, or similar methods, should be used during the construction of any new assessment instrument because it offers particular benefits given the complexity of assessing SUD. Improved validity of SUD diagnosis can advance the precision of prevalence estimates derived via epidemiologic surveys as well as aid in identifying those at risk and in need of treatment.

This study was supported by National Institutes of Health Grant F31AA026177. The funders played no role in the current review.

1 Although other qualitative approaches (e.g., focus groups) can certainly be used for the same purpose of revising items, we believe that cognitive interviewing offers several unique benefits such as its flexible application to different types of surveys and its emphasis on gaining unbiased information from participants. In focus groups, for example, it is well known that biases can arise from the presence of other participants in the group (e.g., Vogt et al., 2004 ). This is an important consideration when asking about SUDs, which can be difficult for participants to talk about openly (e.g., due to stigma). At a minimum, cognitive interviewing could be more readily combined with other qualitative approaches in the measure development process.

2 Heavy drinking was defined as (a) having at least five separate heavy drinking occasions (i.e., 5+ drinks for males; 4+ drinks for females) OR (b) exceeding NIAAA’s weekly limit guidelines (i.e., >14 drinks for males; >7 drinks for females; U.S. Department of Health and Human Services & U.S. Department of Agriculture, 2015 ).

3 Importantly, though, these findings were only from one round of cognitive interviewing. In practice, several rounds of interviewing (e.g., three rounds of 10–15 participants) would be conducted to test item revisions iteratively.

  • Boness C. L., Lane S. P., Sher K. J. Assessment of withdrawal and hangover is confounded in the Alcohol Use Disorder and Associated Disabilities Interview Schedule: Withdrawal prevalence is likely inflated. Alcoholism: Clinical and Experimental Research . 2016; 40 :1691–1699. doi:10.1111/acer.13121. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Boness C. L., Lane S. P., Sher K. J. Not all alcohol use disorder criteria are equally severe: Toward severity grading of individual criteria in college drinkers. Psychology of Addictive Behaviors . 2019; 33 :35–49. doi:10.1037/adb0000443. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Chung T., Martin C. S. What were they thinking? Adolescents’ interpretations of DSM-IV alcohol dependence symptom queries and implications for diagnostic validity. Drug and Alcohol Dependence . 2005; 80 :191–200. doi:10.1016/j.drugalcdep.2005.03.023. [ PubMed ] [ Google Scholar ]
  • Clark L. A., Watson D. Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment . 2019; 31 :1412–1427. doi:10.1037/pas0000626. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Cronbach L. J., Meehl P. E. Construct validity in psychological tests. Psychological Bulletin . 1955; 52 :281–302. doi:10.1037/h0040957. [ PubMed ] [ Google Scholar ]
  • Drummond C., Gual A., Goos C., Godfrey C., Deluca P., Von Der Goltz C., Kaner E. Identifying the gap between need and intervention for alcohol use disorders in Europe. Addiction . 2011; 106 :31–36. doi:10.1111/j.1360-0443.2010.03335.x. [ PubMed ] [ Google Scholar ]
  • Flake J. K., Fried E. I. Measurement schmeasurement: Questionable measurement practices and how to avoid them . 2019, January 17. Retrieved from https://psyarxiv.com/hs7wm/ [ Google Scholar ]
  • Fowler F. Improving survey questions: Design and evaluation . Thousand Oaks, CA: Sage; 1995. [ Google Scholar ]
  • Grant B. F., Amsbary M., Chu A., Sigman R., Kali J., Sugawana Y., Chou P. S. Source and accuracy statement: National Epidemiologic Survey on Alcohol and Related Conditions-III (NESARC-III) Rockville, MD: National Institute on Alcohol Abuse and Alcoholism; 2014. [ Google Scholar ]
  • Hoffman M., Steinley D., Trull T. J., Sher K. J. Criteria definitions and network relations: The importance of criterion thresholds. Clinical Psychological Science . 2018; 6 :506–516. doi:10.1177/2167702617747657. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Kessler R. C., Ustün T. B. The world mental health (WMH) survey initiative version of the World Health Organization (WHO) composite international diagnostic interview (CIDI) International Journal of Methods in Psychiatric Research . 2004; 13 :93–121. doi:10.1002/mpr.168. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Lane S. P., Steinley D., Sher K. J. Meta-analysis of DSM alcohol use disorder criteria severities: Structural consistency is only ‘skin deep. Psychological Medicine . 2016; 46 :1769–1784. doi:10.1017/S0033291716000404. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Mewton L., Slade T., Teesson M., Memedovic S., Krueger R. F. Improving the diagnostic criteria for alcohol use disorders through survey methodology and cognitive interviewing. International Journal of Methods in Psychiatric Research . 2014; 23 :359–371. doi:10.1002/mpr.1448. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Schwarz N. Self-reports: How the questions shape the answers. The American Psychologist . 1999; 54 :93–105. doi:10.1037/0003-066X.54.2.93. [ Google Scholar ]
  • Simms L. J., Zelazny K., Williams T. F., Bernstein L. Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment . 2019; 31 :557–566. doi:10.1037/pas0000648. [ PubMed ] [ Google Scholar ]
  • Slade T., Teesson M., Mewton L., Memedovic S., Krueger R. F. Do young adults interpret the DSM diagnostic criteria for alcohol use disorders as intended? A cognitive interviewing study. Alcoholism: Clinical and Experimental Research . 2013; 37 :1001–1007. doi:10.1111/acer.12063. [ PubMed ] [ Google Scholar ]
  • U.S. Department of Health and Human Services & U.S. Department of Agriculture. Dietary guidelines for Americans . 2015. Retrieved from http://health.gov/dietaryguidelines/2015/guidelines/ [ Google Scholar ]
  • Vogt D. S., King D. W., King L. A. Focus groups in psychological assessment: Enhancing content validity by consulting members of the target population. Psychological Assessment . 2004; 16 :231–243. doi:10.1037/1040-3590.16.3.231. [ PubMed ] [ Google Scholar ]
  • Willis G. B. Cognitive interviewing: A tool for improving questionnaire design . Thousand Oaks, CA: Sage; 2005. [ Google Scholar ]
  • Willis G. B., Artino A. R., Jr. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. Journal of Graduate Medical Education . 2013; 5 :353–356. doi:10.4300/JGME-D-13-00154.1. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Wolf M. G., Ihm E. D., Maul A., Taves A. Survey item validation . 2019, July 23. Retrieved from . [ CrossRef ] [ Google Scholar ]

Introspective interviewing for work activities: applying subjective digital ethnography in a nuclear industry case study

  • Original Article
  • Open access
  • Published: 05 January 2021
  • Volume 23 , pages 625–638, ( 2021 )

Cite this article

You have full access to this open access article

  • Philippe Fauquet-Alekhine   ORCID: orcid.org/0000-0002-3428-9987 1 , 2 , 3 ,
  • Martin W. Bauer 4 &
  • Saadi Lahlou 4  

2095 Accesses

1 Altmetric

Explore all metrics

A Correction to this article was published on 11 April 2021

This article has been updated

Subjective Evidence-Based Ethnography (SEBE) is a family of methods developed in digital ethnography for investigation in social science based on subjective audio–video recordings using first-person perspective. Recordings are used for self-confrontation (collect subjective experience, discussion of findings and final interpretation). Several studies applying SEBE methods mentioned “introspection” as a process occurring during self-confrontation and discussed it without providing evidence of its occurrence. This article aimed at clarifying introspection and its occurrence in SEBE. After a literature review addressing introspection, the process of introspection in SEBE was analyzed, depicted and illustrated by a case study. Conditions for introspection to occur in SEBE and the related mechanisms were proposed: it was found that indirect introspection could actually occur but not frequently and could go unnoticed without lessening the quality of the analysis. A refined analysis of introspection during or after the interviews was not identified as an added-value for the activity analysis.

Similar content being viewed by others

cognitive interview case study

The Ethical Implications of Artificial Intelligence (AI) For Meaningful Work

Sarah Bankins & Paul Formosa

cognitive interview case study

Transcription and Qualitative Methods: Implications for Third Sector Research

Caitlin McMullin

cognitive interview case study

Qualitative Research and Content Analysis

Avoid common mistakes on your manuscript.

1 Introduction

Accessing subjects’ action during activities inevitably refers to activity analysis and thus to the cognitive task analysis paradigm which regroups methodologies for job or task design and analysis. Two reviews attempted to provide an exhaustive state of the art (Wei and Salvendy 2004 ; Tofel-Grehl and Feldon 2013 ) and a categorization of the methods. Among them, process tracing methods have developed with the recent progress of miniaturized camera. Process tracing is capturing expertise during activity performance through audio and/or video recording, followed by an analysis phase of the recordings. When using miniaturized cameras, process tracing methods may be referred to as Subjective Evidence-Based Ethnography (SEBE) as defined by Lahlou ( 2011 ). The SEBE is a family of methods developed in digital ethnography for investigation in social science based on subjective audio–video recordings or subfilms (the first-person perspective: Pea 1994 ; Omodei et al. 2005 ; Knoblauch et al. 2006 ; Goldman et al. 2007 ; Petitmengin 2009 ; Rix-Lièvre and Lièvre 2010 ; Lahlou 2011 ) using miniature video cameras (usually worn at eye-level by subjects: the subcam). Subfilms are then used for self-confrontation with subjects to collect their subjective experience, discussion of findings and final interpretations between researchers and subjects.

Self-confrontation was developed by Von Cranach et al. ( 1982 ), and then, on the basis of this work, by Theureau ( 2002 ) as a method of investigation of human activity in the framework of his theory of goal-oriented activity. Von Cranach identified three inter-dependent levels of action, each being recoverable by a specific method: (1) the ongoing behavior (acts) are recoverable through audio–visual observation techniques; (2) the cognitive guidance of action is recoverable by a self-confrontation of the actor; (3) the organization of social action is recoverable through the confrontation with other actors (social representations). Self-confrontation is a deferred examination of the dynamics of structural coupling subject situation supported jointly by means of reproduction of behavior (e.g., video) and by the researcher as both observer and interlocutor (Theureau 2002 ). Rieken et al. ( 2015 : 255) applying self-confrontation during digital ethnographic studies for day-work analysis (image-recording of an activity with post-analysis during interviews with subjects) mentioned “introspection” as a process occurring during self-confrontation. Similarly, (Lalhou 2011 ; Lalhou et al. 2015 ) and Le Bellu ( 2011 , 2016 ), also applying digital ethnography for activity analysis, mentioned “introspection” when describing psychological processes during interviews. However, the authors did not characterize the process of introspection and did not provide evidence of its occurrence.

The aim of this article is to clarify two points. First, the notion of introspection is not simple and introspection takes different senses from one research to another; furthermore, “introspection is often viewed with suspicion and seen as an expression of flawed thinking” (Weger et al., 2018 : 206): a literature review clarifies this notion in “ Literature background ”. Second, specialists of digital ethnography seem to ask for introspection during the self-confrontation phase of their SEBE-based studies, but it is not clear which forms of introspection occur during these interviews. The research question is thus: does introspection actually occur during SEBE interviews and if yes, in what form? A method for the characterization of introspection during SEBE interviewing is proposed and applied to a case study. Finally, we discuss the added-value of introspection in SEBE.

2 Literature background

2.1 main streams and controversies.

For Danziger ( 2015 : 702), in a first approach, introspection may simply refer to “the self-observation of mental events”. Stated in other words by Vermersch ( 1994 : 203), “the access to knowledge of one’s own cognitive functioning may be in a general manner considered as an act of introspection”. Written this, any attempt to access and additionally to understand what happens or has happened in a subject’s flow of conscious mind ‘as experience’ is introspection. When understood in this way, psychoanalysis is introspection, meditation, as well as is self-confrontation. Introspection in this sense of supported reflecting on one’s life is part of a long tradition of techniques of self-improvement, religious enlightenment and life change or conversion experiences. We refer to this type of introspection as “macro-introspection”, and this will not be our main focus here.

However, for a large part of the scientific community, introspection addresses a more specific psychological process. In his 2006 paper, Overgaard provided an interesting synthesis of what is introspection and how it may be applied. Confronting arguments of renowned pioneers of this concept (Brentano, Comte, James and Wundt), he concluded that ‘introspection’ is the access to one’s inner mental state by oneself (p.630) and “involves an attending to the content of one’s consciousness and nothing more than that” for which “the use of an introspective report about the relevant state seems the only possible methodology” (p.631).

Overgaard ( 2006 ) pointed to two controversies. A first controversy was that introspection could be considered as an active observation vs a “passive inner perception” so as to avoid “destroying the introspective experience” and thus provide reliable reports (p.630–631). This first controversy involved an all or none argument, with, on the one side, introspectionists like Wundt and Titchener, advocating introspection as ‘self-observation’ and as the privileged source of psychological data; through strict and laboriously trained methodological procedures, the trained subject would get access to ‘pure sensation’, purified from all sorts of learned apperceptions. They adhered to the ‘constancy hypothesis’ between stimulus and sensation, which would later be criticized by the Gestalt psychologist (Koehler 1947 : 42ff) to explain perceptual constancy under variables stimulus conditions. Gestalts inquired about phenomenal experience, not introspective accounts. On the other side were Comte, Mills and the positivist tradition, who rejected ‘introspection’ as an impossibility. Considering ‘observation’ as the source of data, they rejected ‘inner observation’ as an illusion because of active attention to mental content, and observation means focused attention, necessarily changing that content.

A second controversy addressed the possibility of accessing inner awareness as retrospective introspection. Overgaard ( 2006 ) summarized it so: while Comte tried to demonstrate its impossibility through the argument that a subject cannot split in two parts “so as one part observes the other”, James bypassed this problem by turning “introspection” into “retrospection” implying “ not to accept introspection of currently conscious states” (p.630). This challenging point led to many studies addressing subjects’ verbal reports as valid introspective reports by trying to clarify various conditions of validity. Many of these studies based on the correlation between verbal reports and objective behavioral data related to the same event. To date, the results remain inconclusive. Gaillard et al. ( 2006 ), referring to Ericsson and Simon’s work ( 1980 ), insisted that the main issue was not acceptability (yes or no) overall, but admonished attention to the circumstances of acceptability. Two difficulties needed to be avoided: on the one hand, subjects’ concurrent verbal reports of their mental states while performing a task interfere with the cognitive process or decrease the speed of performing the activity (Nisbett and Wilson 1977 ; Ericsson and Simon 1980 ; Ericsson 2003 ): if you think about your running you slow down. On the other hand, subjects’ retrospective verbal reports lose in reliability due to an inaccurate or partial recall from memory (Gaillard et al. 2006 ). Many authors advocated combining systematically verbal reports and objective measurements so that to ensure they seized not only what was reported but all that was reportable (Gaillard et al. 2006 : 713). In addition, time-delayed retrospection might generate the subjects’ interpretation or speculation about their own cognitive processes making it difficult to accept verbal reports as objective and thus acceptable data. In conclusion, the discussion has shifted: rather than dogmatically declaring that all concurrent or retrospective verbal reports are invalid on principle, to make this ‘validity’ an empirical question in search for circumstances under which verbal reports can usefully be considered. Three considerations come into play: first, the time-lapse between action and verbal report needs to be considered, the longer the more difficult to obtain reliable retrospective reports; second, retrospective reports can be supported by props such as video replay, of which the actor-perspective video seems particularly effective; and thirdly, distinctions needed to be made between ‘reports of concomitant awareness’ and reports involving increasing mediation including sense making, argumentations and interpretations, inferences and causal attributions or rationalization of actions. Erikson and Simon ( 1980 ) developed a memory-based theory of verbalizations to classify reports that are obtained through ‘introspection’. We might call this ‘micro-introspection’.

Experiments illustrated these empirical efforts such as this of Bechara et al. ( 1997 ). The authors studied the subjects’ consciousness of self-performance during a simple gambling task (the ‘‘Iowa gambling task’’) by comparing their skin conductance (continuous objective somatic measure) to verbal reports obtained through periodical questioning (subjective data). The task consisted in choosing a card among four decks and maximizing gains on a series of trials, a card resulting in a win or a loss when playing for money. Unknown to the participants, some decks were advantageous, as opposed to others. They found out that subjects started selecting cards from advantageous decks and skin conductance increased just before selecting a disadvantageous card before the subjects were able to verbally explain their choice. For Bechara et al., this situation of dissociation between performance and verbal reports could be interpreted as evidence of unconscious knowledge accessed through introspective verbal reports and confirmed by objective data. A replicated experiment by Maia and McClelland ( 2004 ) used a more elaborate questionnaire. They found out that the dissociation vanished and they concluded that there was no evidence of unconscious knowledge and thus no introspective. However, Overgaard ( 2006 ) highlighted that the original experiment complied with the introspection need to put the subjects in a “passive inner perception” using simple questions. On the contrary, using an elaborate questionnaire, researchers of the replicated experiment implemented an active observation of mental states and destroyed the introspective experience. Simple questions induced simple answers conversely to elaborate questionnaires. This echoes Titchener’s viewpoint (as reported by Danziger 2015 ), who was “a major exponent of experimental introspection”, and who “demanded that introspective descriptions should be in terms of simple, irreducible units and should abstract from any meaning that the stimulus might have” (p.703), since “additional cognitive activity must necessarily change the sequence of mediating thoughts” (Ericsson 2003 : 11). Ericsson undertook an analysis regarding factors decreasing validity of verbal reports: “The first arises when the investigators try to obtain more information than the subjects’ thought sequences can provide. […] Second, investigators often ask the subjects to describe their methods for solving problems at the end of the experiment, when they have completed a long series of different tasks. If the subjects generated and consistently applied a general strategy for solving all of the problems, they should be able to respond to such requests easily with a single memory retrieval. But the subjects typically employ many methods and shortcuts and even change their strategies during the experiment, through learning. Under such circumstances, the subjects would have great difficulty describing a single strategy used consistently throughout the experiment, even in the unlikely event that they were motivated and able to recall most of the relevant thought sequences. It is, therefore, not surprising that the subjects’ descriptions are imperfectly related to their averaged performance during the entire experiment” (Ericsson 2003 : 15). This also led to the assumption that analysts may have an introspective approach with the aim of obtaining introspective verbal reports from subjects about whom they intend to access the mental state but they may be unable to create favorable conditions for introspection. According to Gaillard et al. ( 2006 : 714), “authors widely agree on the validity and on the reliability of verbal data as a source of information about cognitive processes as long as they are elicited with care and interpreted with full understanding of the circumstances under which they were obtained”. Introspection is rehabilitated from Positivist denial and rescued from Wundtian dead ends; Brentano ( 1874 : 42ff) showed the middle way of careful methodological procedures (e.g., support for retrospection of awareness, time-lapse considered) and data analysis considering different types of verbal production; not all reports obtained are ‘introspective’, many involve communicative reasoning. Thus introspection also ‘democratised’ in the sense that everybody is capable of doing so; for Wundt and Titchener, only researchers trained for hundreds of hours and had acquired the ‘right attitude’ were allowed to ‘introspect’ (Schwitzgebel 2004 ).

2.2 Characterizing introspection from verbal characteristics

Gaillard et al. ( 2006 ), after reviewing and discussing results regarding the contribution of introspection to implicit learning, suggested three criteria to validate the acceptability of verbal reports as introspective:

Instructions given to subjects for verbalization in either a general or a specific manner must not have any effect on the introspective cognitive processes (limiting the subject’s answers to a few possible categories such as ‘‘yes’’ or ‘‘no’’ or, on the contrary, forcing subjects to produce elaborated and exhaustive description might limit or spoil the introspection process),

Verbal reports must be complete, consisting of all the information the subjects have about their own cognitive processes,

Verbal reports must be consistent with other empirical data on behavior.

The last point should be considered as a bilateral relationship: if empirical data on behaviour leads to validating verbal reports, similarly verbal reports lead to explaining empirical data. This is well illustrated by the ‘‘Iowa gambling task’’ experiment reported above: verbal reports were corroborated by the existence of variations of skin conductance and variations of skin conductance were explained by verbal reports; without verbal reports, researchers would have perhaps never known that variations of skin conductance were related to an unconscious knowledge of advantageous decks.

When this criterion is not respected, Gaillard et al. ( 2006 : 713) use the expression “retrospective verbal reports” (shared by other researchers) potentially suffering from a bias due to the fact that “subjects may forget or inaccurately recall the relevant features of the experimental situation, which of course cannot be interpreted as implying that they were unaware when engaged in the task.” For these retrospective verbal reports to be considered similar to introspective verbal reports, Ericsson ( 2003 : 13), on the basis of a prior review, suggested that the time to generate these reports had to occur less than 5 s after the event described so that “the participants can recall their sequence of thoughts reasonably accurately”. However, Gaillard did not include this time criterion perhaps, because it characterizes more the way verbal reports were obtained than their validity.

2.3 Characterizing introspection from memory aspects

As we can see, applying introspection during work activities implies concurrent verbal reports. Authors qualify the investigation as retrospection or indirect introspection as proposed by (Titchener 1912 and Kriegel 2013 ) when performed during interviews in an attempt to access mental states (see also Piccinini 2003 : concurrent introspective reports and retrospective introspective reports). Nevertheless, implementation does not seem easy whether it be for introspection or retrospection.

As recalled by Kriegel ( 2013 : 1172), direct introspection occurs simultaneously with the introspected event, whereas indirect introspection occurs later and involves recollection of past events; therefore, in terms of memory (Fig.  1 ), the former relates to short term memory (or working memory) when the latter relies on episodic memory (one of the forms of explicit memory) or procedural memory (one of the forms of implicit memory) used to recall past events (Piccinini 2003 ).

Sensory Memory refers to the ability to retain sensory information through the five senses.

Short Term Memory refers to the capacity to keep available information for a short period of time.

Long Term Memory refers to the capacity to hold a large amount of information for a long period.

Encoding process: the perceived item of interest is converted into a construct that can be stored, and then recalled later from Short Term Memory to Long Term Memory)

Storage process: retaining information in either of the three-stage memory, but mostly in Long Term Memory

Consolidation process: stabilizing a memory trace after initial acquisition

Retrieval process: re-accessing events or information from the past previously encoded and stored

Implicit Memory refers to memories storing previous experiences to perform of a task without conscious awareness of these (including Priming Memory, Procedural Memory and (non) Associative Learning Memory).

Explicit Memory refers to memories that can be consciously recalled.

Episodic Memory contains past personal experiences (e.g., time, places, and related emotions).

Semantic Memory refers to meanings, understandings and other concept-based knowledge.

figure 1

Multi-store or modal model of memory (adapted from Atkinson-Shiffrin 1968 ) is the model generally adopted by the scientific community (Velez-Pardo and Jimenez-Del-Rio 2015 ). Memory is a three-stage sequence: Sensory Memory, Short Term Memory and Long Term Memory

2.4 Introspection and the SEBE

In her studies addressing work activity analysis through SEBE-based interviews, Le Bellu ( 2011 , 2016 ) mentioned introspection without developing its contribution to the analysis but another research team (Rieken et al. 2015 ) recently published this kind of study and explicitly referred to introspection in the title of the paper. According to Rieken et al. ( 2015 : 256), “Introspection in digital ethnography does not rely upon a mental representation of events but instead on shareable digital account of events” in that it “gives both the participating introspecting subject and the inquiring interviewer equal access to the first-person perspective digital representations”.

It can be agreed that introspection in digital ethnography relies on shareable digital account of events but, before reaching the “subject-analyst” sharing level, another level, prior to this mentioned by Rieken et al., must be considered: the “subject–subject” level. Indeed, during the replay interview (as for any self-confrontation using video), the first one concerned by sharing events is the subject him/herself. This is why self-confrontation is called “ self ”—“confrontation”. This consideration suggests that the first part of Rieken et al.’s sentence might be wrong as the subject is confronted with events experienced in the past for which the first-person perspective of the subfilm summons mental representations of what happened at this time.

In other words, the SEBE approach deconstructs Comte’s argument saying that the subject cannot split in two (Comte 1830 ), because here, due to video self-confrontation, splitting the subject in two becomes effective. Indeed, if a lapse in time Δt separates the realization of the activity happening at time t and the replay interview, thus happening at time t + Δt, the context of the replay interview brings together the conditions necessary for submitting the subject’s Self at t + Δt (noted Self t + Δt ) watching and subjectively re-experiencing a past activity that might give him/her access to the re-enactment of event experienced by the Self at t (noted Self t ). Hence, provided that the conditions of introspection are effective during the replay interview, the subject may undertake introspection by the Self t + Δt on the Self t . As suggested by Clot ( 2001 : 258) referring to Vygotsky, “ ‘being’ does not coincide with the phenomenon (in other words, the ‘real’ with the ‘realized’) and even introspection does not abolish this difference. Because the mind is not only subject to it. It is divided into object and subject: my joy and my introspective understanding of this joy are different things” (pp. 273–274). Similarly, the Self t ’s event and the Self t + Δt ’s understanding of this event are two different things, because they address two different processes (occurring and understanding) and it involves two different Selfs ( Self t and Self t + Δt ). This refers to the third-person perspective with the Self observing the Self (Libby and Eibach 2014 ) rather than to the objective state of mind proposed by Noa et al. ( 2018 ) in which the Self perceives him/herself from the imagined perspective of another person. This proposal of multiple Selfs was already envisaged by Overgaard ( 2006 : 630): “ ‘The Self’ or the subject is obviously not identical to the content of his or her consciousness; for instance, the subject enjoys an uncountable number of conscious throughout his or her lifetime. Were the subject identical to conscious content, he or she would be as many selves as possible number of contents, continuously beginning and ceasing to exist.”

All this leads to the assumption that introspection by the Self t + Δt on the Self t might be possible during the replay interview. This assumption matches Rieken et al. ( 2015 ) writings; for them, introspection in SEBE effectively occurs during the self-confrontation interview when the subjects view subjective (first-person) recording of their activity. So how was this done in Rieken et al.’s study? The main problem with this work is that authors claimed using introspection and emphasized its social dimension, but they did not provide neither experimental evidence of that nor analysis of this process. Therefore, the contribution of introspection in Rieken et al.’s work and consequently of its social dimension remained at the stage of assumption.

The fact that the subject may undertake introspection by the Self t + Δt on the Self t provided that conditions of introspection are effective during the replay interview is fundamental. According to the literature review characterizing what these conditions may be (see Sect.  2.1.2 ), it is clear that if introspection happens during the replay interview, it is impossible to take place all along the interview, since the interview is mainly a detailed explanation of states of mind, goals, intents and actions that goes beyond the mere description of the mental state. Introspection in replay interview might take place from time to time, it might even be said that it happens very infrequently and lasts very short length of time for each. Indeed, during the replay interview, the Self t + Δt (the subject watching the sub-film) comments and explains most of the time what is doing the Self t (the subject in action in the sub-film) relying mainly on episodic memory; this is neither introspection nor retrospection. If introspection happens, it is when the Self t + Δt provides a comment on the Self t ’s action while viewing the introspected event in the sub-film or immediately after viewing it, combined with the fact that this comment is short and gives information about a Self t ’s mental state or cognitive process about which neither the Self t + Δt nor the Self t are aware of before making the comment.

This description additionally shows that, if it is introspection, it is nevertheless different from the forms proposed by the main streams (see Sect.  2.1.1 ) in the sense of the reviews of Overgaard ( 2006 ) and Danziger ( 2015 ): for these authors, introspection is an interaction by the Self t on the Self t, whereas here the interaction is by the Self t + Δt on the Self t . Instead of introspection (that we should name direct introspection for reasons of accuracy), in this case we had retrospective introspection [or indirect introspection as proposed by Titchener ( 1912 : 491)] which is triggered by self-confrontation and not opposite as claimed by others (see Rieken et al. 2015 : 260). As recalled by Kriegel ( 2013 : 1172), direct introspection occurs simultaneously with the introspected event (summoning short term memory), whereas indirect introspection occurs later and involves recollection of past events, relying on episodic memory or procedural memory (Piccinini 2003 ) (Fig.  1 ).

Lahlou’s analyses of replay interviews in SEBE suggested that indirect introspection might be effective during replay interviews. He noticed that replay interviews could address events that could remain unconscious and could be remembered with accuracy: “as the human machine is designed for economy of attention, most of what we can do without cluttering our limited attention and consciousness span is done automatically (that is: often below the conscious threshold), and not stored in explicit memory” (Lahlou 2011 : 620); “Probably because the recordings contain rich situated visual, auditory and kinetic cues which evoke re-enactment, participants recall with great detail their mental states at the time they acted, and can verbalize them, including their goals and sometimes sensations (e.g., thirst) […] It seems that the more similar the context of memory retrieval is to the context of memory encoding, the better is the recall, and that having multimodal cues helps, especially when they are spatial or motor—see the enactment effect (Engelkamp and Cohen 1991 ). In other words, re-living the situation from first-person perspective would facilitate recalling one’s own actions and mental states/processes” (Lahlou et al. 2015 : 5). What is recalled is better anchored in memory due to the fact that what is recalled was experienced in action involving both mind and body (see for example Engelkamp and Zimmer 1989 ; Cohen 1989 ; Engelkamp and Cohen 1991 ; Engelkamp et al. 2005 ; Madan and Singhal 2012 ; Schult et al. 2014 ). These considerations led us to assume that a kind of indirect introspection might occur during SEBE interviews, a kind of long term indirect introspection involving shifted verbal reports not immediately consecutive to the introspected event but delayed by several hours: ∆t would be several hours instead of several seconds; this might be called introspection provided that the introspected mental state could be recalled at the moment of the introspective act from a non-conscious memory.

3 Materials and methods

The model mobilized for introspection during the SEBE replay interview was this of an interaction of the Self t + Δt on the Self t relying on episodic memory. Δt is the lapse in time separating the realization of the activity and the replay interview. This assumed that introspection should occur in its indirect form.

The replay interview of a case study regarding the application of a SEBE method was considered. The SEBE method was applied to analyze activities at a French nuclear power plant during operating situations (results regarding the work activity analysis were already published (Fauquet-Alekhine and Daviet 2015 ) and performed by Operations shift teams as part of their routine work. The activity chosen for the study was performed by a nuclear reactor pilot in the control room, common to any pilots' work: in French "le tour de bloc" which may be translated as "block watch-around". This consists in watching and checking operating parameters in the control room. This activity lasts usually from 10 to 30 min. The SEBE-based analysis of this activity was undertaken in the framework of an overall improvement of workers’ competencies at Chinon nuclear power plant (France) and the activity was chosen for investigation addressing “introspection”, because it does not last long and does not imply complex technical gestures; the consecutive interview is, therefore, easy too. The protocol of the SEBE method is described in Sect.  3.3 . Among the cases analyzed in the framework of an overall improvement of workers’ competencies, one was chosen as a case study (Sect.  3.5 ) to undertake an analysis of how “introspection” could happen during the interview of self-confrontation in SEBE. Due to the heaviness of the analysis, it was limited to one case study, i.e., one subject and one occurrence of introspection. The protocol of the analysis for introspection during the replay interview is described in Sect.  3.4 .

3.2 Apparatus for the SEBE method applied to the case study

The digital ethnography equipment was made up of three parts linked with cables: (1) a micro audio digital recorder DVR-500-HD2 self-powered by internal batteries, not much bigger than a mobile phone, (2) a 4 mm diameter, 40 mm length miniaturized camera (subcam: Lahlou 1999 ) mounted on safety glasses, (3) a lavaliere microphone. This SEBE equipment was assembled from components produced at Active Media Concept (website: www.amc-tec.com ). This equipment fulfilled the requirements of video quality, energy autonomy, data storage, size and industrial environment disturbance. The main advantage of this equipment was to be adaptable to any kind of glasses (safety or vision).

3.3 Protocol for the SEBE method applied to the case study

The procedure applied for the SEBE-based work activity analysis was developed, tested and validated in a previous work (Fauquet-Alekhine 2016a ) and was applied in other studies (Fauquet-Alekhine and Lahlou 2017a , b ; Fauquet-Alekhine et al. 2018 ).

The schema of the protocol was the following:

At the beginning of the shift, the analyst exchanged with the Operations team during the shift briefing to identify who would be volunteer to be involved in the activity analysis.

Workers were informed of the aim of the study, signed an informed consent and undertook a risk assessment regarding the subcam used in real operating situation (Fauquet-Alekhine 2016b ; Fauquet-Alekhine et al. 2018 ). Then they were equipped with the subcam. This lasted about 5 min. for each subject (preparation phase).

Workers performed their activity (lasting from 10 to 30 min.), recording at the same time a subjective video (subfilm) of their activity (capture phase). During this phase, no verbalization (speaking out loud) was requested but this was not forbidden if the worker was used to doing so.

At the end of the activity, workers gave the subcam equipment back to the analyst and returned to their work; during this time, the analyst performed a pre-analysis of the subjective video for the workers to select video sequences of interest and prepare possible questions.

After the activity or during the following shifts, subjects were met individually to undertake the replay interview (analysis phase). The replay interview was based on self-confrontation and explicitation techniques. A post-analysis of the interviews (recorded) was carried out by the analyst to produce the output data regarding improvement of workers’ competencies (Fauquet-Alekhine and Daviet 2015 ).

These resulting output data were finally discussed with representatives of the professions for validation (validation phase).

As mentioned in Sect. 1 , the replay interview took from techniques of self-confrontation and explicitation interview described hereinafter. This method is similar to the cued recall debrief developed by Omodei and McLennan ( 1994 ) and applied by others (see for example Bentley et al. 2005 ; Rix and Biache 2004 ). In addition, during the replay interview, the work activity analysis was based on the Square of PErcieved ACtion model (SPEAC model) proposed by Fauquet-Alekhine ( 2016a ). The SPEAC model describes the necessary conditions for the subject to successfully put competencies in action and helps the analyst to question the worker regarding the motives of the activity, the professional experience, the means and the motivation. This thus helped the analyst to question the workers during the replay interview.

3.4 Protocol of the analysis for introspection during the replay interview

The SEBE replay interview was video recorded from a third-person perspective. It was then carefully analyzed to identify possible occurrences of introspection. The criteria to select moments of possible introspection were these defined from the literature background, the structure of these moments being compared with the characteristics of introspection provided by the characterization of introspection during SEBE interviews (see Sects.  2.1.2 to 2.1.4 ). A refined description of such a moment was used to understand the introspection process. Due to the heaviness of the analysis, only one occurrence of introspection was examined.

3.5 Subjects

Subjects’ participation to the SEBE-based work activity analysis was voluntary ( N  = 10). They filled in a form regarding individual data: gender, age and professional experience. Age was asked per range; for example, range 21–30 years, range 31–40 years and so on. Professional experience was asked in years according to the time spent in the position. Table 1 presents the average characteristics of the sample of subjects participating in the SEBE-based work activity analysis.

For analysis of the introspection, the selection of a subject was based on the least squares calculated using individual data and the mean values in Table 1 . The subject (reference subject: #OpJ) selected as a study case for introspection was male, presented the professional experience (5 years) closer to the mean value (4.8 years) and was in the second range of age (21–30 years); the selected subject presented the minimum value for least squares thus corresponding to the minimal distance from the mean values of gender, age and experience.

All participants signed an informed consent before undertaking experiments. This study received ethical approval (Code of Approval: DSP/RS/PFA-4) of the Ethics Committee of the Dept. of Social Psychology (LSE, UK) and has, therefore, been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki. The person selected in the case study has agreed to waive anonymity and appear in clear on the photographs. The person signed an informed agreement to this effect.

We considered the replay interview for the activity “block watch-around” undertaken with a subject (reference #OpS, experiment TEST-IND-OP-01) selected as a representative of the workers sample as explained in Sect.  3.5 . The video recording of the replay interview was carefully analyzed in the light of the model proposed to depict introspection in Sect.  3.4 .

During the replay interview which was video recorded, to depict the way he was checking monitors while performing the activity (excerpt of the subjective video in Fig.  2 ), the pilot said “je regarde si ça tire droit” (I watch to see whether it goes straight). Consecutive exchanges between the pilot and the analyst showed that this meant the pilot did not read the values of parameters on the monitors: it was easier and faster to check a signal position rather than read the value according to the scale of the monitors and compare it with the expected values. He said this was done without losing any reliability on values. The analysis for introspection during the replay interview allowed us to assuming an introspection occurrence when detecting this tacit knowledge during the subject-analyst interaction (illustrated in Fig.  3 ).

figure 2

Source: data subcam et al.\simu pil MS1 and MSI-062013\20130606 part1 (Op J)—file 1 Go—t = 07′54”

Excerpt of the pilot’s subfilm, while the video field remained unchanged for a few hundreds of milliseconds during the activity “bloc watch-around”. In the upper part of the picture, five paper monitors provide prints of physical parameters with time.

figure 3

Sequences of replay interview including introspection. a t r  = 18′09″—the subject stopped the video player showing the specific moment (subject’s mouth is closed). b t r  = 18′11″—the subject qualified the specific moment (subject’s mouth is opened). c t r  = 18′14″—the subject spontaneously gave details about the specific moment. d t r  = 18′37″—the subject explained the specific moment (with gesture). Note: t is the time associated with the replay interview recording

The sequences of the replay interview are described in the following (Reference File: interview replay OpJ 20,130,821—file 1 Go):

At t r  = 18′09″: Identification of specific moment for the Self t.

Note: t r is the time associated with the replay interview recording.

While the subject (the Self t + Δt ) and the analyst were together watching the activity video during the replay interview, the subject acting in the subfilm (the Self t ) stopped in front of monitors on the control panel (see excerpt of the subfilm in Fig.  2 : the video field remained unchanged for a few hundreds of milliseconds), suggesting that the subject was looking at or thinking about something. The analyst asked what was going on (t r  = 18′09″) and the subject, who had his right hand on the mouse, stopped the video player (t r  = 18′09″).

At t r  = 18′10″:

The subject said “je regarde si ça tire droit sur les enregistreurs” (I look to see whether it goes straight on the recording monitors). This was verbal reports by the Self t + Δt about the Self t to qualify the specific moment (from t r  = 18′10″ to 18′12″).

At t r  = 18′12″:

Continuing the first comment, the subject spontaneously gave details about the physical quantities monitored. This was verbal reports by the Self t + Δt about the Self t to spontaneously explain the specific moment (from t r  = 18′12″ until t r  = 18′14″).

At t r  = 18′26″:

The analyst took notes and then asked (t r  = 18′26″) what he meant by the expression “goes straight”. The pilot explained (t r  = 18′34″ until 18′54″) the goal of this way of working: he did not read the values of parameters for certain monitors or indicators; it was easier and faster to check a signal position rather than read the value according to the scale of the monitor and compare it with the expected value. Further questions and answers (after t r  = 18′54″ until 20′14″) showed that this was done without losing any reliability on values. When he was asked whether this practice was his own, the subject said that most of his colleagues (even all) did so. When he was asked, where he was taught this practice, he could not find any answer. This was verbal reports by the Self t + Δt about the Self t to explain the specific moment (from t r  = 18′26″ until t r  = 20′14″) when answering questions. During this sequence, verbal reports were not spontaneous but induced by questions, conversely to previous sequences.

This description shows that it followed a process made up of two distinct steps:

A focus of the subject Self t + Δt on a given mental state triggered by the analyst corresponding to the step “Identification of specific moment for the Self t ” at t r  = 18′09”.

A self-description of the mental state by the subject Self t + Δt through an immediate, consecutive brief verbal report corresponding to the step “Verbal reports from the Self t + Δt about the Self t to qualify the specific moment” from t r  = 18′10″ to 18′12”.

This two-time segment (identifying and qualifying) fulfils Gaillard’s criteria described in section.

4.1 Necessary for introspection

Instructions given to subject for verbalization were short enough (“what was going on ?”) to minimize the effect on the cognitive process of introspection: the verbal report was neither limited to a few possible categories such as ‘‘yes’’ or ‘‘no’’ nor forced to be an elaborated and exhaustive description or the introspected event.

Despite its metaphorical form, the verbal report was informative enough to be considered as complete regarding the subject’s own cognitive processes. It was verbalized in a simple form and thus minimized the disturbance of the current cognitive process in progress.

The verbal report was consistent with the third-person observation and analysis of the video undertaken by the analyst.

These elements match the model proposed to describe introspection during SEBE interview; an indirect introspection of the Selft+Δt on the Selft, where Δt is the lapse in time between the realization of the activity and the replay interview.

However, the indirect introspection was shifted in time by several hours when compared with the task carried out (average delay (in hours) for subjects involved in the SEBE-based work activity analysis was M  = 59.1 with SD = 39.6). When comparing the time related to the action (performing the block watch-around) and the time related to the occurrence of introspection, it is obvious that this did not comply with the maximum 5 s-criterion suggested by Ericsson for verbal reports (see Sect.  2.1.2 and Ericsson 2003 ). Measuring the time criterion this way means that the action relates to the Selft+Δt, while the occurrence of introspection relates to the Selft. The fact that the two points relate to different Selves need to be discussed.

Regarding the memory aspects, episodic memory is obviously called upon during the phase of verbal report to qualify the specific moment. Identifying the implication of other forms of memory should be discussed.

5 Discussion

5.1 process of introspection.

Section  4.1 characterized how indirect introspection could occur during a SEBE replay interview. Results obtained from the case study showed that indirect introspection occurred according to a two phase segment (identifying and qualifying):

“Identification of specific moment for Self t ” at t r  = 18′09”.

At this time of the replay interview, there was a focus of the subject Self t + Δt on a given mental state triggered by the analyst. This corresponded to the first introspective phase: the identification of the mental state. This was triggered by the analyst’s question, because the action going on at this moment on the subfilm was just a usual action in the subject’s opinion. This is why the subject neither stopped the video by himself nor made any spontaneous comment. While watching the video, the subject probably re-enacted the activity due to “situated visual, auditory and kinetic cues” contributing to “recall with great detail their mental states at the time they acted” as suggested by Lahlou et al. ( 2015 : 220). We may suggest here two possibilities regarding the cognitive process underpinning this identification: (1) the re-enactment effect recreated the same unconscious mental state for the subject while viewing the sub-film and the mind was made conscious at the moment of the analysts’ question, (2) the analysts’ question summoned the subject’s implicit memory storing previous experiences to carry out a task without conscious awareness of this (see Fig.  1 ), i.e., the procedural memory. Whatever the possibility explaining the process, it is highly likely that the mental state identified was unconscious for the subject before identification.

“Verbal reports from the Self t + Δt about the Self t ” to qualify the time segment t r  = 18′10″ to 18′12”.

At this time of the replay interview, there was a second introspective phase: the self-description of the mental state by the subject Self t + Δt .

The subject said “je regarde si ça tire droit sur les enregistreurs” (I watch to see whether it goes straight on the recording monitors). The metaphorical expression used for this immediate consecutive verbal report was natural, simple, and contained all the information necessary to depict the mental state, thus minimizing the disturbance of the current cognitive process (Titchener 1912 ; Ericsson 2003 ; Danziger 2015 ). The problem here was that all the information necessary to depict the mental state was concentrated and hidden within this simple metaphorical verbal report.

This two-time segment was then complemented by an explanation phase:

Verbal reports from the Self t + Δt about the Self t for explanation after t r  = 18′12”.

At this point, the indirect introspection was completed and the subject entered a post-introspective phase: the introspected mental state had become conscious and the subject was definitely engaged in a cognitive process of description. This phase relied on procedural memory to describe the metaphorical introspective verbal report and relied on episodic memory when the subject gave details regarding the parameters checked, the context of checking and the subject’s goals (Fig.  1 ).

This kind of long term indirect introspection might have happened several times per interview. They were so short (in the case study, the two-time segment did not last more than 3 s.) that they have gone unnoticed.

5.2 Characteristics of introspection

Given this, it is interesting to compare what was observed during the episode indirect introspection in the case study to the key points of the literature background (Sect.  2 ).

The necessity to implement strict and laboriously trained methodological procedures in the aim to access introspection is not verified for the subject: in the study case, the subject was a passive interviewee. The finding is less clear for the analyst: although not trained to investigate introspection, the analyst was an interviewer trained to apply self-confrontation with explicitation and a SPEAC-based questioning; this has nothing to do directly with introspection; however, this might have been a set a favoring factors.

The opposite effect of simple vs elaborate questions was verified: in the case study, a simple question led to a simple and short answer that gave rise to the occurrence of introspection (the two-time segment) and the following elaborate questions triggered an explanation phase, the subject being engaged in a cognitive process of description.

Finally, the time lapse suggested by Ericsson was verified provided that the correct elements are considered: as demonstrated in Sect.  2.1.4 , what must be compared in SEBE interviews is not the time related to the action vs the time related to the occurrence of introspection; this would result in comparing action and introspection from the only perspective of the Self t . Doing so, action and occurrence of introspection are related to the same Self. What must be considered is the action from the perspective of the Self t and the introspection from the perspective of the Self t + Δt . This lapse of time was estimated at 3 s, value of the same magnitude of this of Ericsson (about 5 s).

5.3 Discussion regarding benefits of introspection for SEBE

The fact that introspection occurs one or several times during a SEBE-based interview is interesting in terms of process. Provided that the analyst is able to seize such a moment (even when not being aware that introspection is in progress) and to obtain clarifications about what is described during the activity and what is felt by the subject, the moment of introspection is a benefit for the work activity analysis, because it focuses the analyst on a point of interest during the activity. This process may go unnoticed: it works unbeknownst to the analyst and the subject as demonstrated by the study case.

Similarly, identifying precisely moments of introspection and undertaking an in-depth analysis of the related process does not yet present any added-value for the SEBE-based interview during work activity analysis. This is illustrated when comparing the results presented in Sect.  4.2 and these of the SEBE-based work analysis in Fauquet-Alekhine and Daviet ( 2015 ): detecting introspection did neither change the results nor the conclusions of the study.

5.4 Metaphors

In addition, the study case associated a metaphorical expression with the introspection occurrence. We even argue that this metaphorical expression indicated that introspection was actual: the subject’s verbal report was informative enough to be considered as complete and was verbalized simply and thus minimized the disturbance of the current cognitive process. Perhaps this would not have been possible with a literal expression. This remark does not mean that all introspections are metaphorical or that, every time a metaphor is noted in the speech, introspection occurs.

As the language used by subjects to describe their work activity is not innocuous, the results of the case study show that metaphors may be a relevant source for understanding the activity beyond the literal meaning of the words or sentences. Psycholinguistic studies have been undertaken regarding discourse analysis and communication in the workplace (see for example: Roth 2004 ; Andrén et al. 2010 ; Fauquet-Alekhine 2010 , 2017a , b ) and have contributed to the demonstration that language through the discourse or the narrative may carry implicit depictions of the subjects’ activities. Among these implicit elements of language, metaphors may offer new angles of comprehension of the subjects’ activity (Glucksberg 2003 ; Steen 2011 ; Veraksa 2013 ). Therefore, the detection and analysis of metaphorical language as a tool for work analysis during the replay interview is an aspect which should be studied in more depth.

6 Conclusions

We reviewed controversies over introspection and followed the Brentano middle way between a Positivist (Comte, Mills et al.) categorical denial of validity and Wundtian dead ends of methodology to isolate ‘pure sensation’ with the right attitude. The key question was not whether introspection is valid, but under which circumstances. This posed the need for supporting methodology such as video replay and self-confrontation interviewing (e.g., SEBE method) which elicit verbal reports that are eminently informative for the analysis, understanding and improvement of work and other everyday activities.

The notion of introspection (direct and indirect) was clarified when occurring during a SEBE interview and criteria specifying circumstances of introspection occurrence were given. For the first time, an in-depth analysis of introspection during replay interviews was undertaken. It was found that introspection could occur during the SEBE-based interview and could remain unnoticed. Also, the form of introspection was found indirect ( Self t + Δt / Self t ) and the process of introspection was described as a two-time segment (identifying and qualifying) lasting a few seconds.

In the case study, we found that being aware of the occurrence of introspection did not give any added-value to the SEBE analysis; however, this occurrence, even being unnoticed, was an actual benefit by giving access to a non-explicit aspect of the work activity analyzed. Furthermore, metaphorical characters of the subjects’ speech could be a relevant indicator for introspection detection, easier to detect than introspection.

The association metaphor/introspection occurrence seems important: in the case study, introspection might have not happened without the metaphor. This remark highlights two points: (1) might introspection occurrence be always related to a metaphorical expression during a SEBE-based interview? (2) considering the fact that it helps the analysts to identify a tacit know-how, might introspection and/or metaphor be an indicator of a possible tacit knowledge or know-how? Both needs further experiments.

Acknowledgements

The authors thank all participants from the Nuclear Power Plant and the Training Center of Chinon (France) for their contribution. The authors thank especially Julien Castellarin, nuclear reactor pilot, who accepted to illustrate this paper with his image. The authors thank Active Media Concept ( www.amc-tec.com ) for efficient technical support. Research was financially supported by Electricité de France.

Author information

Authors and affiliations.

SEBE-Lab, Department of Psychological and Behavioural Sc., London School of Economics and Political Science, Houghton St., London, WC2A 2AE, UK

Philippe Fauquet-Alekhine

Lab. for Research in Science of Energy, H. Sc., Avoine, France

INTRA Robotics Lab, NPP Chinon, BP61, 37420, Avoine, France

Department of Psychological and Behavioural Sc., London School of Economics and Political Science, Houghton St., London, WC2A 2AE, UK

Martin W. Bauer & Saadi Lahlou

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Philippe Fauquet-Alekhine .

Ethics declarations

Conflict of interest.

Philippe Fauquet-Alekhine is Scientific Director INTRA Robotics lab (a branch of Electricité de France) and is former Human Factors Consultant, expert in Innovative Development for the Operational Professionalization at Chinon Nuclear Power Plant, Electricité de France (France).

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised due to retrospective open access.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Fauquet-Alekhine, P., Bauer, M.W. & Lahlou, S. Introspective interviewing for work activities: applying subjective digital ethnography in a nuclear industry case study. Cogn Tech Work 23 , 625–638 (2021). https://doi.org/10.1007/s10111-020-00662-9

Download citation

Received : 15 November 2019

Accepted : 08 December 2020

Published : 05 January 2021

Issue Date : August 2021

DOI : https://doi.org/10.1007/s10111-020-00662-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Activity analysis
  • Digital ethnography
  • Introspection
  • Find a journal
  • Publish with us
  • Track your research

Law Enforcement Learning

Cognitive Interviewing

Cognitive Interviewing is one of the most highly researched methods of investigative interviewing, drawing international support from the academic community as well as investigators within the law enforcement and security fields. This course uses videos, presentations, discussions, and real-world activities to teach cognitive interviewing skills, and students who complete the course will be able to employ cognitive interviewing techniques in a variety of law enforcement and security applications. We’ll start the course by examining the basics of a cognitive interview and observing an interview case study. Next, we’ll analyze the mechanics of a cognitive interview and explore essential interview elements such as setting, timing, and victimization. From there, we’ll learn how memory impacts cognition, explore the strengths and weaknesses of human memory, and gain insight into how individuals process events and how we can identify deceptive or unreliable information. After exploring memory, we’ll delve into questioning techniques and cover areas such as rapport, active listening, personalizing, open questioning, synchrony, and more. Finally, through various assignments, we’ll apply course information in real-world situations, hone our skills, and become competent cognitive interviews through trial and application. At the conclusion of this class, students will be able to:

  • Understand how human memory works
  • Use techniques to enhance memory and recall from victims and witnesses
  • Identify deceptive and unreliable information in interviews
  • Gather accurate facts
  • Distinguish between information coming from memory and information that is fabricated
  • Conduct cognitive interviews in a variety of contexts

This training will develop essential skills for patrol, first responders, and investigators. Detectives working “cold cases” may also find these skills helpful as memory can degrade over time.

About the Instructor

Wesley Clark

Wesley Clark

Wesley Clark retired from the Connecticut State Police Department after more than 22 years of service. The majority of his time was within the Major Crime Squad as a Detective and Detective Sergeant. He is the founder and President of LIES, LLC Linguistic Interrogation Expert Services, a company dedicated to training, research and consultation within the field of investigative interviewing, criminal interrogation, statement analysis, detecting deception and credibility assessment.

Change Password

Your password must have 6 characters or more:.

  • a lower case character, 
  • an upper case character, 
  • a special character 

Password Changed Successfully

Your password has been changed

Create your account

Forget yout password.

Enter your email address below and we will send you the reset instructions

If the address matches an existing account you will receive an email with instructions to reset your password

Forgot your Username?

Enter your email address below and we will send you your username

If the address matches an existing account you will receive an email with instructions to retrieve your username

Psychiatry Online

  • Winter 2024 | VOL. 36, NO. 1 CURRENT ISSUE pp.A5-81

The American Psychiatric Association (APA) has updated its Privacy Policy and Terms of Use , including with new information specifically addressed to individuals in the European Economic Area. As described in the Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences.

Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

Case Study 1: A 55-Year-Old Woman With Progressive Cognitive, Perceptual, and Motor Impairments

  • Scott M. McGinnis , M.D. ,
  • Andrew M. Stern , M.D., Ph.D. ,
  • Jared K. Woods , M.D., Ph.D. ,
  • Matthew Torre , M.D. ,
  • Mel B. Feany , M.D., Ph.D. ,
  • Michael B. Miller , M.D., Ph.D. ,
  • David A. Silbersweig , M.D. ,
  • Seth A. Gale , M.D. ,
  • Kirk R. Daffner , M.D.

Search for more papers by this author

CASE PRESENTATION

A 55-year-old right-handed woman presented with a 3-year history of cognitive changes. Early symptoms included mild forgetfulness—for example, forgetting where she left her purse or failing to remember to retrieve a take-out order her family placed—and word-finding difficulties. Problems with depth perception affected her ability to back her car out of the driveway. When descending stairs, she had to locate her feet visually in order to place them correctly, such that when she carried her dog and her view was obscured, she had difficulty managing this activity. She struggled to execute relatively simple tasks, such as inserting a plug into an outlet. She lost the ability to type on a keyboard, despite being able to move her fingers quickly. Her symptoms worsened progressively for 3 years, over which time she developed a sad mood and anxiety. She was laid off from work as a nurse administrator. Her family members assumed responsibility for paying her bills, and she ceased driving.

Her past medical history included high blood pressure, Hashimoto’s thyroiditis with thyroid peroxidase antibodies, remote history of migraine, and anxiety. Medications included mirtazapine, levothyroxine, calcium, and vitamin D. She had no history of smoking, drinking alcohol, or recreational drug use. There was no known family history of neurologic diseases.

What Are Diagnostic Considerations Based on the History? How Might a Clinical Examination Help to Narrow the Differential Diagnosis?

Insidious onset and gradual progression of cognitive symptoms over the course of several years raise concern for a neurodegenerative disorder. It is helpful to consider whether or not the presentation fits with a recognized neurodegenerative clinical syndrome, a judgment based principally on familiarity with syndromes and pattern recognition. Onset of symptoms before age 65 should prompt consideration of syndromes in the spectrum of frontotemporal dementia (FTD) and atypical (nonamnesic) presentations of Alzheimer’s disease (AD) ( 1 , 2 ). This patient’s symptoms reflect relatively prominent early dysfunction in visual-spatial processing and body schema, as might be observed in posterior cortical atrophy (PCA), although the history also includes mention of forgetfulness and word-retrieval difficulties. A chief goal of the cognitive examination would be to survey major domains of cognition—attention, executive functioning, memory, language, visual-spatial functioning, and higher somatosensory and motor functioning—to determine whether any domains stand out as more prominently affected. In addition to screening for evidence of focal signs, a neurological examination in this context should assess for evidence of parkinsonism or motor neuron disease, which can coexist with cognitive changes in neurodegenerative presentations.

The patient’s young age and history of Hashimoto’s thyroiditis might also prompt consideration of Hashimoto’s encephalopathy (HE; also known as steroid-responsive encephalopathy), associated with autoimmune thyroiditis. This syndrome is most likely attributable to an autoimmune or inflammatory process affecting the central nervous system. The time course of HE is usually more subacute and rapidly progressive or relapsing-remitting, as opposed to the gradual progression over months to years observed in the present case ( 3 ).

The patient’s mental status examination included the Montreal Cognitive Assessment (MoCA), a brief global screen of cognition ( 4 ), on which she scored 12/30. There was evidence of dysfunction across multiple cognitive domains ( Figure 1 ). She was fully oriented to location, day, month, year, and exact date. When asked to describe a complex scene from a picture in a magazine, she had great difficulty doing so, focusing on different details but having trouble directing her saccades to pertinent visual information. She likewise had problems directing her gaze to specified objects in the room and problems reaching in front of her to touch target objects in either visual field. In terms of other symptoms of higher order motor and somatosensory functioning, she had difficulty demonstrating previously learned actions—for example, positioning her hand correctly to pantomime holding a brush and combing her hair. She was confused about which side of her body was the left and which was the right. She had difficulty with mental calculations, even relatively simple ones such as “18 minus 12.” In addition, she had problems writing a sentence in terms of both grammar and the appropriate spacing of words and letters on the page.

FIGURE 1. Selected elements of a 55-year-old patient’s cognitive examination at presentation a

a BNT-15=Boston Naming Test (15-Item); MoCA=Montreal Cognitive Assessment.

On elementary neurologic examination she had symmetrically brisk reflexes, with spread. She walked steadily with a narrow base, but when asked to pass through a doorway she had difficulty finding her way through it and bumped into the door jamb. Her elemental neurological examination was otherwise normal, including but not limited to brisk, full-amplitude vertical eye movements, normal visual fields, no evidence of peripheral neuropathy, and no parkinsonian signs such as slowness of movement, tremor, or rigidity.

How Does the Examination Contribute to Our Understanding of Diagnostic Considerations? What Additional Tests or Studies Are Indicated?

The most prominent early symptoms and signs localize predominantly to the parietal association cortex: The patient has impairments in visual construction, ability to judge spatial relationships, ability to synthesize component parts of a visual scene into a coherent whole (simultanagnosia or asimultagnosia), impaired visually guided reaching for objects (optic ataxia), and most likely impaired ability to shift her visual attention so as to direct saccades to targets in her field of view (oculomotor apraxia or ocular apraxia). The last three signs constitute Bálint syndrome, which localizes to disruption of dorsal visual networks (i.e., dorsal stream) with key nodes in the posterior parietal and prefrontal cortices bilaterally ( 5 ). She has additional salient symptoms and signs suggesting left inferior parietal dysfunction, including ideomotor limb apraxia and elements of Gerstmann syndrome, which comprises dysgraphia, acalculia, left-right confusion, and finger agnosia ( 6 ). Information was not included about whether she was explicitly examined for finger agnosia, but elements of her presentation suggested a more generalized disruption of body schema (i.e., her representation of the position and configuration of her body in space). Her less prominent impairment in lexical-semantic retrieval evidenced by impaired confrontation naming and category fluency likely localizes to the language network in the left hemisphere. Her impairments in attention and executive functions have less localizing value but would plausibly arise in the context of frontoparietal network dysfunction. At this point, it is unclear whether her impairment in episodic memory mostly reflects encoding and activation versus a rapid rate of forgetting (storage), as occurs in temporolimbic amnesia. Regardless, it does not appear to be the most salient feature of her presentation.

This localization, presenting with insidious onset and gradual progression, is characteristic of a PCA syndrome. If we apply consensus clinical diagnostic criteria proposed by a working group of experts, we find that our patient has many of the representative features of early disturbance of visual functions plus or minus other cognitive functions mediated by the posterior cerebral cortex ( Table 1 ) ( 7 ). Some functions such as limb apraxia also occur in corticobasal syndrome (CBS), a clinical syndrome defined initially in association with corticobasal degeneration (CBD) neuropathology, a 4-repeat tauopathy characterized by achromatic ballooned neurons, neuropil threads, and astrocytic plaques. However, our patient lacks other suggestive features of CBS, including extrapyramidal motor dysfunction (e.g., limb rigidity, bradykinesia, dystonia), myoclonus, and alien limb phenomenon ( Table 1 ) ( 8 ).

a Consensus diagnostic criteria for posterior cortical atrophy per Crutch et al. ( 7 ) require at least three cognitive features and relative sparing of anterograde memory, speech-nonvisual language functions, executive functions, behavior, and personality. Diagnostic criteria for probable corticobasal syndrome per Armstrong et al. ( 8 ) require asymmetric presentation of at least two motor features and at least two higher cortical features. AD=Alzheimer’s disease; CBD=corticobasal degeneration; FDG-PET=[ 18 ]F-fluorodexoxyglucose positron emission tomography; JCD=Jakob-Creutzfeldt disease; LBD=Lewy body disease; PSP=progressive supranuclear palsy; SPECT=single-photon emission computed tomography; TDP=TDP–43 proteinopathy.

TABLE 1. Clinical features and neuropathological associations of posterior cortical atrophy and corticobasal syndrome a

In addition to a standard laboratory work-up for cognitive impairment, it is important to determine whether imaging of the brain provides evidence of neurodegeneration in a topographical distribution consistent with the clinical presentation. A first step in most cases would be to obtain an MRI of the brain that includes a high-resolution T 1 -weighted MRI sequence to assess potential atrophy, a T 2 /fluid-attenuated inversion recovery (FLAIR) sequence to assess the burden of vascular disease and rule out less likely etiological considerations (e.g., infection, autoimmune-inflammatory, neoplasm), a diffusion-weighted sequence to rule out subacute infarcts and prion disease (more pertinent to subacute or rapidly progressive cases), and a T 2 *-gradient echo or susceptibility weighted sequence to examine for microhemorrhages and superficial siderosis.

A lumbar puncture would serve two purposes. First, it would allow for the assessment of inflammation that might occur in HE, as approximately 80% of cases have some abnormality of CSF (i.e., elevated protein, lymphocytic pleiocytosis, or oligoclonal bands) ( 9 ). Second, in selected circumstances—particularly in cases with atypical nonamnesic clinical presentations or early-onset dementia in which AD is in the neuropathological differential diagnosis—we frequently pursue AD biomarkers of molecular neuropathology ( 10 , 11 ). This is most frequently accomplished with CSF analysis of amyloid-β-42, total tau, and phosphorylated tau levels. Amyloid positron emission tomography (PET) imaging, and most recently tau PET imaging, represent additional options that are approved by the U.S. Food and Drug Administration for clinical use. However, insurance often does not cover amyloid PET and currently does not reimburse tau PET imaging. [ 18 ]-F-fluorodeoxyglucose (FDG) PET and perfusion single-photon emission computed tomography imaging may provide indirect evidence for AD neuropathology via a pattern of hypometabolism or hypoperfusion involving the temporoparietal and posterior cingulate regions, though without molecular specificity. Pertinent to this case, a syndromic diagnosis of PCA is most commonly associated with underlying AD neuropathology—that is, plaques containing amyloid-β and neurofibrillary tangles containing tau ( 12 – 15 ).

The patient underwent MRI, demonstrating a minimal burden of T 2 /FLAIR hyperintensities and some degree of bilateral parietal volume loss with a left greater than right predominance ( Figure 2A ). There was relatively minimal medial temporal volume loss. Her basic laboratory work-up, including thyroid function, vitamin B 12 level, and treponemal antibody, was normal. She underwent a lumbar puncture; CSF studies revealed normal cell counts, protein, and glucose levels and low amyloid-β-42 levels at 165.9 pg/ml [>500 pg/ml] and elevated total and phosphorylated tau levels at 1,553 pg/ml [<350 pg/ml] and 200.4 pg/ml [<61 pg/ml], respectively.

FIGURE 2. MRI brain scan of the patient at presentation and 4 years later a

a Arrows denote regions of significant atrophy.

Considering This Additional Data, What Would Be an Appropriate Diagnostic Formulation?

For optimal clarity, we aim to provide a three-tiered approach to diagnosis comprising neurodegenerative clinical syndrome (e.g., primary amnesic, mixed amnesic and dysexecutive, primary progressive aphasia), level of severity (i.e., mild cognitive impairment; mild, moderate or severe dementia), and predicted underlying neuropathology (e.g., AD, Lewy body disease [LBD], frontotemporal lobar degeneration) ( 16 ). This approach avoids problematic conflations that cause confusion, for example when people equate AD with memory loss or dementia, whereas AD most strictly describes the neuropathology of plaques and tangles, regardless of the patient’s clinical symptoms and severity. This framework is important because there is never an exclusive, one-to-one correspondence between syndromic and neuropathological diagnosis. Syndromes arise from neurodegeneration that starts focally and progresses along the anatomical lines of large-scale brain networks that can be defined on the basis of both structural and functional connectivity, a concept detailed in the network degeneration hypothesis ( 17 ). It is important to note that neuropathologies defined on the basis of specific misfolded protein inclusions can target more than one large-scale network, and any given large-scale network can degenerate in association with more than one neuropathology.

The MRI results in this case support a syndromic diagnosis of PCA, with a posteriorly predominant pattern of atrophy. Given the patient’s loss of independent functioning in instrumental activities of daily living (ADLs), including driving and managing her finances, the patient would be characterized as having a dementia (also known as major neurocognitive disorder). The preservation of basic ADLs would suggest that the dementia was of mild severity. The CSF results provide supportive evidence for AD amyloid plaque and tau neurofibrillary tangle (NFT) neuropathology over other pathologies potentially associated with PCA syndrome (i.e., CBD, LBD, TDP-43 proteinopathy, and Jakob-Creutzfeldt disease) ( 13 , 14 ). The patient’s formulation would thus be best summarized as PCA at a level of mild dementia, likely associated with underlying AD neuropathology.

The patient’s symptoms progressed. One year after initial presentation, she had difficulty locating the buttons on her clothing or the food on her plate. Her word-finding difficulties worsened. Others observed stiffness of her right arm, a new symptom that was not present initially. She also had decreased ability using her right hand to hold everyday objects such as a comb, a brush, or a pen. On exam, she was noted to have rigidity of her right arm, impaired dexterity with her right hand for fine motor tasks, and a symmetrical tremor of the arms, apparent when holding objects or reaching. Her right hand would also intermittently assume a flexed, dystonic posture and would sometime move in complex ways without her having a sense of volitional control.

Four to 5 years after initial presentation, her functional status declined to the point where she was unable to feed, bathe, or dress herself. She was unable to follow simple instructions. She developed neuropsychiatric symptoms, including compulsive behaviors, anxiety, and apathy. Her right-sided motor symptoms progressed; she spent much of the time with her right arm flexed in abnormal postures or moving abnormally. She developed myoclonus of both arms. Her speech became slurred and monosyllabic. Her gait became less steady. She underwent a second MRI of the brain, demonstrating progressive bilateral atrophy involving the frontal and occipital lobes in addition to the parietal lobes and with more left > right asymmetry than was previously apparent ( Figure 2B ). Over time, she exhibited increasing weight loss. She was enrolled in hospice and ultimately passed away 8 years from the onset of symptoms.

Does Information About the Longitudinal Course of Her Illness Alter the Formulation About the Most Likely Underlying Neuropathological Process?

This patient developed clinical features characteristic of corticobasal syndrome over the longitudinal course of her disease. With time, it became apparent that she had lost volitional control over her right arm (characteristic of an alien limb phenomenon), and she developed signs more suggestive of basal ganglionic involvement (i.e., limb rigidity and possible dystonia). This presentation highlights the frequent overlap between neurodegenerative clinical syndromes; any given person may have elements of more than one syndrome, especially later in the course of a disease. In many instances, symptomatic features that are less prominent at presentation but evolve and progress can provide clues regarding the underlying neuropathological diagnosis. For example, a patient with primary progressive apraxia of speech or nonfluent-agrammatic primary progressive aphasia could develop the motor features of a progressive supranuclear palsy (PSP) clinical syndrome (e.g., supranuclear gaze impairment, axial rigidity, postural instability), which would suggest underlying PSP neuropathology (4-repeat tauopathy characterized by globose neurofibrillary tangles, tufted astrocytes, and oligodendroglial coiled bodies).

If CSF biomarker data were not suggestive of AD, the secondary elements of CBS would substantially increase the likelihood of underlying CBD neuropathology presenting with a PCA syndrome and evolving to a mixed PCA-CBS. But the CSF amyloid and tau levels are unambiguously suggestive of AD (i.e., very low amyloid-β-42 and very high p-tau levels), the neuropathology of which accounts for not only a vast majority of PCA presentations but also roughly a quarter of cases presenting with CBS ( 18 , 19 ). Thus, underlying AD appears most likely.

NEUROPATHOLOGY

On gross examination, the brain weighed 1,150 g, slightly less than the lower end of normal at 1,200 g. External examination demonstrated mild cortical atrophy with widening of the sulci, relatively symmetrical and uniform throughout the brain ( Figure 3A ). There was no evidence of atrophy of the brainstem or cerebellum. On cut sections, the hippocampus was mildly atrophic. The substantia nigra in the midbrain was intact, showing appropriate dark pigmentation as would be seen in a relatively normal brain. The remainder of the gross examination was unremarkable.

FIGURE 3. Mild cortical atrophy with posterior predominance and neurofibrillary tangles, granulovacuolar degeneration, and a Hirano body a

a Panel A shows the gross view of the brain, demonstrating mild cortical atrophy with posterior predominance (arrow). Panel B shows the hematoxylin and eosin of the hippocampus at high power, demonstrating neurofibrillary tangles, granulovacuolar degeneration, and a Hirano body.

Histological examination confirmed that the neurons in the substantia nigra were appropriately pigmented, with occasional extraneuronal neuromelanin and moderate neuronal loss. In the nucleus basalis of Meynert, NFTs were apparent on hematoxylin and eosin staining as dense fibrillar eosinophilic structures in the neuronal cytoplasm, confirmed by tau immunohistochemistry (IHC; Figure 4 ). Low-power examination of the hippocampus revealed neuronal loss in the subiculum and in Ammon’s horn, most pronounced in the cornu ammonis 1 (CA1) subfield, with a relatively intact neuronal population in the dentate gyrus. Higher power examination with hematoxylin and eosin demonstrated numerous NFTs, neurons exhibiting granulovacuolar degeneration, and Hirano bodies ( Figure 3B ). Tau IHC confirmed numerous NFTs in the CA1 region and the subiculum. Amyloid-β IHC demonstrated occasional amyloid plaques in this region, less abundant than tau pathology. An α-synuclein stain revealed scattered Lewy bodies in the hippocampus and in the amygdala.

FIGURE 4. Tau immunohistochemistry demonstrating neurofibrillary tangles (staining brown) in the nucleus basalis of Meynert, in the hippocampus, and in the cerebral cortex of the frontal, temporal, parietal, and occipital lobes

In the neocortex, tau IHC highlighted the extent of the NFTs, which were very prominent in all of the lobes from which sections were taken: frontal, temporal, parietal and occipital. Numerous plaques on amyloid-β stain were likewise present in all cortical regions examined. The tau pathology was confined to the gray matter, sparing white matter. There were no ballooned neurons and no astrocytic plaques—two findings one would expect to see in CBD ( Table 2 ).

a AD=Alzheimer’s disease; CBD=corticobasal degeneration; CBS=corticobasal syndrome; PCA=posterior cortical atrophy.

TABLE 2. Neuropathological features of this case compared with a case of corticobasal degeneration a

The case was designated by the neuropathology division as Alzheimer’s-type pathology, Braak stage V–VI (of VI), due to the widespread neocortical tau pathology, with LBD primarily in the limbic areas.

Our patient had AD neuropathology presenting atypically with a young age at onset (52 years old) and a predominantly visual-spatial and corticobasal syndrome as opposed to prominent amnesia. Syndromic diversity is a well-recognized phenomenon in AD. Nonamnesic presentations include not only PCA and CBS but also the logopenic variant of primary progressive aphasia and a behavioral-dysexecutive syndrome ( 20 ). Converging lines of evidence link the topographical distribution of NFTs with syndromic presentations and the pattern of hypometabolism and cortical atrophy. Neuropathological case reports and case series suggest that atypical AD syndromes arise in the setting of higher than normal densities of NFTs in networks subserving the functions compromised, including visual association areas in PCA-AD ( 21 ), the language network in PPA-AD ( 22 ), and frontal regions in behavioral-dysexecutive AD ( 23 ). In a large sample of close to 900 cases of pathologically diagnosed AD employing quantitative assessment of NFT density and distribution in selected neocortical and hippocampal regions, 25% of cases did not conform to a typical distribution of NFTs characterized in the Braak staging scheme ( 24 ). A subset of cases classified as hippocampal sparing with higher density of NFTs in the neocortex and lower density of NFTs in the hippocampus had a younger mean age at onset, higher frequency of atypical (nonamnesic) presentations, and more rapid rate of longitudinal decline than subsets defined as typical or limbic-predominant.

Tau PET, which detects the spatial distribution of fibrillary tau present in NFTs, has corroborated postmortem work in demonstrating distinct patterns of tracer uptake in different subtypes of AD defined by clinical symptoms and topographical distributions of atrophy ( 25 – 28 ). Amyloid PET, which detects the spatial distribution of fibrillar amyloid- β found in amyloid plaques, does not distinguish between typical and atypical AD ( 29 , 30 ). In a longitudinal study of 32 patients at early symptomatic stages of AD, the baseline topography of tau PET signal predicted subsequent atrophy on MRI at the single patient level, independent of baseline cortical thickness ( 31 ). This correlation was strongest in early-onset AD patients, who also tended to have higher tau signal and more rapid progression of atrophy than late-onset AD patients.

Differential vulnerability of selected large-scale brain networks in AD and in neurodegenerative disease more broadly remains poorly understood. There is evidence to support multiple mechanisms that are not mutually exclusive, including metabolic stress to key network nodes, trophic failure, transneuronal spread of pathological proteins (i.e., prion-like mechanisms), and shared vulnerability within network regions based on genetic or developmental factors ( 32 ). In the case of AD, cortical hub regions with high intrinsic functional connectivity to other regions across the brain appear to have high metabolic rates across the lifespan and to be foci of convergence of amyloid-β and tau accumulation ( 33 , 34 ). Tau NFT pathology appears to spread temporally along connected networks within the brain ( 35 ). Patients with primary progressive aphasia are more likely to have a personal or family history of developmental language-based learning disability ( 36 ), and patients with PCA are more likely to have a personal history of mathematical or visuospatial learning disability ( 37 ).

This case highlights the symptomatic heterogeneity in AD and the value of a three-tiered approach to diagnostic formulation in neurodegenerative presentations. It is important to remember that not all AD presents with amnesia and that early-onset AD tends to be more atypical and to progress more rapidly than late-onset AD. Multiple lines of evidence support a relationship between the burden and topographical distribution of tau NFT neuropathology and clinical symptomatology in AD, instantiating network-based neurodegeneration via mechanisms under ongoing investigation.

The authors report no financial relationships with commercial interests.

Supported by NIH grants K08 AG065502 (to Dr. Miller) and T32 HL007627 (to Dr. Miller).

The authors have confirmed that details of the case have been disguised to protect patient privacy.

1 Balasa M, Gelpi E, Antonell A, et al. : Clinical features and APOE genotype of pathologically proven early-onset Alzheimer disease . Neurology 2011 ; 76:1720–1725 Crossref , Medline ,  Google Scholar

2 Mercy L, Hodges JR, Dawson K, et al. : Incidence of early-onset dementias in Cambridgeshire, United Kingdom . Neurology 2008 ; 71:1496–1499 Crossref , Medline ,  Google Scholar

3 Kothbauer-Margreiter I, Sturzenegger M, Komor J, et al. : Encephalopathy associated with Hashimoto thyroiditis: diagnosis and treatment . J Neurol 1996 ; 243:585–593 Crossref , Medline ,  Google Scholar

4 Nasreddine ZS, Phillips NA, Bédirian V, et al. : The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment . J Am Geriatr Soc 2005 ; 53:695–699 Crossref , Medline ,  Google Scholar

5 Rizzo M, Vecera SP : Psychoanatomical substrates of Bálint’s syndrome . J Neurol Neurosurg Psychiatry 2002 ; 72:162–178 Crossref , Medline ,  Google Scholar

6 Rusconi E : Gerstmann syndrome: historic and current perspectives . Handb Clin Neurol 2018 ; 151:395–411 Crossref , Medline ,  Google Scholar

7 Crutch SJ, Schott JM, Rabinovici GD, et al. : Consensus classification of posterior cortical atrophy . Alzheimers Dement 2017 ; 13:870–884 Crossref , Medline ,  Google Scholar

8 Armstrong MJ, Litvan I, Lang AE, et al. : Criteria for the diagnosis of corticobasal degeneration . Neurology 2013 ; 80:496–503 Crossref , Medline ,  Google Scholar

9 Marshall GA, Doyle JJ : Long-term treatment of Hashimoto’s encephalopathy . J Neuropsychiatry Clin Neurosci 2006 ; 18:14–20 Link ,  Google Scholar

10 Johnson KA, Minoshima S, Bohnen NI, et al. : Appropriate use criteria for amyloid PET: a report of the Amyloid Imaging Task Force, the Society of Nuclear Medicine and Molecular Imaging, and the Alzheimer’s Association . Alzheimers Dement 2013 ; 9:e-1–e-16 Crossref , Medline ,  Google Scholar

11 Shaw LM, Arias J, Blennow K, et al. : Appropriate use criteria for lumbar puncture and cerebrospinal fluid testing in the diagnosis of Alzheimer’s disease . Alzheimers Dement 2018 ; 14:1505–1521 Crossref , Medline ,  Google Scholar

12 Alladi S, Xuereb J, Bak T, et al. : Focal cortical presentations of Alzheimer’s disease . Brain 2007 ; 130:2636–2645 Crossref , Medline ,  Google Scholar

13 Renner JA, Burns JM, Hou CE, et al. : Progressive posterior cortical dysfunction: a clinicopathologic series . Neurology 2004 ; 63:1175–1180 Crossref , Medline ,  Google Scholar

14 Tang-Wai DF, Graff-Radford NR, Boeve BF, et al. : Clinical, genetic, and neuropathologic characteristics of posterior cortical atrophy . Neurology 2004 ; 63:1168–1174 Crossref , Medline ,  Google Scholar

15 Victoroff J, Ross GW, Benson DF, et al. : Posterior cortical atrophy: neuropathologic correlations . Arch Neurol 1994 ; 51:269–274 Crossref , Medline ,  Google Scholar

16 Dickerson BC, McGinnis SM, Xia C, et al. : Approach to atypical Alzheimer’s disease and case studies of the major subtypes . CNS Spectr 2017 ; 22:439–449 Crossref , Medline ,  Google Scholar

17 Seeley WW, Crawford RK, Zhou J, et al. : Neurodegenerative diseases target large-scale human brain networks . Neuron 2009 ; 62:42–52 Crossref , Medline ,  Google Scholar

18 Lee SE, Rabinovici GD, Mayo MC, et al. : Clinicopathological correlations in corticobasal degeneration . Ann Neurol 2011 ; 70:327–340 Crossref , Medline ,  Google Scholar

19 Whitwell JL, Jack CR Jr, Boeve BF, et al. : Imaging correlates of pathology in corticobasal syndrome . Neurology 2010 ; 75:1879–1887 Crossref , Medline ,  Google Scholar

20 Warren JD, Fletcher PD, Golden HL : The paradox of syndromic diversity in Alzheimer disease . Nat Rev Neurol 2012 ; 8:451–464 Crossref , Medline ,  Google Scholar

21 Hof PR, Archin N, Osmand AP, et al. : Posterior cortical atrophy in Alzheimer’s disease: analysis of a new case and re-evaluation of a historical report . Acta Neuropathol 1993 ; 86:215–223 Crossref , Medline ,  Google Scholar

22 Mesulam MM, Weintraub S, Rogalski EJ, et al. : Asymmetry and heterogeneity of Alzheimer’s and frontotemporal pathology in primary progressive aphasia . Brain 2014 ; 137:1176–1192 Crossref , Medline ,  Google Scholar

23 Blennerhassett R, Lillo P, Halliday GM, et al. : Distribution of pathology in frontal variant Alzheimer’s disease . J Alzheimers Dis 2014 ; 39:63–70 Crossref , Medline ,  Google Scholar

24 Murray ME, Graff-Radford NR, Ross OA, et al. : Neuropathologically defined subtypes of Alzheimer’s disease with distinct clinical characteristics: a retrospective study . Lancet Neurol 2011 ; 10:785–796 Crossref , Medline ,  Google Scholar

25 Ossenkoppele R, Lyoo CH, Sudre CH, et al. : Distinct tau PET patterns in atrophy-defined subtypes of Alzheimer’s disease . Alzheimers Dement 2020 ; 16:335–344 Crossref , Medline ,  Google Scholar

26 Phillips JS, Das SR, McMillan CT, et al. : Tau PET imaging predicts cognition in atypical variants of Alzheimer’s disease . Hum Brain Mapp 2018 ; 39:691–708 Crossref , Medline ,  Google Scholar

27 Tetzloff KA, Graff-Radford J, Martin PR, et al. : Regional distribution, asymmetry, and clinical correlates of tau uptake on [18F]AV-1451 PET in atypical Alzheimer’s disease . J Alzheimers Dis 2018 ; 62:1713–1724 Crossref , Medline ,  Google Scholar

28 Xia C, Makaretz SJ, Caso C, et al. : Association of in vivo [18F]AV-1451 tau PET imaging results with cortical atrophy and symptoms in typical and atypical Alzheimer disease . JAMA Neurol 2017 ; 74:427–436 Crossref , Medline ,  Google Scholar

29 Formaglio M, Costes N, Seguin J, et al. : In vivo demonstration of amyloid burden in posterior cortical atrophy: a case series with PET and CSF findings . J Neurol 2011 ; 258:1841–1851 Crossref , Medline ,  Google Scholar

30 Lehmann M, Ghosh PM, Madison C, et al. : Diverging patterns of amyloid deposition and hypometabolism in clinical variants of probable Alzheimer’s disease . Brain 2013 ; 136:844–858 Crossref , Medline ,  Google Scholar

31 La Joie R, Visani AV, Baker SL, et al. : Prospective longitudinal atrophy in Alzheimer’s disease correlates with the intensity and topography of baseline tau-PET . Sci Transl Med 2020 ; 12:12 Crossref ,  Google Scholar

32 Zhou J, Gennatas ED, Kramer JH, et al. : Predicting regional neurodegeneration from the healthy brain functional connectome . Neuron 2012 ; 73:1216–1227 Crossref , Medline ,  Google Scholar

33 Buckner RL, Sepulcre J, Talukdar T, et al. : Cortical hubs revealed by intrinsic functional connectivity: mapping, assessment of stability, and relation to Alzheimer’s disease . J Neurosci 2009 ; 29:1860–1873 Crossref , Medline ,  Google Scholar

34 Hoenig MC, Bischof GN, Seemiller J, et al. : Networks of tau distribution in Alzheimer’s disease . Brain 2018 ; 141:568–581 Crossref , Medline ,  Google Scholar

35 Liu L, Drouet V, Wu JW, et al. : Trans-synaptic spread of tau pathology in vivo . PLoS One 2012 ; 7:e31302 Crossref , Medline ,  Google Scholar

36 Rogalski E, Johnson N, Weintraub S, et al. : Increased frequency of learning disability in patients with primary progressive aphasia and their first-degree relatives . Arch Neurol 2008 ; 65:244–248 Crossref , Medline ,  Google Scholar

37 Miller ZA, Rosenberg L, Santos-Santos MA, et al. : Prevalence of mathematical and visuospatial learning disabilities in patients with posterior cortical atrophy . JAMA Neurol 2018 ; 75:728–737 Crossref , Medline ,  Google Scholar

  • Jeffrey Maneval , M.D. ,
  • Kirk R. Daffner , M.D. ,
  • Scott M. McGinnis , M.D.
  • Seth A. Gale , M.A., M.D. ,
  • C. Alan Anderson , M.D. ,
  • David B. Arciniegas , M.D.

cognitive interview case study

  • Posterior Cortical Atrophy
  • Corticobasal Syndrome
  • Atypical Alzheimer Disease
  • Network Degeneration
  • Bipolar Disorder
  • Therapy Center
  • When To See a Therapist
  • Types of Therapy
  • Best Online Therapy
  • Best Couples Therapy
  • Best Family Therapy
  • Managing Stress
  • Sleep and Dreaming
  • Understanding Emotions
  • Self-Improvement
  • Healthy Relationships
  • Student Resources
  • Personality Types
  • Verywell Mind Insights
  • 2023 Verywell Mind 25
  • Mental Health in the Classroom
  • Editorial Process
  • Meet Our Review Board
  • Crisis Support

What Is a Case Study?

Weighing the pros and cons of this method of research

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

cognitive interview case study

Cara Lustik is a fact-checker and copywriter.

cognitive interview case study

Verywell / Colleen Tighe

  • Pros and Cons

What Types of Case Studies Are Out There?

Where do you find data for a case study, how do i write a psychology case study.

A case study is an in-depth study of one person, group, or event. In a case study, nearly every aspect of the subject's life and history is analyzed to seek patterns and causes of behavior. Case studies can be used in many different fields, including psychology, medicine, education, anthropology, political science, and social work.

The point of a case study is to learn as much as possible about an individual or group so that the information can be generalized to many others. Unfortunately, case studies tend to be highly subjective, and it is sometimes difficult to generalize results to a larger population.

While case studies focus on a single individual or group, they follow a format similar to other types of psychology writing. If you are writing a case study, we got you—here are some rules of APA format to reference.  

At a Glance

A case study, or an in-depth study of a person, group, or event, can be a useful research tool when used wisely. In many cases, case studies are best used in situations where it would be difficult or impossible for you to conduct an experiment. They are helpful for looking at unique situations and allow researchers to gather a lot of˜ information about a specific individual or group of people. However, it's important to be cautious of any bias we draw from them as they are highly subjective.

What Are the Benefits and Limitations of Case Studies?

A case study can have its strengths and weaknesses. Researchers must consider these pros and cons before deciding if this type of study is appropriate for their needs.

One of the greatest advantages of a case study is that it allows researchers to investigate things that are often difficult or impossible to replicate in a lab. Some other benefits of a case study:

  • Allows researchers to capture information on the 'how,' 'what,' and 'why,' of something that's implemented
  • Gives researchers the chance to collect information on why one strategy might be chosen over another
  • Permits researchers to develop hypotheses that can be explored in experimental research

On the other hand, a case study can have some drawbacks:

  • It cannot necessarily be generalized to the larger population
  • Cannot demonstrate cause and effect
  • It may not be scientifically rigorous
  • It can lead to bias

Researchers may choose to perform a case study if they want to explore a unique or recently discovered phenomenon. Through their insights, researchers develop additional ideas and study questions that might be explored in future studies.

It's important to remember that the insights from case studies cannot be used to determine cause-and-effect relationships between variables. However, case studies may be used to develop hypotheses that can then be addressed in experimental research.

Case Study Examples

There have been a number of notable case studies in the history of psychology. Much of  Freud's work and theories were developed through individual case studies. Some great examples of case studies in psychology include:

  • Anna O : Anna O. was a pseudonym of a woman named Bertha Pappenheim, a patient of a physician named Josef Breuer. While she was never a patient of Freud's, Freud and Breuer discussed her case extensively. The woman was experiencing symptoms of a condition that was then known as hysteria and found that talking about her problems helped relieve her symptoms. Her case played an important part in the development of talk therapy as an approach to mental health treatment.
  • Phineas Gage : Phineas Gage was a railroad employee who experienced a terrible accident in which an explosion sent a metal rod through his skull, damaging important portions of his brain. Gage recovered from his accident but was left with serious changes in both personality and behavior.
  • Genie : Genie was a young girl subjected to horrific abuse and isolation. The case study of Genie allowed researchers to study whether language learning was possible, even after missing critical periods for language development. Her case also served as an example of how scientific research may interfere with treatment and lead to further abuse of vulnerable individuals.

Such cases demonstrate how case research can be used to study things that researchers could not replicate in experimental settings. In Genie's case, her horrific abuse denied her the opportunity to learn a language at critical points in her development.

This is clearly not something researchers could ethically replicate, but conducting a case study on Genie allowed researchers to study phenomena that are otherwise impossible to reproduce.

There are a few different types of case studies that psychologists and other researchers might use:

  • Collective case studies : These involve studying a group of individuals. Researchers might study a group of people in a certain setting or look at an entire community. For example, psychologists might explore how access to resources in a community has affected the collective mental well-being of those who live there.
  • Descriptive case studies : These involve starting with a descriptive theory. The subjects are then observed, and the information gathered is compared to the pre-existing theory.
  • Explanatory case studies : These   are often used to do causal investigations. In other words, researchers are interested in looking at factors that may have caused certain things to occur.
  • Exploratory case studies : These are sometimes used as a prelude to further, more in-depth research. This allows researchers to gather more information before developing their research questions and hypotheses .
  • Instrumental case studies : These occur when the individual or group allows researchers to understand more than what is initially obvious to observers.
  • Intrinsic case studies : This type of case study is when the researcher has a personal interest in the case. Jean Piaget's observations of his own children are good examples of how an intrinsic case study can contribute to the development of a psychological theory.

The three main case study types often used are intrinsic, instrumental, and collective. Intrinsic case studies are useful for learning about unique cases. Instrumental case studies help look at an individual to learn more about a broader issue. A collective case study can be useful for looking at several cases simultaneously.

The type of case study that psychology researchers use depends on the unique characteristics of the situation and the case itself.

There are a number of different sources and methods that researchers can use to gather information about an individual or group. Six major sources that have been identified by researchers are:

  • Archival records : Census records, survey records, and name lists are examples of archival records.
  • Direct observation : This strategy involves observing the subject, often in a natural setting . While an individual observer is sometimes used, it is more common to utilize a group of observers.
  • Documents : Letters, newspaper articles, administrative records, etc., are the types of documents often used as sources.
  • Interviews : Interviews are one of the most important methods for gathering information in case studies. An interview can involve structured survey questions or more open-ended questions.
  • Participant observation : When the researcher serves as a participant in events and observes the actions and outcomes, it is called participant observation.
  • Physical artifacts : Tools, objects, instruments, and other artifacts are often observed during a direct observation of the subject.

If you have been directed to write a case study for a psychology course, be sure to check with your instructor for any specific guidelines you need to follow. If you are writing your case study for a professional publication, check with the publisher for their specific guidelines for submitting a case study.

Here is a general outline of what should be included in a case study.

Section 1: A Case History

This section will have the following structure and content:

Background information : The first section of your paper will present your client's background. Include factors such as age, gender, work, health status, family mental health history, family and social relationships, drug and alcohol history, life difficulties, goals, and coping skills and weaknesses.

Description of the presenting problem : In the next section of your case study, you will describe the problem or symptoms that the client presented with.

Describe any physical, emotional, or sensory symptoms reported by the client. Thoughts, feelings, and perceptions related to the symptoms should also be noted. Any screening or diagnostic assessments that are used should also be described in detail and all scores reported.

Your diagnosis : Provide your diagnosis and give the appropriate Diagnostic and Statistical Manual code. Explain how you reached your diagnosis, how the client's symptoms fit the diagnostic criteria for the disorder(s), or any possible difficulties in reaching a diagnosis.

Section 2: Treatment Plan

This portion of the paper will address the chosen treatment for the condition. This might also include the theoretical basis for the chosen treatment or any other evidence that might exist to support why this approach was chosen.

  • Cognitive behavioral approach : Explain how a cognitive behavioral therapist would approach treatment. Offer background information on cognitive behavioral therapy and describe the treatment sessions, client response, and outcome of this type of treatment. Make note of any difficulties or successes encountered by your client during treatment.
  • Humanistic approach : Describe a humanistic approach that could be used to treat your client, such as client-centered therapy . Provide information on the type of treatment you chose, the client's reaction to the treatment, and the end result of this approach. Explain why the treatment was successful or unsuccessful.
  • Psychoanalytic approach : Describe how a psychoanalytic therapist would view the client's problem. Provide some background on the psychoanalytic approach and cite relevant references. Explain how psychoanalytic therapy would be used to treat the client, how the client would respond to therapy, and the effectiveness of this treatment approach.
  • Pharmacological approach : If treatment primarily involves the use of medications, explain which medications were used and why. Provide background on the effectiveness of these medications and how monotherapy may compare with an approach that combines medications with therapy or other treatments.

This section of a case study should also include information about the treatment goals, process, and outcomes.

When you are writing a case study, you should also include a section where you discuss the case study itself, including the strengths and limitiations of the study. You should note how the findings of your case study might support previous research. 

In your discussion section, you should also describe some of the implications of your case study. What ideas or findings might require further exploration? How might researchers go about exploring some of these questions in additional studies?

Need More Tips?

Here are a few additional pointers to keep in mind when formatting your case study:

  • Never refer to the subject of your case study as "the client." Instead, use their name or a pseudonym.
  • Read examples of case studies to gain an idea about the style and format.
  • Remember to use APA format when citing references .

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach .  BMC Med Res Methodol . 2011;11:100.

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach . BMC Med Res Methodol . 2011 Jun 27;11:100. doi:10.1186/1471-2288-11-100

Gagnon, Yves-Chantal.  The Case Study as Research Method: A Practical Handbook . Canada, Chicago Review Press Incorporated DBA Independent Pub Group, 2010.

Yin, Robert K. Case Study Research and Applications: Design and Methods . United States, SAGE Publications, 2017.

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

IMAGES

  1. PPT

    cognitive interview case study

  2. Cognitive Interview

    cognitive interview case study

  3. Cognitive Interviewing Methodologies

    cognitive interview case study

  4. Cognitive Interviewing: A Few Best Practices

    cognitive interview case study

  5. A Guide for Case Study Interview Presentations for Beginners

    cognitive interview case study

  6. PPT

    cognitive interview case study

VIDEO

  1. CASE STUDY OF AN COGNITIVE ADOLESCENCE

  2. Research Methods-Chapter 5 (Focus Group, Intensive Interview, Case Study)

  3. Solution focused Interview first attempt

  4. What skills are relevant in case interviews? #shorts

  5. Cognitive Interview Webinar: A Sneak Peek

  6. Ignoring partners in an interview (Case Interview & Management Consulting classics)

COMMENTS

  1. Cognitive Interview Technique

    Geiselman (1985) Cognitive Interview Evaluation Findings concerning the unreliability of eyewitness accounts have led researchers to attempt to devise methods for improving retrieval. One of these methods is the cognitive interview (Fisher & Geiselman, 1992).

  2. Cognitive Interviewing Methodologies

    Introduction Cognitive interviewing (CI), also known as cognitive testing (CT) or cognitive debriefing (CD), is a qualitative survey development method used in questionnaire design and should not be confused with cognitive interviewing to assess mental status.

  3. What do you think it means? Using cognitive interviewing to improve

    Cognitive interviews can be tailored to the goals of an implementation study. Given implementation research often includes a broad range of academic and community partners, interviews can be tailored for specific partner groups, to assess specific parts of a measure (e.g., instructions, terms, response options), to examine the relevance of the ...

  4. Cleo Smith case: how 'cognitive interviewing' can help police compile

    Building the case. Child interviewing techniques are based on research about memory and cognition. Creating a safe and comfortable space for the child will help them recall crucial details, and ...

  5. Research Synthesis: The Practice of Cognitive Interviewing

    This synthesis reviews the range of current cognitive interviewing practices, focusing on three considerations: (1) what are the dominant paradigms of cognitive interviewing—what is produced under each, and what are their apparent advantages; (2) what key decisions about cognitive interview study design need to be made once the general approach ...

  6. PDF Cognitive Interviewing

    Cognitive interviewer training. Cognitive interviewing is an acquired skill, consisting of a number of separate sub-skills. Optimally, good interviewers can serve as "detectives" who can find problems in survey questions, and as "engineers" who can work toward developing workable solutions to the problems defined.

  7. Cognitive Interviewing: Lessons Learned and Recommendations for

    Cognitive interviewing is a structured research method that assists researchers in designing survey questions that clearly and accurately operationalize construct (s) of interest for use with one or more sub-populations.

  8. Cognitive Interviewing Methodologies

    Cognitive interviewing, one of the most widely used qualitative methods for evaluating how individuals mentally process and respond to survey questionnaires and covers a set of techniques (e.g., think aloud protocols, verbal probes) that enable a researcher to analyze how respondents understand the survey questions ( Ryan et al., 2012 ).

  9. An experimental test of the effectiveness of cognitive interviewing in

    Abstract Pretesting survey questions via cognitive interviewing is based on the assumptions that the problems identified by the method truly exist in a later survey and that question revisions based on cognitive interviewing findings produce higher-quality data than the original questions.

  10. Using cognitive interviews and think-aloud protocols to understand

    SJTs are case-based tests where examinees evaluate potential responses to dilemmas or difficult scenarios, and these responses are designed to reflect attributes such as empathy, integrity, and adaptability. SJTs often require complex decision-making or problem-solving processes.

  11. Effective questionnaire design: How to use cognitive interviews to

    Background: Cognitive interviewing is a technique that can be used to improve and refine questionnaire items. We describe the basic methodology of cognitive interviewing and illustrate its utility through our experience using cognitive interviews to refine a questionnaire assessing parental understanding of concepts related to preterm birth.

  12. Cognitive Interviewing Methodology

    With a primary focus on question evaluation, Cognitive Interviewing Methodology also includes: Step-by-step procedures for conducting cognitive interviewing studies, which includes the various aspects of data collection, questionnaire design, and data interpretation Newly developed tools to benefit cognitive interviewing studies as well as...

  13. The cognitive interview: Its origins, empirical support, evaluation and

    Abstract This paper provides a background to current research on the cognitive interview (CI), which is a set of cognitive retrieval techniques designed to facilitate memory search (for example, via reinstatement of contextual cues).

  14. What do you think it means? Using cognitive interviewing to improve

    Pragmatic measures are essential to evaluate the implementation of evidence-based interventions. Cognitive interviewing, a qualitative method that collects partner feedback throughout measure development, is particularly useful for developing pragmatic implementation measures. Measure developers can use cognitive interviewing to increase a measure's fit within a particular implementation ...

  15. Cognitive interview

    The cognitive interview ( CI) is a method of interviewing eyewitnesses and victims about what they remember from a crime scene. Using four retrievals, the primary focus of the cognitive interview is to make witnesses and victims of a situation aware of all the events that transpired.

  16. The Case for Cognitive Interviewing in Survey Item Validation: A Useful

    An example of cognitive interviewing with alcohol tolerance. To offer an applied example, we conducted cognitive interviews with a small group of participants (N = 10) to evaluate how tolerance items from the National Epidemiologic Survey on Alcohol and Related Conditions-III (NESARC-III; Grant et al., 2014) perform and to recommend revisions.NESARC-III was chosen because it is used to derive ...

  17. Rethinking the Reliability of Eyewitness Memory

    Although most studies are concerned with the ability of a structured interview such as the cognitive interview to elicit more information than alternative ... van Koppen P. J. (2009). Eyewitness memory of a supermarket robbery: A case study of accuracy and confidence after 3 months. Law and Human Behavior, 33, 506-514. Crossref. PubMed. ISI.

  18. Introspective interviewing for work activities: applying subjective

    The replay interview of a case study regarding the application of a SEBE method was considered. The SEBE method was applied to analyze activities at a French nuclear power plant during operating situations (results regarding the work activity analysis were already published (Fauquet-Alekhine and Daviet 2015 ) and performed by Operations shift ...

  19. Cognitive Interview

    Cognitive interviews are structured investigative interviews that use non-suggestive, non-leading questions designed to tap into an individual's free-call memory without tainting or...

  20. Cognitive Interviewing » Law Enforcement Learning

    We'll start the course by examining the basics of a cognitive interview and observing an interview case study. Next, we'll analyze the mechanics of a cognitive interview and explore essential interview elements such as setting, timing, and victimization. From there, we'll learn how memory impacts cognition, explore the strengths and ...

  21. The cognitive interview: Inexperienced police officers' perceptions of

    Purpose. The primary objectives of the study reported here were twofold. First, to investigate less experienced frontline police officers' perceptions of their witness interviewing practices with specific reference to their use of the ten cognitive interview components taught during initial PEACE (a mnemonic for the stages of the interview; Planning and preparation, Engage and explain, Account ...

  22. Case Study 1: A 55-Year-Old Woman With Progressive Cognitive

    Case Study 1: A 55-Year-Old Woman With Progressive Cognitive, Perceptual, and Motor Impairments Scott M. McGinnis , M.D., Andrew M. Stern , M.D., Ph.D., Jared K. Woods , M.D., Ph.D., Matthew Torre , M.D., Mel B. Feany , M.D., Ph.D., Michael B. Miller , M.D., Ph.D., David A. Silbersweig , M.D., … See all authors

  23. Case Study: Definition, Examples, Types, and How to Write

    Pros One of the greatest advantages of a case study is that it allows researchers to investigate things that are often difficult or impossible to replicate in a lab. Some other benefits of a case study: Allows researchers to capture information on the 'how,' 'what,' and 'why,' of something that's implemented