a research strategy that simultaneously

• Privacy Policy

Home » Mixed Methods Research – Types & Analysis

Mixed Methods Research – Types & Analysis

Table of Contents

Mixed Methods Research

Mixed methods research is an approach to research that combines both quantitative and qualitative research methods in a single study or research project. It is a methodological approach that involves collecting and analyzing both numerical (quantitative) and narrative (qualitative) data to gain a more comprehensive understanding of a research problem.

Types of Mixed Research

There are different types of mixed methods research designs that researchers can use, depending on the research question, the available data, and the resources available. Here are some common types:

Convergent Parallel Design

This design involves collecting both qualitative and quantitative data simultaneously, analyzing them separately, and then merging the findings to draw conclusions. The qualitative and quantitative data are given equal weight, and the findings are integrated during the interpretation phase.

Sequential Explanatory Design

In this design, the researcher collects and analyzes quantitative data first, and then uses qualitative data to explain or elaborate on the quantitative findings. The researcher may use the qualitative data to clarify unexpected or contradictory results from the quantitative analysis.

Sequential Exploratory Design

This design involves collecting qualitative data first, analyzing it, and then collecting and analyzing quantitative data to confirm or refute the qualitative findings. Qualitative data are used to generate hypotheses that are tested using quantitative data.

Concurrent Triangulation Design

This design involves collecting both qualitative and quantitative data concurrently and then comparing the results to find areas of agreement and disagreement. The findings are integrated during the interpretation phase to provide a more comprehensive understanding of the research question.

Concurrent Nested Design

This design involves collecting one type of data as the primary method and then using the other type of data to elaborate or clarify the primary data. For example, a researcher may use quantitative data as the primary method and qualitative data as a secondary method to provide more context and detail.

Transformative Design

This design involves using mixed methods research to not only understand the research question but also to bring about social change or transformation. The research is conducted in collaboration with stakeholders and aims to generate knowledge that can be used to improve policies, programs, and practices.

Concurrent Embedded Design

Concurrent embedded design is a type of mixed methods research design in which one type of data is embedded within another type of data. This design involves collecting both quantitative and qualitative data simultaneously, with one type of data being the primary method and the other type of data being the secondary method. The secondary method is embedded within the primary method, meaning that it is used to provide additional information or to clarify the primary data.

Data Collection Methods

Here are some common data collection methods used in mixed methods research:

Surveys are a common quantitative data collection method used in mixed methods research. Surveys involve collecting standardized responses to a set of questions from a sample of participants. Surveys can be conducted online, in person, or over the phone.

Interviews are a qualitative data collection method that involves asking open-ended questions to gather in-depth information about a participant’s experiences, perspectives, and opinions. Interviews can be conducted in person, over the phone, or online.

Focus groups

Focus groups are a qualitative data collection method that involves bringing together a small group of participants to discuss a topic or research question. The group is facilitated by a researcher, and the discussion is recorded and analyzed for themes and patterns.

Observations

Observations are a qualitative data collection method that involves systematically watching and recording behavior in a natural setting. Observations can be structured or unstructured and can be used to gather information about behavior, interactions, and context.

Document Analysis

Document analysis is a qualitative data collection method that involves analyzing existing documents, such as reports, policy documents, or media articles. Document analysis can be used to gather information about trends, policy changes, or public attitudes.

Experimentation

Experimentation is a quantitative data collection method that involves manipulating one or more variables and measuring their effects on an outcome. Experiments can be conducted in a laboratory or in a natural setting.

Data Analysis Methods

Mixed methods research involves using both quantitative and qualitative data analysis methods to analyze data collected through different methods. Here are some common data analysis methods used in mixed methods research:

Quantitative Data Analysis

Quantitative data collected through surveys or experiments can be analyzed using statistical methods. Statistical analysis can be used to identify relationships between variables, test hypotheses, and make predictions. Common statistical methods used in quantitative data analysis include regression analysis, t-tests, ANOVA, and correlation analysis.

Qualitative Data Analysis

Qualitative data collected through interviews, focus groups, or observations can be analyzed using a variety of qualitative data analysis methods. These methods include content analysis, thematic analysis, narrative analysis, and grounded theory. Qualitative data analysis involves identifying themes and patterns in the data, interpreting the meaning of the data, and drawing conclusions based on the findings.

Integration of Data

The integration of quantitative and qualitative data involves combining the results from both types of data analysis to gain a more comprehensive understanding of the research question. Integration can involve either a concurrent or sequential approach. Concurrent integration involves analyzing quantitative and qualitative data at the same time, while sequential integration involves analyzing one type of data first and then using the results to inform the analysis of the other type of data.

Triangulation

Triangulation involves using multiple sources or types of data to validate or corroborate findings. This can involve using both quantitative and qualitative data or multiple qualitative methods. Triangulation can enhance the credibility and validity of the research findings.

Mixed Methods Meta-analysis

Mixed methods meta-analysis involves the systematic review and synthesis of findings from multiple studies that use mixed methods designs. This involves combining quantitative and qualitative data from multiple studies to gain a broader understanding of a research question.

How to conduct Mixed Methods Research

Here are some general steps for conducting mixed methods research:

• Identify the research problem: The first step is to clearly define the research problem and determine if mixed methods research is appropriate for addressing it.
• Design the study: The research design should include both qualitative and quantitative data collection and analysis methods. The specific design will depend on the research question and the purpose of the study.
• Collect data : Data collection involves collecting both qualitative and quantitative data through various methods such as surveys, interviews, observations, and document analysis.
• Analyze data: Both qualitative and quantitative data need to be analyzed separately and then integrated. Analysis methods may include coding, statistical analysis, and thematic analysis.
• Interpret results: The results of the analysis should be interpreted, taking into account both the quantitative and qualitative findings. This involves integrating the results and identifying any patterns, themes, or discrepancies.
• Draw conclusions : Based on the interpretation of the results, conclusions should be drawn that address the research question and objectives.
• Report findings: Finally, the findings should be reported in a clear and concise manner, using both quantitative and qualitative data to support the conclusions.

Applications of Mixed Methods Research

Mixed methods research can be applied to a wide range of research fields and topics, including:

• Education : Mixed methods research can be used to evaluate educational programs, assess the effectiveness of teaching methods, and investigate student learning experiences.
• Health and social sciences: Mixed methods research can be used to study health interventions, understand the experiences of patients and their families, and assess the effectiveness of social programs.
• Business and management: Mixed methods research can be used to investigate customer satisfaction, assess the impact of marketing campaigns, and analyze the effectiveness of management strategies.
• Psychology : Mixed methods research can be used to explore the experiences and perspectives of individuals with mental health issues, investigate the impact of psychological interventions, and assess the effectiveness of therapy.
• Sociology : Mixed methods research can be used to study social phenomena, investigate the experiences and perspectives of marginalized groups, and assess the impact of social policies.
• Environmental studies: Mixed methods research can be used to assess the impact of environmental policies, investigate public perceptions of environmental issues, and analyze the effectiveness of conservation strategies.

Examples of Mixed Methods Research

Here are some examples of Mixed-Methods research:

• Evaluating a school-based mental health program: A researcher might use a concurrent embedded design to evaluate a school-based mental health program. The researcher might collect quantitative data through surveys and qualitative data through interviews with students and teachers. The quantitative data might be analyzed using statistical methods, while the qualitative data might be analyzed using thematic analysis. The results of the two types of data analysis could be integrated to provide a comprehensive evaluation of the program’s effectiveness.
• Understanding patient experiences of chronic illness: A researcher might use a sequential explanatory design to investigate patient experiences of chronic illness. The researcher might collect quantitative data through surveys and then use the results of the survey to inform the selection of participants for qualitative interviews. The qualitative data might be analyzed using content analysis to identify common themes in the patients’ experiences.
• Assessing the impact of a new public transportation system : A researcher might use a concurrent triangulation design to assess the impact of a new public transportation system. The researcher might collect quantitative data through surveys and qualitative data through focus groups with community members. The results of the two types of data analysis could be triangulated to provide a more comprehensive understanding of the impact of the new transportation system on the community.
• Exploring teacher perceptions of technology integration in the classroom: A researcher might use a sequential exploratory design to investigate teacher perceptions of technology integration in the classroom. The researcher might collect qualitative data through in-depth interviews with teachers and then use the results of the interviews to develop a survey. The quantitative data might be analyzed using descriptive statistics to identify trends in teacher perceptions.

When to use Mixed Methods Research

Mixed methods research is typically used when a research question cannot be fully answered by using only quantitative or qualitative methods. Here are some common situations where mixed methods research is appropriate:

• When the research question requires a more comprehensive understanding than can be achieved by using only quantitative or qualitative methods.
• When the research question requires both an exploration of individuals’ experiences, perspectives, and attitudes, as well as the measurement of objective outcomes and variables.
• When the research question requires the examination of a phenomenon in its natural setting and context, which can be achieved by collecting rich qualitative data, as well as the generalization of findings to a larger population, which can be achieved through the use of quantitative methods.
• When the research question requires the integration of different types of data or perspectives, such as combining data collected from participants with data collected from stakeholders or experts.
• When the research question requires the validation of findings obtained through one method by using another method.
• When the research question involves studying a complex phenomenon that cannot be understood by using only one method, such as studying the impact of a policy on a community’s well-being.
• When the research question involves studying a topic that has not been well-researched, and using mixed methods can help provide a more comprehensive understanding of the topic.

Purpose of Mixed Methods Research

The purpose of mixed methods research is to provide a more comprehensive understanding of a research problem than can be obtained through either quantitative or qualitative methods alone.

Mixed methods research is particularly useful when the research problem is complex and requires a deep understanding of the context and subjective experiences of participants, as well as the ability to generalize findings to a larger population. By combining both qualitative and quantitative methods, researchers can obtain a more complete picture of the research problem and its underlying mechanisms, as well as test hypotheses and identify patterns that may not be apparent with only one method.

Overall, mixed methods research aims to provide a more holistic and nuanced understanding of the research problem, allowing researchers to draw more valid and reliable conclusions, make more informed decisions, and develop more effective interventions and policies.

Advantages of Mixed Methods Research

Mixed methods research offers several advantages over using only qualitative or quantitative research methods. Here are some of the main advantages of mixed methods research:

• Comprehensive understanding: Mixed methods research provides a more comprehensive understanding of the research problem by combining both qualitative and quantitative data, which allows for a more nuanced interpretation of the data.
• Triangulation : Mixed methods research allows for triangulation, which is the use of multiple sources of data to verify findings. This improves the validity and reliability of the research.
• Addressing limitations: Mixed methods research can address the limitations of qualitative or quantitative research by compensating for the weaknesses of each method.
• Flexibility : Mixed methods research is flexible, allowing researchers to adapt the research design and methods as needed to best address the research question.
• Validity : Mixed methods research can increase the validity of the research by using multiple methods to measure the same concept.
• Generalizability : Mixed methods research can improve the generalizability of the findings by using quantitative data to test the applicability of qualitative findings to a larger population.
• Practical applications: Mixed methods research is useful for developing practical applications, such as interventions or policies, as it provides a more comprehensive understanding of the research problem.

Limitations of Mixed Methods Research

Here are some of the main limitations of mixed methods research:

• Time-consuming: Mixed methods research can be time-consuming and may require more resources than using only one research method.
• Complex data analysis: Integrating qualitative and quantitative data can be challenging and requires specialized skills for data analysis.
• Sampling bias: Mixed methods research can be subject to sampling bias, particularly if the sampling strategies for the qualitative and quantitative components are not aligned.
• Validity and reliability: Mixed methods research requires careful attention to the validity and reliability of both the qualitative and quantitative data, as well as the integration of the two data types.
• Difficulty in balancing the two methods: Mixed methods research can be difficult to balance the qualitative and quantitative methods effectively, particularly if one method dominates the other.
• Theoretical and philosophical issues: Mixed methods research raises theoretical and philosophical questions about the compatibility of qualitative and quantitative research methods and the underlying assumptions about the nature of reality and knowledge.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Questionnaire – Definition, Types, and Examples

Case Study – Methods, Examples and Guide

Observational Research – Methods and Guide

Quantitative Research – Methods, Types and...

Qualitative Research Methods

Explanatory Research – Types, Methods, Guide

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Med Libr Assoc
• v.106(4); 2018 Oct

A systematic approach to searching: an efficient and complete method to develop literature searches

Associated data.

Creating search strategies for systematic reviews, finding the best balance between sensitivity and specificity, and translating search strategies between databases is challenging. Several methods describe standards for systematic search strategies, but a consistent approach for creating an exhaustive search strategy has not yet been fully described in enough detail to be fully replicable. The authors have established a method that describes step by step the process of developing a systematic search strategy as needed in the systematic review. This method describes how single-line search strategies can be prepared in a text document by typing search syntax (such as field codes, parentheses, and Boolean operators) before copying and pasting search terms (keywords and free-text synonyms) that are found in the thesaurus. To help ensure term completeness, we developed a novel optimization technique that is mainly based on comparing the results retrieved by thesaurus terms with those retrieved by the free-text search words to identify potentially relevant candidate search terms. Macros in Microsoft Word have been developed to convert syntaxes between databases and interfaces almost automatically. This method helps information specialists in developing librarian-mediated searches for systematic reviews as well as medical and health care practitioners who are searching for evidence to answer clinical questions. The described method can be used to create complex and comprehensive search strategies for different databases and interfaces, such as those that are needed when searching for relevant references for systematic reviews, and will assist both information specialists and practitioners when they are searching the biomedical literature.

INTRODUCTION

Librarians and information specialists are often involved in the process of preparing and completing systematic reviews (SRs), where one of their main tasks is to identify relevant references to include in the review [ 1 ]. Although several recommendations for the process of searching have been published [ 2 – 6 ], none describe the development of a systematic search strategy from start to finish.

Traditional methods of SR search strategy development and execution are highly time consuming, reportedly requiring up to 100 hours or more [ 7 , 8 ]. The authors wanted to develop systematic and exhaustive search strategies more efficiently, while preserving the high sensitivity that SR search strategies necessitate. In this article, we describe the method developed at Erasmus University Medical Center (MC) and demonstrate its use through an example search. The efficiency of the search method and outcome of 73 searches that have resulted in published reviews are described in a separate article [ 9 ].

As we aimed to describe the creation of systematic searches in full detail, the method starts at a basic level with the analysis of the research question and the creation of search terms. Readers who are new to SR searching are advised to follow all steps described. More experienced searchers can consider the basic steps to be existing knowledge that will already be part of their normal workflow, although step 4 probably differs from general practice. Experienced searchers will gain the most from reading about the novelties in the method as described in steps 10–13 and comparing the examples given in the supplementary appendix to their own practice.

CREATING A SYSTEMATIC SEARCH STRATEGY

Our methodology for planning and creating a multi-database search strategy consists of the following steps:

• Determine a clear and focused question
• Describe the articles that can answer the question
• Decide which key concepts address the different elements of the question
• Decide which elements should be used for the best results
• Choose an appropriate database and interface to start with
• Document the search process in a text document
• Identify appropriate index terms in the thesaurus of the first database
• Identify synonyms in the thesaurus
• Add variations in search terms
• Use database-appropriate syntax, with parentheses, Boolean operators, and field codes
• Optimize the search
• Evaluate the initial results
• Check for errors
• Translate to other databases
• Test and reiterate

Each step in the process is reflected by an example search described in the supplementary appendix .

1. Determine a clear and focused question

A systematic search can best be applied to a well-defined and precise research or clinical question. Questions that are too broad or too vague cannot be answered easily in a systematic way and will generally result in an overwhelming number of search results. On the other hand, a question that is too specific will result into too few or even zero search results. Various papers describe this process in more detail [ 10 – 12 ].

2. Describe the articles that can answer the question

Although not all clinical or research questions can be answered in the literature, the next step is to presume that the answer can indeed be found in published studies. A good starting point for a search is hypothesizing what the research that can answer the question would look like. These hypothetical (when possible, combined with known) articles can be used as guidance for constructing the search strategy.

3. Decide which key concepts address the different elements of the question

Key concepts are the topics or components that the desired articles should address, such as diseases or conditions, actions, substances, settings, domains (e.g., therapy, diagnosis, etiology), or study types. Key concepts from the research question can be grouped to create elements in the search strategy.

Elements in a search strategy do not necessarily follow the patient, intervention, comparison, outcome (PICO) structure or any other related structure. Using the PICO or another similar framework as guidance can be helpful to consider, especially in the inclusion and exclusion review stage of the SR, but this is not necessary for good search strategy development [ 13 – 15 ]. Sometimes concepts from different parts of the PICO structure can be grouped together into one search element, such as when the desired outcome is frequently described in a certain study type.

4. Decide which elements should be used for the best results

Not all elements of a research question should necessarily be used in the search strategy. Some elements are less important than others or may unnecessarily complicate or restrict a search strategy. Adding an element to a search strategy increases the chance of missing relevant references. Therefore, the number of elements in a search strategy should remain as low as possible to optimize recall.

Using the schema in Figure 1 , elements can be ordered by their specificity and importance to determine the best search approach. Whether an element is more specific or more general can be measured objectively by the number of hits retrieved in a database when searching for a key term representing that element. Depending on the research question, certain elements are more important than others. If articles (hypothetically or known) exist that can answer the question but lack a certain element in their titles, abstracts, or keywords, that element is unimportant to the question. An element can also be unimportant because of expected bias or an overlap with another element.

Schema for determining the optimal order of elements

Bias in elements

The choice of elements in a search strategy can introduce bias through use of overly specific terminology or terms often associated with positive outcomes. For the question “does prolonged breastfeeding improve intelligence outcomes in children?,” searching specifically for the element of duration will introduce bias, as articles that find a positive effect of prolonged breastfeeding will be much more likely to mention time factors in their titles or abstracts.

Overlapping elements

Elements in a question sometimes overlap in their meaning. Sometimes certain therapies are interventions for one specific disease. The Lichtenstein technique, for example, is a repair method for inguinal hernias. There is no need to include an element of “inguinal hernias” to a search for the effectiveness of the Lichtenstein therapy. Likewise, sometimes certain diseases are only found in certain populations. Adding such an overlapping element could lead to missing relevant references.

The elements to use in a search strategy can be found in the plot of elements in Figure 1 , by following the top row from left to right. For this method, we recommend starting with the most important and specific elements. Then, continue with more general and important elements until the number of results is acceptable for screening. Determining how many results are acceptable for screening is often a matter of negotiation with the SR team.

5. Choose an appropriate database and interface to start with

Important factors for choosing databases to use are the coverage and the presence of a thesaurus. For medically oriented searches, the coverage and recall of Embase, which includes the MEDLINE database, are superior to those of MEDLINE [ 16 ]. Each of these two databases has its own thesaurus with its own unique definitions and structure. Because of the complexity of the Embase thesaurus, Emtree, which contains much more specific thesaurus terms than the MEDLINE Medical Subject Headings (MeSH) thesaurus, translation from Emtree to MeSH is easier than the other way around. Therefore, we recommend starting in Embase.

MEDLINE and Embase are available through many different vendors and interfaces. The choice of an interface and primary database is often determined by the searcher’s accessibility. For our method, an interface that allows searching with proximity operators is desirable, and full functionality of the thesaurus, including explosion of narrower terms, is crucial. We recommend developing a personal workflow that always starts with one specific database and interface.

6. Document the search process in a text document

We advise designing and creating the complete search strategies in a log document, instead of directly in the database itself, to register the steps taken and to make searches accountable and reproducible. The developed search strategies can be copied and pasted into the desired databases from the log document. This way, the searcher is in control of the whole process. Any change to the search strategy should be done in the log document, assuring that the search strategy in the log is always the most recent.

7. Identify appropriate index terms in the thesaurus of the first database

Searches should start by identifying appropriate thesaurus terms for the desired elements. The thesaurus of the database is searched for matching index terms for each key concept. We advise restricting the initial terms to the most important and most relevant terms. Later in the process, more general terms can be added in the optimization process, in which the effect on the number of hits, and thus the desirability of adding these terms, can be evaluated more easily.

Several factors can complicate the identification of thesaurus terms. Sometimes, one thesaurus term is found that exactly describes a specific element. In contrast, especially in more general elements, multiple thesaurus terms can be found to describe one element. If no relevant thesaurus terms have been found for an element, free-text terms can be used, and possible thesaurus terms found in the resulting references can be added later (step 11).

Sometimes, no distinct thesaurus term is available for a specific key concept that describes the concept in enough detail. In Emtree, one thesaurus term often combines two or more elements. The easiest solution for combining these terms for a sensitive search is to use such a thesaurus term in all elements where it is relevant. Examples are given in the supplementary appendix .

8. Identify synonyms in the thesaurus

Most thesauri offer a list of synonyms on their term details page (named Synonyms in Emtree and Entry Terms in MeSH). To create a sensitive search strategy for SRs, these terms need to be searched as free-text keywords in the title and abstract fields, in addition to searching their associated thesaurus terms.

The Emtree thesaurus contains more synonyms (300,000) than MeSH does (220,000) [ 17 ]. The difference in number of terms is even higher considering that many synonyms in MeSH are permuted terms (i.e., inversions of phrases using commas).

Thesaurus terms are ordered in a tree structure. When searching for a more general thesaurus term, the more specific (narrower) terms in the branches below that term will also be searched (this is frequently referred to as “exploding” a thesaurus term). However, to perform a sensitive search, all relevant variations of the narrower terms must be searched as free-text keywords in the title or abstract, in addition to relying on the exploded thesaurus term. Thus, all articles that describe a certain narrower topic in their titles and abstracts will already be retrieved before MeSH terms are added.

9. Add variations in search terms (e.g., truncation, spelling differences, abbreviations, opposites)

Truncation allows a searcher to search for words beginning with the same word stem. A search for therap* will, thus, retrieve therapy, therapies, therapeutic, and all other words starting with “therap.” Do not truncate a word stem that is too short. Also, limitations of interfaces should be taken into account, especially in PubMed, where the number of search term variations that can be found by truncation is limited to 600.

Databases contain references to articles using both standard British and American English spellings. Both need to be searched as free-text terms in the title and abstract. Alternatively, many interfaces offer a certain code to replace zero or one characters, allowing a search for “pediatric” or “paediatric” as “p?ediatric.” Table 1 provides a detailed description of the syntax for different interfaces.

Field codes in five most used interfaces for biomedical literature searching

Searching for abbreviations can identify extra, relevant references and retrieve more irrelevant ones. The search can be more focused by combining the abbreviation with an important word that is relevant to its meaning or by using the Boolean “NOT” to exclude frequently observed, clearly irrelevant results. We advise that searchers do not exclude all possible irrelevant meanings, as it is very time consuming to identify all the variations, it will result in unnecessarily complicated search strategies, and it may lead to erroneously narrowing the search and, thereby, reduce recall.

Searching partial abbreviations can be useful for retrieving relevant references. For example, it is very likely that an article would mention osteoarthritis (OA) early in the abstract, replacing all further occurrences of osteoarthritis with OA . Therefore, it may not contain the phrase “hip osteoarthritis” but only “hip oa.”

It is also important to search for the opposites of search terms to avoid bias. When searching for “disease recurrence,” articles about “disease free” may be relevant as well. When the desired outcome is survival , articles about mortality may be relevant.

10. Use database-appropriate syntax, with parentheses, Boolean operators, and field codes

Different interfaces require different syntaxes, the special set of rules and symbols unique to each database that define how a correctly constructed search operates. Common syntax components include the use of parentheses and Boolean operators such as “AND,” “OR,” and “NOT,” which are available in all major interfaces. An overview of different syntaxes for four major interfaces for bibliographic medical databases (PubMed, Ovid, EBSCOhost, Embase.com, and ProQuest) is shown in Table 1 .

Creating the appropriate syntax for each database, in combination with the selected terms as described in steps 7–9, can be challenging. Following the method outlined below simplifies the process:

• Create single-line queries in a text document (not combining multiple record sets), which allows immediate checking of the relevance of retrieved references and efficient optimization.
• Type the syntax (Boolean operators, parentheses, and field codes) before adding terms, which reduces the chance that errors are made in the syntax, especially in the number of parentheses.
• Use predefined proximity structures including parentheses, such as (() ADJ3 ()) in Ovid, that can be reused in the query when necessary.
• Use thesaurus terms separately from free-text terms of each element. Start an element with all thesaurus terms (using “OR”) and follow with the free-text terms. This allows the unique optimization methods as described in step 11.
• When adding terms to an existing search strategy, pay close attention to the position of the cursor. Make sure to place it appropriately either in the thesaurus terms section, in the title/abstract section, or as an addition (broadening) to an existing proximity search.

The supplementary appendix explains the method of building a query in more detail, step by step for different interfaces: PubMed, Ovid, EBSCOhost, Embase.com, and ProQuest. This method results in a basic search strategy designed to retrieve some relevant references upon which a more thorough search strategy can be built with optimization such as described in step 11.

11. Optimize the search

The most important question when performing a systematic search is whether all (or most) potentially relevant articles have been retrieved by the search strategy. This is also the most difficult question to answer, since it is unknown which and how many articles are relevant. It is, therefore, wise first to broaden the initial search strategy, making the search more sensitive, and then check if new relevant articles are found by comparing the set results (i.e., search for Strategy #2 NOT Strategy #1 to see the unique results).

A search strategy should be tested for completeness. Therefore, it is necessary to identify extra, possibly relevant search terms and add them to the test search in an OR relationship with the already used search terms. A good place to start, and a well-known strategy, is scanning the top retrieved articles when sorted by relevance, looking for additional relevant synonyms that could be added to the search strategy.

We have developed a unique optimization method that has not been described before in the literature. This method often adds valuable extra terms to our search strategy and, therefore, extra, relevant references to our search results. Extra synonyms can be found in articles that have been assigned a certain set of thesaurus terms but that lack synonyms in the title and/or abstract that are already present in the current search strategy. Searching for thesaurus terms NOT free-text terms will help identify missed free-text terms in the title or abstract. Searching for free-text terms NOT thesaurus terms will help identify missed thesaurus terms. If this is done repeatedly for each element, leaving the rest of the query unchanged, this method will help add numerous relevant terms to the query. These steps are explained in detail for five different search platforms in the supplementary appendix .

12. Evaluate the initial results

The results should now contain relevant references. If the interface allows relevance ranking, use that in the evaluation. If you know some relevant references that should be included in the research, search for those references specifically; for example, combine a specific (first) author name with a page number and the publication year. Check whether those references are retrieved by the search. If the known relevant references are not retrieved by the search, adapt the search so that they are. If it is unclear which element should be adapted to retrieve a certain article, combine that article with each element separately.

Different outcomes are desired for different types of research questions. For instance, in the case of clinical question answering, the researcher will not be satisfied with many references that contain a lot of irrelevant references. A clinical search should be rather specific and is allowed to miss a relevant reference. In the case of an SR, the researchers do not want to miss any relevant reference and are willing to handle many irrelevant references to do so. The search for references to include in an SR should be very sensitive: no included reference should be missed. A search that is too specific or too sensitive for the intended goal can be adapted to become more sensitive or specific. Steps to increase sensitivity or specificity of a search strategy can be found in the supplementary appendix .

13. Check for errors

Errors might not be easily detected. Sometimes clues can be found in the number of results, either when the number of results is much higher or lower than expected or when many retrieved references are not relevant. However, the number expected is often unknown, and very sensitive search strategies will always retrieve many irrelevant articles. Each query should, therefore, be checked for errors.

One of the most frequently occurring errors is missing the Boolean operator “OR.” When no “OR” is added between two search terms, many interfaces automatically add an “AND,” which unintentionally reduces the number of results and likely misses relevant references. One good strategy to identify missing “OR”s is to go to the web page containing the full search strategy, as translated by the database, and using Ctrl-F search for “AND.” Check whether the occurrences of the “AND” operator are deliberate.

Ideally, search strategies should be checked by other information specialists [ 18 ]. The Peer Review of Electronic Search Strategies (PRESS) checklist offers good guidance for this process [ 4 ]. Apart from the syntax (especially Boolean operators and field codes) of the search strategy, it is wise to have the search terms checked by the clinician or researcher familiar with the topic. At Erasmus MC, researchers and clinicians are involved during the complete process of structuring and optimizing the search strategy. Each word is added after the combined decision of the searcher and the researcher, with the possibility of directly comparing results with and without the new term.

14. Translate to other databases

To retrieve as many relevant references as possible, one has to search multiple databases. Translation of complex and exhaustive queries between different databases can be very time consuming and cumbersome. The single-line search strategy approach detailed above allows quick translations using the find and replace method in Microsoft Word (<Ctrl-H>).

At Erasmus MC, macros based on the find-and-replace method in Microsoft Word have been developed for easy and fast translation between the most used databases for biomedical and health sciences questions. The schema that is followed for the translation between databases is shown in Figure 2 . Most databases simply follow the structure set by the Embase.com search strategy. The translation from Emtree terms to MeSH terms for MEDLINE in Ovid often identifies new terms that need to be added to the Embase.com search strategy before the translation to other databases.

Schematic representation of translation between databases used at Erasmus University Medical Center

Dotted lines represent databases that are used in less than 80% of the searches.

Using five different macros, a thoroughly optimized query in Embase.com can be relatively quickly translated into eight major databases. Basic search strategies will be created to use in many, mostly smaller, databases, because such niche databases often do not have extensive thesauri or advanced syntax options. Also, there is not much need to use extensive syntax because the number of hits and, therefore, the amount of noise in these databases is generally low. In MEDLINE (Ovid), PsycINFO (Ovid), and CINAHL (EBSCOhost), the thesaurus terms must be adapted manually, as each database has its own custom thesaurus. These macros and instructions for their installation, use, and adaptation are available at bit.ly/databasemacros.

15. Test and reiterate

Ideally, exhaustive search strategies should retrieve all references that are covered in a specific database. For SR search strategies, checking searches for their recall is advised. This can be done after included references have been determined by the authors of the systematic review. If additional papers have been identified through other non-database methods (i.e., checking references in included studies), results that were not identified by the database searches should be examined. If these results were available in the databases but not located by the search strategy, the search strategy should be adapted to try to retrieve these results, as they may contain terms that were omitted in the original search strategies. This may enable the identification of additional relevant results.

A methodology for creating exhaustive search strategies has been created that describes all steps of the search process, starting with a question and resulting in thorough search strategies in multiple databases. Many of the steps described are not new, but together, they form a strong method creating high-quality, robust searches in a relatively short time frame.

Our methodology is intended to create thoroughness for literature searches. The optimization method, as described in step 11, will identify missed synonyms or thesaurus terms, unlike any other method that largely depends on predetermined keywords and synonyms. Using this method results in a much quicker search process, compared to traditional methods, especially because of the easier translation between databases and interfaces (step 13). The method is not a guarantee for speed, since speed depends on many factors, including experience. However, by following the steps and using the tools as described above, searchers can gain confidence first and increase speed through practice.

What is new?

This method encourages searchers to start their search development process using empty syntax first and later adding the thesaurus terms and free-text synonyms. We feel this helps the searcher to focus on the search terms, instead of on the structure of the search query. The optimization method in which new terms are found in the already retrieved articles is used in some other institutes as well but has to our knowledge not been described in the literature. The macros to translate search strategies between interfaces are unique in this method.

What is different compared to common practice?

Traditionally, librarians and information specialists have focused on creating complex, multi-line (also called line-by-line) search strategies, consisting of multiple record sets, and this method is frequently advised in the literature and handbooks [ 2 , 19 – 21 ]. Our method, instead, uses single-line searches, which is critical to its success. Single-line search strategies can be easily adapted by adding or dropping a term without having to recode numbers of record sets, which would be necessary in multi-line searches. They can easily be saved in a text document and repeated by copying and pasting for search updates. Single-line search strategies also allow easy translation to other syntaxes using find-and-replace technology to update field codes and other syntax elements or using macros (step 13).

When constructing a search strategy, the searcher might experience that certain parentheses in the syntax are unnecessary, such as parentheses around all search terms in the title/abstract portion, if there is only one such term, there are double parentheses in the proximity statement, or one of the word groups exists for only one word. One might be tempted to omit those parentheses for ease of reading and management. However, during the optimization process, the searcher is likely to find extra synonyms that might consist of one word. To add those terms to the first query (with reduced parentheses) requires adding extra parentheses (meticulously placing and counting them), whereas, in the latter search, it only requires proper placement of those terms.

Many search methods highly depend on the PICO framework. Research states that often PICO or PICOS is not suitable for every question [ 22 , 23 ]. There are other acronyms than PICO—such as sample, phenomenon of interest, design, evaluation, research type (SPIDER) [ 24 ]—but each is just a variant. In our method, the most important and specific elements of a question are being analyzed for building the best search strategy.

Though it is generally recommended that searchers search both MEDLINE and Embase, most use MEDLINE as the starting point. It is considered the gold standard for biomedical searching, partially due to historical reasons, since it was the first of its kind, and more so now that it is freely available via the PubMed interface. Our method can be used with any database as a starting point, but we use Embase instead of MEDLINE or another database for a number of reasons. First, Embase provides both unique content and the complete content of MEDLINE. Therefore, searching Embase will be, by definition, more complete than searching MEDLINE only. Second, the number of terms in Emtree (the Embase thesaurus) is three times as high as that of MeSH (the MEDLINE thesaurus). It is easier to find MeSH terms after all relevant Emtree terms have been identified than to start with MeSH and translate to Emtree.

At Erasmus MC, the researchers sit next to the information specialist during most of the search strategy design process. This way, the researchers can deliver immediate feedback on the relevance of proposed search terms and retrieved references. The search team then combines knowledge about databases with knowledge about the research topic, which is an important condition to create the highest quality searches.

Limitations of the method

One disadvantage of single-line searches compared to multi-line search strategies is that errors are harder to recognize. However, with the methods for optimization as described (step 11), errors are recognized easily because missed synonyms and spelling errors will be identified during the process. Also problematic is that more parentheses are needed, making it more difficult for the searcher and others to assess the logic of the search strategy. However, as parentheses and field codes are typed before the search terms are added (step 10), errors in parentheses can be prevented.

Our methodology works best if used in an interface that allows proximity searching. It is recommended that searchers with access to an interface with proximity searching capabilities select one of those as the initial database to develop and optimize the search strategy. Because the PubMed interface does not allow proximity searches, phrases or Boolean “AND” combinations are required. Phrase searching complicates the process and is more specific, with the higher risk of missing relevant articles, and using Boolean “AND” combinations increases sensitivity but at an often high loss of specificity. Due to some searchers’ lack of access to expensive databases or interfaces, the freely available PubMed interface may be necessary to use, though it should never be the sole database used for an SR [ 2 , 16 , 25 ]. A limitation of our method is that it works best with subscription-based and licensed resources.

Another limitation is the customization of the macros to a specific institution’s resources. The macros for the translation between different database interfaces only work between the interfaces as described. To mitigate this, we recommend using the find-and-replace functionality of text editors like Microsoft Word to ease the translation of syntaxes between other databases. Depending on one’s institutional resources, custom macros can be developed using similar methods.

Results of the method

Whether this method results in exhaustive searches where no important article is missed is difficult to determine, because the number of relevant articles is unknown for any topic. A comparison of several parameters of 73 published reviews that were based on a search developed with this method to 258 reviews that acknowledged information specialists from other Dutch academic hospitals shows that the performance of the searches following our method is comparable to those performed in other institutes but that the time needed to develop the search strategies was much shorter than the time reported for the other reviews [ 9 ].

CONCLUSIONS

With the described method, searchers can gain confidence in their search strategies by finding many relevant words and creating exhaustive search strategies quickly. The approach can be used when performing SR searches or for other purposes such as answering clinical questions, with different expectations of the search’s precision and recall. This method, with practice, provides a stepwise approach that facilitates the search strategy development process from question clarification to final iteration and beyond.

SUPPLEMENTAL FILE

Acknowledgments.

We highly appreciate the work that was done by our former colleague Louis Volkers, who in his twenty years as an information specialist in Erasmus MC laid the basis for our method. We thank Professor Oscar Franco for reviewing earlier drafts of this article.

Research Strategies and Methods

• First Online: 22 July 2021

Cite this chapter

• Paul Johannesson 3 &
• Erik Perjons 3  

2498 Accesses

Researchers have since centuries used research methods to support the creation of reliable knowledge based on empirical evidence and logical arguments. This chapter offers an overview of established research strategies and methods with a focus on empirical research in the social sciences. We discuss research strategies, such as experiment, survey, case study, ethnography, grounded theory, action research, and phenomenology. Research methods for data collection are also described, including questionnaires, interviews, focus groups, observations, and documents. Qualitative and quantitative methods for data analysis are discussed. Finally, the use of research strategies and methods within design science is investigated.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Bhattacherjee A (2012) Social science research: principles, methods, and practices, 2 edn. CreateSpace Independent Publishing Platform, Tampa, FL

Google Scholar  

Blake W (2012) Delphi complete works of William Blake, 2nd edn. Delphi Classics

Bradburn NM, Sudman S, Wansink B (2004) Asking questions: the definitive guide to questionnaire design – for market research, political polls, and social and health questionnaires, revised edition. Jossey-Bass, San Francisco

Bryman A (2016) Social research methods, 5th edn. Oxford University Press, Oxford

Charmaz K (2014) Constructing grounded theory, 2nd edn. SAGE Publications, Thousand Oaks, CA

Coghlan D (2019) Doing action research in your own organization, 5th edn. SAGE Publications, Thousand Oaks, CA

Creswell JW, Creswell JD (2017) Research design: qualitative, quantitative, and mixed methods approaches, 5th edn. SAGE Publications, Thousand Oaks, CA

MATH   Google Scholar  

Denscombe M (2017) The good research guide, 6th edn. Open University Press, London

Dey I (2003) Qualitative data analysis: a user friendly guide for social scientists. Routledge, London

Book   Google Scholar  

Fairclough N (2013) Critical discourse analysis: the critical study of language, 2 edn. Routledge, London

Field A, Hole GJ (2003) How to design and report experiments, 1st edn. Sage Publications, Thousand Oaks, CA

Fowler FJ (2013) Survey research methods, 5th edn. SAGE Publications, Thousand Oaks, CA

Glaser B, Strauss A (1999) The discovery of grounded theory: strategies for qualitative research. Routledge, London

Kemmis S, McTaggart R, Nixon R (2016) The action research planner: doing critical participatory action research, 1st edn. Springer, Berlin

Krippendorff K (2018) Content analysis: an introduction to its methodology, 4th edn. SAGE Publications, Thousand Oaks, CA

LeCompte MD, Schensul JJ (2010) Designing and conducting ethnographic research: an introduction, 2nd edn. AltaMira Press, Lanham, MD

McNiff J (2013) Action research: principles and practice, 3rd edn. Routledge, London

Oates BJ (2006) Researching information systems and computing. SAGE Publications, Thousand Oaks, CA

Peterson RA (2000) Constructing effective questionnaires. Sage Publications, Thousand Oaks

Prior L (2008) Repositioning documents in social research. Sociology 42(5):821–836

Article   Google Scholar  

Seidman I (2019) Interviewing as qualitative research: a guide for researchers in education and the social sciences, 5th edn. Teachers College Press, New York

Silverman D (2018) Doing qualitative research, 5th edn. SAGE Publications, Thousand Oaks, CA

Stephens L (2004) Advanced statistics demystified, 1st edn. McGraw-Hill Professional, New York

Urdan TC (2016) Statistics in plain English, 4th ed. Routledge, London

Yin RK (2017) Case study research and applications: design and methods, 6th edn. SAGE Publications, Thousand Oaks, CA

Download references

Author information

Authors and affiliations.

Stockholm University, Kista, Sweden

Paul Johannesson & Erik Perjons

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Johannesson, P., Perjons, E. (2021). Research Strategies and Methods. In: An Introduction to Design Science. Springer, Cham. https://doi.org/10.1007/978-3-030-78132-3_3

Download citation

DOI : https://doi.org/10.1007/978-3-030-78132-3_3

Published : 22 July 2021

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-78131-6

Online ISBN : 978-3-030-78132-3

eBook Packages : Computer Science Computer Science (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

An official website of the United States government

Here’s how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

2024 National Strategy for Suicide Prevention

Suicide is an urgent and growing public health crisis. More than 49,000 people in the United States died by suicide in 2022. That’s one death every 11 minutes.

National Strategy for Suicide Prevention

The 2024 National Strategy for Suicide Prevention is a bold new 10-year, comprehensive, whole-of-society approach to suicide prevention that provides concrete recommendations for addressing gaps in the suicide prevention field. This coordinated and comprehensive approach to suicide prevention at the national, state, tribal, local, and territorial levels relies upon critical partnerships across the public and private sectors. People with lived experience are critical to the success of this work. 

 The National Strategy seeks to prevent suicide risk in the first place; identify and support people with increased risk through treatment and crisis intervention; prevent reattempts; promote long-term recovery; and support survivors of suicide loss. 

Four strategic directions guide the National Strategy:

Strategic Direction 1: Community-Based Suicide Prevention

Goal 1: Establish effective, broad-based, collaborative, and sustainable suicide prevention partnerships.

Goal 2: Support upstream comprehensive community-based suicide prevention.

Goal 3: Reduce access to lethal means among people at risk of suicide.

Goal 4: Conduct postvention and support people with suicide-centered lived experience.

Goal 5: Integrate suicide prevention into the culture of the workplace and into other community settings.

Goal 6: Build and sustain suicide prevention infrastructure at the state, tribal, local, and territorial levels.

Goal 7: Implement research-informed suicide prevention communication activities in diverse populations using best practices from communication science.

Strategic Direction 2: Treatment and Crisis Services

Goal 8: Implement effective suicide prevention services as a core component of health care.

Goal 9: Improve the quality and accessibility of crisis care services across all communities.

Strategic Direction 3: Surveillance, Quality Improvement, and Research

Goal 10: Improve the quality, timeliness, scope, usefulness, and accessibility of data needed for suicide-related surveillance, research, evaluation, and quality improvement.

Goal 11: Promote and support research on suicide prevention.

Strategic Direction 4: Health Equity in Suicide Prevention

Goal 12: Embed health equity into all comprehensive suicide prevention activities.

Goal 13: Implement comprehensive suicide prevention strategies for populations disproportionately affected by suicide, with a focus on historically marginalized communities, persons with suicide-centered lived experience, and youth.

Goal 14: Create an equitable and diverse suicide prevention workforce that is equipped and supported to address the needs of the communities they serve.

Goal 15: Improve and expand effective suicide prevention programs for populations disproportionately impacted by suicide across the life span through improved data, research, and evaluation.

Federal Action Plan

The Federal Action Plan identifies more than 200 actions across the federal government to be taken over the next three years in support of those goals. These actions include:

• Evaluating promising community-based suicide prevention strategies
• Identifying ways to address substance use/overdose and suicide risk together in the clinical setting
• Funding a mobile crisis locator for use by 988 crisis centers
• Increasing support for survivors of suicide loss and others whose lives have been impacted by suicide

These actions will be monitored and evaluated regularly to determine progress and success, and to further identify barriers to suicide prevention.

Get Involved

Join the conversation. Everyone has a role to play in preventing the tragedy of suicide. Find social media material, templates, and other resources to support and participate in the shared effort.

Read the press release

* This content is undergoing Section 508 remediation. For immediate assistance, contact [email protected] .

An official website of the United States government

Here’s how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

JavaScript appears to be disabled on this computer. Please click here to see any active alerts .

PFAS Enforcement Discretion and Settlement Policy Under CERCLA

Memorandum provides direction about how the EPA will exercise its enforcement discretion under the Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA) in matters involving per- and polyfluoroalkyl substances (PFAS). The memorandum describes how EPA will focus on holding responsible entities who significantly contributed to the release of PFAS contamination into the environment, including parties that have manufactured PFAS or used PFAS in the manufacturing process, federal facilities, and other industrial parties.

The policy also reflects that EPA does not intend to pursue entities where equitable factors do not support seeking response actions or costs under CERCLA, including farmers, municipal landfills, water utilities, municipal airports, and local fire departments. Issued: April 19, 2024

PFAS Enforcement Discretion Settlement Policy Under CERCLA (pdf) (372.5 KB, 4/19/2024)

On April 19, 2024, the EPA announced that it was designating two types of PFAS, perfluorooctanoic acid (PFOA) and perfluorooctanesulfonic acid (PFOS), as CERCLA hazardous substances. Information regarding the rulemaking is available from the Agency's  Designation of Perfluorooctanoic Acid (PFOA) and Perfluorooctanesulfonic Acid (PFOS) as CERCLA Hazardous Substances web page. 

• Enforcement Home
• Enforcement Basics
• Enforcement and Compliance Annual Results for FY 2023
• Air Enforcement
• Water Enforcement
• Waste, Chemical and Cleanup Enforcement
• Criminal Enforcement
• Enforcement at Federal Facilities
• Data and Results
• Policy, Guidance and Publications

Poll: Election interest hits new low in tight Biden-Trump race

The share of voters who say they have high interest in the 2024 election has hit a nearly 20-year low at this point in a presidential race, according to the latest national NBC News poll , with majorities holding negative views of both President Joe Biden and former President Donald Trump.

The poll also shows Biden trimming Trump’s previous lead to just 2 points in a head-to-head contest, an improvement within the margin of error compared to the previous survey, as Biden bests Trump on the issues of abortion and uniting the country, while Trump is ahead on competency and dealing with inflation.

And it finds inflation and immigration topping the list of most important issues facing the country, as just one-third of voters give Biden credit for an improving economy.

But what also stands out in the survey is how the low voter interest and the independent candidacy of Robert F. Kennedy Jr. could scramble what has been a stable presidential contest with more than six months until Election Day. While Trump holds a 2-point edge over Biden head to head, Biden leads Trump by 2 points in a five-way ballot test including Kennedy and other third-party candidates.

“I don’t think Biden has done much as a president. And if Trump gets elected, I just feel like it’s going to be the same thing as it was before Biden got elected,” said poll respondent Devin Fletcher, 37, of Wayne, Michigan, a Democrat who said he’s still voting for Biden.

“I just don’t feel like I have a candidate that I’m excited to vote for,” Fletcher added.

Another poll respondent from New Jersey, who declined to provide her name and voted for Biden in 2020, said she wouldn’t be voting in November.

“Our candidates are horrible. I have no interest in voting for Biden. He did nothing. And I absolutely will not vote for Trump,” she said.

Democratic pollster Jeff Horwitt of Hart Research Associates, who conducted the survey with Republican pollster Bill McInturff of Public Opinion Strategies, said, “Americans don’t agree on much these days, but nothing unites the country more than voters’ desire to tune this election out.”

The poll was conducted April 12-16, during yet another turbulent time in American politics, including the  beginning of Trump’s criminal trial  in New York and new attacks and heightened tensions  in the Middle East.

According to the poll, 64% of registered voters say they have high levels of interest in November’s election — registering either a “9” or a 10” on a 10-point scale of interest.

That’s lower than what the NBC News poll showed at this time in the 2008 (74%), 2012 (67%), 2016 (69%) and 2020 (77%) presidential contests.

The question dates to the 2008 election cycle. The lowest level of high election interest in the poll during a presidential cycle was in March 2012 — at 59%. But it quickly ticked up in the next survey.

This election cycle, high interest has been both low and relatively flat for months, according to the poll.

McInturff, the Republican pollster, says the high level of interest in the poll has “always been a signal for the level of turnout” for a presidential contest.

“It makes it very hard for us to predict turnout this far in advance of November, but every signal is turnout will be a lower percentage of eligible voters than in 2020,” he said.

By party, the current poll shows 70% of self-identified Republicans saying they have high interest in the coming election, compared with 65% of Democrats who say so.

Independents are at 48%, while only 36% of voters ages 18 to 34 rate themselves as highly interested in the election.

“They just aren’t low interest,” McInturff said of young voters. “They are off-the-charts low.”

NBC News poll: Frequently asked questions

Professional pollsters at a Democratic polling firm (Hart Research Associates) and a Republican firm (Public Opinion Strategies) have worked together to conduct and operate this poll since 1989. (Coldwater Corporation served as the Republican firm from 1989-2004.)

The polling firms employ a call center, where live interviewers speak by cell phone and telephone with a cross section of (usually) 1,000 respondents. The respondents are randomly selected from national lists of households and cell numbers. Respondents are asked for by name, starting with the youngest male adult or female adult in the household.

One of the common questions that critics ask of polls is, "I wasn't interviewed, so why should this poll matter?” By interviewing 1,000 respondents and applying minimal weights based on race, ethnicity, age, gender, education and the 2020 presidential vote, the poll achieves a representative sample of the nation at large – with a margin of error at a 95% confidence level.

NBC News editors and reporters — along with the pollsters at Hart Research and Public Opinion Strategies — all work to formulate the questions to try to capture the news and current events NBC is trying to gauge. Both Hart Research and Public Opinion Strategies work to ensure the language and placement of the questions are as neutral as possible.

Biden trims Trump’s lead

The poll also finds Trump narrowly ahead of Biden by 2 points among registered voters in a head-to-head matchup, 46% to 44% — down from Trump’s 5-point advantage in January, 47% to 42%.

The movement, which is within the poll’s margin of error of plus or minus 3.1 percentage points, is consistent with what other national polls have found in the Trump-Biden race.

Trump’s biggest advantages are among men (53% to 37%), white voters (54% to 37%) and white voters without college degrees (65% to 25%).

Biden’s top advantages are among Black voters (71% to 13%), women (50% to 39%) and Latinos (49% to 39%).

The poll shows the two candidates are essentially tied among independents (Biden 36%, Trump 34%) and voters ages 18-34 (Biden 44%, Trump 43%). One of the big polling mysteries this cycle is whether young voters have defected from Biden (as the NBC News poll has found over multiple surveys) or whether Democrats have maintained their advantage among that demographic.

When the ballot is expanded to five named candidates, Biden takes a 2-point lead over Trump: Biden 39%, Trump 37%, Kennedy 13%, Jill Stein 3% and Cornel West 2%.

Again, the result between Biden and Trump is within the poll’s margin of error.

Notably, the poll finds a greater share of Trump voters from the head-to-head matchup supporting Kennedy in the expanded ballot compared with Biden voters, different from the results of some other surveys.

(Read more here about how Kennedy's candidacy affe cts the 2024 race, according to the poll.)

The president’s approval rating ticks up to 42%

In addition, the poll found 42% of registered voters approving of Biden’s overall job performance — up 5 points since January’s NBC News poll, which found Biden at the lowest point of his presidency.

Fifty-six percent of voters say they disapprove of the job he has done, which is down 4 points from January.

Biden’s gains over the past few months have come from key parts of his 2020   base, especially among Democrats and Black voters. But he continues to hold low ratings among Latinos (40% approval), young voters (37%) and independents (36%).

“The data across this poll show that Joe Biden has begun to gain some ground in rebuilding his coalition from 2020,” said Horwitt, the Democratic pollster. “The question is whether he can build upon this momentum and make inroads with the groups of voters that still are holding back support.”

But McInturff, the GOP pollster, points out that the only recent presidents   who lost re-election had approval ratings higher than Biden’s at this point in the election cycle: George H.W. Bush (43%) and Trump (46%).

“President Biden has a precarious hold on the presidency and is in a difficult position as it relates to his re-election,” McInturff said.

On the issues, 39% of voters say they approve of Biden’s handling of the economy (up from 36% in January), 28% approve of his handling of border security and immigration, and just 27% approve of his handling of the Israel-Hamas war (down from 29% in January).

Voters gave Biden his highest issue rating on   addressing student loan debt, with 44% approving of his handling of the issue, compared with 51% who say they disapprove.

Biden leads on abortion and unity; Trump leads on inflation and competency

The NBC News poll asked voters to determine which candidate they thought is better on several different issues and attributes.

Biden holds a 15-point advantage over Trump on dealing with the issue of abortion, and he is ahead by 9 points on having the ability to bring the country together — though that is down from his 24-point advantage on that issue in the September 2020 NBC News poll.

Trump, meanwhile, leads in having the ability to handle a crisis (by 4 points), in having a strong record of accomplishments (by 7 points), in being competent and effective (by 11 points), in having the necessary mental and physical health to be president (by 19 points) and in dealing with inflation and the cost of living (by 22 points).

Inflation, immigration are the top 2024 issues

Inflation and the cost of living top the list of issues in the poll, with 23% of voters saying they’re the most important issue facing the country.

The other top voters is   immigration and the situation at the border (22%) — followed by   threats to democracy (16%), jobs and the economy (11%), abortion (6%) and health care (6%).

In addition, 63% of voters say their families’ incomes are falling behind the cost of living — essentially unchanged from what the poll found in 2022 and 2023.

And 53% of voters say the country’s economy hasn’t improved, compared with 33% who say that it has improved and that Biden deserves some credit for it and another 8% who agree the economy has improved but don’t give him credit for it.

“If I look back to when I had all three of my children in the house — we only have one child left in the house now, and we’re spending more now than what we did when we had a family of five,” said poll respondent Art Fales, 45, of Florida, who says he’s most likely voting for Trump.

But on a separate question — is there an issue so important that you’ll vote for or against a candidate solely on that basis? — the top responses are protecting democracy and constitutional rights (28%), immigration and border security (20%) and abortion (19%).

Indeed, 30% of Democrats, 29% of young voters and 27% of women say they are single-issue voters on abortion.

“I have a right to what I do with my body,” said poll respondent Amanda Willis, 28, of Louisiana, who said she’s voting for Biden. “And I don’t believe that other people should have the ability to determine that.”

Other poll findings

• With Trump’s first criminal trial underway, 50% of voters say he is being held to the same standard as anyone else when it comes to his multiple legal challenges. That compares with 43% who believe he’s being unfairly targeted in the trials. 
• 52% of voters have unfavorable views of Biden, while 53% share the same views of Trump.
• And Democrats and Republicans are essentially tied in congressional preference, with 47% of voters preferring Republicans to control Congress and 46% wanting Democrats in charge. Republicans held a 4-point lead on this question in January.

The NBC News poll of 1,000 registered voters nationwide — 891 contacted via cellphone — was conducted April 12-16, and it has an overall margin of error of plus or minus 3.1 percentage points.

Mark Murray is a senior political editor at NBC News.

Sarah Dean is a 2024 NBC News campaign embed.