limitations of new research

21 Research Limitations Examples

Research limitations refer to the potential weaknesses inherent in a study. All studies have limitations of some sort, meaning declaring limitations doesn’t necessarily need to be a bad thing, so long as your declaration of limitations is well thought-out and explained.

Rarely is a study perfect. Researchers have to make trade-offs when developing their studies, which are often based upon practical considerations such as time and monetary constraints, weighing the breadth of participants against the depth of insight, and choosing one methodology or another.

In research, studies can have limitations such as limited scope, researcher subjectivity, and lack of available research tools.

Acknowledging the limitations of your study should be seen as a strength. It demonstrates your willingness for transparency, humility, and submission to the scientific method and can bolster the integrity of the study. It can also inform future research direction.

Typically, scholars will explore the limitations of their study in either their methodology section, their conclusion section, or both.

Research Limitations Examples

Qualitative and quantitative research offer different perspectives and methods in exploring phenomena, each with its own strengths and limitations. So, I’ve split the limitations examples sections into qualitative and quantitative below.

Qualitative Research Limitations

Qualitative research seeks to understand phenomena in-depth and in context. It focuses on the ‘why’ and ‘how’ questions.

It’s often used to explore new or complex issues, and it provides rich, detailed insights into participants’ experiences, behaviors, and attitudes. However, these strengths also create certain limitations, as explained below.

1. Subjectivity

Qualitative research often requires the researcher to interpret subjective data. One researcher may examine a text and identify different themes or concepts as more dominant than others.

Close qualitative readings of texts are necessarily subjective – and while this may be a limitation, qualitative researchers argue this is the best way to deeply understand everything in context.

Suggested Solution and Response: To minimize subjectivity bias, you could consider cross-checking your own readings of themes and data against other scholars’ readings and interpretations. This may involve giving the raw data to a supervisor or colleague and asking them to code the data separately, then coming together to compare and contrast results.

2. Researcher Bias

The concept of researcher bias is related to, but slightly different from, subjectivity.

Researcher bias refers to the perspectives and opinions you bring with you when doing your research.

For example, a researcher who is explicitly of a certain philosophical or political persuasion may bring that persuasion to bear when interpreting data.

In many scholarly traditions, we will attempt to minimize researcher bias through the utilization of clear procedures that are set out in advance or through the use of statistical analysis tools.

However, in other traditions, such as in postmodern feminist research , declaration of bias is expected, and acknowledgment of bias is seen as a positive because, in those traditions, it is believed that bias cannot be eliminated from research, so instead, it is a matter of integrity to present it upfront.

Suggested Solution and Response: Acknowledge the potential for researcher bias and, depending on your theoretical framework , accept this, or identify procedures you have taken to seek a closer approximation to objectivity in your coding and analysis.

3. Generalizability

If you’re struggling to find a limitation to discuss in your own qualitative research study, then this one is for you: all qualitative research, of all persuasions and perspectives, cannot be generalized.

This is a core feature that sets qualitative data and quantitative data apart.

The point of qualitative data is to select case studies and similarly small corpora and dig deep through in-depth analysis and thick description of data.

Often, this will also mean that you have a non-randomized sample size.

While this is a positive – you’re going to get some really deep, contextualized, interesting insights – it also means that the findings may not be generalizable to a larger population that may not be representative of the small group of people in your study.

Suggested Solution and Response: Suggest future studies that take a quantitative approach to the question.

4. The Hawthorne Effect

The Hawthorne effect refers to the phenomenon where research participants change their ‘observed behavior’ when they’re aware that they are being observed.

This effect was first identified by Elton Mayo who conducted studies of the effects of various factors ton workers’ productivity. He noticed that no matter what he did – turning up the lights, turning down the lights, etc. – there was an increase in worker outputs compared to prior to the study taking place.

Mayo realized that the mere act of observing the workers made them work harder – his observation was what was changing behavior.

So, if you’re looking for a potential limitation to name for your observational research study , highlight the possible impact of the Hawthorne effect (and how you could reduce your footprint or visibility in order to decrease its likelihood).

Suggested Solution and Response: Highlight ways you have attempted to reduce your footprint while in the field, and guarantee anonymity to your research participants.

5. Replicability

Quantitative research has a great benefit in that the studies are replicable – a researcher can get a similar sample size, duplicate the variables, and re-test a study. But you can’t do that in qualitative research.

Qualitative research relies heavily on context – a specific case study or specific variables that make a certain instance worthy of analysis. As a result, it’s often difficult to re-enter the same setting with the same variables and repeat the study.

Furthermore, the individual researcher’s interpretation is more influential in qualitative research, meaning even if a new researcher enters an environment and makes observations, their observations may be different because subjectivity comes into play much more. This doesn’t make the research bad necessarily (great insights can be made in qualitative research), but it certainly does demonstrate a weakness of qualitative research.

6. Limited Scope

“Limited scope” is perhaps one of the most common limitations listed by researchers – and while this is often a catch-all way of saying, “well, I’m not studying that in this study”, it’s also a valid point.

No study can explore everything related to a topic. At some point, we have to make decisions about what’s included in the study and what is excluded from the study.

So, you could say that a limitation of your study is that it doesn’t look at an extra variable or concept that’s certainly worthy of study but will have to be explored in your next project because this project has a clearly and narrowly defined goal.

Suggested Solution and Response: Be clear about what’s in and out of the study when writing your research question.

7. Time Constraints

This is also a catch-all claim you can make about your research project: that you would have included more people in the study, looked at more variables, and so on. But you’ve got to submit this thing by the end of next semester! You’ve got time constraints.

And time constraints are a recognized reality in all research.

But this means you’ll need to explain how time has limited your decisions. As with “limited scope”, this may mean that you had to study a smaller group of subjects, limit the amount of time you spent in the field, and so forth.

Suggested Solution and Response: Suggest future studies that will build on your current work, possibly as a PhD project.

8. Resource Intensiveness

Qualitative research can be expensive due to the cost of transcription, the involvement of trained researchers, and potential travel for interviews or observations.

So, resource intensiveness is similar to the time constraints concept. If you don’t have the funds, you have to make decisions about which tools to use, which statistical software to employ, and how many research assistants you can dedicate to the study.

Suggested Solution and Response: Suggest future studies that will gain more funding on the back of this ‘ exploratory study ‘.

9. Coding Difficulties

Data analysis in qualitative research often involves coding, which can be subjective and complex, especially when dealing with ambiguous or contradicting data.

After naming this as a limitation in your research, it’s important to explain how you’ve attempted to address this. Some ways to ‘limit the limitation’ include:

• Triangulation: Have 2 other researchers code the data as well and cross-check your results with theirs to identify outliers that may need to be re-examined, debated with the other researchers, or removed altogether.
• Procedure: Use a clear coding procedure to demonstrate reliability in your coding process. I personally use the thematic network analysis method outlined in this academic article by Attride-Stirling (2001).

Suggested Solution and Response: Triangulate your coding findings with colleagues, and follow a thematic network analysis procedure.

10. Risk of Non-Responsiveness

There is always a risk in research that research participants will be unwilling or uncomfortable sharing their genuine thoughts and feelings in the study.

This is particularly true when you’re conducting research on sensitive topics, politicized topics, or topics where the participant is expressing vulnerability .

This is similar to the Hawthorne effect (aka participant bias), where participants change their behaviors in your presence; but it goes a step further, where participants actively hide their true thoughts and feelings from you.

Suggested Solution and Response: One way to manage this is to try to include a wider group of people with the expectation that there will be non-responsiveness from some participants.

11. Risk of Attrition

Attrition refers to the process of losing research participants throughout the study.

This occurs most commonly in longitudinal studies , where a researcher must return to conduct their analysis over spaced periods of time, often over a period of years.

Things happen to people over time – they move overseas, their life experiences change, they get sick, change their minds, and even die. The more time that passes, the greater the risk of attrition.

Suggested Solution and Response: One way to manage this is to try to include a wider group of people with the expectation that there will be attrition over time.

12. Difficulty in Maintaining Confidentiality and Anonymity

Given the detailed nature of qualitative data , ensuring participant anonymity can be challenging.

If you have a sensitive topic in a specific case study, even anonymizing research participants sometimes isn’t enough. People might be able to induce who you’re talking about.

Sometimes, this will mean you have to exclude some interesting data that you collected from your final report. Confidentiality and anonymity come before your findings in research ethics – and this is a necessary limiting factor.

Suggested Solution and Response: Highlight the efforts you have taken to anonymize data, and accept that confidentiality and accountability place extremely important constraints on academic research.

13. Difficulty in Finding Research Participants

A study that looks at a very specific phenomenon or even a specific set of cases within a phenomenon means that the pool of potential research participants can be very low.

Compile on top of this the fact that many people you approach may choose not to participate, and you could end up with a very small corpus of subjects to explore. This may limit your ability to make complete findings, even in a quantitative sense.

You may need to therefore limit your research question and objectives to something more realistic.

Suggested Solution and Response: Highlight that this is going to limit the study’s generalizability significantly.

14. Ethical Limitations

Ethical limitations refer to the things you cannot do based on ethical concerns identified either by yourself or your institution’s ethics review board.

This might include threats to the physical or psychological well-being of your research subjects, the potential of releasing data that could harm a person’s reputation, and so on.

Furthermore, even if your study follows all expected standards of ethics, you still, as an ethical researcher, need to allow a research participant to pull out at any point in time, after which you cannot use their data, which demonstrates an overlap between ethical constraints and participant attrition.

Suggested Solution and Response: Highlight that these ethical limitations are inevitable but important to sustain the integrity of the research.

For more on Qualitative Research, Explore my Qualitative Research Guide

Quantitative Research Limitations

Quantitative research focuses on quantifiable data and statistical, mathematical, or computational techniques. It’s often used to test hypotheses, assess relationships and causality, and generalize findings across larger populations.

Quantitative research is widely respected for its ability to provide reliable, measurable, and generalizable data (if done well!). Its structured methodology has strengths over qualitative research, such as the fact it allows for replication of the study, which underpins the validity of the research.

However, this approach is not without it limitations, explained below.

1. Over-Simplification

Quantitative research is powerful because it allows you to measure and analyze data in a systematic and standardized way. However, one of its limitations is that it can sometimes simplify complex phenomena or situations.

In other words, it might miss the subtleties or nuances of the research subject.

For example, if you’re studying why people choose a particular diet, a quantitative study might identify factors like age, income, or health status. But it might miss other aspects, such as cultural influences or personal beliefs, that can also significantly impact dietary choices.

When writing about this limitation, you can say that your quantitative approach, while providing precise measurements and comparisons, may not capture the full complexity of your subjects of study.

Suggested Solution and Response: Suggest a follow-up case study using the same research participants in order to gain additional context and depth.

2. Lack of Context

Another potential issue with quantitative research is that it often focuses on numbers and statistics at the expense of context or qualitative information.

Let’s say you’re studying the effect of classroom size on student performance. You might find that students in smaller classes generally perform better. However, this doesn’t take into account other variables, like teaching style , student motivation, or family support.

When describing this limitation, you might say, “Although our research provides important insights into the relationship between class size and student performance, it does not incorporate the impact of other potentially influential variables. Future research could benefit from a mixed-methods approach that combines quantitative analysis with qualitative insights.”

3. Applicability to Real-World Settings

Oftentimes, experimental research takes place in controlled environments to limit the influence of outside factors.

This control is great for isolation and understanding the specific phenomenon but can limit the applicability or “external validity” of the research to real-world settings.

For example, if you conduct a lab experiment to see how sleep deprivation impacts cognitive performance, the sterile, controlled lab environment might not reflect real-world conditions where people are dealing with multiple stressors.

Therefore, when explaining the limitations of your quantitative study in your methodology section, you could state:

“While our findings provide valuable information about [topic], the controlled conditions of the experiment may not accurately represent real-world scenarios where extraneous variables will exist. As such, the direct applicability of our results to broader contexts may be limited.”

Suggested Solution and Response: Suggest future studies that will engage in real-world observational research, such as ethnographic research.

4. Limited Flexibility

Once a quantitative study is underway, it can be challenging to make changes to it. This is because, unlike in grounded research, you’re putting in place your study in advance, and you can’t make changes part-way through.

Your study design, data collection methods, and analysis techniques need to be decided upon before you start collecting data.

For example, if you are conducting a survey on the impact of social media on teenage mental health, and halfway through, you realize that you should have included a question about their screen time, it’s generally too late to add it.

When discussing this limitation, you could write something like, “The structured nature of our quantitative approach allows for consistent data collection and analysis but also limits our flexibility to adapt and modify the research process in response to emerging insights and ideas.”

Suggested Solution and Response: Suggest future studies that will use mixed-methods or qualitative research methods to gain additional depth of insight.

5. Risk of Survey Error

Surveys are a common tool in quantitative research, but they carry risks of error.

There can be measurement errors (if a question is misunderstood), coverage errors (if some groups aren’t adequately represented), non-response errors (if certain people don’t respond), and sampling errors (if your sample isn’t representative of the population).

For instance, if you’re surveying college students about their study habits , but only daytime students respond because you conduct the survey during the day, your results will be skewed.

In discussing this limitation, you might say, “Despite our best efforts to develop a comprehensive survey, there remains a risk of survey error, including measurement, coverage, non-response, and sampling errors. These could potentially impact the reliability and generalizability of our findings.”

Suggested Solution and Response: Suggest future studies that will use other survey tools to compare and contrast results.

6. Limited Ability to Probe Answers

With quantitative research, you typically can’t ask follow-up questions or delve deeper into participants’ responses like you could in a qualitative interview.

For instance, imagine you are surveying 500 students about study habits in a questionnaire. A respondent might indicate that they study for two hours each night. You might want to follow up by asking them to elaborate on what those study sessions involve or how effective they feel their habits are.

However, quantitative research generally disallows this in the way a qualitative semi-structured interview could.

When discussing this limitation, you might write, “Given the structured nature of our survey, our ability to probe deeper into individual responses is limited. This means we may not fully understand the context or reasoning behind the responses, potentially limiting the depth of our findings.”

Suggested Solution and Response: Suggest future studies that engage in mixed-method or qualitative methodologies to address the issue from another angle.

7. Reliance on Instruments for Data Collection

In quantitative research, the collection of data heavily relies on instruments like questionnaires, surveys, or machines.

The limitation here is that the data you get is only as good as the instrument you’re using. If the instrument isn’t designed or calibrated well, your data can be flawed.

For instance, if you’re using a questionnaire to study customer satisfaction and the questions are vague, confusing, or biased, the responses may not accurately reflect the customers’ true feelings.

When discussing this limitation, you could say, “Our study depends on the use of questionnaires for data collection. Although we have put significant effort into designing and testing the instrument, it’s possible that inaccuracies or misunderstandings could potentially affect the validity of the data collected.”

Suggested Solution and Response: Suggest future studies that will use different instruments but examine the same variables to triangulate results.

8. Time and Resource Constraints (Specific to Quantitative Research)

Quantitative research can be time-consuming and resource-intensive, especially when dealing with large samples.

It often involves systematic sampling, rigorous design, and sometimes complex statistical analysis.

If resources and time are limited, it can restrict the scale of your research, the techniques you can employ, or the extent of your data analysis.

For example, you may want to conduct a nationwide survey on public opinion about a certain policy. However, due to limited resources, you might only be able to survey people in one city.

When writing about this limitation, you could say, “Given the scope of our research and the resources available, we are limited to conducting our survey within one city, which may not fully represent the nationwide public opinion. Hence, the generalizability of the results may be limited.”

Suggested Solution and Response: Suggest future studies that will have more funding or longer timeframes.

How to Discuss Your Research Limitations

1. in your research proposal and methodology section.

In the research proposal, which will become the methodology section of your dissertation, I would recommend taking the four following steps, in order:

• Be Explicit about your Scope – If you limit the scope of your study in your research question, aims, and objectives, then you can set yourself up well later in the methodology to say that certain questions are “outside the scope of the study.” For example, you may identify the fact that the study doesn’t address a certain variable, but you can follow up by stating that the research question is specifically focused on the variable that you are examining, so this limitation would need to be looked at in future studies.
• Acknowledge the Limitation – Acknowledging the limitations of your study demonstrates reflexivity and humility and can make your research more reliable and valid. It also pre-empts questions the people grading your paper may have, so instead of them down-grading you for your limitations; they will congratulate you on explaining the limitations and how you have addressed them!
• Explain your Decisions – You may have chosen your approach (despite its limitations) for a very specific reason. This might be because your approach remains, on balance, the best one to answer your research question. Or, it might be because of time and monetary constraints that are outside of your control.
• Highlight the Strengths of your Approach – Conclude your limitations section by strongly demonstrating that, despite limitations, you’ve worked hard to minimize the effects of the limitations and that you have chosen your specific approach and methodology because it’s also got some terrific strengths. Name the strengths.

Overall, you’ll want to acknowledge your own limitations but also explain that the limitations don’t detract from the value of your study as it stands.

2. In the Conclusion Section or Chapter

In the conclusion of your study, it is generally expected that you return to a discussion of the study’s limitations. Here, I recommend the following steps:

• Acknowledge issues faced – After completing your study, you will be increasingly aware of issues you may have faced that, if you re-did the study, you may have addressed earlier in order to avoid those issues. Acknowledge these issues as limitations, and frame them as recommendations for subsequent studies.
• Suggest further research – Scholarly research aims to fill gaps in the current literature and knowledge. Having established your expertise through your study, suggest lines of inquiry for future researchers. You could state that your study had certain limitations, and “future studies” can address those limitations.
• Suggest a mixed methods approach – Qualitative and quantitative research each have pros and cons. So, note those ‘cons’ of your approach, then say the next study should approach the topic using the opposite methodology or could approach it using a mixed-methods approach that could achieve the benefits of quantitative studies with the nuanced insights of associated qualitative insights as part of an in-study case-study.

Overall, be clear about both your limitations and how those limitations can inform future studies.

In sum, each type of research method has its own strengths and limitations. Qualitative research excels in exploring depth, context, and complexity, while quantitative research excels in examining breadth, generalizability, and quantifiable measures. Despite their individual limitations, each method contributes unique and valuable insights, and researchers often use them together to provide a more comprehensive understanding of the phenomenon being studied.

Attride-Stirling, J. (2001). Thematic networks: an analytic tool for qualitative research. Qualitative research , 1 (3), 385-405. ( Source )

Atkinson, P., Delamont, S., Cernat, A., Sakshaug, J., & Williams, R. A. (2021).  SAGE research methods foundations . London: Sage Publications.

Clark, T., Foster, L., Bryman, A., & Sloan, L. (2021).  Bryman’s social research methods . Oxford: Oxford University Press.

Köhler, T., Smith, A., & Bhakoo, V. (2022). Templates in qualitative research methods: Origins, limitations, and new directions.  Organizational Research Methods ,  25 (2), 183-210. ( Source )

Lenger, A. (2019). The rejection of qualitative research methods in economics.  Journal of Economic Issues ,  53 (4), 946-965. ( Source )

Taherdoost, H. (2022). What are different research approaches? Comprehensive review of qualitative, quantitative, and mixed method research, their applications, types, and limitations.  Journal of Management Science & Engineering Research ,  5 (1), 53-63. ( Source )

Walliman, N. (2021).  Research methods: The basics . New York: Routledge.

Chris Drew (PhD)

Dr. Chris Drew is the founder of the Helpful Professor. He holds a PhD in education and has published over 20 articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education. [Image Descriptor: Photo of Chris]

• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 5 Top Tips for Succeeding at University
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 50 Durable Goods Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 100 Consumer Goods Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 30 Globalization Pros and Cons

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• Limitations of the Study
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The limitations of the study are those characteristics of design or methodology that impacted or influenced the interpretation of the findings from your research. Study limitations are the constraints placed on the ability to generalize from the results, to further describe applications to practice, and/or related to the utility of findings that are the result of the ways in which you initially chose to design the study or the method used to establish internal and external validity or the result of unanticipated challenges that emerged during the study.

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Theofanidis, Dimitrios and Antigoni Fountouki. "Limitations and Delimitations in the Research Process." Perioperative Nursing 7 (September-December 2018): 155-163. .

Importance of...

Always acknowledge a study's limitations. It is far better that you identify and acknowledge your study’s limitations than to have them pointed out by your professor and have your grade lowered because you appeared to have ignored them or didn't realize they existed.

Keep in mind that acknowledgment of a study's limitations is an opportunity to make suggestions for further research. If you do connect your study's limitations to suggestions for further research, be sure to explain the ways in which these unanswered questions may become more focused because of your study.

Acknowledgment of a study's limitations also provides you with opportunities to demonstrate that you have thought critically about the research problem, understood the relevant literature published about it, and correctly assessed the methods chosen for studying the problem. A key objective of the research process is not only discovering new knowledge but also to confront assumptions and explore what we don't know.

Claiming limitations is a subjective process because you must evaluate the impact of those limitations . Don't just list key weaknesses and the magnitude of a study's limitations. To do so diminishes the validity of your research because it leaves the reader wondering whether, or in what ways, limitation(s) in your study may have impacted the results and conclusions. Limitations require a critical, overall appraisal and interpretation of their impact. You should answer the question: do these problems with errors, methods, validity, etc. eventually matter and, if so, to what extent?

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com.

Descriptions of Possible Limitations

All studies have limitations . However, it is important that you restrict your discussion to limitations related to the research problem under investigation. For example, if a meta-analysis of existing literature is not a stated purpose of your research, it should not be discussed as a limitation. Do not apologize for not addressing issues that you did not promise to investigate in the introduction of your paper.

Here are examples of limitations related to methodology and the research process you may need to describe and discuss how they possibly impacted your results. Note that descriptions of limitations should be stated in the past tense because they were discovered after you completed your research.

Possible Methodological Limitations

• Sample size -- the number of the units of analysis you use in your study is dictated by the type of research problem you are investigating. Note that, if your sample size is too small, it will be difficult to find significant relationships from the data, as statistical tests normally require a larger sample size to ensure a representative distribution of the population and to be considered representative of groups of people to whom results will be generalized or transferred. Note that sample size is generally less relevant in qualitative research if explained in the context of the research problem.
• Lack of available and/or reliable data -- a lack of data or of reliable data will likely require you to limit the scope of your analysis, the size of your sample, or it can be a significant obstacle in finding a trend and a meaningful relationship. You need to not only describe these limitations but provide cogent reasons why you believe data is missing or is unreliable. However, don’t just throw up your hands in frustration; use this as an opportunity to describe a need for future research based on designing a different method for gathering data.
• Lack of prior research studies on the topic -- citing prior research studies forms the basis of your literature review and helps lay a foundation for understanding the research problem you are investigating. Depending on the currency or scope of your research topic, there may be little, if any, prior research on your topic. Before assuming this to be true, though, consult with a librarian! In cases when a librarian has confirmed that there is little or no prior research, you may be required to develop an entirely new research typology [for example, using an exploratory rather than an explanatory research design ]. Note again that discovering a limitation can serve as an important opportunity to identify new gaps in the literature and to describe the need for further research.
• Measure used to collect the data -- sometimes it is the case that, after completing your interpretation of the findings, you discover that the way in which you gathered data inhibited your ability to conduct a thorough analysis of the results. For example, you regret not including a specific question in a survey that, in retrospect, could have helped address a particular issue that emerged later in the study. Acknowledge the deficiency by stating a need for future researchers to revise the specific method for gathering data.
• Self-reported data -- whether you are relying on pre-existing data or you are conducting a qualitative research study and gathering the data yourself, self-reported data is limited by the fact that it rarely can be independently verified. In other words, you have to the accuracy of what people say, whether in interviews, focus groups, or on questionnaires, at face value. However, self-reported data can contain several potential sources of bias that you should be alert to and note as limitations. These biases become apparent if they are incongruent with data from other sources. These are: (1) selective memory [remembering or not remembering experiences or events that occurred at some point in the past]; (2) telescoping [recalling events that occurred at one time as if they occurred at another time]; (3) attribution [the act of attributing positive events and outcomes to one's own agency, but attributing negative events and outcomes to external forces]; and, (4) exaggeration [the act of representing outcomes or embellishing events as more significant than is actually suggested from other data].

Possible Limitations of the Researcher

• Access -- if your study depends on having access to people, organizations, data, or documents and, for whatever reason, access is denied or limited in some way, the reasons for this needs to be described. Also, include an explanation why being denied or limited access did not prevent you from following through on your study.
• Longitudinal effects -- unlike your professor, who can literally devote years [even a lifetime] to studying a single topic, the time available to investigate a research problem and to measure change or stability over time is constrained by the due date of your assignment. Be sure to choose a research problem that does not require an excessive amount of time to complete the literature review, apply the methodology, and gather and interpret the results. If you're unsure whether you can complete your research within the confines of the assignment's due date, talk to your professor.
• Cultural and other type of bias -- we all have biases, whether we are conscience of them or not. Bias is when a person, place, event, or thing is viewed or shown in a consistently inaccurate way. Bias is usually negative, though one can have a positive bias as well, especially if that bias reflects your reliance on research that only support your hypothesis. When proof-reading your paper, be especially critical in reviewing how you have stated a problem, selected the data to be studied, what may have been omitted, the manner in which you have ordered events, people, or places, how you have chosen to represent a person, place, or thing, to name a phenomenon, or to use possible words with a positive or negative connotation. NOTE :   If you detect bias in prior research, it must be acknowledged and you should explain what measures were taken to avoid perpetuating that bias. For example, if a previous study only used boys to examine how music education supports effective math skills, describe how your research expands the study to include girls.
• Fluency in a language -- if your research focuses , for example, on measuring the perceived value of after-school tutoring among Mexican-American ESL [English as a Second Language] students and you are not fluent in Spanish, you are limited in being able to read and interpret Spanish language research studies on the topic or to speak with these students in their primary language. This deficiency should be acknowledged.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Senunyeme, Emmanuel K. Business Research Methods. Powerpoint Presentation. Regent University of Science and Technology; ter Riet, Gerben et al. “All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies.” PLOS One 8 (November 2013): 1-6.

Structure and Writing Style

Information about the limitations of your study are generally placed either at the beginning of the discussion section of your paper so the reader knows and understands the limitations before reading the rest of your analysis of the findings, or, the limitations are outlined at the conclusion of the discussion section as an acknowledgement of the need for further study. Statements about a study's limitations should not be buried in the body [middle] of the discussion section unless a limitation is specific to something covered in that part of the paper. If this is the case, though, the limitation should be reiterated at the conclusion of the section.

If you determine that your study is seriously flawed due to important limitations , such as, an inability to acquire critical data, consider reframing it as an exploratory study intended to lay the groundwork for a more complete research study in the future. Be sure, though, to specifically explain the ways that these flaws can be successfully overcome in a new study.

But, do not use this as an excuse for not developing a thorough research paper! Review the tab in this guide for developing a research topic . If serious limitations exist, it generally indicates a likelihood that your research problem is too narrowly defined or that the issue or event under study is too recent and, thus, very little research has been written about it. If serious limitations do emerge, consult with your professor about possible ways to overcome them or how to revise your study.

When discussing the limitations of your research, be sure to:

• Describe each limitation in detailed but concise terms;
• Explain why each limitation exists;
• Provide the reasons why each limitation could not be overcome using the method(s) chosen to acquire or gather the data [cite to other studies that had similar problems when possible];
• Assess the impact of each limitation in relation to the overall findings and conclusions of your study; and,
• If appropriate, describe how these limitations could point to the need for further research.

Remember that the method you chose may be the source of a significant limitation that has emerged during your interpretation of the results [for example, you didn't interview a group of people that you later wish you had]. If this is the case, don't panic. Acknowledge it, and explain how applying a different or more robust methodology might address the research problem more effectively in a future study. A underlying goal of scholarly research is not only to show what works, but to demonstrate what doesn't work or what needs further clarification.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Ioannidis, John P.A. "Limitations are not Properly Acknowledged in the Scientific Literature." Journal of Clinical Epidemiology 60 (2007): 324-329; Pasek, Josh. Writing the Empirical Social Science Research Paper: A Guide for the Perplexed. January 24, 2012. Academia.edu; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com; What Is an Academic Paper? Institute for Writing Rhetoric. Dartmouth College; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University.

Writing Tip

Don't Inflate the Importance of Your Findings!

After all the hard work and long hours devoted to writing your research paper, it is easy to get carried away with attributing unwarranted importance to what you’ve done. We all want our academic work to be viewed as excellent and worthy of a good grade, but it is important that you understand and openly acknowledge the limitations of your study. Inflating the importance of your study's findings could be perceived by your readers as an attempt hide its flaws or encourage a biased interpretation of the results. A small measure of humility goes a long way!

Another Writing Tip

Negative Results are Not a Limitation!

Negative evidence refers to findings that unexpectedly challenge rather than support your hypothesis. If you didn't get the results you anticipated, it may mean your hypothesis was incorrect and needs to be reformulated. Or, perhaps you have stumbled onto something unexpected that warrants further study. Moreover, the absence of an effect may be very telling in many situations, particularly in experimental research designs. In any case, your results may very well be of importance to others even though they did not support your hypothesis. Do not fall into the trap of thinking that results contrary to what you expected is a limitation to your study. If you carried out the research well, they are simply your results and only require additional interpretation.

Lewis, George H. and Jonathan F. Lewis. “The Dog in the Night-Time: Negative Evidence in Social Research.” The British Journal of Sociology 31 (December 1980): 544-558.

Yet Another Writing Tip

Sample Size Limitations in Qualitative Research

Sample sizes are typically smaller in qualitative research because, as the study goes on, acquiring more data does not necessarily lead to more information. This is because one occurrence of a piece of data, or a code, is all that is necessary to ensure that it becomes part of the analysis framework. However, it remains true that sample sizes that are too small cannot adequately support claims of having achieved valid conclusions and sample sizes that are too large do not permit the deep, naturalistic, and inductive analysis that defines qualitative inquiry. Determining adequate sample size in qualitative research is ultimately a matter of judgment and experience in evaluating the quality of the information collected against the uses to which it will be applied and the particular research method and purposeful sampling strategy employed. If the sample size is found to be a limitation, it may reflect your judgment about the methodological technique chosen [e.g., single life history study versus focus group interviews] rather than the number of respondents used.

Boddy, Clive Roland. "Sample Size for Qualitative Research." Qualitative Market Research: An International Journal 19 (2016): 426-432; Huberman, A. Michael and Matthew B. Miles. "Data Management and Analysis Methods." In Handbook of Qualitative Research . Norman K. Denzin and Yvonna S. Lincoln, eds. (Thousand Oaks, CA: Sage, 1994), pp. 428-444; Blaikie, Norman. "Confounding Issues Related to Determining Sample Size in Qualitative Research." International Journal of Social Research Methodology 21 (2018): 635-641; Oppong, Steward Harrison. "The Problem of Sampling in qualitative Research." Asian Journal of Management Sciences and Education 2 (2013): 202-210.

• << Previous: 8. The Discussion
• Next: 9. The Conclusion >>
• Last Updated: Apr 5, 2024 1:38 PM
• URL: https://libguides.usc.edu/writingguide

How to present limitations in research

Last updated

30 January 2024

Reviewed by

Limitations don’t invalidate or diminish your results, but it’s best to acknowledge them. This will enable you to address any questions your study failed to answer because of them.

In this guide, learn how to recognize, present, and overcome limitations in research.

• What is a research limitation?

Research limitations are weaknesses in your research design or execution that may have impacted outcomes and conclusions. Uncovering limitations doesn’t necessarily indicate poor research design—it just means you encountered challenges you couldn’t have anticipated that limited your research efforts.

Does basic research have limitations?

Basic research aims to provide more information about your research topic. It requires the same standard research methodology and data collection efforts as any other research type, and it can also have limitations.

• Common research limitations

Researchers encounter common limitations when embarking on a study. Limitations can occur in relation to the methods you apply or the research process you design. They could also be connected to you as the researcher.

Methodology limitations

Not having access to data or reliable information can impact the methods used to facilitate your research. A lack of data or reliability may limit the parameters of your study area and the extent of your exploration.

Your sample size may also be affected because you won’t have any direction on how big or small it should be and who or what you should include. Having too few participants won’t adequately represent the population or groups of people needed to draw meaningful conclusions.

Research process limitations

The study’s design can impose constraints on the process. For example, as you’re conducting the research, issues may arise that don’t conform to the data collection methodology you developed. You may not realize until well into the process that you should have incorporated more specific questions or comprehensive experiments to generate the data you need to have confidence in your results.

Constraints on resources can also have an impact. Being limited on participants or participation incentives may limit your sample sizes. Insufficient tools, equipment, and materials to conduct a thorough study may also be a factor.

Common researcher limitations

Here are some of the common researcher limitations you may encounter:

Time: some research areas require multi-year longitudinal approaches, but you might not be able to dedicate that much time. Imagine you want to measure how much memory a person loses as they age. This may involve conducting multiple tests on a sample of participants over 20–30 years, which may be impossible.

Bias: researchers can consciously or unconsciously apply bias to their research. Biases can contribute to relying on research sources and methodologies that will only support your beliefs about the research you’re embarking on. You might also omit relevant issues or participants from the scope of your study because of your biases.

Limited access to data : you may need to pay to access specific databases or journals that would be helpful to your research process. You might also need to gain information from certain people or organizations but have limited access to them. These cases require readjusting your process and explaining why your findings are still reliable.

• Why is it important to identify limitations?

Identifying limitations adds credibility to research and provides a deeper understanding of how you arrived at your conclusions.

Constraints may have prevented you from collecting specific data or information you hoped would prove or disprove your hypothesis or provide a more comprehensive understanding of your research topic.

However, identifying the limitations contributing to your conclusions can inspire further research efforts that help gather more substantial information and data.

• Where to put limitations in a research paper

A research paper is broken up into different sections that appear in the following order:

Introduction

Methodology

The discussion portion of your paper explores your findings and puts them in the context of the overall research. Either place research limitations at the beginning of the discussion section before the analysis of your findings or at the end of the section to indicate that further research needs to be pursued.

What not to include in the limitations section

Evidence that doesn’t support your hypothesis is not a limitation, so you shouldn’t include it in the limitation section. Don’t just list limitations and their degree of severity without further explanation.

• How to present limitations

You’ll want to present the limitations of your study in a way that doesn’t diminish the validity of your research and leave the reader wondering if your results and conclusions have been compromised.

Include only the limitations that directly relate to and impact how you addressed your research questions. Following a specific format enables the reader to develop an understanding of the weaknesses within the context of your findings without doubting the quality and integrity of your research.

Identify the limitations specific to your study

You don’t have to identify every possible limitation that might have occurred during your research process. Only identify those that may have influenced the quality of your findings and your ability to answer your research question.

Explain study limitations in detail

This explanation should be the most significant portion of your limitation section.

Link each limitation with an interpretation and appraisal of their impact on the study. You’ll have to evaluate and explain whether the error, method, or validity issues influenced the study’s outcome and how.

Propose a direction for future studies and present alternatives

In this section, suggest how researchers can avoid the pitfalls you experienced during your research process.

If an issue with methodology was a limitation, propose alternate methods that may help with a smoother and more conclusive research project. Discuss the pros and cons of your alternate recommendation.

Describe steps taken to minimize each limitation

You probably took steps to try to address or mitigate limitations when you noticed them throughout the course of your research project. Describe these steps in the limitation section.

• Limitation example

“Approaches like stem cell transplantation and vaccination in AD [Alzheimer’s disease] work on a cellular or molecular level in the laboratory. However, translation into clinical settings will remain a challenge for the next decade.”

The authors are saying that even though these methods showed promise in helping people with memory loss when conducted in the lab (in other words, using animal studies), more studies are needed. These may be controlled clinical trials, for example. 

However, the short life span of stem cells outside the lab and the vaccination’s severe inflammatory side effects are limitations. Researchers won’t be able to conduct clinical trials until these issues are overcome.

• How to overcome limitations in research

You’ve already started on the road to overcoming limitations in research by acknowledging that they exist. However, you need to ensure readers don’t mistake weaknesses for errors within your research design.

To do this, you’ll need to justify and explain your rationale for the methods, research design, and analysis tools you chose and how you noticed they may have presented limitations.

Your readers need to know that even when limitations presented themselves, you followed best practices and the ethical standards of your field. You didn’t violate any rules and regulations during your research process.

You’ll also want to reinforce the validity of your conclusions and results with multiple sources, methods, and perspectives. This prevents readers from assuming your findings were derived from a single or biased source.

• Learning and improving starts with limitations in research

Dealing with limitations with transparency and integrity helps identify areas for future improvements and developments. It’s a learning process, providing valuable insights into how you can improve methodologies, expand sample sizes, or explore alternate approaches to further support the validity of your findings.

Get started today

Go from raw data to valuable insights with a flexible research platform

Editor’s picks

Last updated: 21 December 2023

Last updated: 16 December 2023

Last updated: 6 October 2023

Last updated: 5 March 2024

Last updated: 25 November 2023

Last updated: 15 February 2024

Last updated: 11 March 2024

Last updated: 12 December 2023

Last updated: 6 March 2024

Last updated: 10 April 2023

Last updated: 20 December 2023

Latest articles

Related topics, log in or sign up.

Get started for free

How to Write Limitations of the Study (with examples)

This blog emphasizes the importance of recognizing and effectively writing about limitations in research. It discusses the types of limitations, their significance, and provides guidelines for writing about them, highlighting their role in advancing scholarly research.

Updated on August 24, 2023

No matter how well thought out, every research endeavor encounters challenges. There is simply no way to predict all possible variances throughout the process.

These uncharted boundaries and abrupt constraints are known as limitations in research . Identifying and acknowledging limitations is crucial for conducting rigorous studies. Limitations provide context and shed light on gaps in the prevailing inquiry and literature.

This article explores the importance of recognizing limitations and discusses how to write them effectively. By interpreting limitations in research and considering prevalent examples, we aim to reframe the perception from shameful mistakes to respectable revelations.

What are limitations in research?

In the clearest terms, research limitations are the practical or theoretical shortcomings of a study that are often outside of the researcher’s control . While these weaknesses limit the generalizability of a study’s conclusions, they also present a foundation for future research.

Sometimes limitations arise from tangible circumstances like time and funding constraints, or equipment and participant availability. Other times the rationale is more obscure and buried within the research design. Common types of limitations and their ramifications include:

• Theoretical: limits the scope, depth, or applicability of a study.
• Methodological: limits the quality, quantity, or diversity of the data.
• Empirical: limits the representativeness, validity, or reliability of the data.
• Analytical: limits the accuracy, completeness, or significance of the findings.
• Ethical: limits the access, consent, or confidentiality of the data.

Regardless of how, when, or why they arise, limitations are a natural part of the research process and should never be ignored . Like all other aspects, they are vital in their own purpose.

Why is identifying limitations important?

Whether to seek acceptance or avoid struggle, humans often instinctively hide flaws and mistakes. Merging this thought process into research by attempting to hide limitations, however, is a bad idea. It has the potential to negate the validity of outcomes and damage the reputation of scholars.

By identifying and addressing limitations throughout a project, researchers strengthen their arguments and curtail the chance of peer censure based on overlooked mistakes. Pointing out these flaws shows an understanding of variable limits and a scrupulous research process.

Showing awareness of and taking responsibility for a project’s boundaries and challenges validates the integrity and transparency of a researcher. It further demonstrates the researchers understand the applicable literature and have thoroughly evaluated their chosen research methods.

Presenting limitations also benefits the readers by providing context for research findings. It guides them to interpret the project’s conclusions only within the scope of very specific conditions. By allowing for an appropriate generalization of the findings that is accurately confined by research boundaries and is not too broad, limitations boost a study’s credibility .

Limitations are true assets to the research process. They highlight opportunities for future research. When researchers identify the limitations of their particular approach to a study question, they enable precise transferability and improve chances for reproducibility. 

Simply stating a project’s limitations is not adequate for spurring further research, though. To spark the interest of other researchers, these acknowledgements must come with thorough explanations regarding how the limitations affected the current study and how they can potentially be overcome with amended methods.

How to write limitations

Typically, the information about a study’s limitations is situated either at the beginning of the discussion section to provide context for readers or at the conclusion of the discussion section to acknowledge the need for further research. However, it varies depending upon the target journal or publication guidelines. 

Don’t hide your limitations

It is also important to not bury a limitation in the body of the paper unless it has a unique connection to a topic in that section. If so, it needs to be reiterated with the other limitations or at the conclusion of the discussion section. Wherever it is included in the manuscript, ensure that the limitations section is prominently positioned and clearly introduced.

While maintaining transparency by disclosing limitations means taking a comprehensive approach, it is not necessary to discuss everything that could have potentially gone wrong during the research study. If there is no commitment to investigation in the introduction, it is unnecessary to consider the issue a limitation to the research. Wholly consider the term ‘limitations’ and ask, “Did it significantly change or limit the possible outcomes?” Then, qualify the occurrence as either a limitation to include in the current manuscript or as an idea to note for other projects. 

Writing limitations

Once the limitations are concretely identified and it is decided where they will be included in the paper, researchers are ready for the writing task. Including only what is pertinent, keeping explanations detailed but concise, and employing the following guidelines is key for crafting valuable limitations:

1) Identify and describe the limitations : Clearly introduce the limitation by classifying its form and specifying its origin. For example:

• An unintentional bias encountered during data collection
• An intentional use of unplanned post-hoc data analysis

2) Explain the implications : Describe how the limitation potentially influences the study’s findings and how the validity and generalizability are subsequently impacted. Provide examples and evidence to support claims of the limitations’ effects without making excuses or exaggerating their impact. Overall, be transparent and objective in presenting the limitations, without undermining the significance of the research. 

3) Provide alternative approaches for future studies : Offer specific suggestions for potential improvements or avenues for further investigation. Demonstrate a proactive approach by encouraging future research that addresses the identified gaps and, therefore, expands the knowledge base.

Whether presenting limitations as an individual section within the manuscript or as a subtopic in the discussion area, authors should use clear headings and straightforward language to facilitate readability. There is no need to complicate limitations with jargon, computations, or complex datasets.

Examples of common limitations

Limitations are generally grouped into two categories , methodology and research process .

Methodology limitations

Methodology may include limitations due to:

• Sample size
• Lack of available or reliable data
• Lack of prior research studies on the topic
• Measure used to collect the data
• Self-reported data

The researcher is addressing how the large sample size requires a reassessment of the measures used to collect and analyze the data.

Research process limitations

Limitations during the research process may arise from:

• Access to information
• Longitudinal effects
• Cultural and other biases
• Language fluency
• Time constraints

The author is pointing out that the model’s estimates are based on potentially biased observational studies.

Final thoughts

Successfully proving theories and touting great achievements are only two very narrow goals of scholarly research. The true passion and greatest efforts of researchers comes more in the form of confronting assumptions and exploring the obscure.

In many ways, recognizing and sharing the limitations of a research study both allows for and encourages this type of discovery that continuously pushes research forward. By using limitations to provide a transparent account of the project's boundaries and to contextualize the findings, researchers pave the way for even more robust and impactful research in the future.

Charla Viera, MS

See our "Privacy Policy"

Ensure your structure and ideas are consistent and clearly communicated

Pair your Premium Editing with our add-on service Presubmission Review for an overall assessment of your manuscript.

What are the limitations in research and how to write them?

Learn about the potential limitations in research and how to appropriately address them in order to deliver honest and ethical research.

It is fairly uncommon for researchers to stumble into the term research limitations when working on their research paper. Limitations in research can arise owing to constraints on design, methods, materials, and so on, and these aspects, unfortunately, may have an influence on your subject’s findings.

In this Mind The Graph’s article, we’ll discuss some recommendations for writing limitations in research , provide examples of various common types of limitations, and suggest how to properly present this information.

What are the limitations in research?

The limitations in research are the constraints in design, methods or even researchers’ limitations that affect and influence the interpretation of your research’s ultimate findings. These are limitations on the generalization and usability of findings that emerge from the design of the research and/or the method employed to ensure validity both internally and externally. 

Researchers are usually cautious to acknowledge the limitations of their research in their publications for fear of undermining the research’s scientific validity. No research is faultless or covers every possible angle. As a result, addressing the constraints of your research exhibits honesty and integrity .

Why should include limitations of research in my paper?

Though limitations tackle potential flaws in research, commenting on them at the conclusion of your paper, by demonstrating that you are aware of these limitations and explaining how they impact the conclusions that may be taken from the research, improves your research by disclosing any issues before other researchers or reviewers do . 

Additionally, emphasizing research constraints implies that you have thoroughly investigated the ramifications of research shortcomings and have a thorough understanding of your research problem. 

Limits exist in any research; being honest about them and explaining them would impress researchers and reviewers more than disregarding them. 

Remember that acknowledging a research’s shortcomings offers a chance to provide ideas for future research, but be careful to describe how your study may help to concentrate on these outstanding problems.

Possible limitations examples

Here are some limitations connected to methodology and the research procedure that you may need to explain and discuss in connection to your findings.

Methodological limitations

Sample size.

The number of units of analysis used in your study is determined by the sort of research issue being investigated. It is important to note that if your sample is too small, finding significant connections in the data will be challenging, as statistical tests typically require a larger sample size to ensure a fair representation and this can be limiting. 

Lack of available or reliable data

A lack of data or trustworthy data will almost certainly necessitate limiting the scope of your research or the size of your sample, or it can be a substantial impediment to identifying a pattern and a relevant connection.

Lack of prior research on the subject

Citing previous research papers forms the basis of your literature review and aids in comprehending the research subject you are researching. Yet there may be little if any, past research on your issue.

The measure used to collect data

After finishing your analysis of the findings, you realize that the method you used to collect data limited your capacity to undertake a comprehensive evaluation of the findings. Recognize the flaw by mentioning that future researchers should change the specific approach for data collection.

Issues with research samples and selection

Sampling inaccuracies arise when a probability sampling method is employed to choose a sample, but that sample does not accurately represent the overall population or the relevant group. As a result, your study suffers from “sampling bias” or “selection bias.”

Limitations of the research

When your research requires polling certain persons or a specific group, you may have encountered the issue of limited access to these interviewees. Because of the limited access, you may need to reorganize or rearrange your research. In this scenario, explain why access is restricted and ensure that your findings are still trustworthy and valid despite the constraint.

Time constraints

Practical difficulties may limit the amount of time available to explore a research issue and monitor changes as they occur. If time restrictions have any detrimental influence on your research, recognize this impact by expressing the necessity for a future investigation.

Due to their cultural origins or opinions on observed events, researchers may carry biased opinions, which can influence the credibility of a research. Furthermore, researchers may exhibit biases toward data and conclusions that only support their hypotheses or arguments.

The structure of the limitations section 

The limitations of your research are usually stated at the beginning of the discussion section of your paper so that the reader is aware of and comprehends the limitations prior to actually reading the rest of your findings, or they are stated at the end of the discussion section as an acknowledgment of the need for further research.

The ideal way is to divide your limitations section into three steps: 

1. Identify the research constraints; 

2. Describe in great detail how they affect your research; 

3. Mention the opportunity for future investigations and give possibilities. 

By following this method while addressing the constraints of your research, you will be able to effectively highlight your research’s shortcomings without jeopardizing the quality and integrity of your research.

Present your research or paper in an innovative way

If you want your readers to be engaged and participate in your research, try Mind The Graph tool to add visual assets to your content. Infographics may improve comprehension and are easy to read, just as the Mind The Graph tool is simple to use and offers a variety of templates from which you can select the one that best suits your information.

Subscribe to our newsletter

Exclusive high quality content about effective visual communication in science.

About Jessica Abbadia

Jessica Abbadia is a lawyer that has been working in Digital Marketing since 2020, improving organic performance for apps and websites in various regions through ASO and SEO. Currently developing scientific and intellectual knowledge for the community's benefit. Jessica is an animal rights activist who enjoys reading and drinking strong coffee.

Content tags

Research Limitations & Delimitations

What they are and how they’re different (with examples)

By: Derek Jansen (MBA) | Expert Reviewed By: David Phair (PhD) | September 2022

If you’re new to the world of research, you’ve probably heard the terms “ research limitations ” and “ research delimitations ” being thrown around, often quite loosely. In this post, we’ll unpack what both of these mean, how they’re similar and how they’re different – so that you can write up these sections the right way.

Overview: Limitations vs Delimitations

• Are they the same?
• What are research limitations
• What are research delimitations
• Limitations vs delimitations

First things first…

Let’s start with the most important takeaway point of this post – research limitations and research delimitations are not the same – but they are related to each other (we’ll unpack that a little later). So, if you hear someone using these two words interchangeably, be sure to share this post with them!

Research Limitations

Research limitations are, at the simplest level, the weaknesses of the study, based on factors that are often outside of your control as the researcher. These factors could include things like time , access to funding, equipment , data or participants . For example, if you weren’t able to access a random sample of participants for your study and had to adopt a convenience sampling strategy instead, that would impact the generalizability of your findings and therefore reflect a limitation of your study.

Research limitations can also emerge from the research design itself . For example, if you were undertaking a correlational study, you wouldn’t be able to infer causality (since correlation doesn’t mean certain causation). Similarly, if you utilised online surveys to collect data from your participants, you naturally wouldn’t be able to get the same degree of rich data that you would from in-person interviews .

Simply put, research limitations reflect the shortcomings of a study , based on practical (or theoretical) constraints that the researcher faced. These shortcomings limit what you can conclude from a study, but at the same time, present a foundation for future research . Importantly, all research has limitations , so there’s no need to hide anything here – as long as you discuss how the limitations might affect your findings, it’s all good.

Research Delimitations

Alright, now that we’ve unpacked the limitations, let’s move on to the delimitations .

Research delimitations are similar to limitations in that they also “ limit ” the study, but their focus is entirely different. Specifically, the delimitations of a study refer to the scope of the research aims and research questions . In other words, delimitations reflect the choices you, as the researcher, intentionally make in terms of what you will and won’t try to achieve with your study. In other words, what your research aims and research questions will and won’t include.

As we’ve spoken about many times before, it’s important to have a tight, narrow focus for your research, so that you can dive deeply into your topic, apply your energy to one specific area and develop meaningful insights. If you have an overly broad scope or unfocused topic, your research will often pull in multiple, even opposing directions, and you’ll just land up with a muddy mess of findings .

So, the delimitations section is where you’ll clearly state what your research aims and research questions will focus on – and just as importantly, what they will exclude . For example, you might investigate a widespread phenomenon, but choose to focus your study on a specific age group, ethnicity or gender. Similarly, your study may focus exclusively on one country, city or even organization. As long as the scope is well justified (in other words, it represents a novel, valuable research topic), this is perfectly acceptable – in fact, it’s essential. Remember, focus is your friend.

Need a helping hand?

Conclusion: Limitations vs Delimitations

Ok, so let’s recap.

Research limitations and research delimitations are related in that they both refer to “limits” within a study. But, they are distinctly different. Limitations reflect the shortcomings of your study, based on practical or theoretical constraints that you faced.

Contrasted to that, delimitations reflect the choices that you made in terms of the focus and scope of your research aims and research questions. If you want to learn more about research aims and questions, you can check out this video post , where we unpack those concepts in detail.

Psst… there’s more (for free)

This post is part of our dissertation mini-course, which covers everything you need to get started with your dissertation, thesis or research project. 

You Might Also Like:

17 Comments

Good clarification of ideas on how a researcher ought to do during Process of choice

Thank you so much for this very simple but explicit explanation on limitation and delimitation. It has so helped me to develop my masters proposal. hope to recieve more from your site as time progresses

Thank you for this explanation – very clear.

Thanks for the explanation, really got it well.

This website is really helpful for my masters proposal

Thank you very much for helping to explain these two terms

I spent almost the whole day trying to figure out the differences

when I came across your notes everything became very clear

thanks for the clearly outlined explanation on the two terms, limitation and delimitation.

Very helpful Many thanks 🙏

Excellent it resolved my conflict .

I would like you to assist me please. If in my Research, I interviewed some participants and I submitted Questionnaires to other participants to answered to the questions, in the same organization, Is this a Qualitative methodology , a Quantitative Methodology or is it a Mixture Methodology I have used in my research? Please help me

How do I cite this article in APA format

Really so great ,finally have understood it’s difference now

Getting more clear regarding Limitations and Delimitation and concepts

I really appreciate your apt and precise explanation of the two concepts namely ; Limitations and Delimitations.

thank you for this, very helpful to researchers

Very good explained

Great and clear explanation, after a long confusion period on the two words, i can now explain to someone with ease.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Writing Limitations of Research Study — 4 Reasons Why It Is Important!

It is not unusual for researchers to come across the term limitations of research during their academic paper writing. More often this is interpreted as something terrible. However, when it comes to research study, limitations can help structure the research study better. Therefore, do not underestimate significance of limitations of research study.

Allow us to take you through the context of how to evaluate the limits of your research and conclude an impactful relevance to your results.

Table of Contents

What Are the Limitations of a Research Study?

Every research has its limit and these limitations arise due to restrictions in methodology or research design.  This could impact your entire research or the research paper you wish to publish. Unfortunately, most researchers choose not to discuss their limitations of research fearing it will affect the value of their article in the eyes of readers.

However, it is very important to discuss your study limitations and show it to your target audience (other researchers, journal editors, peer reviewers etc.). It is very important that you provide an explanation of how your research limitations may affect the conclusions and opinions drawn from your research. Moreover, when as an author you state the limitations of research, it shows that you have investigated all the weaknesses of your study and have a deep understanding of the subject. Being honest could impress your readers and mark your study as a sincere effort in research.

Why and Where Should You Include the Research Limitations?

The main goal of your research is to address your research objectives. Conduct experiments, get results and explain those results, and finally justify your research question . It is best to mention the limitations of research in the discussion paragraph of your research article.

At the very beginning of this paragraph, immediately after highlighting the strengths of the research methodology, you should write down your limitations. You can discuss specific points from your research limitations as suggestions for further research in the conclusion of your thesis.

1. Common Limitations of the Researchers

Limitations that are related to the researcher must be mentioned. This will help you gain transparency with your readers. Furthermore, you could provide suggestions on decreasing these limitations in you and your future studies.

2. Limited Access to Information

Your work may involve some institutions and individuals in research, and sometimes you may have problems accessing these institutions. Therefore, you need to redesign and rewrite your work. You must explain your readers the reason for limited access.

3. Limited Time

All researchers are bound by their deadlines when it comes to completing their studies. Sometimes, time constraints can affect your research negatively. However, the best practice is to acknowledge it and mention a requirement for future study to solve the research problem in a better way.

4. Conflict over Biased Views and Personal Issues

Biased views can affect the research. In fact, researchers end up choosing only those results and data that support their main argument, keeping aside the other loose ends of the research.

Types of Limitations of Research

Before beginning your research study, know that there are certain limitations to what you are testing or possible research results. There are different types that researchers may encounter, and they all have unique characteristics, such as:

1. Research Design Limitations

Certain restrictions on your research or available procedures may affect your final results or research outputs. You may have formulated research goals and objectives too broadly. However, this can help you understand how you can narrow down the formulation of research goals and objectives, thereby increasing the focus of your study.

2. Impact Limitations

Even if your research has excellent statistics and a strong design, it can suffer from the influence of the following factors:

• Presence of increasing findings as researched
• Being population specific
• A strong regional focus.

3. Data or statistical limitations

In some cases, it is impossible to collect sufficient data for research or very difficult to get access to the data. This could lead to incomplete conclusion to your study. Moreover, this insufficiency in data could be the outcome of your study design. The unclear, shabby research outline could produce more problems in interpreting your findings.

How to Correctly Structure Your Research Limitations?

There are strict guidelines for narrowing down research questions, wherein you could justify and explain potential weaknesses of your academic paper. You could go through these basic steps to get a well-structured clarity of research limitations:

• Declare that you wish to identify your limitations of research and explain their importance,
• Provide the necessary depth, explain their nature, and justify your study choices.
• Write how you are suggesting that it is possible to overcome them in the future.

In this section, your readers will see that you are aware of the potential weaknesses in your business, understand them and offer effective solutions, and it will positively strengthen your article as you clarify all limitations of research to your target audience.

Know that you cannot be perfect and there is no individual without flaws. You could use the limitations of research as a great opportunity to take on a new challenge and improve the future of research. In a typical academic paper, research limitations may relate to:

1. Formulating your goals and objectives

If you formulate goals and objectives too broadly, your work will have some shortcomings. In this case, specify effective methods or ways to narrow down the formula of goals and aim to increase your level of study focus.

2. Application of your data collection methods in research

If you do not have experience in primary data collection, there is a risk that there will be flaws in the implementation of your methods. It is necessary to accept this, and learn and educate yourself to understand data collection methods.

3. Sample sizes

This depends on the nature of problem you choose. Sample size is of a greater importance in quantitative studies as opposed to qualitative ones. If your sample size is too small, statistical tests cannot identify significant relationships or connections within a given data set.

You could point out that other researchers should base the same study on a larger sample size to get more accurate results.

4. The absence of previous studies in the field you have chosen

Writing a literature review is an important step in any scientific study because it helps researchers determine the scope of current work in the chosen field. It is a major foundation for any researcher who must use them to achieve a set of specific goals or objectives.

However, if you are focused on the most current and evolving research problem or a very narrow research problem, there may be very little prior research on your topic. For example, if you chose to explore the role of Bitcoin as the currency of the future, you may not find tons of scientific papers addressing the research problem as Bitcoins are only a new phenomenon.

It is important that you learn to identify research limitations examples at each step. Whatever field you choose, feel free to add the shortcoming of your work. This is mainly because you do not have many years of experience writing scientific papers or completing complex work. Therefore, the depth and scope of your discussions may be compromised at different levels compared to academics with a lot of expertise. Include specific points from limitations of research. Use them as suggestions for the future.

Have you ever faced a challenge of writing the limitations of research study in your paper? How did you overcome it? What ways did you follow? Were they beneficial? Let us know in the comments below!

Frequently Asked Questions

Setting limitations in our study helps to clarify the outcomes drawn from our research and enhance understanding of the subject. Moreover, it shows that the author has investigated all the weaknesses in the study.

Scope is the range and limitations of a research project which are set to define the boundaries of a project. Limitations are the impacts on the overall study due to the constraints on the research design.

Limitation in research is an impact of a constraint on the research design in the overall study. They are the flaws or weaknesses in the study, which may influence the outcome of the research.

1. Limitations in research can be written as follows: Formulate your goals and objectives 2. Analyze the chosen data collection method and the sample sizes 3. Identify your limitations of research and explain their importance 4. Provide the necessary depth, explain their nature, and justify your study choices 5. Write how you are suggesting that it is possible to overcome them in the future

Excellent article ,,,it has helped me big

This is very helpful information. It has given me an insight on how to go about my study limitations.

Good comments and helpful

Rate this article Cancel Reply

Your email address will not be published.

Enago Academy's Most Popular Articles

• Diversity and Inclusion
• Trending Now

The Silent Struggle: Confronting gender bias in science funding

In the 1990s, Dr. Katalin Kariko’s pioneering mRNA research seemed destined for obscurity, doomed by…

• Promoting Research

Plain Language Summary — Communicating your research to bridge the academic-lay gap

Science can be complex, but does that mean it should not be accessible to the…

Addressing Barriers in Academia: Navigating unconscious biases in the Ph.D. journey

In the journey of academia, a Ph.D. marks a transitional phase, like that of a…

• Manuscripts & Grants
• Reporting Research

Unraveling Research Population and Sample: Understanding their role in statistical inference

Research population and sample serve as the cornerstones of any scientific inquiry. They hold the…

• Manuscript Preparation
• Publishing Research

Research Problem Statement — Find out how to write an impactful one!

What Is a Research Problem Statement? A research problem statement is a clear, concise, and…

How to Develop a Good Research Question? — Types & Examples

5 Effective Ways to Avoid Ghostwriting for Busy Researchers

Top 5 Key Differences Between Methods and Methodology

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

• 2000+ blog articles
• 50+ Webinars
• 10+ Expert podcasts
• 50+ Infographics
• 10+ Checklists
• Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

What should universities' stance be on AI tools in research and academic writing?

• Affiliate Program

• UNITED STATES
• 台灣 (TAIWAN)
• TÜRKIYE (TURKEY)
• Academic Editing Services
• - Research Paper
• - Journal Manuscript
• - Dissertation
• - College & University Assignments
• Admissions Editing Services
• - Application Essay
• - Personal Statement
• - Recommendation Letter
• - Cover Letter
• - CV/Resume
• Business Editing Services
• - Business Documents
• - Report & Brochure
• - Website & Blog
• Writer Editing Services
• - Script & Screenplay
• Our Editors
• Client Reviews
• Editing & Proofreading Prices
• Wordvice Points
• Partner Discount
• Plagiarism Checker

APA Citation Generator

MLA Citation Generator

Chicago Citation Generator

Vancouver Citation Generator

• - APA Style
• - MLA Style
• - Chicago Style
• - Vancouver Style
• Writing & Editing Guide
• Academic Resources
• Admissions Resources

How to Present the Limitations of the Study Examples

What are the limitations of a study?

The limitations of a study are the elements of methodology or study design that impact the interpretation of your research results. The limitations essentially detail any flaws or shortcomings in your study. Study limitations can exist due to constraints on research design, methodology, materials, etc., and these factors may impact the findings of your study. However, researchers are often reluctant to discuss the limitations of their study in their papers, feeling that bringing up limitations may undermine its research value in the eyes of readers and reviewers.

In spite of the impact it might have (and perhaps because of it) you should clearly acknowledge any limitations in your research paper in order to show readers—whether journal editors, other researchers, or the general public—that you are aware of these limitations and to explain how they affect the conclusions that can be drawn from the research.

In this article, we provide some guidelines for writing about research limitations, show examples of some frequently seen study limitations, and recommend techniques for presenting this information. And after you have finished drafting and have received manuscript editing for your work, you still might want to follow this up with academic editing before submitting your work to your target journal.

Why do I need to include limitations of research in my paper?

Although limitations address the potential weaknesses of a study, writing about them toward the end of your paper actually strengthens your study by identifying any problems before other researchers or reviewers find them.

Furthermore, pointing out study limitations shows that you’ve considered the impact of research weakness thoroughly and have an in-depth understanding of your research topic. Since all studies face limitations, being honest and detailing these limitations will impress researchers and reviewers more than ignoring them.

Where should I put the limitations of the study in my paper?

Some limitations might be evident to researchers before the start of the study, while others might become clear while you are conducting the research. Whether these limitations are anticipated or not, and whether they are due to research design or to methodology, they should be clearly identified and discussed in the discussion section —the final section of your paper. Most journals now require you to include a discussion of potential limitations of your work, and many journals now ask you to place this “limitations section” at the very end of your article. 

Some journals ask you to also discuss the strengths of your work in this section, and some allow you to freely choose where to include that information in your discussion section—make sure to always check the author instructions of your target journal before you finalize a manuscript and submit it for peer review .

Limitations of the Study Examples

There are several reasons why limitations of research might exist. The two main categories of limitations are those that result from the methodology and those that result from issues with the researcher(s).

Common Methodological Limitations of Studies

Limitations of research due to methodological problems can be addressed by clearly and directly identifying the potential problem and suggesting ways in which this could have been addressed—and SHOULD be addressed in future studies. The following are some major potential methodological issues that can impact the conclusions researchers can draw from the research.

Issues with research samples and selection

Sampling errors occur when a probability sampling method is used to select a sample, but that sample does not reflect the general population or appropriate population concerned. This results in limitations of your study known as “sample bias” or “selection bias.”

For example, if you conducted a survey to obtain your research results, your samples (participants) were asked to respond to the survey questions. However, you might have had limited ability to gain access to the appropriate type or geographic scope of participants. In this case, the people who responded to your survey questions may not truly be a random sample.

Insufficient sample size for statistical measurements

When conducting a study, it is important to have a sufficient sample size in order to draw valid conclusions. The larger the sample, the more precise your results will be. If your sample size is too small, it will be difficult to identify significant relationships in the data.

Normally, statistical tests require a larger sample size to ensure that the sample is considered representative of a population and that the statistical result can be generalized to a larger population. It is a good idea to understand how to choose an appropriate sample size before you conduct your research by using scientific calculation tools—in fact, many journals now require such estimation to be included in every manuscript that is sent out for review.

Lack of previous research studies on the topic

Citing and referencing prior research studies constitutes the basis of the literature review for your thesis or study, and these prior studies provide the theoretical foundations for the research question you are investigating. However, depending on the scope of your research topic, prior research studies that are relevant to your thesis might be limited.

When there is very little or no prior research on a specific topic, you may need to develop an entirely new research typology. In this case, discovering a limitation can be considered an important opportunity to identify literature gaps and to present the need for further development in the area of study.

Methods/instruments/techniques used to collect the data

After you complete your analysis of the research findings (in the discussion section), you might realize that the manner in which you have collected the data or the ways in which you have measured variables has limited your ability to conduct a thorough analysis of the results.

For example, you might realize that you should have addressed your survey questions from another viable perspective, or that you were not able to include an important question in the survey. In these cases, you should acknowledge the deficiency or deficiencies by stating a need for future researchers to revise their specific methods for collecting data that includes these missing elements.

Common Limitations of the Researcher(s)

Study limitations that arise from situations relating to the researcher or researchers (whether the direct fault of the individuals or not) should also be addressed and dealt with, and remedies to decrease these limitations—both hypothetically in your study, and practically in future studies—should be proposed.

Limited access to data

If your research involved surveying certain people or organizations, you might have faced the problem of having limited access to these respondents. Due to this limited access, you might need to redesign or restructure your research in a different way. In this case, explain the reasons for limited access and be sure that your finding is still reliable and valid despite this limitation.

Time constraints

Just as students have deadlines to turn in their class papers, academic researchers might also have to meet deadlines for submitting a manuscript to a journal or face other time constraints related to their research (e.g., participants are only available during a certain period; funding runs out; collaborators move to a new institution). The time available to study a research problem and to measure change over time might be constrained by such practical issues. If time constraints negatively impacted your study in any way, acknowledge this impact by mentioning a need for a future study (e.g., a longitudinal study) to answer this research problem.

Conflicts arising from cultural bias and other personal issues

Researchers might hold biased views due to their cultural backgrounds or perspectives of certain phenomena, and this can affect a study’s legitimacy. Also, it is possible that researchers will have biases toward data and results that only support their hypotheses or arguments. In order to avoid these problems, the author(s) of a study should examine whether the way the research problem was stated and the data-gathering process was carried out appropriately.

Steps for Organizing Your Study Limitations Section

When you discuss the limitations of your study, don’t simply list and describe your limitations—explain how these limitations have influenced your research findings. There might be multiple limitations in your study, but you only need to point out and explain those that directly relate to and impact how you address your research questions.

We suggest that you divide your limitations section into three steps: (1) identify the study limitations; (2) explain how they impact your study in detail; and (3) propose a direction for future studies and present alternatives. By following this sequence when discussing your study’s limitations, you will be able to clearly demonstrate your study’s weakness without undermining the quality and integrity of your research.

Step 1. Identify the limitation(s) of the study

• This part should comprise around 10%-20% of your discussion of study limitations.

The first step is to identify the particular limitation(s) that affected your study. There are many possible limitations of research that can affect your study, but you don’t need to write a long review of all possible study limitations. A 200-500 word critique is an appropriate length for a research limitations section. In the beginning of this section, identify what limitations your study has faced and how important these limitations are.

You only need to identify limitations that had the greatest potential impact on: (1) the quality of your findings, and (2) your ability to answer your research question.

Step 2. Explain these study limitations in detail

• This part should comprise around 60-70% of your discussion of limitations.

After identifying your research limitations, it’s time to explain the nature of the limitations and how they potentially impacted your study. For example, when you conduct quantitative research, a lack of probability sampling is an important issue that you should mention. On the other hand, when you conduct qualitative research, the inability to generalize the research findings could be an issue that deserves mention.

Explain the role these limitations played on the results and implications of the research and justify the choice you made in using this “limiting” methodology or other action in your research. Also, make sure that these limitations didn’t undermine the quality of your dissertation .

Step 3. Propose a direction for future studies and present alternatives (optional)

• This part should comprise around 10-20% of your discussion of limitations.

After acknowledging the limitations of the research, you need to discuss some possible ways to overcome these limitations in future studies. One way to do this is to present alternative methodologies and ways to avoid issues with, or “fill in the gaps of” the limitations of this study you have presented.  Discuss both the pros and cons of these alternatives and clearly explain why researchers should choose these approaches.

Make sure you are current on approaches used by prior studies and the impacts they have had on their findings. Cite review articles or scientific bodies that have recommended these approaches and why. This might be evidence in support of the approach you chose, or it might be the reason you consider your choices to be included as limitations. This process can act as a justification for your approach and a defense of your decision to take it while acknowledging the feasibility of other approaches.

P hrases and Tips for Introducing Your Study Limitations in the Discussion Section

The following phrases are frequently used to introduce the limitations of the study:

• “There may be some possible limitations in this study.”
• “The findings of this study have to be seen in light of some limitations.”
•  “The first is the…The second limitation concerns the…”
•  “The empirical results reported herein should be considered in the light of some limitations.”
• “This research, however, is subject to several limitations.”
• “The primary limitation to the generalization of these results is…”
• “Nonetheless, these results must be interpreted with caution and a number of limitations should be borne in mind.”
• “As with the majority of studies, the design of the current study is subject to limitations.”
• “There are two major limitations in this study that could be addressed in future research. First, the study focused on …. Second ….”

For more articles on research writing and the journal submissions and publication process, visit Wordvice’s Academic Resources page.

And be sure to receive professional English editing and proofreading services , including paper editing services , for your journal manuscript before submitting it to journal editors.

Wordvice Resources

Proofreading & Editing Guide

Writing the Results Section for a Research Paper

How to Write a Literature Review

Research Writing Tips: How to Draft a Powerful Discussion Section

How to Captivate Journal Readers with a Strong Introduction

Tips That Will Make Your Abstract a Success!

APA In-Text Citation Guide for Research Writing

Additional Resources

• Diving Deeper into Limitations and Delimitations (PhD student)
• Organizing Your Social Sciences Research Paper: Limitations of the Study (USC Library)
• Research Limitations (Research Methodology)
• How to Present Limitations and Alternatives (UMASS)

Article References

Pearson-Stuttard, J., Kypridemos, C., Collins, B., Mozaffarian, D., Huang, Y., Bandosz, P.,…Micha, R. (2018). Estimating the health and economic effects of the proposed US Food and Drug Administration voluntary sodium reformulation: Microsimulation cost-effectiveness analysis. PLOS. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002551

Xu, W.L, Pedersen, N.L., Keller, L., Kalpouzos, G., Wang, H.X., Graff, C,. Fratiglioni, L. (2015). HHEX_23 AA Genotype Exacerbates Effect of Diabetes on Dementia and Alzheimer Disease: A Population-Based Longitudinal Study. PLOS. Retrieved from https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001853

CPS Online Graduate Studies Research Paper (UNH Manchester Library): Limitations of the Study

• Overview of the Research Process for Capstone Projects
• Types of Research Design
• Selecting a Research Problem
• The Title of Your Research Paper
• Before You Begin Writing
• 7 Parts of the Research Paper
• Background Information
• Quanitative and Qualitative Methods
• Qualitative Methods
• Quanitative Methods
• Resources to Help You With the Literature Review
• Non-Textual Elements

Limitations of the Study

• Format of Capstone Research Projects at GSC
• Editing and Proofreading Your Paper
• Acknowledgements
• UNH Scholar's Repository

The limitations of the study are those characteristics of design or methodology that impacted or influenced the interpretation of the findings from your research. They are the constraints on generalizability, applications to practice, and/or utility of findings that are the result of the ways in which you initially chose to design the study and/or the method used to establish internal and external validity.

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67.

Always acknowledge a study's limitations. It is far better that you identify and acknowledge your study’s limitations than to have them pointed out by your professor and be graded down because you appear to have ignored them.

Keep in mind that acknowledgement of a study's limitations is an opportunity to make suggestions for further research. If you do connect your study's limitations to suggestions for further research, be sure to explain the ways in which these unanswered questions may become more focused because of your study.

Acknowledgement of a study's limitations also provides you with an opportunity to demonstrate that you have thought critically about the research problem, understood the relevant literature published about it, and correctly assessed the methods chosen for studying the problem. A key objective of the research process is not only discovering new knowledge but to also confront assumptions and explore what we don't know.

Claiming limitations is a subjective process because you must evaluate the impact of those limitations . Don't just list key weaknesses and the magnitude of a study's limitations. To do so diminishes the validity of your research because it leaves the reader wondering whether, or in what ways, limitation(s) in your study may have impacted the results and conclusions. Limitations require a critical, overall appraisal and interpretation of their impact. You should answer the question: do these problems with errors, methods, validity, etc. eventually matter and, if so, to what extent?

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Structure: How to Structure the Research Limitations Section of Your Dissertation . Dissertations and Theses: An Online Textbook. Laerd.com.

Descriptions of Possible Limitations

All studies have limitations . However, it is important that you restrict your discussion to limitations related to the research problem under investigation. For example, if a meta-analysis of existing literature is not a stated purpose of your research, it should not be discussed as a limitation. Do not apologize for not addressing issues that you did not promise to investigate in the introduction of your paper.

Here are examples of limitations related to methodology and the research process you may need to describe and to discuss how they possibly impacted your results. Descriptions of limitations should be stated in the past tense because they were discovered after you completed your research.

Possible Methodological Limitations

• Sample size -- the number of the units of analysis you use in your study is dictated by the type of research problem you are investigating. Note that, if your sample size is too small, it will be difficult to find significant relationships from the data, as statistical tests normally require a larger sample size to ensure a representative distribution of the population and to be considered representative of groups of people to whom results will be generalized or transferred. Note that sample size is less relevant in qualitative research.
• Lack of available and/or reliable data -- a lack of data or of reliable data will likely require you to limit the scope of your analysis, the size of your sample, or it can be a significant obstacle in finding a trend and a meaningful relationship. You need to not only describe these limitations but to offer reasons why you believe data is missing or is unreliable. However, don’t just throw up your hands in frustration; use this as an opportunity to describe the need for future research.
• Lack of prior research studies on the topic -- citing prior research studies forms the basis of your literature review and helps lay a foundation for understanding the research problem you are investigating. Depending on the currency or scope of your research topic, there may be little, if any, prior research on your topic. Before assuming this to be true, though, consult with a librarian. In cases when a librarian has confirmed that there is no prior research, you may be required to develop an entirely new research typology [for example, using an exploratory rather than an explanatory research design]. Note again that discovering a limitation can serve as an important opportunity to identify new gaps in the literature and to describe the need for further research.
• Measure used to collect the data -- sometimes it is the case that, after completing your interpretation of the findings, you discover that the way in which you gathered data inhibited your ability to conduct a thorough analysis of the results. For example, you regret not including a specific question in a survey that, in retrospect, could have helped address a particular issue that emerged later in the study. Acknowledge the deficiency by stating a need for future researchers to revise the specific method for gathering data.
• Self-reported data -- whether you are relying on pre-existing data or you are conducting a qualitative research study and gathering the data yourself, self-reported data is limited by the fact that it rarely can be independently verified. In other words, you have to take what people say, whether in interviews, focus groups, or on questionnaires, at face value. However, self-reported data can contain several potential sources of bias that you should be alert to and note as limitations. These biases become apparent if they are incongruent with data from other sources. These are: (1) selective memory [remembering or not remembering experiences or events that occurred at some point in the past]; (2) telescoping [recalling events that occurred at one time as if they occurred at another time]; (3) attribution [the act of attributing positive events and outcomes to one's own agency but attributing negative events and outcomes to external forces]; and, (4) exaggeration [the act of representing outcomes or embellishing events as more significant than is actually suggested from other data].

Possible Limitations of the Researcher

• Access -- if your study depends on having access to people, organizations, or documents and, for whatever reason, access is denied or limited in some way, the reasons for this need to be described.
• Longitudinal effects -- unlike your professor, who can literally devote years [even a lifetime] to studying a single topic, the time available to investigate a research problem and to measure change or stability over time is pretty much constrained by the due date of your assignment. Be sure to choose a research problem that does not require an excessive amount of time to complete the literature review, apply the methodology, and gather and interpret the results. If you're unsure whether you can complete your research within the confines of the assignment's due date, talk to your professor.
• Cultural and other type of bias -- we all have biases, whether we are conscience of them or not. Bias is when a person, place, or thing is viewed or shown in a consistently inaccurate way. Bias is usually negative, though one can have a positive bias as well, especially if that bias reflects your reliance on research that only support for your hypothesis. When proof-reading your paper, be especially critical in reviewing how you have stated a problem, selected the data to be studied, what may have been omitted, the manner in which you have ordered events, people, or places, how you have chosen to represent a person, place, or thing, to name a phenomenon, or to use possible words with a positive or negative connotation.

NOTE:   If you detect bias in prior research, it must be acknowledged and you should explain what measures were taken to avoid perpetuating that bias.

• Fluency in a language -- if your research focuses on measuring the perceived value of after-school tutoring among Mexican-American ESL [English as a Second Language] students, for example, and you are not fluent in Spanish, you are limited in being able to read and interpret Spanish language research studies on the topic. This deficiency should be acknowledged.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Senunyeme, Emmanuel K. Business Research Methods . Powerpoint Presentation. Regent University of Science and Technology; ter Riet, Gerben et al. “All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies.” PLOS One 8 (November 2013): 1-6.

Structure and Writing Style

Information about the limitations of your study are generally placed either at the beginning of the discussion section of your paper so the reader knows and understands the limitations before reading the rest of your analysis of the findings, or, the limitations are outlined at the conclusion of the discussion section as an acknowledgement of the need for further study. Statements about a study's limitations should not be buried in the body [middle] of the discussion section unless a limitation is specific to something covered in that part of the paper. If this is the case, though, the limitation should be reiterated at the conclusion of the section. If you determine that your study is seriously flawed due to important limitations, such as, an inability to acquire critical data, consider reframing it as an exploratory study intended to lay the groundwork for a more complete research study in the future. Be sure, though, to specifically explain the ways that these flaws can be successfully overcome in a new study. But, do not use this as an excuse for not developing a thorough research paper! Review the tab in this guide for developing a research topic. If serious limitations exist, it generally indicates a likelihood that your research problem is too narrowly defined or that the issue or event under study is too recent and, thus, very little research has been written about it. If serious limitations do emerge, consult with your professor about possible ways to overcome them or how to revise your study. When discussing the limitations of your research, be sure to: Describe each limitation in detailed but concise terms; Explain why each limitation exists; Provide the reasons why each limitation could not be overcome using the method(s) chosen to acquire or gather the data [cite to other studies that had similar problems when possible]; Assess the impact of each limitation in relation to the overall findings and conclusions of your study; and, If appropriate, describe how these limitations could point to the need for further research. Remember that the method you chose may be the source of a significant limitation that has emerged during your interpretation of the results [for example, you didn't interview a group of people that you later wish you had]. If this is the case, don't panic. Acknowledge it, and explain how applying a different or more robust methodology might address the research problem more effectively in a future study. A underlying goal of scholarly research is not only to show what works, but to demonstrate what doesn't work or what needs further clarification. Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Ioannidis, John P.A. "Limitations are not Properly Acknowledged in the Scientific Literature." Journal of Clinical Epidemiology 60 (2007): 324-329; Pasek, Josh. Writing the Empirical Social Science Research Paper: A Guide for the Perplexed. January 24, 2012. Academia.edu; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com; What Is an Academic Paper? Institute for Writing Rhetoric. Dartmouth College; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University.

Information about the limitations of your study are generally placed either at the beginning of the discussion section of your paper so the reader knows and understands the limitations before reading the rest of your analysis of the findings, or, the limitations are outlined at the conclusion of the discussion section as an acknowledgement of the need for further study. Statements about a study's limitations should not be buried in the body [middle] of the discussion section unless a limitation is specific to something covered in that part of the paper. If this is the case, though, the limitation should be reiterated at the conclusion of the section.

If you determine that your study is seriously flawed due to important limitations , such as, an inability to acquire critical data, consider reframing it as an exploratory study intended to lay the groundwork for a more complete research study in the future. Be sure, though, to specifically explain the ways that these flaws can be successfully overcome in a new study.

But, do not use this as an excuse for not developing a thorough research paper! Review the tab in this guide for developing a research topic . If serious limitations exist, it generally indicates a likelihood that your research problem is too narrowly defined or that the issue or event under study is too recent and, thus, very little research has been written about it. If serious limitations do emerge, consult with your professor about possible ways to overcome them or how to revise your study.

When discussing the limitations of your research, be sure to:

• Describe each limitation in detailed but concise terms;
• Explain why each limitation exists;
• Provide the reasons why each limitation could not be overcome using the method(s) chosen to acquire or gather the data [cite to other studies that had similar problems when possible];
• Assess the impact of each limitation in relation to the overall findings and conclusions of your study; and,
• If appropriate, describe how these limitations could point to the need for further research.

Remember that the method you chose may be the source of a significant limitation that has emerged during your interpretation of the results [for example, you didn't interview a group of people that you later wish you had]. If this is the case, don't panic. Acknowledge it, and explain how applying a different or more robust methodology might address the research problem more effectively in a future study. A underlying goal of scholarly research is not only to show what works, but to demonstrate what doesn't work or what needs further clarification.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Ioannidis, John P.A. "Limitations are not Properly Acknowledged in the Scientific Literature." Journal of Clinical Epidemiology 60 (2007): 324-329; Pasek, Josh. Writing the Empirical Social Science Research Paper: A Guide for the Perplexed . January 24, 2012. Academia.edu; Structure: How to Structure the Research Limitations Section of Your Dissertation . Dissertations and Theses: An Online Textbook. Laerd.com; What Is an Academic Paper? Institute for Writing Rhetoric. Dartmouth College; Writing the Experimental Report: Methods, Results, and Discussion . The Writing Lab and The OWL. Purdue University.

• << Previous: The Discussion
• Next: Conclusion >>
• Last Updated: Nov 6, 2023 1:43 PM
• URL: https://libraryguides.unh.edu/cpsonlinegradpaper

Limited by our limitations

• Open access
• Published: 25 July 2019
• Volume 8 , pages 261–264, ( 2019 )

Cite this article

You have full access to this open access article

• Paula T. Ross   ORCID: orcid.org/0000-0001-7751-784X 1 &
• Nikki L. Bibler Zaidi 1  

49k Accesses

24 Citations

28 Altmetric

Explore all metrics

Study limitations represent weaknesses within a research design that may influence outcomes and conclusions of the research. Researchers have an obligation to the academic community to present complete and honest limitations of a presented study. Too often, authors use generic descriptions to describe study limitations. Including redundant or irrelevant limitations is an ineffective use of the already limited word count. A meaningful presentation of study limitations should describe the potential limitation, explain the implication of the limitation, provide possible alternative approaches, and describe steps taken to mitigate the limitation. This includes placing research findings within their proper context to ensure readers do not overemphasize or minimize findings. A more complete presentation will enrich the readers’ understanding of the study’s limitations and support future investigation.

Similar content being viewed by others

Writing Quantitative Research Studies

The Specific Aims

Avoid common mistakes on your manuscript.

Introduction

Regardless of the format scholarship assumes, from qualitative research to clinical trials, all studies have limitations. Limitations represent weaknesses within the study that may influence outcomes and conclusions of the research. The goal of presenting limitations is to provide meaningful information to the reader; however, too often, limitations in medical education articles are overlooked or reduced to simplistic and minimally relevant themes (e.g., single institution study, use of self-reported data, or small sample size) [ 1 ]. This issue is prominent in other fields of inquiry in medicine as well. For example, despite the clinical implications, medical studies often fail to discuss how limitations could have affected the study findings and interpretations [ 2 ]. Further, observational research often fails to remind readers of the fundamental limitation inherent in the study design, which is the inability to attribute causation [ 3 ]. By reporting generic limitations or omitting them altogether, researchers miss opportunities to fully communicate the relevance of their work, illustrate how their work advances a larger field under study, and suggest potential areas for further investigation.

Goals of presenting limitations

Medical education scholarship should provide empirical evidence that deepens our knowledge and understanding of education [ 4 , 5 ], informs educational practice and process, [ 6 , 7 ] and serves as a forum for educating other researchers [ 8 ]. Providing study limitations is indeed an important part of this scholarly process. Without them, research consumers are pressed to fully grasp the potential exclusion areas or other biases that may affect the results and conclusions provided [ 9 ]. Study limitations should leave the reader thinking about opportunities to engage in prospective improvements [ 9 , 10 , 11 ] by presenting gaps in the current research and extant literature, thereby cultivating other researchers’ curiosity and interest in expanding the line of scholarly inquiry [ 9 ].

Presenting study limitations is also an ethical element of scientific inquiry [ 12 ]. It ensures transparency of both the research and the researchers [ 10 , 13 , 14 ], as well as provides transferability [ 15 ] and reproducibility of methods. Presenting limitations also supports proper interpretation and validity of the findings [ 16 ]. A study’s limitations should place research findings within their proper context to ensure readers are fully able to discern the credibility of a study’s conclusion, and can generalize findings appropriately [ 16 ].

Why some authors may fail to present limitations

As Price and Murnan [ 8 ] note, there may be overriding reasons why researchers do not sufficiently report the limitations of their study. For example, authors may not fully understand the importance and implications of their study’s limitations or assume that not discussing them may increase the likelihood of publication. Word limits imposed by journals may also prevent authors from providing thorough descriptions of their study’s limitations [ 17 ]. Still another possible reason for excluding limitations is a diffusion of responsibility in which some authors may incorrectly assume that the journal editor is responsible for identifying limitations. Regardless of reason or intent, researchers have an obligation to the academic community to present complete and honest study limitations.

A guide to presenting limitations

The presentation of limitations should describe the potential limitations, explain the implication of the limitations, provide possible alternative approaches, and describe steps taken to mitigate the limitations. Too often, authors only list the potential limitations, without including these other important elements.

Describe the limitations

When describing limitations authors should identify the limitation type to clearly introduce the limitation and specify the origin of the limitation. This helps to ensure readers are able to interpret and generalize findings appropriately. Here we outline various limitation types that can occur at different stages of the research process.

Study design

Some study limitations originate from conscious choices made by the researcher (also known as delimitations) to narrow the scope of the study [ 1 , 8 , 18 ]. For example, the researcher may have designed the study for a particular age group, sex, race, ethnicity, geographically defined region, or some other attribute that would limit to whom the findings can be generalized. Such delimitations involve conscious exclusionary and inclusionary decisions made during the development of the study plan, which may represent a systematic bias intentionally introduced into the study design or instrument by the researcher [ 8 ]. The clear description and delineation of delimitations and limitations will assist editors and reviewers in understanding any methodological issues.

Data collection

Study limitations can also be introduced during data collection. An unintentional consequence of human subjects research is the potential of the researcher to influence how participants respond to their questions. Even when appropriate methods for sampling have been employed, some studies remain limited by the use of data collected only from participants who decided to enrol in the study (self-selection bias) [ 11 , 19 ]. In some cases, participants may provide biased input by responding to questions they believe are favourable to the researcher rather than their authentic response (social desirability bias) [ 20 , 21 , 22 ]. Participants may influence the data collected by changing their behaviour when they are knowingly being observed (Hawthorne effect) [ 23 ]. Researchers—in their role as an observer—may also bias the data they collect by allowing a first impression of the participant to be influenced by a single characteristic or impression of another characteristic either unfavourably (horns effect) or favourably (halo effort) [ 24 ].

Data analysis

Study limitations may arise as a consequence of the type of statistical analysis performed. Some studies may not follow the basic tenets of inferential statistical analyses when they use convenience sampling (i.e. non-probability sampling) rather than employing probability sampling from a target population [ 19 ]. Another limitation that can arise during statistical analyses occurs when studies employ unplanned post-hoc data analyses that were not specified before the initial analysis [ 25 ]. Unplanned post-hoc analysis may lead to statistical relationships that suggest associations but are no more than coincidental findings [ 23 ]. Therefore, when unplanned post-hoc analyses are conducted, this should be clearly stated to allow the reader to make proper interpretation and conclusions—especially when only a subset of the original sample is investigated [ 23 ].

Study results

The limitations of any research study will be rooted in the validity of its results—specifically threats to internal or external validity [ 8 ]. Internal validity refers to reliability or accuracy of the study results [ 26 ], while external validity pertains to the generalizability of results from the study’s sample to the larger, target population [ 8 ].

Examples of threats to internal validity include: effects of events external to the study (history), changes in participants due to time instead of the studied effect (maturation), systematic reduction in participants related to a feature of the study (attrition), changes in participant responses due to repeatedly measuring participants (testing effect), modifications to the instrument (instrumentality) and selecting participants based on extreme scores that will regress towards the mean in repeat tests (regression to the mean) [ 27 ].

Threats to external validity include factors that might inhibit generalizability of results from the study’s sample to the larger, target population [ 8 , 27 ]. External validity is challenged when results from a study cannot be generalized to its larger population or to similar populations in terms of the context, setting, participants and time [ 18 ]. Therefore, limitations should be made transparent in the results to inform research consumers of any known or potentially hidden biases that may have affected the study and prevent generalization beyond the study parameters.

Explain the implication(s) of each limitation

Authors should include the potential impact of the limitations (e.g., likelihood, magnitude) [ 13 ] as well as address specific validity implications of the results and subsequent conclusions [ 16 , 28 ]. For example, self-reported data may lead to inaccuracies (e.g. due to social desirability bias) which threatens internal validity [ 19 ]. Even a researcher’s inappropriate attribution to a characteristic or outcome (e.g., stereotyping) can overemphasize (either positively or negatively) unrelated characteristics or outcomes (halo or horns effect) and impact the internal validity [ 24 ]. Participants’ awareness that they are part of a research study can also influence outcomes (Hawthorne effect) and limit external validity of findings [ 23 ]. External validity may also be threatened should the respondents’ propensity for participation be correlated with the substantive topic of study, as data will be biased and not represent the population of interest (self-selection bias) [ 29 ]. Having this explanation helps readers interpret the results and generalize the applicability of the results for their own setting.

Provide potential alternative approaches and explanations

Often, researchers use other studies’ limitations as the first step in formulating new research questions and shaping the next phase of research. Therefore, it is important for readers to understand why potential alternative approaches (e.g. approaches taken by others exploring similar topics) were not taken. In addition to alternative approaches, authors can also present alternative explanations for their own study’s findings [ 13 ]. This information is valuable coming from the researcher because of the direct, relevant experience and insight gained as they conducted the study. The presentation of alternative approaches represents a major contribution to the scholarly community.

Describe steps taken to minimize each limitation

No research design is perfect and free from explicit and implicit biases; however various methods can be employed to minimize the impact of study limitations. Some suggested steps to mitigate or minimize the limitations mentioned above include using neutral questions, randomized response technique, force choice items, or self-administered questionnaires to reduce respondents’ discomfort when answering sensitive questions (social desirability bias) [ 21 ]; using unobtrusive data collection measures (e.g., use of secondary data) that do not require the researcher to be present (Hawthorne effect) [ 11 , 30 ]; using standardized rubrics and objective assessment forms with clearly defined scoring instructions to minimize researcher bias, or making rater adjustments to assessment scores to account for rater tendencies (halo or horns effect) [ 24 ]; or using existing data or control groups (self-selection bias) [ 11 , 30 ]. When appropriate, researchers should provide sufficient evidence that demonstrates the steps taken to mitigate limitations as part of their study design [ 13 ].

In conclusion, authors may be limiting the impact of their research by neglecting or providing abbreviated and generic limitations. We present several examples of limitations to consider; however, this should not be considered an exhaustive list nor should these examples be added to the growing list of generic and overused limitations. Instead, careful thought should go into presenting limitations after research has concluded and the major findings have been described. Limitations help focus the reader on key findings, therefore it is important to only address the most salient limitations of the study [ 17 , 28 ] related to the specific research problem, not general limitations of most studies [ 1 ]. It is important not to minimize the limitations of study design or results. Rather, results, including their limitations, must help readers draw connections between current research and the extant literature.

The quality and rigor of our research is largely defined by our limitations [ 31 ]. In fact, one of the top reasons reviewers report recommending acceptance of medical education research manuscripts involves limitations—specifically how the study’s interpretation accounts for its limitations [ 32 ]. Therefore, it is not only best for authors to acknowledge their study’s limitations rather than to have them identified by an editor or reviewer, but proper framing and presentation of limitations can actually increase the likelihood of acceptance. Perhaps, these issues could be ameliorated if academic and research organizations adopted policies and/or expectations to guide authors in proper description of limitations.

Connelly LM. Limitation Section. Medsurg Nurs. 2013;22:325–36.

Google Scholar  

Puhan M, Akl E, Bryant D, Zie F, Apolone G, ter Riet G. Discussing study limitations in reports of biomedical studies-the need for more transparency. Health Qual Life Outcomes. 2012;10(23) https://doi.org/10.1186/1477-7525-10-23 .

Wang M, Bolland M, Grey A. Reporting of limitations of observational research. JAMA Intern Med. 2015;175:1571–2.

Article   Google Scholar  

Ringsted C, Hodges B, Scherpbier A. The research compass: an introduction to research in medical education: AMEE guide no. 56. Med Teach. 2001;33:695–709.

Norman G. Research in medical education: three decades of progress. BMJ. 2002;324:1560–2.

Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. 2008;42:128–33.

Gruppen LD. Is medical education research ‘hard’ or ‘soft’ research? Adv Health Sci Educ Theory Pract. 2008;13:1–2.

Price JH, Murnan J. Research limitations and the necessity of reporting them. Am J Health Educ. 2004;35:66–7.

Greener S. Research limitations: the need for honesty and common sense. Inter Learn Environ. 2018;26:567–8.

Bunniss S, Kelly DR. Research paradigms in medical education research. Med Educ. 2010;44:358–66.

Adler ES, Clark R. How it’s done: an invitation to social research. 2nd ed. Belmont: Wadsworth/Thomson Learning; 2003.

Drotar D. Thoughts on establishing research significance and preserving scientific integrity. J Pediatr Psychol. 2008;33:1–5.

Chasan-Taber L. Writing Dissertation Proposals and Grants: epidemiology, preventative medicine, and biostatistics. 1st ed. Boca Raton: Taylor & Francis Group; 2014.

Book   Google Scholar  

Anesley TM. The discussion section: your closing argument. Clin Chem. 2010;56:1671–4.

Eva K, Lingard L. What’s next? A guiding question for educators engaged in educational research. Med Educ. 2008;42:752–4.

Ioannidis JPA. Limitations are not properly acknowledged in the scientific literature. J Clinc Epid. 2007;60:324–9.

Lingard L. The art of limitations. Perspect Med Educ. 2015;4:136–7.

Creswell JW. Research design. Qualitative, quantitative, and mixed methods approaches. 2nd ed. Thousand Oaks: SAGE; 2003.

Lavrakas PJ. Encylopedia of survey research methods. Thousand Oaks: SAGE; 2008.

Kenrick DT, Neuberg SL, Social Psychology CRB. Unraveling the mystery. Boston: Allyn & Bacon; 2002.

Nederhof A. Methods of coping with social desirability bias: a review. Eur J Soc Psychol. 1985;15:263–80.

Krumpal I. Determinants of social desirability bias in sensitive surveys: a literature review. Qual Quant Intern J Method. 2013;47:2025–47.

Segwick P, Greenwood N. Understanding the hawthorne Effec. BMJ. 2015. https://doi.org/10.1136/bmj.h4672 .

Law J. A dictionary of business and management (Oxford quick reference). 6th ed. Oxford: University Press; 2009.

Curran-Deverett D, Milgrom H. Post-hoc data analysis: benefits and limitations. Curr Opin Allergy Clin Immunol. 2013;13:223–4.

Pannucci CJ, Wilkins EG. Identifying and avoiding bias in research. Plast Reconstr Surg. 2010;126:619–25.

Bagg A. Discussion: the heart of the paper. Indian Pediatr. 2016;53:901–4.

Drotar D. How to write an effective results and discussion for the Journal of Pediatric Psychology. J Pediatr Psychol. 2009;34:339–43.

Olsen R. Self-selection bias. In: Lavrakas PJ, editor. Encyclopedia of survey research methods. Thousand Oaks: SAGE; 2008. pp. 809–10.

Babbie E. The practice of social research. 8th ed. Belmont: Wadsworth Publishing Company; 1998.

Resnik DB, Shamoo AE. Reproducibility and research integrity. Account Res. 2017;24:116–23.

Bordage G. Reasons reviewers reject and accept manuscripts: The strengths and weaknesses in medical education reports. Acad Med. 2001;76:889–96.

Download references

Author information

Authors and affiliations.

Medical School, University of Michigan, Ann Arbor, MI, USA

Paula T. Ross & Nikki L. Bibler Zaidi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Paula T. Ross .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Ross, P.T., Bibler Zaidi, N.L. Limited by our limitations. Perspect Med Educ 8 , 261–264 (2019). https://doi.org/10.1007/s40037-019-00530-x

Download citation

Published : 25 July 2019

Issue Date : 01 August 2019

DOI : https://doi.org/10.1007/s40037-019-00530-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Limitations
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024 Jan-.

StatPearls [Internet].

Common pitfalls in the research process.

Jacob Shreffler ; Martin R. Huecker .

Affiliations

Last Update: March 6, 2023 .

• Definition/Introduction

Conducting research from planning to publication can be a very rewarding process. However, multiple preventable setbacks can occur within each stage of research. While these inefficiencies are an inevitable part of the research process, understanding common pitfalls can limit those hindrances. Many issues can present themselves throughout the research process. It has been said about academics that “the politics are so harsh because the stakes are so low.” Beyond interpersonal and political / funding concerns, prospective authors may encounter some disenchantment with the publish or perish culture. With a metric of (any) publication, the motivation to contribute meaningfully to science can be overshadowed by a compulsive drive to publish. [1]  We believe in quality over quantity and highlight the importance of channeling creativity when pursuing scholarly work.

When considering embarking on a medical research project, one must begin with detailed planning. Do not underestimate the amount of time a project can take, often spanning years from conception to manuscript preparation. Will you conduct a retrospective chart review, a prospective study, or a true clinical trial with randomization and blinding? Will you systematically seek out and remove sources of bias from the study design and interpretation of results? Will you ensure the study is powered properly to justify conclusions? Will you eliminate or explain any conflicts of interest occurring among your author group? Will you fall victim to the temptation of frivolous subgroup analyses, or will you stick with the original plan? Will your study have a realistic chance at publication in a journal within your specialty, or perhaps another subfield? The study results may prove the null hypothesis, a ‘negative study,’ and therefore be difficult to publish. [2]  Additionally, the intervention you find beneficial may subsequently be proven unhelpful or even dangerous, leading to prudent medical reversal. [3]

These considerations and more necessitate meticulous planning and vigilant adherence to a sound protocol. Along the way, you will encounter obstacles, pitfalls, some of which are presented in this article. But remain persistent, and your efforts will be rewarded with publication and contribution to science. This review covers common pitfalls researchers encounter and suggested strategies to avoid them.

• Issues of Concern

There are five phases of research: planning phase, data collection/analysis phase, writing phase, journal submission phase, and rejections/revisions/acceptance phase.

Phase I Pitfalls: Planning a Study

The highest yield preempting of pitfalls in the research process occurs in the planning phase. This is when a researcher can set the stage for an optimal research process. Below are pitfalls that can occur during the planning phase.

Pitfall: Underestimating what committing to a research project requires

Conducting a research study and achieving publication sounds fulfilling, right?

Consider the many steps: conducting a literature search, writing an IRB proposal, planning and having research meetings, long and cumbersome data collection processes, working with statisticians or analyzing complex data, having unexpected research setbacks (e.g., subjects drop out, newly published papers on same topic, etc.), the possibility that after data collection you have no statistically (or clinically) significant findings, conducting an updated literature search, writing introduction, methods, results, and discussion sections of a paper, going through the many journal options to determine best fit while aiming for high impact factors, adhering to journal guidelines/fixing drafts, writing cover letters stating importance of the topic to respective journals, creating journal portal accounts, possibly being rejected numerous times, waiting months for journal decisions, working on numerous revisions and being informed by numerous individuals about all of the flaws in your writing and research.

Does it sound, maybe less fulfilling ?

Conducting a research project from inception to publication can be a rewarding experience. Research requires significant time. Setbacks are normal. To produce an important and sought-after research product, an individual must understand the magnitude of commitment required.

Pitfall: Choosing the wrong research pursuit/topic lacks precision

Consider an investigator interested in substance use research. The first challenge is the immense amount of research already published on this topic. Fortunately, there is still a massive amount of uncharted territory in substance use research.

It is important to understand what has been done and what is still undiscovered in your area of research. Do not simply study a topic because you find it interesting; passion is advantageous, but you should ensure that your study will contribute to some field/specialty or research in a significant way.

How does your research differ from what has been done?

How will it impact practice in a way that no previous study has?

Consider these questions when choosing a topic for research. Otherwise, you may struggle to get the work published. It can be demoralizing if you have already written your paper and realize that your paper is not going to get accepted by a reputable journal due to the presence of other papers already describing the same concepts you have.

As always, the first step is a thorough literature search.

Pitfall: Not considering research bias

A common theme noted in literature is that bias can, unfortunately, lead to failure to reproduce results, raising concerns regarding the integrity of science. [4]  Bias can be considered various (inadvertent) poor strategies related to data design, analysis, and results reporting that produce spurious results and papers that perhaps should not be published. [5]

While one cannot completely eliminate bias from the research process, researchers should take steps to understand research bias in study endeavors and determine how to minimize bias during the planning phase of the study.  

Pitfall: Not focusing on which variables to collect

Researchers often want to collect as much data as possible but should not build a list of variables that includes every single detail about subjects if the variables collected are unlikely to yield insight into the topic of research. The longer the data collection instrument, the higher likelihood of (human) errors (if manually data entry) and the longer duration of the data collection phase. Instead of taking time to build a database with many variables, consider cutting irrelevant variables and use that time to increase the sample size. Determine, based on your own clinical knowledge and published empirical works, which variables are most crucial. 

Pitfall: Worrying about the statistics after the data has been collected

A vital part of the research process is ensuring you have a rigorous statistical approach. Involve your statistician very early in the project, preferably in the planning stages. They will have insight into the types of variables to collect and help shape the research methods. Statistical power is an important concept to consider before data collection to avoid false-negative results (Zlowodzki et al., 2006). Furthermore, other concepts, such as covariates, need to be part of the planning phase. Do not wait until after the data collection phase to give data to the statistician who cannot transform the data you have into outputs you want.

Pitfall: Not setting defined author roles

It is important to define who will be declared authors at the beginning of the research process to avoid conflict. Do most people want to be an author? Sure. Does everybody do the work worthy of authorship? No. While placing general comments in a shared document's margin may make the paper slightly better, it probably should not qualify for authorship. Review authorship criteria to determine what constitutes authorship. Clear expectations can ensure that everyone is on the same page and that everyone feels the process is fair, especially for individuals who plan to invest significant time in the project. Clear expectations for each author should occur before any writing begins, including deadlines and specific contributions. [6] [7] [6]

Pitfall: Not considering limitations of work before the paper is written

Avoid this pitfall by reviewing recent manuscripts and reading the limitations sections of these papers. Many of these limitations sections will make notions about generalizability to other populations. Some will discuss low power. Even the best papers in the top journals have many limitations. The best way to avoid or mitigate your work's limitations is to consider them during the planning phase.

How can you set up your project to limit your limitations section?

What (types of) samples should you include in your study?

Were you originally thinking of retrospective design, but it could be prospective?

What steps can you utilize to control baseline characteristics between groups?

Consider all limitations and think about how you can control these before data collection.

Phase II Pitfalls: Data Collection and Analysis

After the planning has occurred, typically after institutional review board (IRB) approval, the data collection and analysis phase can transpire. The entire team should typically stay involved throughout these phases. Below are pitfalls to avoid.

Pitfall: Not being involved in the data collection phase

It is important to be involved with the data collection phase, even if you do not personally collect data. Train the individuals who collect data to ensure all are on the same page and provide periodic oversight to ensure accuracy and quality of the data over time. [8]  Do not assume the data collection phase is going smoothly – you may find yourself with a huge dataset riddled with inconsistencies or errors. Schedule periodic meetings to review data.

Pitfall: Not being involved with the statistical analysis phase

If you are not conducting the statistical analysis, do not assume that the person who is analyzing the data is 100% on the same page. Have meetings about the data, how to interpret the data, and the limitations of the data. Ask what other ways the data could be analyzed and how reviewers might negatively critique the data itself or the statistical methods.

The person conducting the analysis will not have the same familiarity with the topic. You are not going to be as familiar with the outputs. By understanding each other, you will a) have clearer, more robust methods and results in sections of the paper, b) limit critiques regarding the statistical approach/data outcomes, c) understand your research better for any presentations, discussion, or future work, and d) develop a positive collaboration for future work.

Phase III Pitfalls: The Writing Phase

The next phase is the writing phase. While this section covers pitfalls during the writing phase, for recommendations on conducting a literature search, writing, and publishing research, see StatPearls Evidence-Base Medicine Chapter: How to Write and Publish a Scientific Manuscript. [9]  Below are pitfalls that can occur during the writing phase. 

Pitfall: Poor or outdated references

When writing your paper, perform multiple literature searches to ensure all recent, salient references are covered—claims about recent similar work or research that frames your study if the references are outdated. Journals may even ask reviewers to comment on the presence or absence of up-to-date/suitable references. Conduct a literature search prior to data collection and stay on top of references throughout the research process as new papers become available.

Pitfall: No clearly defined purpose of the paper

Many aspects of manuscripts can get overlooked. Lack of a clear purpose statement can doom a paper to futility. Remind the readers of the goal of the project. You do not want consumers of your research to read the results section and forget what the goals/main outcomes are. The purpose statement should be located at the end of the introduction section.  

Pitfall: Unclear methods making research hard to reproduce

A common concern in science is the lack of transparency in methods for reproducibility. The methods section should allow a reader to understand exactly what was done and conduct the study. Consider examining the S treng T hening the R eporting of OB servational studies in E pidemiology (STROBE) checklist for the methods (as well as other paper sections) to ensure best reporting practices for reproducibility. [10]

Pitfall: The tables and narratives are the same

Reviewers prefer you not to state findings in narratives that are in tables. Tables focus readers on the most important results and are not redundant with the written content. Make call-outs to the table in the paper's narrative sections, but do not state information found in tables.  

Pitfall: Not reporting all data/outcomes

Some authors will state the main outcome of interest or have a statement such as “there were no other statistically significant findings between other groups.” Authors must report all outcomes and statistical analyses for reproducibility of the research. While this may be difficult to do with a broad approach, utilize tables and appendices to report all outcomes to show transparency and limit researcher bias.

Pitfall: Repeating results in discussion

Do not simply restate in the discussion what you already have in the results section. Utilize this section of the paper to link other references to your work and reflect on other empirical investigations' similarities or differences. Explain why your research provides an impactful contribution to the topic.  

Pitfall: Making conclusions that do not align with your work

Authors sometimes note in their conclusions how the work impacts a topic due to X reason when X may be too broad a claim and the work doesn’t really support or prove that notion. Researchers should align their conclusions to their own results and highlight the significance of their findings.

Pitfall: Thinking the title is not a big deal

A strong title will help with the impact/readership of your paper. Consider keeping a short title that provides the main takeaway. Papers with more concise titles and present the study conclusion result in a bigger impact/receive more citations. [11]

Pitfall: Completing the abstract last minute

Similar to the title, do not underestimate an abstract. Journal and conference reviewers (and the general audience) may only read your abstract. The abstract must have the key results and contributions of the study and be well-written.

Phase IV Pitfalls: Submitting to a Journal

After the paper has been written, it is time to choose the journal. This phase also has numerous pitfalls. Below are pitfalls that can occur during this phase.  

Pitfall: Choosing the wrong journal

Choosing the journal for your work can be overwhelming due to the number of options. Always look at the aims and scope of prospective journals. Look through the author guidelines to ensure that your manuscript adheres. This will save time. Review your reference list for any journals that appear more than once; if so, consider submitting to that journal. You do not want to submit your paper, wait two weeks, and then get a desk rejection because the editors state the paper is not aligned to the journal's aims and scope.

Additionally, researchers can aim too high and spend months (and numerous hours in journal submission portals) trying to publish a manuscript in a journal with a very large impact factor. Though admirable, if the research design and results lacking “gold standard” reporting, authors should consider a journal that is more likely to accept. Find a balance between the quality of your paper and the quality of the journal. Seek feedback from the other authors and/or senior colleagues who can provide honest feedback.

Pitfall: Poor cover letter on journal submission

Do not submit work with a flawed cover letter (errors or lack of clarity in how your work contributes to the body of literature). Spend time writing a detailed cover letter once, have it edited by someone else, and utilize that for all future projects. You can highlight the differences (e.g., the purpose of this work, our results showed) with each project. Use the cover letter to highlight the significance of the study while adhering to the disclosure guidelines (e.g., conflicts of interests, authors contributions, data releases, etc.), which will help the editorial board determine not only the suitability of the paper for the journal but also streamline the review process. [12]

Pitfall: Assuming that after the paper has been submitted to a journal, the work is done             

The paper has been submitted! You think you are finished…but, unfortunately, the publishing game may still be far from over. Researchers often do not recognize the amount of time going into the submission/rejection/revisions phases. Revisions can sometimes be total overhauls, more work than writing a whole new paper. Be prepared to continue working.

Phase V Pitfalls: The Rejections, Revisions, and Acceptance Phase

Finally, perhaps the most unpredictable phase, the rejections, revisions, and acceptance phase, has unique pitfalls and other obstacles.

Pitfall: Mourning rejections too long/ “sitting on” a rejected paper             

Did you get a desk to reject (i.e., the manuscript was not even sent for blind review)? That is unfortunate but common. You do not have time to sulk. Get that paper submitted somewhere else. The older the data, the less desirable your paper becomes. If the paper went in for a full review and was rejected, that may be even tougher than a desk reject because more time has elapsed. The good news is that (hopefully) you received feedback to incorporate in a revision. Do not spend too much time grieving rejections.

Pitfall: Not laying to rest rejected papers when it is indeed their time to go

Did you write a paper a couple of years ago, and you’ve submitted it to 20 different journals? The data is getting old. The topic wasn’t focused on. The sample size was small. Perhaps the project is not worth pursuing any longer. Do not give in to the sunk cost fallacy. If, however, you are proud of the work and stand by the paper, do not give up. If you believe after the numerous rejections that the topic/project is flawed, you can use this failure as a personal learning/growth opportunity. Do not repeat controllable mistakes on future projects.

Pitfall: Not addressing all of reviewer feedback

Did you get a revise and resubmit? Great news! The reviewers and editors will likely ask you to respond to each comment when you resubmit. Address all of the reviewer feedback. Take your time reading through the feedback, digest it, and re-read it. Carefully respond and decide how to revise your manuscript based on the feedback. Share the reviews and the duties of revision with coauthors. In your response to reviewers, stay professional and address each statement, even if you disagree with what is stated. If you do not respond to each statement, the reviewers often highlight the concern(s) again.

Pitfall: Thinking you know what the reviewers are going to say

Research reviewers are like a box of chocolates. You never know what you are going to get. You may be worried about a section of your paper/research approach, and the reviewers do not mention it at all in their review; instead, they criticize a section of your manuscript that you are most proud of.

In some reviews, you may get feedback like the following:

Reviewer #1

Please change lines 104-108 as I believe they are irrelevant to your study.

Reviewer #2

Please build on lines 104-108, as I believe they are the foundation of your study.

Sometimes, after multiple revisions, there are new concerns presented by the reviewers. This can be disheartening. Should some regulations restrict reviewers from bringing up new ideas/concerns during revision #7? Perhaps. Does any current rule prevent them from doing this? No.

During the review process, we must have faith that the reviewers are knowledgeable and provide fair, insightful, and constructive feedback. While the review process can be arbitrary or frustrating in some cases, peer review remains the gold standard in a scientific publication. Stay positive and persistent. Stay professional in responses to the reviewers. Remember that the review process can be very beneficial as it often leads to feedback that truly elevates your work and makes the product (and you) look better. [13]

Pitfall: Not rewarding yourself for a published paper

You did it! Celebrate your accomplishment. Reflect on the merit of your effort before you move on to other work or re-enter the cycle of IRBs, data coding, journal submissions, etc. Remember and appreciate how remarkable it is that you just contributed knowledge to the world.

• Clinical Significance

Many pitfalls can occur throughout the research process. Researchers should understand these pitfalls and utilize strategies to avoid them to produce high-quality, sought-after research results that are useful for basic science and clinical practice.

• Review Questions
• Access free multiple choice questions on this topic.
• Comment on this article.

Disclosure: Jacob Shreffler declares no relevant financial relationships with ineligible companies.

Disclosure: Martin Huecker declares no relevant financial relationships with ineligible companies.

This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ), which permits others to distribute the work, provided that the article is not altered or used commercially. You are not required to obtain permission to distribute this article, provided that you credit the author and journal.

• Cite this Page Shreffler J, Huecker MR. Common Pitfalls In The Research Process. [Updated 2023 Mar 6]. In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024 Jan-.

In this Page

Bulk download.

• Bulk download StatPearls data from FTP

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Similar articles in PubMed

• Publish or perish: ensuring longevity in nurse education-evaluation of a strategy to engage academics, students, and clinicians in publication activity. [J Prof Nurs. 2013] Publish or perish: ensuring longevity in nurse education-evaluation of a strategy to engage academics, students, and clinicians in publication activity. Wilson A, Sharrad S, Rasmussen P, Kernick J. J Prof Nurs. 2013 Jul-Aug; 29(4):210-6.
• Extent of publishing in predatory journals by academics in higher education institutions in Zimbabwe: A case study of a university. [Account Res. 2023] Extent of publishing in predatory journals by academics in higher education institutions in Zimbabwe: A case study of a university. Jingura RM, Chigwada J, Diver T, Shangwa D. Account Res. 2023 Sep 11; :1-15. Epub 2023 Sep 11.
• Has "Publish or Perish" Become "Publish and Payment"? Navigating Neurosurgical Research in an Innovative Industry. [World Neurosurg. 2017] Has "Publish or Perish" Become "Publish and Payment"? Navigating Neurosurgical Research in an Innovative Industry. Fraser JF. World Neurosurg. 2017 Aug; 104:987-989. Epub 2017 May 17.
• Review [Publish &amp; Perish; research on research and researchers]. [Tijdschr Psychiatr. 2017] Review [Publish &amp; Perish; research on research and researchers]. Tijdink JK. Tijdschr Psychiatr. 2017; 59(7):406-413.
• Review Publication Practices and Responsible Authorship: A Review Article. [J Public Health Afr. 2017] Review Publication Practices and Responsible Authorship: A Review Article. Tarkang EE, Kweku M, Zotor FB. J Public Health Afr. 2017 Jun 23; 8(1):723. Epub 2017 Jun 27.

Recent Activity

• Common Pitfalls In The Research Process - StatPearls Common Pitfalls In The Research Process - StatPearls

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Research Limitations

It is for sure that your research will have some limitations and it is normal. However, it is critically important for you to be striving to minimize the range of scope of limitations throughout the research process.  Also, you need to provide the acknowledgement of your research limitations in conclusions chapter honestly.

It is always better to identify and acknowledge shortcomings of your work, rather than to leave them pointed out to your by your dissertation assessor. While discussing your research limitations, don’t just provide the list and description of shortcomings of your work. It is also important for you to explain how these limitations have impacted your research findings.

Your research may have multiple limitations, but you need to discuss only those limitations that directly relate to your research problems. For example, if conducting a meta-analysis of the secondary data has not been stated as your research objective, no need to mention it as your research limitation.

Research limitations in a typical dissertation may relate to the following points:

1. Formulation of research aims and objectives . You might have formulated research aims and objectives too broadly. You can specify in which ways the formulation of research aims and objectives could be narrowed so that the level of focus of the study could be increased.

2. Implementation of data collection method . Because you do not have an extensive experience in primary data collection (otherwise you would not be reading this book), there is a great chance that the nature of implementation of data collection method is flawed.

3. Sample size. Sample size depends on the nature of the research problem. If sample size is too small, statistical tests would not be able to identify significant relationships within data set. You can state that basing your study in larger sample size could have generated more accurate results. The importance of sample size is greater in quantitative studies compared to qualitative studies.

4. Lack of previous studies in the research area . Literature review is an important part of any research, because it helps to identify the scope of works that have been done so far in research area. Literature review findings are used as the foundation for the researcher to be built upon to achieve her research objectives.

However, there may be little, if any, prior research on your topic if you have focused on the most contemporary and evolving research problem or too narrow research problem. For example, if you have chosen to explore the role of Bitcoins as the future currency, you may not be able to find tons of scholarly paper addressing the research problem, because Bitcoins are only a recent phenomenon.

5. Scope of discussions . You can include this point as a limitation of your research regardless of the choice of the research area. Because (most likely) you don’t have many years of experience of conducing researches and producing academic papers of such a large size individually, the scope and depth of discussions in your paper is compromised in many levels compared to the works of experienced scholars.

You can discuss certain points from your research limitations as the suggestion for further research at conclusions chapter of your dissertation.

My e-book,  The Ultimate Guide to Writing a Dissertation in Business Studies: a step by step assistance  offers practical assistance to complete a dissertation with minimum or no stress. The e-book covers all stages of writing a dissertation starting from the selection to the research area to submitting the completed version of the work within the deadline. John Dudovskiy

• Privacy Policy

Buy Me a Coffee

Home » Implications in Research – Types, Examples and Writing Guide

Implications in Research – Types, Examples and Writing Guide

Table of Contents

Implications in Research

Implications in research refer to the potential consequences, applications, or outcomes of the findings and conclusions of a research study. These can include both theoretical and practical implications that extend beyond the immediate scope of the study and may impact various stakeholders, such as policymakers, practitioners, researchers , or the general public.

Structure of Implications

The format of implications in research typically follows the structure below:

• Restate the main findings: Begin by restating the main findings of the study in a brief summary .
• Link to the research question/hypothesis : Clearly articulate how the findings are related to the research question /hypothesis.
• Discuss the practical implications: Discuss the practical implications of the findings, including their potential impact on the field or industry.
• Discuss the theoretical implications : Discuss the theoretical implications of the findings, including their potential impact on existing theories or the development of new ones.
• Identify limitations: Identify the limitations of the study and how they may affect the generalizability of the findings.
• Suggest directions for future research: Suggest areas for future research that could build on the current study’s findings and address any limitations.

Types of Implications in Research

Types of Implications in Research are as follows:

Theoretical Implications

These are the implications that a study has for advancing theoretical understanding in a particular field. For example, a study that finds a new relationship between two variables can have implications for the development of theories and models in that field.

Practical Implications

These are the implications that a study has for solving practical problems or improving real-world outcomes. For example, a study that finds a new treatment for a disease can have implications for improving the health of patients.

Methodological Implications

These are the implications that a study has for advancing research methods and techniques. For example, a study that introduces a new method for data analysis can have implications for how future research in that field is conducted.

Ethical Implications

These are the implications that a study has for ethical considerations in research. For example, a study that involves human participants must consider the ethical implications of the research on the participants and take steps to protect their rights and welfare.

Policy Implications

These are the implications that a study has for informing policy decisions. For example, a study that examines the effectiveness of a particular policy can have implications for policymakers who are considering whether to implement or change that policy.

Societal Implications

These are the implications that a study has for society as a whole. For example, a study that examines the impact of a social issue such as poverty or inequality can have implications for how society addresses that issue.

Forms of Implications In Research

Forms of Implications are as follows:

Positive Implications

These refer to the positive outcomes or benefits that may result from a study’s findings. For example, a study that finds a new treatment for a disease can have positive implications for patients, healthcare providers, and the wider society.

Negative Implications

These refer to the negative outcomes or risks that may result from a study’s findings. For example, a study that finds a harmful side effect of a medication can have negative implications for patients, healthcare providers, and the wider society.

Direct Implications

These refer to the immediate consequences of a study’s findings. For example, a study that finds a new method for reducing greenhouse gas emissions can have direct implications for policymakers and businesses.

Indirect Implications

These refer to the broader or long-term consequences of a study’s findings. For example, a study that finds a link between childhood trauma and mental health issues can have indirect implications for social welfare policies, education, and public health.

Importance of Implications in Research

The following are some of the reasons why implications are important in research:

• To inform policy and practice: Research implications can inform policy and practice decisions by providing evidence-based recommendations for actions that can be taken to address the issues identified in the research. This can lead to more effective policies and practices that are grounded in empirical evidence.
• To guide future research: Implications can also guide future research by identifying areas that need further investigation, highlighting gaps in current knowledge, and suggesting new directions for research.
• To increase the impact of research : By communicating the practical and theoretical implications of their research, researchers can increase the impact of their work by demonstrating its relevance and importance to a wider audience.
• To enhance the credibility of research : Implications can help to enhance the credibility of research by demonstrating that the findings have practical and theoretical significance and are not just abstract or academic exercises.
• To foster collaboration and engagement : Implications can also foster collaboration and engagement between researchers, practitioners, policymakers, and other stakeholders by providing a common language and understanding of the practical and theoretical implications of the research.

Example of Implications in Research

Here are some examples of implications in research:

• Medical research: A study on the efficacy of a new drug for a specific disease can have significant implications for medical practitioners, patients, and pharmaceutical companies. If the drug is found to be effective, it can be used to treat patients with the disease, improve their health outcomes, and generate revenue for the pharmaceutical company.
• Educational research: A study on the impact of technology on student learning can have implications for educators and policymakers. If the study finds that technology improves student learning outcomes, educators can incorporate technology into their teaching methods, and policymakers can allocate more resources to technology in schools.
• Social work research: A study on the effectiveness of a new intervention program for individuals with mental health issues can have implications for social workers, mental health professionals, and policymakers. If the program is found to be effective, social workers and mental health professionals can incorporate it into their practice, and policymakers can allocate more resources to the program.
• Environmental research: A study on the impact of climate change on a particular ecosystem can have implications for environmentalists, policymakers, and industries. If the study finds that the ecosystem is at risk, environmentalists can advocate for policy changes to protect the ecosystem, policymakers can allocate resources to mitigate the impact of climate change, and industries can adjust their practices to reduce their carbon footprint.
• Economic research: A study on the impact of minimum wage on employment can have implications for policymakers and businesses. If the study finds that increasing the minimum wage does not lead to job losses, policymakers can implement policies to increase the minimum wage, and businesses can adjust their payroll practices.

How to Write Implications in Research

Writing implications in research involves discussing the potential outcomes or consequences of your findings and the practical applications of your study’s results. Here are some steps to follow when writing implications in research:

• Summarize your key findings: Before discussing the implications of your research, briefly summarize your key findings. This will provide context for your implications and help readers understand how your research relates to your conclusions.
• Identify the implications: Identify the potential implications of your research based on your key findings. Consider how your results might be applied in the real world, what further research might be necessary, and what other areas of study could be impacted by your research.
• Connect implications to research question: Make sure that your implications are directly related to your research question or hypotheses. This will help to ensure that your implications are relevant and meaningful.
• Consider limitations : Acknowledge any limitations or weaknesses of your research, and discuss how these might impact the implications of your research. This will help to provide a more balanced view of your findings.
• Discuss practical applications : Discuss the practical applications of your research and how your findings could be used in real-world situations. This might include recommendations for policy or practice changes, or suggestions for future research.
• Be clear and concise : When writing implications in research, be clear and concise. Use simple language and avoid jargon or technical terms that might be confusing to readers.
• Provide a strong conclusion: Provide a strong conclusion that summarizes your key implications and leaves readers with a clear understanding of the significance of your research.

Purpose of Implications in Research

The purposes of implications in research include:

• Informing practice: The implications of research can provide guidance for practitioners, policymakers, and other stakeholders about how to apply research findings in practical settings.
• Generating new research questions: Implications can also inspire new research questions that build upon the findings of the original study.
• Identifying gaps in knowledge: Implications can help to identify areas where more research is needed to fully understand a phenomenon.
• Promoting scientific literacy: Implications can also help to promote scientific literacy by communicating research findings in accessible and relevant ways.
• Facilitating decision-making : The implications of research can assist decision-makers in making informed decisions based on scientific evidence.
• Contributing to theory development : Implications can also contribute to the development of theories by expanding upon or challenging existing theories.

When to Write Implications in Research

Here are some specific situations of when to write implications in research:

• Research proposal : When writing a research proposal, it is important to include a section on the potential implications of the research. This section should discuss the potential impact of the research on the field and its potential applications.
• Literature review : The literature review is an important section of the research paper where the researcher summarizes existing knowledge on the topic. This is also a good place to discuss the potential implications of the research. The researcher can identify gaps in the literature and suggest areas for further research.
• Conclusion or discussion section : The conclusion or discussion section is where the researcher summarizes the findings of the study and interprets their meaning. This is a good place to discuss the implications of the research and its potential impact on the field.

Advantages of Implications in Research

Implications are an important part of research that can provide a range of advantages. Here are some of the key advantages of implications in research:

• Practical applications: Implications can help researchers to identify practical applications of their research findings, which can be useful for practitioners and policymakers who are interested in applying the research in real-world contexts.
• Improved decision-making: Implications can also help decision-makers to make more informed decisions based on the research findings. By clearly identifying the implications of the research, decision-makers can understand the potential outcomes of their decisions and make better choices.
• Future research directions : Implications can also guide future research directions by highlighting areas that require further investigation or by suggesting new research questions. This can help to build on existing knowledge and fill gaps in the current understanding of a topic.
• Increased relevance: By highlighting the implications of their research, researchers can increase the relevance of their work to real-world problems and challenges. This can help to increase the impact of their research and make it more meaningful to stakeholders.
• Enhanced communication : Implications can also help researchers to communicate their findings more effectively to a wider audience. By highlighting the practical applications and potential benefits of their research, researchers can engage with stakeholders and communicate the value of their work more clearly.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How to Cite Research Paper – All Formats and...

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Paper Format – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Psychology: Research and Review

• Open access
• Published: 25 January 2017

Scale development: ten main limitations and recommendations to improve future research practices

• Fabiane F. R. Morgado 1 ,
• Juliana F. F. Meireles 2 ,
• Clara M. Neves 2 ,
• Ana C. S. Amaral 3 &
• Maria E. C. Ferreira 2  

Psicologia: Reflexão e Crítica volume  30 , Article number:  3 ( 2018 ) Cite this article

99k Accesses

272 Citations

2 Altmetric

Metrics details

An Erratum to this article was published on 03 March 2017

The scale development process is critical to building knowledge in human and social sciences. The present paper aimed (a) to provide a systematic review of the published literature regarding current practices of the scale development process, (b) to assess the main limitations reported by the authors in these processes, and (c) to provide a set of recommendations for best practices in future scale development research. Papers were selected in September 2015, with the search terms “scale development” and “limitations” from three databases: Scopus, PsycINFO, and Web of Science, with no time restriction. We evaluated 105 studies published between 1976 and 2015. The analysis considered the three basic steps in scale development: item generation, theoretical analysis, and psychometric analysis. The study identified ten main types of limitation in these practices reported in the literature: sample characteristic limitations, methodological limitations, psychometric limitations, qualitative research limitations, missing data, social desirability bias, item limitations, brevity of the scale, difficulty controlling all variables, and lack of manual instructions. Considering these results, various studies analyzed in this review clearly identified methodological weaknesses in the scale development process (e.g., smaller sample sizes in psychometric analysis), but only a few researchers recognized and recorded these limitations. We hope that a systematic knowledge of the difficulties usually reported in scale development will help future researchers to recognize their own limitations and especially to make the most appropriate choices among different conceptions and methodological strategies.

Introduction

In recent years, numerous measurement scales have been developed to assess attitudes, techniques, and interventions in a variety of scientific applications (Meneses et al. 2014 ). Measurement is a fundamental activity of science, since it enables researchers to acquire knowledge about people, objects, events, and processes. Measurement scales are useful tools to attribute scores in some numerical dimension to phenomena that cannot be measured directly. They consist of sets of items revealing levels of theoretical variables otherwise unobservable by direct means (DeVellis 2003 ).

A variety of authors (Clark and Watson 1995 ; DeVellis 2003 ; Nunnally 1967 ; Pasquali 2010 ) have agreed that the scale development process involves complex and systematic procedures that require theoretical and methodological rigor. According to these authors, the scale development process can be carried out in three basic steps.

In the first step, commonly referred as “item generation,” the researcher provides theoretical support for the initial item pool (Hutz et al. 2015 ). Methods for the initial item generation can be classified as deductive, inductive, or a combination of the two. Deductive methods involve item generation based on an extensive literature review and pre-existing scales (Hinkin 1995 ). On the other hand, inductive methods base item development on qualitative information regarding a construct obtained from opinions gathered from the target population—e.g., focus groups, interviews, expert panels, and qualitative exploratory research methodologies (Kapuscinski and Masters 2010 ). The researcher is also concerned with a variety of parameters that regulate the setting of each item and of the scale as a whole. For example, suitable scale instructions, an appropriate number of items, adequate display format, appropriate item redaction (all items should be simple, clear, specific, ensure the variability of response, remain unbiased, etc.), among other parameters (DeVellis 2003 ; Pasquali 2010 ).

In the second step, usually referred to as the “theoretical analysis,” the researcher assesses the content validity of the new scale, ensuring that the initial item pool reflects the desired construct (Arias et al. 2014 ). A content validity assessment is required, since inferences are made based on the final scale items. The item content must be deemed valid to instill confidence in all consequent inferences. In order to ensure the content validity, the researcher seeks other opinions about the operationalized items. The opinions can be those of expert judges (experts in the development scales or experts in the target construct) or target population judges (potential users of the scale), enabling the researcher to ensure that the hypothesis elaborated in the research appropriately represents the construct of interest (Nunnally 1967 ).

In the last step, psychometric analysis, the researcher should assess whether the new scale has construct validity and reliability. Construct validity is most directly related to the question of what the instrument is in fact measuring—what construct, trait, or concept underlies an individual’s performance or score on a measure (Churchill 1979 ). This refers to the degree to which inferences can be legitimately made from the observed scores to the theoretical constructs about which these observations are supposed to contain information (Podsakoff et al. 2013 ). Construct validity can be assessed with the use of exploratory factor analysis (EFA), confirmatory factor analysis (CFA), or with convergent, discriminant, predictive/nomological, criterion, internal, and external validity. In turn, reliability is a measure of score consistency, usually measured by use of internal consistency, test-retest reliability, split-half, item-total correlation/inter-item reliability, and inter-observer reliability (DeVellis 2003 ). To ensure construct validity and reliability, the data should be collected in a large and appropriately representative sample of the target population. It is a common rule of thumb that there should be at least 10 participants for each item of the scale, making an ideal of 15:1 or 20:1 (Clark and Watson 1995 ; DeVellis 2003 ; Hair Junior et al. 2009 ).

Although the literature on theoretical and methodological care in scale development is extensive, many limitations have been identified in the process. These include failure to adequately define the construct domain, failure to correctly specify the measurement model, underutilization of some techniques that are helpful in establishing construct validity (MacKenzie et al. 2011 ), relatively weak psychometric properties, applicability to only a single form of treatment or manual, extensive time required to fill out the questionnaire (Hilsenroth et al. 2005 ), inappropriate item redaction, too few items and participants in the construction and analysis, an imbalance between items that assess positive beliefs and those that assess negative beliefs (Prados 2007 ), social desirability bias (King and Bruner 2000 ), among others.

These limitations in the scale development process weaken the obtained psychometric results, limiting the future applicability of the new scale and hindering its generalizability. In this sense, knowledge of the most often reported limitations is fundamental in providing essential information to help develop best practices for future research in this area. The purpose of this article is threefold: (a) to provide a systematic review of the published literature regarding some current practices of the scale development process, (b) to assess the main limitations reported by the authors in this process, and (c) to provide a set of recommendations for best practices in future scale development research.

This systematic review identified and selected papers from three databases: Scopus, PsycINFO, and Web of Science. There was no time restriction in the literature search, which was completed in September 1, 2015. The following search term was used: “scale development.” In the set of databases analyzed, the search was done inclusively in “Any Field” (PsycINFO), in “Article Title, Abstract, Keywords” (Scopus), or in any “Topic” (Web of Science). In addition, we used an advanced search to filter the articles in (search within results), with the search term “limitations” identified in “Any Field” in all databases. Both terms were used in English only. Four reviewers evaluated the papers in an independent and blinded way. Any disagreements on eligibility of a particular study were resolved through consensus among reviewers.

Figure  1 shows a flowchart summarizing the strategy adopted for identification and selection of studies. We used only one inclusion criteria for the evaluation of the studies: (a) articles that aim to develop and validate self-administered measurement scales for humans. We excluded (a) unavailable full-text papers in the analyzed databases, (b) papers in languages other than English, Portuguese, or Spanish, (c) articles which were not clearly aimed at the development of a new scale (i.e., we excluded articles investigating only the reliability, validity, or revisions of existing scales and studies that describe the validation of instruments for other languages), (d) papers with unvalidated scales, and (e) articles that did not declare the limitations of the study.

Flowchart showing summary of the systematic process of identifying and selecting article

In all, this systematic review evaluated 105 studies published between 1976 and 2015. Most (88.5%) was published between 2005 and 2015, and only two studies date from the last century. We analyzed two major issues: (a) current practices of the scale development process —considering the three steps usually reported in the literature (step 1—item generation, step 2—theoretical analysis, step 3—psychometric analysis), the number of participants in step 3, the number of items in the beginning scale, and the number of items in the final scale; (b) m ain limitations reported by the authors in the scale development process —considering the limitations observed and recorded by the authors during the scale development process. The description of these results can be found in Table  1 .

Current practices of the scale development process

Step 1—item generation.

In the first step, 35.2% ( n  = 37) of the studies reported using exclusively deductive methods to write items, 7.6% ( n  = 8) used only inductive methods, and 56.2% ( n  = 59) combined deductive and inductive strategies. The majority of the studies used a literature review (84.7%, n  = 89) as the deductive method in item generation. In inductive methods, 26.6% of studies ( n  = 28) chose to conduct an interview.

Step 2—theoretical analysis

In order to theoretically refine the items, several studies used opinions of experts (74.2%, n  = 78), whereas others used target population opinions (43.8%, n  = 46). In addition, 63.8% ( n  = 67) of the studies used only one of these approaches (expert or population judges).

Step 3—psychometric analysis

The most common analyses that have been used to assess construct validity are EFA (88.6%, n  = 93), CFA (72.3%, n  = 76), convergent validity (72.3%, n  = 76), and discriminant validity (56.2%, n  = 59). Most studies opted to combine EFA and CFA (65.7%, n  = 69). Only 4.7% ( n  = 5) failed to use factor analysis in their research. In relation to study reliability, internal consistency checks were used by all studies and test-retest reliability was the second most commonly used technique (22.8%, n  = 24).

Sample size in step 3 and number of items

Interestingly, 50.4% ( n  = 53) of the studies used sample sizes smaller than the rule of thumb, which is a minimum of 10 participants for each item in the scale. Regarding number of items, the majority of the studies (49.6%, n  = 52) lost more than 50% of the initial item pool during the validation process.

Table  2 summarizes and provides more details on our findings regarding the current practices in the scale development.

Main limitations reported in the scale development process

As result of this systematic review, we found ten main limitations commonly referenced in the scale development process: (1) sample characteristic limitations—cited by 81% of the studies, (2) methodological limitations—33.2%, (3) psychometric limitations—30.4%, (4) qualitative research limitations—5.6%, (5) missing data—2.8%, (6) social desirability bias—1.9%, (7) item limitations—1.9%, (8) brevity of the scale—1.9%, (9) difficulty controlling all variables—0.9%, and (10) lack of manual instructions—0.9%. Table  3 summarizes these findings.

This systematic review was primarily directed at identifying the published literature regarding current practices of the scale development. The results show a variety of practices that have been used to generate and assess items, both theoretically and psychometrically. We evaluated these current practices, considering three distinct steps (item generation, theoretical analysis, and psychometric analysis). We also considered the relationship between sample size and number of items, since this is considered an important methodological aspect to be evaluated during the scale development process. The results are discussed together with recommendations for best practices in future scale development research.

Current practices of the scale development process—findings and research implications

Regarding step 1, item generation, our results show that, although several studies used exclusively deductive methods (e.g., Henderson-King and Henderson-King 2005 ; Kim et al. 2011 ), the majority (e.g., Bakar and Mustaffa 2013 ; Uzunboylu and Ozdamli 2011 ) combined deductive and inductive methods, a combination consistent with the recommended strategy for the creation of new measures (DeVellis 2003 ). These findings, however, differ from previous critical reviews of scale development practices, which found that most of the reported studies used exclusively deductive methods (Hinkin 1995 ; Kapuscinski and Masters 2010 ; Ladhari 2010 ). This is particularly important since the quality of generated items depends on the way that the construct is defined. Failing to adequately define the conceptual domain of a construct causes several problems related to poor construct definition, leading to, for example, (a) confusion about what the construct does and does not refer to, including the similarities and differences between it and other constructs that already exist in the field, (b) indicators that may either be deficient or contaminated, and (c) invalid conclusions about relationships with other constructs (MacKenzie et al. 2011 ). Considering that item generation may be the most important part of the scale development process, future measures should be developed using the appropriate definition of the conceptual domain based on the combination of both deductive and inductive approaches.

Our results suggest that literature review was the most widely used deductive method (e.g., Bolton and Lane 2012 ; Henderson-King and Henderson-King 2005 ). This is consistent with the views of several other researchers who have systematically reviewed scales (Bastos et al. 2010 ; Ladhari 2010 ; Sveinbjornsdottir and Thorsteinsson 2008 ). Nevertheless, this finding differs from another study (Kapuscinski and Masters 2010 ) that found that the most common deductive strategies were reading works by spiritual leaders, theory written by psychologists, and discussion among authors. Literature review should be considered central for the enumeration of the constructs. It also serves to clarify the nature and variety of the target construct content. In addition, literature reviews help to identify existing measures that can be used as references to create new scales (Clark and Watson 1995 ; DeVellis 2003 ). In this sense, future research should consider the literature review as the initial and necessary deductive step foundational to building a new scale.

This review also highlights the fact that interviews and focus groups were the most widely used inductive methods (e.g., Lin and Hsieh 2011 ; Sharma 2010 ). Similar results were found in the systematic review by Kapuscinski and Masters ( 2010 ), Sveinbjornsdottir and Thorsteinsson ( 2008 ), and Ladhari ( 2010 ). These findings have particular relevance to future researchers, since they emphasize the importance of using methodological strategies that consider the opinions of the target population. Despite the fact that a panel of experts contributes widely to increasing the researchers’ confidence in the content validity of the new scale, it is important to also consider the most original and genuine information about the construct of interest, which can be best obtained through reports obtained from interviews and focus groups with the target population.

Related to step 2, theoretical analysis, the results of this review indicate that expert judges have been the most widely utilized tool for analyzing content validity (e.g., Uzunboylu and Ozdamli 2011 ; Zheng et al. 2010 ). Previous studies have also found expert opinion to be the most common qualitative method for the elimination of unsuitable items (Kapuscinski and Masters 2010 ; Ladhari 2010 ). In the literature review conducted by Hardesty and Bearden ( 2004 ), the authors highlighted the importance of these experts to carefully analyze the initial item pool. They suggested that any research using new, changed, or previously unexamined scale items, should at a minimum be judged by a panel of experts. However, the authors also point out the apparent lack of consistency in the literature in terms of how researchers use the opinions of expert judges in aiding the decision of whether or not to retain items for a scale. Given this inconsistency, the authors developed guidelines regarding the application of different decision rules to use for item retention. For example, the “sumscore decision rule,” defined as the total score for an item across all judges, is considered by the authors to be the most effective in predicting whether an item should be included in a scale and appears, therefore, to be a reasonable rule for researchers to employ.

Future research in developing scales should be concerned, not only with opinions from experts but also with the opinions of the target population. The results of this review show that only a minority of studies considered the review of the scales’ items by members of the target population (e.g., Uzunboylu and Ozdamli 2011 ; Zheng et al. 2010 ). In addition, a smaller minority combined the two approaches in the assessment of item content (e.g., Mahudin et al. 2012 ; Morgado et al. 2014 ). The limited use of target population opinions is a problem. A previous study of systematic scale development reviews found that the opinion of these people is the basis for content validity (Bastos et al. 2010 ). As highlighted by Clark and Watson ( 1995 ) and Malhotra ( 2004 ), it is essential for the new scale to undergo prior review by members of the target population. Pre-test or pilot study procedures make it possible to determine respondents’ opinions of, and reactions to, each item on the scale, enabling researchers to identify and eliminate potential problems in the scale before it is applied at large.

Another problem noted in this systematic review was that some studies failed to clearly report how they performed the theoretical analysis of the items (e.g., Glynn et al. 2015 ; Gottlieb et al. 2014 ). We hypothesized that the authors either did not perform this analysis or found it unimportant to record. Future research should consider this analysis, as well as all subsequent analyses, necessary and relevant for reporting.

Almost all studies (95.3%) reported using at least one type of factor analysis—EFA or CFA—in step 3, psychometric analysis (e.g., Sewitch et al. 2003 ; Tanimura et al. 2011 ). Clark and Watson ( 1995 ) consider that “unfortunately, many test developers are hesitant to use factor analysis, either because it requires a relatively large number of respondents or because it involves several perplexing decisions” (p. 17). They emphasized the importance of the researcher’s need to understand and apply this analysis, “it is important that test developers either learn about the technique or consult with a psychometrician during the scale development process” (Clark and Watson 1995 , p. 17). This question seems to have been almost overcome in recent studies, since the vast majority of the analyzed studies used the factor analysis method.

Among the studies than used factor analysis, the majority chose to use EFA (e.g., Bakar and Mustaffa 2013 ; Turker 2009 ). Similar to our findings, Bastos et al. ( 2010 ) and Ladhari ( 2010 ) found EFA to be the more commonly utilized construct validity method when compared to CFA. EFA has extensive value because it is considered to be effective in identifying the underlying latent variables or factors of a measure by exploring relationships among observed variables. However, it allows for more subjectivity in the decision-making process than many other statistical procedures, which can be considered a problem (Roberson et al. 2014 ).

For more consistent results on the psychometric indices of the new scale, DeVellis ( 2003 ) indicates the combined use of EFA and CFA, as was performed with most studies evaluated in this review. In CFA, the specific hypothesized factor structure proposed in EFA (including the correlations among the factors) is statistically evaluated. If the estimated model fits the data, then a researcher concludes that the factor structure replicates. If not, the modification indices are used to identify where constraints placed on the factor pattern are causing a misfit (Reise et al. 2000 ). Future studies should consider the combined use of EFA and CFA during the evaluation of construct validity of the new measure, and should also apply a combination of multiple fit indices (e.g., modification indices) in order to provide more consistent psychometric results.

After EFA and CFA, convergent validity was the preferred technique used in the vast majority of the studies included in this review (e.g., Brun et al. 2014 ; Cicero et al. 2010 ). This finding is consistent with prior research (Bastos et al. 2010 ). Convergent validity consists in examining whether a scale’s score is associated with the other variables and measures of the same construct to which it should be related. It is verified either by calculating the average variance extracted for each factor when the shared variance accounted for 0.50 or more of the total variance or by correlating their scales with a measure of overall quality (Ladhari 2010 ). In the sequence of convergent validity, the following methods were identified as favorites in the assessment of construct validity: discriminant validity (the extent to which the scale’s score does not correlate with unrelated constructs) (e.g., Coker et al. 2011 ), predictive/nomological validity (the extent to which the scores of one construct are empirically related to the scores of other conceptually related constructs) (e.g., Sharma 2010 ), criterion validity (the empirical association that the new scale has with a gold standard criterion concerned with the prediction of a certain behavior) (e.g., Tanimura et al. 2011 ), internal (signifies whether the study results and conclusions are valid for the study population), and external validity (generalizability of study) (e.g., Bolton and Lane 2012 ; Khorsan and Crawford 2014 ). Considering the importance of validity to ensure the quality of the collected data and the generalized potential of the new instrument, future studies should allow different ways to assess the validity of the new scale, thus increasing the psychometric rigor of the analysis.

With regard to reliability, all studies reported internal consistency statistics (Cronbach’s alpha) for all subscales and/or the final version of the full scale (e.g., Schlosser and McNaughton 2009 ; Sewitch et al. 2003 ). These findings are consistent with those of previous review studies (Bastos et al. 2010 ; Kapuscinski and Masters 2010 ). DeVellis ( 2003 ) explains that internal consistency is the most widely used measure of reliability. It is concerned with the homogeneity of the items within a scale. Given its importance, future studies should to consider alpha evaluation as a central point of measurement reliability, and yet, as much as possible, involve the assessment of internal consistency with other measures of reliability. In the sequence of internal consistency, the following methods were identified by this review: test-retest reliability (analysis of the temporal stability; items are applied on two separate occasions, and the scores could be correlated) (e.g., Forbush et al. 2013 ), item-total/inter-item correlation reliability (analysis of the correlation of each item with the total score of the scale or subscales/analysis of the correlation of each item with another item) (e.g., Rodrigues and Bastos 2012 ), split-half reliability (the scale is split in half and the first half of the items are compared to the second half) (e.g., Uzunboylu and Ozdamli 2011 ), and inter-judge reliability (analysis of the consistency between two different observers when they assess the same measure in the same individual) (e.g., Akter et al. 2013 ; DeVellis 2003 ; Nunnally 1967 ).

Regarding sample size in step 3 and number of items, a particularly noteworthy finding was that most studies utilized sample sizes smaller than the rule of thumb that the minimum required ratio should be 10:1 (e.g., Turker 2009 ; Zheng et al. 2010 ). DeVellis ( 2003 ) and Hair Junior et al. ( 2009 ) comment that the sample size should be as large as possible to ensure factor stability. The ‘observations to variables’ ratio is ideal at 15:1, or even 20:1. However, most of the studies included in this review failed to adopt this rule. Some studies looked for justification on evidence related to the effectiveness of much smaller observations to variables ratios. For example, Nagy et al. ( 2014 ) justified the small sample size used in their investigation based on the findings of Barrett and Kline ( 1981 ), concluding that the difference in ratios 1.25:1 and 31:1 was not a significant contributor to results obtained in the factor stability. Additionally, Arrindell and van der Ende ( 1985 ) concluded that ratios of 1.3:1 and 19.8:1 did not impact the factor stability. Although the rules of thumb vary enormously, ten participants to each item has widely been considered safe recommended (Sveinbjornsdottir and Thorsteinsson 2008 ).

Finally, several studies had their number final of items reduced by more than 50%. For example, Flight et al. ( 2011 ) developed an initial item pool composed of 122 items and finished the scale with only 43. Pommer et al. ( 2013 ) developed 391 initial items and finished with only 18. Our findings clearly indicate that a significant amount of items can get lost during the development of a new scale. These results are consistent with previous literature which states both that the initial number of items must be twice the desired number in the final scale, since, during the process of analysis of the items, many may be excluded for inadequacy (Nunnally 1967 ), and that the initial set of items should be three or four times more numerous than the number of items desired, as a good way to ensure internal consistency of the scale (DeVellis 2003 ). Future research should consider these issues and expect significant loss of items during the scale development process.

Ten main limitations reported in the scale development process—findings and research implications

In addition to identifying the current practices of the scale development process, this review also aims to assess the main limitations reported by the authors. Ten limitations were found, which will be discussed together with recommendations for best practices in future scale development research (Table  3 ).

Sample characteristic limitations

The above-mentioned limitations were recorded in the majority of the studies, in two main ways. The first and the most representative way was related to the sample type. Several studies used homogeneous sampling (e.g., Forbush et al. 2013 ; Morean et al. 2012 ), whereas others used convenience sampling (e.g., Coker et al. 2011 ; Flight et al. 2011 ). Both homogeneous and convenience samples were related to limitations of generalization. For example, Atkins and Kim ( 2012 ) pointed out that “the participants for all stages of the study were US consumers; therefore, this study cannot be generalized to other cultural contexts.” Or indeed, “convenience samples are weaknesses of this study, as they pose generalizability questions,” as highlighted by Blankson et al. ( 2012 ). Nunnally ( 1967 ) suggested that, to extend the generalizability of the new scale, sample diversification should be considered in terms of data collection, particularly in the psychometric evaluation step. Future studies should consider this suggestion, recruiting heterogeneous and truly random samples for the evaluation of construct validity and the reliability of the new measure.

The second way was related to small sample size. As previously described, most of the analyzed studies utilized sample sizes less than 10:1. Only some of the authors recognized this flaw. For example, Nagy et al. ( 2014 ) reported that “the sample size employed in conducting the exploratory factor analysis is another potential limitation of the study,” Rosenthal ( 2011 ) described, “the current study was limited by the relatively small nonprobability sample of university students,” and Ho and Lin ( 2010 ) recognized that “the respondent sample size was small.” Based in these results, we emphasize that future research should seek a larger sample size (minimum ratio of 10:1) to increase the credibility of the results and thus obtain a more exact outcome in the psychometric analysis.

Methodological limitations

Cross-sectional methods were the main methodological limitations reported by other studies (e.g., Schlosser and McNaughton 2009 ; Tombaugh et al. 2011 ). Data collected under a cross-sectional study design contains the typical limitation associated with this type of research methodology, namely inability to determine the causal relationship. If cross-sectional methods are used to estimate models whose parameters do in fact vary over time, the resulting estimation may fail to yield statistically valid results, fail to identify the true model parameters, and produce inefficient estimates (Bowen and Wiersema 1999 ). In this way, different authors (e.g., Akter et al. 2013 ; Boyar et al. 2014 ) recognized that employing instruments at one point in time limits the ability to assess causal relationships. With the goal of remediating these issues and gaining a deeper understanding of the construct of interest, different studies (e.g., Morean et al. 2012 ; Schlosser and McNaughton 2009 ) suggest conducting a longitudinal study during the scale development. Using the longitudinal studies in this process may also allow the assessment of the scale’s predictive validity, since longitudinal designs evaluate whether the proposed interpretation of test scores can predict outcomes of interest over time. Therefore, future studies should consider the longitudinal approach in the scale development, both to facilitate greater understanding of the analyzed variables and to assess the predictive validity.

Self-reporting methodologies were also cited as limitations in some studies (e.g., Fisher et al. 2014 ; Pan et al. 2013 ). Mahudin et al. ( 2012 ) clarified that the self-reporting nature of quantitative studies raises the possibility of participant bias, social desirability, demand characteristics, and response sets. Such possibilities may, in turn, affect the validity of the findings. We agree with the authors’ suggestion that future research may also incorporate other objective or independent measures to supplement the subjective evaluation of the variables studied in the development of the new scale and to improve the interpretation of findings.

In addition, web-based surveys were another methodological limitation reported in some studies (e.g., Kim et al. 2011 ; Reed et al. 2011 ). Although this particular method has time- and cost-saving elements for data collection, its limitations are also highlighted. Researchers have observed that important concerns include coverage bias (bias due to sampled individuals not having—or choosing not to access—the Internet) and nonresponse bias (bias due to participants of a survey differing from those who did not respond in terms of demographic or attitudinal variables) (Kim et al. 2011 ). Alternatives to minimize the problem in future research would be in-person surveys or survey interviews. Although more costly and more time consuming, these methods reduce problems related to concerns about confidentiality and the potential for coverage and nonresponse bias (Reed et al. 2011 ). Therefore, whenever possible, in-person surveys or survey interviews should be given priority in future research rather than web surveys.

Psychometric limitations

Consistent with previous reports (MacKenzie et al. 2011 ; Prados 2007 ), this systematic review found distinct psychometric limitations reported in the scale development process. The lack of a more robust demonstration of construct validity and/or reliability was the most often mentioned limitation in the majority of the analyzed studies. For example, Alvarado-Herrera et al. ( 2015 ) reported the lack of a more robust demonstration of the predictive validity whereas Kim et al. ( 2011 ) of the nomological validity. Caro and Garcia ( 2007 ) noted that the relationships of the scale with other constructs were not analyzed. Saxena et al. ( 2015 ) and Pan et al. ( 2013 ) described the lack of demonstrable temporal stability (e.g., test-retest reliability). Imprecise or incomplete psychometric procedures that are employed during scale development are likely to obscure the outcome. Therefore, it is necessary for future research to consider adverse consequences for the reliability and validity of any construct, caused by poor test-theoretical practices. Only through detailed information and explanation of the rationale for statistical choices can the new measures be shown to have sufficient psychometric adjustments (Sveinbjornsdottir and Thorsteinsson 2008 ).

Additionally, the inadequate choice of the instruments or variables to be correlated with the variable of interest was another psychometric limitation cited in some studies (e.g., Bakar and Mustaffa 2013 ; Tanimura et al. 2011 ). This kind of limitation directly affects the convergent validity, which is a problem since, as has already been shown in this review, this type of validity has been one of the most recurrent practices in scale development. One hypothesis for this limitation may be the lack of gold standard measures to assess similar constructs as those of a new scale. In such cases, a relatively recent study by Morgado et al. ( 2014 ) offers a valid alternative. The authors used information collected on sociodemographic questionnaires (e.g., level of education and intensity of physical activity) to correlate with the constructs of interest. Future researchers should seek support from the literature on the constructs that would be theoretically associated with the construct of interest, searching for alternatives in information collected on, for example, sociodemographic questionnaires, to assess the convergent validity of the new scale.

Another psychometric limitation reported in some studies was related to factor analysis. These limitations were identified in five main forms: (1) EFA and CFA were conducted using the data from the same sample (Zheng et al. 2010 )—when this occurs, good model fit in the CFA is expected, as a consequence, the added strength of the CFA in testing a hypothesized structure for a new data set based on theory or previous findings is lost (Khine 2008 ); (2) lack of CFA (Bolton and Lane 2012 )—if this happens, the researcher loses the possibility of assigning items to factors, testing the hypothesized structure of the data, and statistically comparing alternative models (Khine 2008 ); (3) a certain amount of subjectivity was necessary in identifying and labeling factors in EFA (Lombaerts et al. 2009 )—since a factor is qualitative, it is common practice to label each factor based on an interpretation of the variables loading most heavily on it; the problem is that these labels are subjective in nature, represent the authors’ interpretation, and can vary typically from 0.30 to 0.50 (Gottlieb et al. 2014 ; Khine 2008 ); (4) the initial unsatisfactory factor analysis output (Lombaerts et al. 2009 ); and (5) lack of a more robust CFA level (Jong et al. 2014 ) taken together—when the study result distances itself from statistical results expected for EFA (e.g., KMO, Bartlett test of sphericity) and/or CFA (e.g., CFI, GFI, RMSEA), it results in an important limitation, since the tested exploratory and theoretical models are not considered valid (Khine 2008 ). Taking these results, future studies should consider the use of separate samples for EFA and CFA, the combination of EFA and CFA, the definition of objective parameters to label factors, and about the consideration for unsatisfactory results of EFA and CFA, seeking alternatives to better fit the model.

Qualitative research limitations

This review also found reported limitations on the qualitative approach of the analyzed studies. The first limitation was related to the exclusive use of the deductive method to generate items. It is noteworthy that, although most of the studies included in this review used exclusively deductive methods to generate items, only two studies recognized this as a limitation (Coleman et al. 2011 ; Song et al. 2011 ). Both studies used only the literature review to generate and operationalize the initial item pool. The authors recognized the importance of this deductive method to theoretically operationalize the target construct, but they noted that, “for further research, more diverse views should be considered to reflect more comprehensive perspectives of human knowledge-creating behaviors to strengthen the validity of the developed scales” (Song et al. 2011 , p. 256) and, “a qualitative stage could have been used to generate additional items […]. This could also have reduced measurement error by using specific language the population used to communicate” (Coleman et al. 2011 ; p. 1069). Thus, the combination of deductive and inductive approaches (e.g., focus groups or interviews) in item generation is again suggested in future research.

In addition, it is also necessary that the researcher consider the quality of the reviewed literature. Napoli et al. ( 2014 , p. 1096) reported limitations related to the loss of a more robust literature review, suggesting that the scale developed in the study may have been incorrectly operationalized: “Yet some question remains as to whether cultural symbolism should form part of this scale. Perhaps the way in which the construct was initially conceptualized and operationalized was incorrect.” The incorrect operation of the construct compromises the psychometric results of scale and its applicability in future studies.

Another limitation involves the subjective analysis of the qualitative research. Fisher et al. ( 2014 , p. 488) pointed out that the qualitative methods (literature reviews and interviews) used to develop and conceptualize the construct were the main weaknesses of the study, “this research is limited by […] the nature of qualitative research in which the interpretations of one researcher may not reflect those of another.” The authors explained that, due to the potential for researcher bias when interpreting data, it has been recognized that credible results are difficult to achieve. Nevertheless, subjective analysis is the essence and nature of qualitative studies. Some precautions in future studies can be taken to rule out potential researcher bias, such as attempts at neutrality. This is not always possible, however, and this limitation will remain a common problem in any qualitative study.

In turn, Sewitch et al. ( 2003 , p. 260) reported that failure to formally assess content validity was a limitation. The reason given was budgetary constraints. It is worthwhile to remember that the content validity is an important step to ensure confidence in any inferences made using the final scale form. Therefore, it is necessarily required in any scale development process.

An additional limitation was reported by Lucas-Carrasco et al. ( 2011 ) in the recruitment of a larger number of interviewers, which may have affected the quality of the data collected. In order to minimize this limitation, the authors reported, “all interviewers had sufficient former education, received training on the study requirements, and were provided with a detailed guide” (p. 1223). Future studies planning the use of multiple interviewers should consider potential resulting bias.

Missing data

In connection, missing data was another issue reported by some studies included in this systematic review (e.g., Glynn et al. 2015 ; Ngorsuraches et al. 2007 ). Such limitations typically occur across different fields of scientific research. Missing data includes numbers that have been grouped, aggregated, rounded, censored, or truncated, resulting in partial loss of information (Schafer and Graham 2002 ). Collins et al. ( 2001 ) clarified that when researchers are confronted with missing data, they run an increased risk of reaching incorrect conclusions. This is because missing data may bias parameter estimates, inflate type I and type II error rates, and degrade the performance of confidence intervals. The authors also explained that, “because a loss of data is nearly always accompanied by a loss of information, missing values may dramatically reduce statistical power” (p. 330). Therefore, future researchers who wish to mitigate these risks during the scale development must pay close attention to the missing data aspect of the analysis and choose their strategy carefully.

Statistical methods to solve the problem of missing data have improved significantly, as demonstrated by Schafer and Graham ( 2002 ), although misconceptions still remain abundant. Several methods to deal with missing data were reviewed, issues raised, and advice offered for those that remain unresolved. Considering the fact that a more detailed discussion of the statistics dealing with missing data is beyond of the scope of this article, more details about missing data analysis can be found in Schafer and Graham ( 2002 ).

Social desirability bias

Another limitation reported in some studies (Bova et al. 2006 ; Ngorsuraches et al. 2007 ) and identified in this systematic review is social desirability bias. This type of bias is considered to be a systematic error in self-reporting measures resulting from the desire of respondents to avoid embarrassment and project a favorable image to others (Fisher 1993 ). According to King and Bruner ( 2000 ), social desirability bias is an important threat to the validity of research employing multi-item scales. Provision of socially desirable responses in self-reported data may lead to spurious correlations between variables, as well as the suppression or moderation of relationships between the constructs of interest. Thus, one aspect of scale validity, which should be of particular concern to researchers, is the potential threat of contamination due to social-desirability response bias. To remedy this problem, we agree with the authors that it is incumbent upon researchers to identify situations in which data may be systematically biased toward the respondents’ perceptions of what is socially acceptable, to determine the extent to which this represents contamination of the data, and to implement the most appropriate methods of control. Details on methods for identifying, testing for, and/or preventing social desirability bias are beyond the scope of this article, but can be found at King and Bruner ( 2000 ).

Item limitations

In comparison with at least one previous study (Prados 2007 ), our findings reflect some potential item limitations. Firstly, items that were ambiguous or difficult to answer were the main weaknesses reported by Gottlieb et al. ( 2014 ). On this issue, the literature dealing with the necessary caution in wording the items is extensive. For example, items must clearly define the problem being addressed, must be as simple as possible, express a single idea, and use common words that reflect the vocabulary level of the target population. Items should not be inductors or have alternative or underlying assumptions. They must be free of generalizations and estimates, and be written to ensure the variability of responses. In writing the items, the researcher should avoid using fashionable expressions and colloquialisms or other words or phrases that impair understanding for groups of varying ages, ethnicities, religions, or genders. Furthermore, the items should be organized properly. For example, the opening questions should be simple and interesting to win the trust of the subjects. The most delicate, complex, or dull questions should be asked at the end of the sequence (Clark and Watson 1995 ; Malhotra 2004 ; Pasquali 2010 ).

Furthermore, Cicero et al. ( 2010 ) reported that the main limitation of their study was the fact that none of the items were reverse-scored. Although some methodologists claim that reverse scoring is necessary to avoid acquiescence among participants, this advice should be taken with caution. There are reports that the reverse-scored items may be confusing to participants, that the opposite of a construct reverse-scored may be fundamentally different than the construct, that reverse-scored items tend to be the worst fitting items in factor analyses, or that the factor structure of scales includes a factor with straightforward wording compared to a reverse-scored factor (Cicero et al. 2010 ). Awareness of these issues is necessary for future researchers to choose between avoiding acquiescence among participants or preventing a number of other problems related to the use of reverse scores.

Brevity of the scale

Limitations on the scale size were also identified in this review. Studies by Negra and Mzoughi ( 2012 ) and Tombaugh et al. ( 2011 ) mentioned the short version of the scale as their main limitation. In both studies, the final version of the new scale included only five items. Generally, short scales are good, because they require less time from respondents. However, very short scales can in fact seriously compromise the reliability of the instrument (Raykov 2008 ). To the extent that the researcher removes items of the scale, the Cronbach’s alpha tends to decrease. It is valuable to remember that the minimum acceptable alpha should be at least 0.7, while an alpha value between 0.8 and 0.9 is considered ideal. Scales with many items tend to be more reliable, with higher alpha values (DeVellis 2003 ). In this context, future researchers should prioritize scales with enough items to keep the alpha within the acceptable range. Although many items may be lost during theoretical and psychometric analysis, an alternative already mentioned in this study would be to begin the initial item pool with at least twice the desired items of the final scale.

Difficulty controlling all variables

In addition to all limitations reported, Gottlieb et al. ( 2014 ) mentioned a common limitation in different research fields—the difficulty of controlling all the variables that could influence the central construct of the study. The authors reported that “it may be that there are other variables that influence visitors’ perception of trade show effectiveness that were not uncovered in the research” and suggest “future research might yield insights that are not provided here” (p. 104). The reported limitation calls attention to the importance of the first step—item generation—in the scale development process. A possible remedy to this issue would be to know the target construct in detail during the item generation, allowing for all possible and important variables to be investigated and controlled. However, this is not always possible. Even using inductive and deductive approaches to generate items (literature review and interview), the authors still reported that limitation. In this light, future researchers must use care in hypothesizing and testing potential variables that could be controlled during construction of the scale development process.

Lack of manual instructions

Finally, this review found a weakness reported on the loss of manualized instructions that regulate the data analysis. Saxena et al. ( 2015 , p. 492) pointed out that the initial version of the new scale “did not contain manualized instructions for raters, so it lacked objective anchor points for choosing specific ratings on many of its questions”. Therefore, an important detail that should have the attention of future researchers are instructions that determine the application methods of the new scale. Pasquali ( 2010 ) suggests that when drafting the instructions, the researcher should define the development of operational strategies that will enable the application of the instrument and the format in which it will be presented and decide both how the subject’s response will be given for each item and the way that the respondent should answer each item. The researcher should also define how the scale scores would be analyzed. In addition, the instructions need to be as short as possible without confusion to the subjects of the target population, should contain one or more examples of how the items should be answered, and should ensure that the subject is free of any related tension or anxiety.

Study limitations and strengths

This review itself is subject to some limitations that should be taken into consideration. First, during the selection of the articles included in the analysis, we may have missed some studies that could have been identified by using other terms related to “scale development.” This may have impacted our findings. However, application of this term alone was recommended by its widespread use by researchers in the area (Clark and Watson 1995 ; DeVellis 2003 ; Hinkin 1995 ; Nunnally 1967 ) and by the large number of publications identified with this descriptor in the period evaluated, as compared with those screened with correlates (e.g., “development of questionnaire” and “development of measure”). In the same way, we may also have missed numerous studies that, despite recording their weaknesses, did not have the search term “limitations” indexed in the analyzed databases. We could have reduced this limitation by also using the search term ‘weakness’ or a similar word for selection and inclusion of several other articles. However, a larger number of included studies would hinder the operationalization of our findings.

Second, particularly regarding analysis of items and reliability, we lost information about the basic theories that support the scale development process: classical test theory (CTT)—known as classical psychometry—and item response theory (IRT)—known as modern psychometry (PASQUALI 2010 ). Although it was beyond the scope of this article to examine these theories, information on the employability of one or the other could contribute to a deeper understanding of their main limitations. Future studies could focus on CTT and IRT, compare the applicability of both, and identify their main limitations in the scale development process.

Still, our review is current with studies published until September 2015. As new evidence emerges on current practices and limitations reported in the scale development process, revisions to this systematic review and practice guideline would be required in future studies.

Despite its weaknesses, the strengths of this study should be highlighted. First, this study reviews the updated and consistent literature on scale development practices to be applied in, not only a specific field of knowledge as carried out in most systematic review studies, but across various fields. With this variety of conceptions, we hope to assist future researchers in different areas of human and social sciences in making the most appropriate choice between strategies.

Second, this study differs from most studies of scale development revision, since it primarily considers the conceptions of the authors themselves about the main difficulties and mistakes made during the scale development process in their own studies. We hope to contribute to the efforts of future researchers, based on the knowledge of previous mistakes. While several weaknesses in scale development research were identified, specific recommendations for future research relevant to particular previously dimensions discussed were embedded within the appropriate sections throughout the article.

We observe that, although some weaknesses have been clearly identified in the scale development practices of many studies, only a few researchers recognized and recorded these limitations. This was evidenced in the large number of studies using exclusively deductive approaches to generate the initial item pool and the limited number of studies that recognized this as a limitation, or there were a large number of studies using smaller sample sizes than recommended in the literature for psychometric analysis and the limited number of studies that reported this issue as a limitation. Considering the observed distance between the limitation and its recognition, it is important that future researchers are comfortable with the detailed process of developing a new measure, especially as it pertains to avoiding theoretical and/or methodological mistakes, or at least, if they occur, to mention them as limitations.

Conclusions

In conclusion, the present research reviews numerous studies that both proposed current practices of the scale development process and also reported its main limitations. A variety of conceptions and methodological strategies and ten mains limitations were identified and discussed along with suggestions for future research. In this way, we believe that this paper makes important contributions to the literature, especially because it provides a comprehensive set of recommendations to increase the quality of future practices in the scale development process.

Aagja, J. P., & Garg, R. (2010). Measuring perceived service quality for public hospitals (PubHosQual) in the Indian context. International Journal of Pharmaceutical and Healthcare Marketing, 4 (10), 60–83. http://dx.doi.org/10.1108/17506121011036033 .

Article   Google Scholar  

Ahmad, N., Awan, M. U., Raouf, A., & Sparks, L. (2009). Development of a service quality scale for pharmaceutical supply chains. International Journal of Pharmaceutical and Healthcare Marketing, 3 (1), 26–45. http://dx.doi.org/10.1108/17506120910948494 .

Akter, S., D’Ambra, J., & Ray, P. (2013). Development and validation of an instrument to measure user perceived service quality of mHealth. Information and Management, 50 , 181–195. http://dx.doi.org/10.1016/j.im.2013.03.001 .

Alvarado-Herrera, A, Bigne, E, Aldas-Manzano, J, & Curras-Perez, R. (2015). A scale for measuring consumer perceptions of corporate social responsibility following the sustainable development paradigm. Journal of Business Ethics , 1-20. doi: http://dx.doi.org/10.1007/s10551-015-2654-9

Arias, M. R. M., Lloreda, M. J. H., & Lloreda, M. V. H. (2014). Psicometría . S.A.: Alianza Editorial

Armfield, J. M. (2010). Development and psychometric evaluation of the Index of Dental Anxiety and Fear (IDAF-4C + ). Psychological Assessment, 22 (2), 279–287. http://dx.doi.org/10.1037/a0018678 .

Article   PubMed   Google Scholar  

Arrindell, W. A., & van der Ende, J. (1985). An empirical-test of the utility of the observations-to-variables ratio in factor and components analysis. Applied Psychological Measurement, 9 (2), 165–178. http://dx.doi.org/10.1177/014662168500900205 .

Atkins, K. G., & Kim, Y. (2012). Smart shopping: conceptualization and measurement. International Journal of Retail and Distribution Management, 40 (5), 360–375. http://dx.doi.org/10.1108/09590551211222349 .

Bagdare, S., & Jain, R. (2013). Measuring retail customer experience. International Journal of Retail and Distribution Management, 41 (10), 790–804. http://dx.doi.org/10.1108/IJRDM-08-2012-0084 .

Bakar, H. A., & Mustaffa, C. S. (2013). Organizational communication in Malaysia organizations. Corporate Communications: An International Journal, 18 (1), 87–109. http://dx.doi.org/10.1108/13563281311294146 .

Barrett, P. T., & Kline, P. (1981). The observation to variable ratio in factor analysis. Personality Study and Group Behavior, 1 , 23–33.

Google Scholar  

Bastos, J. L., Celeste, R. K., Faerstein, E., & Barros, A. J. D. (2010). Racial discrimination and health: a systematic review of scales with a focus on their psychometric properties. Social Science and Medicine, 70 , 1091–1099. http://dx.doi.org/10.1016/j.socscimed.2009.12.20 .

Beaudreuil, J, Allard, A, Zerkak, D, Gerber, RA, Cappelleri, JC, Quintero, N, Lasbleiz, S, … Bardin, T. (2011). Unite’ Rhumatologique des Affections de la Main (URAM) Scale: development and validation of a tool to assess Dupuytren’s disease–specific disability. Arthritis Care & Research, 63 (10), 1448-1455. doi: http://dx.doi.org/10.1002/acr.20564

Bhattacherjee, A. (2002). Individual trust in online firms: scale development and initial test. Journal of Management Information Systems, 19 (1), 211–241. http://dx.doi.org/10.1080/07421222.2002.11045715 .

Blankson, C., Cheng, J. M., & Spears, N. (2007). Determinants of banks selection in USA, Taiwan and Ghana. International Journal of Bank Marketing, 25 (7), 469–489. http://dx.doi.org/10.1108/02652320710832621 .

Blankson, C., Paswan, A., & Boakye, K. G. (2012). College students’ consumption of credit cards. International Journal of Bank Marketing, 30 (7), 567–585. http://dx.doi.org/10.1108/02652321211274327 .

Bolton, D. L., & Lane, M. D. (2012). Individual entrepreneurial orientation: development of a measurement instrument. Education + Training, 54 (2/3), 219–233. http://dx.doi.org/10.1108/00400911211210314 .

Bova, C., Fennie, K. P., Watrous, E., Dieckhaus, K., & Williams, A. B. (2006). The health care relationship (HCR) trust scale: development and psychometric evaluation. Research in Nursing and Health, 29 , 477–488. http://dx.doi.org/10.1002/nur.20158 .

Bowen, H. P., & Wiersema, M. F. (1999). Matching method to paradigm in strategy research: limitations of cross-sectional analysis and some methodological alternatives. Strategic Management Journal, 20 , 625–636.

Boyar, S. L., Campbell, N. S., Mosley, D. C., Jr., & Carson, C. M. (2014). Development of a work/family social support measure. Journal of Managerial Psychology, 29 (7), 901–920. http://dx.doi.org/10.1108/JMP-06-2012-0189 .

Brock, J. K., & Zhou, Y. (2005). Organizational use of the internet. Internet Research, 15 (1), 67–87. http://dx.doi.org/10.1108/10662240510577077 .

Brun, I., Rajaobelina, L., & Ricard, L. (2014). Online relationship quality: scale development and initial testing. International Journal of Bank Marketing, 32 (1), 5–27. http://dx.doi.org/10.1108/IJBM-02-2013-0022 .

Butt, M. M., & Run, E. C. (2010). Private healthcare quality: applying a SERVQUAL model. International Journal of Health Care Quality Assurance, 23 (7), 658–673. http://dx.doi.org/10.1108/09526861011071580 .

Caro, L. M., & García, J. A. M. (2007). Measuring perceived service quality in urgent transport service. Journal of Retailing and Consumer Services, 14 , 60–72. http://dx.doi.org/10.1016/j.jretconser.2006.04.001 .

Chahal, H., & Kumari, N. (2012). Consumer perceived value. International Journal of Pharmaceutical and Healthcare Marketing, 6 (2), 167–190. http://dx.doi.org/10.1108/17506121211243086 .

Chen, H., Tian, Y., & Daugherty, P. J. (2009). Measuring process orientation. The International Journal of Logistics Management, 20 (2), 213–227. http://dx.doi.org/10.1108/09574090910981305 .

Choi, S. W., Victorson, D. E., Yount, S., Anton, S., & Cella, D. (2011). Development of a conceptual framework and calibrated item banks to measure patient-reported dyspnea severity and related functional limitations. Value in Health, 14 , 291–306. http://dx.doi.org/10.1016/j.jval.2010.06.001 .

Christophersen, T., & Konradt, U. (2012). Development and validation of a formative and a reflective measure for the assessment of online store usability. Behaviour and Information Technology, 31 (9), 839–857. http://dx.doi.org/10.1080/0144929X.2010.529165 .

Churchill, G. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16 (1), 64–73. http://dx.doi.org/10.2307/3150876 .

Cicero, D. C., Kerns, J. G., & McCarthy, D. M. (2010). The Aberrant Salience Inventory: a new measure of psychosis proneness. Psychological Assessment, 22 (3), 688–701. http://dx.doi.org/10.1037/a0019913 .

Clark, L. A., & Watson, D. (1995). Constructing validity: basic issues in objective scale development. Psychological Assessment, 7 (3), 309–319. http://dx.doi.org/10.1037/1040-3590.7.3.309 .

Coker, B. L. S., Ashill, N. J., & Hope, B. (2011). Measuring internet product purchase risk. European Journal of Marketing, 45 (7/8), 1130–1151. http://dx.doi.org/10.1108/03090561111137642 .

Coleman, D., Chernatony, L., & Christodoulides, G. (2011). B2B service brand identity: scale development and validation. Industrial Marketing Management, 40 , 1063–1071. http://dx.doi.org/10.1016/j.indmarman.2011.09.010 .

Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6 (4), 330–351. http://dx.doi.org/10.1037/1082-989X.6.4.330 .

Colwell, S. R., Aung, M., Kanetkar, V., & Holden, A. L. (2008). Toward a measure of service convenience: multiple-item scale development and empirical test. Journal of Services Marketing, 22 (2), 160–169. http://dx.doi.org/10.1108/08876040810862895 .

Cossette, S., Cara, C., Ricard, N., & Pepin, J. (2005). Assessing nurse–patient interactions from a caring perspective: report of the development and preliminary psychometric testing of the Caring Nurse–Patient Interactions Scale. International Journal of Nursing Studies, 42 , 673–686. http://dx.doi.org/10.1016/j.ijnurstu.2004.10.004 .

Dennis, R. S., & Bocarnea, M. (2005). Development of the servant leadership assessment instrument. Leadership and Organization Development Journal, 26 (8), 600–615. http://dx.doi.org/10.1108/01437730510633692 .

DeVellis, R. F. (2003). Scale development: theory and applications (2nd ed.). Newbury Park: Sage Publications.

Devlin, J. F., Roy, S. K., & Sekhon, H. (2014). Perceptions of fair treatment in financial services. European Journal of Marketing, 48 (7/8), 1315–1332. http://dx.doi.org/10.1108/EJM-08-2012-0469 .

Dunham, A., & Burt, C. (2014). Understanding employee knowledge: the development of an organizational memory scale. The Learning Organization, 21 (2), 126–145. http://dx.doi.org/10.1108/TLO-04-2011-0026 .

Edwards, J. R., Knight, D. K., Broome, K. M., & Flynn, P. M. (2010). The development and validation of a transformational leadership survey for substance use treatment programs. Substance Use and Misuse, 45 , 1279–1302. http://dx.doi.org/10.3109/10826081003682834 .

Article   PubMed   PubMed Central   Google Scholar  

Feuerstein, M., Nicholas, R. A., Huang, G. D., Haufler, A. J., Pransky, G., & Robertson, M. (2005). Workstyle: development of a measure of response to work in those with upper extremity pain. Journal of Occupational Rehabilitation, 15 (2), 87–104. http://dx.doi.org/10.1007/s10926-005-3420-0 .

Fisher, R. J. (1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20 (2), 303–315. http://dx.doi.org/10.1086/209351 .

Fisher, R., Maritz, A., & Lobo, A. (2014). Evaluating entrepreneurs’ perception of success. International Journal of Entrepreneurial Behavior and Research, 20 (5), 478–492. http://dx.doi.org/10.1108/IJEBR-10-2013-0157 .

Flight, R. L., D’Souza, G., & Allaway, A. W. (2011). Characteristics-based innovation adoption: scale and model validation. Journal of Product and Brand Management, 20 (5), 343–355. http://dx.doi.org/10.1108/10610421111157874 .

Forbush, KT, Wildes, JE, Pollack, LO, Dunbar, D, Luo, J, Patterson, P, Petruzzi, L, … Watson, D. (2013). Development and validation of the Eating Pathology Symptoms Inventory (EPSI). Psychological Assessment, 25 (3), 859-878. doi: http://dx.doi.org/10.1037/a0032639 .

Foster, J. D., McCain, J. L., Hibberts, M. F., Brunell, A. B., & Johnson, B. (2015). The grandiose narcissism scale: a global and facet-level measure of grandiose narcissism. Personality and Individual Differences, 73 , 12–16. http://dx.doi.org/10.1016/j.paid.2014.08.042 .

Franche, R., Corbière, M., Lee, H., Breslin, F. C., & Hepburn, G. (2007). The readiness for return-to-work (RRTW) scale: development and validation of a self-report staging scale in lost-time claimants with musculoskeletal disorders. Journal of Occupational Rehabilitation, 17 , 450–472. http://dx.doi.org/10.1007/s10926-007-9097-9 .

Gesten, E. L. (1976). A health resources inventory: the development of a measure of the personal and social competence of primary-grade children. Journal of Consulting and Clinical Psychology, 44 (5), 775–786. http://dx.doi.org/10.1037/0022-006X.44.5.775 .

Gibbons, C. J., Kenning, C., Coventry, P. A., Bee, P., Bundy, C., Fisher, L., & Bower, P. (2013). Development of a Multimorbidity Illness Perceptions Scale (MULTIPleS). PloS One, 8 (12), e81852. http://dx.doi.org/10.1371/journal.pone.0081852 .

Gligor, D. M., & Holcomb, M. (2014). The road to supply chain agility: an RBV perspective on the role of logistics capabilities. The International Journal of Logistics Management, 25 (1), 160–179. http://dx.doi.org/10.1108/IJLM-07-2012-0062 .

Glynn, N. W., Santanasto, A. J., Simonsick, E. M., Boudreau, R. M., Beach, S. R., Schulz, R., & Newman, A. B. (2015). The Pittsburgh fatigability scale for older adults: development and validation. Journal of American Geriatrics Society, 63 , 130–135. http://dx.doi.org/10.1111/jgs.13191 .

Gottlieb, U., Brown, M., & Ferrier, L. (2014). Consumer perceptions of trade show effectiveness. European Journal of Marketing, 48 (1/2), 89–107. http://dx.doi.org/10.1108/EJM-06-2011-0310 .

Hair Junior, J. F., Black, W. C., Babin, N. J., Anderson, R. E., & Tatham, R. L. (2009). Análise multivariada de dados (6ath ed.). São Paulo: Bookman.

Hall, M. A., Camacho, F., Dugan, E., & Balkrishnan, R. (2002). Trust in the medical profession: conceptual and measurement issues. Health Services Research, 37 (5), 1419–1439. http://dx.doi.org/10.1111/1475-6773.01070 .

Han, H., Back, K., & Kim, Y. (2011). A multidimensional scale of switching barriers in the full-service restaurant industry. Cornell Hospitality Quarterly, 52 (1), 54–-63. http://dx.doi.org/10.1177/1938965510389261 .

Hardesty, D. M., & Bearden, W. O. (2004). The use of expert judges in scale development Implications for improving face validity of measures of unobservable constructs. Journal of Business Research, 57 , 98–107. http://dx.doi.org/10.1016/S0148-2963(01)00295-8 .

Henderson-King, D., & Henderson-King, E. (2005). Acceptance of cosmetic surgery: scale development and validation. Body Image, 2 , 137–149. http://dx.doi.org/10.1016/j.bodyim.2005.03.003 .

Hernandez, J. M. C., & Santos, C. C. (2010). Development-based trust: proposing and validating a new trust measurement model for buyer-seller relationships. Brazilian Administration Review, 7 (2), 172–197. http://dx.doi.org/10.1590/S1807-76922010000200005 .

Hildebrandt, T., Langenbucher, J., & Schlundt, D. G. (2004). Muscularity concerns among men: development of attitudinal and perceptual measures. Body Image, 1 , 169–181. http://dx.doi.org/10.1016/j.bodyim.2004.01.001 .

Hilsenroth, M. J., Blagys, M. D., Ackerman, S. J., Bonge, D. R., & Blais, M. A. (2005). Measuring psychodynamic-interpersonal and cognitive-behavioral techniques: development of the comparative psychotherapy process scale. Psychotherapy: Theory, Research, Practice, Training, 42 (3), 340–356. http://dx.doi.org/10.1037/0033-3204.42.3.340 .

Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21 (5), 967–988. http://dx.doi.org/10.1177/014920639502100509 .

Ho, C. B., & Lin, W. (2010). Measuring the service quality of internet banking: scale development and validation. European Business Review, 22 (1), 5–24. http://dx.doi.org/10.1108/09555341011008981 .

Hutz, CS, Bandeira, DR, & Trentini, CM. (Org.). (2015). Psicometria . Porto Alegre, Artmed

Jong, N., Van Leeuwen, R. G. J., Hoekstra, H. A., & van der Zee, K. I. (2014). CRIQ: an innovative measure using comparison awareness to avoid self-presentation tactics. Journal of Vocational Behavior, 84 , 199–214. http://dx.doi.org/10.1016/j.jvb.2014.01.003 .

Kapuscinski, A. N., & Masters, K. S. (2010). The current status of measures of spirituality: a critical review of scale development. Psychology of Religion and Spirituality, 2 (4), 191–205. http://dx.doi.org/10.1037/a0020498 .

Khine, M. S. (2008). Knowing, knowledge and beliefs: epistemological studies across diverse cultures . New York: Springer.

Book   Google Scholar  

Khorsan, R, & Crawford, C. (2014). External validity and model validity: a conceptual approach for systematic review methodology. Evidence-Based Complementary and Alternative Medicine, 2014 , Article ID 694804, 12 pages. doi: http://dx.doi.org/10.1155/2014/694804

Kim, S., Cha, J., Knutson, B. J., & Beck, J. A. (2011). Development and testing of the Consumer Experience Index (CEI). Managing Service Quality: An International Journal, 21 (2), 112–132. http://dx.doi.org/10.1108/09604521111113429 .

Kim, D., Lee, Y., Lee, J., Nam, J. K., & Chung, Y. (2014). Development of Korean smartphone addiction proneness scale for youth. PloS One, 9 (5), e97920. http://dx.doi.org/10.1371/journal.pone.0097920 .

King, M. F., & Bruner, G. C. (2000). Social desirability bias: a neglected aspect of validity testing. Psychology and Marketing, 17 (2), 79–103. http://dx.doi.org/10.1002/(SICI)1520-6793(200002)17:2<79::AID-MAR2>3.0.CO;2-0 .

Kwon, W., & Lennon, S. J. (2011). Assessing college women’s associations of American specialty apparel brands. Journal of Fashion Marketing and Management: An International Journal, 15 (2), 242–256. http://dx.doi.org/10.1108/13612021111132663 .

Ladhari, R. (2010). Developing e-service quality scales: a literature review. Journal of Retailing and Consumer Services, 17 , 464–477. http://dx.doi.org/10.1016/j.jretconser.2010.06.003 .

Lin, J. C., & Hsieh, P. (2011). Assessing the self-service technology encounters: development and validation of SSTQUAL scale. Journal of Retailing, 87 (2), 194–206. http://dx.doi.org/10.1016/j.jretai.2011.02.006 .

Lombaerts, K., Backer, F., Engels, N., Van Braak, J., & Athanasou, J. (2009). Development of the self-regulated learning teacher belief scale. European Journal of Psychology of Education, 24 (1), 79–96. http://dx.doi.org/10.1007/BF03173476 .

Lucas-Carrasco, R., Eser, E., Hao, Y., McPherson, K. M., Green, A., & Kullmann, L. (2011). The quality of care and support (QOCS) for people with disability scale: development and psychometric properties. Research in Developmental Disabilities, 32 , 1212–1225. http://dx.doi.org/10.1016/j.ridd.2010.12.030 .

MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: integrating new and existing techniques. MIS Quarterly, 35 (2), 293–334.

Mahudin, N. D. M., Cox, T., & Griffiths, A. (2012). Measuring rail passenger crowding: scale development and psychometric properties. Transportation Research Part, F 15 , 38–51. http://dx.doi.org/10.1016/j.trf.2011.11.006 .

Malhotra, N. K. (2004). Pesquisa de marketing: Uma orientação aplicada (4ath ed.). Porto Alegre: Bookman.

Medina-Pradas, C., Navarro, J. B., López, S. R., Grau, A., & Obiols, J. E. (2011). Further development of a scale of perceived expressed emotion and its evaluation in a sample of patients with eating disorders. Psychiatry Research, 190 , 291–296. http://dx.doi.org/10.1016/j.psychres.2011.06.011 .

Meneses, J., Barrios, M., Bonillo, A., Cosculluela, A., Lozano, L. M., Turbany, J., & Valero, S. (2014). Psicometría . Barcelona: Editorial UOC.

Morean, M. E., Corbin, W. R., & Treat, T. A. (2012). The anticipated effects of alcohol scale: development and psychometric evaluation of a novel assessment tool for measuring alcohol expectancies. Psychological Assessment, 24 (4), 1008–1023. http://dx.doi.org/10.1037/a0028982 .

Morgado, F. F. R., Campana, A. N. N. B., & Tavares, M. C. G. C. F. (2014). Development and validation of the self-acceptance scale for persons with early blindness: the SAS-EB. PloS One, 9 (9), e106848. http://dx.doi.org/10.1371/journal.pone.0106848 .

Nagy, B. G., Blair, E. S., & Lohrke, F. T. (2014). Developing a scale to measure liabilities and assets of newness after start-up. International Entrepreneurship and Management Journal, 10 , 277–295. http://dx.doi.org/10.1007/s11365-012-0219-2 .

Napoli, J., Dickinson, S. J., Beverland, M. B., & Farrelly, F. (2014). Measuring consumer-based brand authenticity. Journal of Business Research, 67 , 1090–1098. http://dx.doi.org/10.1016/j.jbusres.2013.06.001 .

Negra, A., & Mzoughi, M. N. (2012). How wise are online procrastinators? A scale development. Internet Research, 22 (4), 426–442. http://dx.doi.org/10.1108/10662241211250971 .

Ngorsuraches, S., Lerkiatbundit, S., Li, S. C., Treesak, C., Sirithorn, R., & Korwiwattanakarn, M. (2007). Development and validation of the patient trust in community pharmacists (TRUST-Ph) scale: results from a study conducted in Thailand. Research in Social and Administrative Pharmacy, 4 , 272–283. http://dx.doi.org/10.1016/j.sapharm.2007.10.002 .

Nunnally, J. C. (1967). Psychometric theory . New York: McGraw Hill.

Oh, H. (2005). Measuring affective reactions to print apparel advertisements: a scale development. Journal of Fashion Marketing and Management: An International Journal, 9 (3), 283–305. http://dx.doi.org/10.1108/13612020510610426 .

Olaya, B, Marsà, F, Ochoa, S, Balanzá-Martínez, V, Barbeito, S, González-Pinto, A, … Haro, JM. (2012). Development of the insight scale for affective disorders (ISAD): modification from the scale to assess unawareness of mental disorder . Journal of Affective Disorders, 142 , 65-71. doi: http://dx.doi.org/10.1016/j.jad.2012.03.041 .

Omar, N. A., & Musa, R. (2011). Measuring service quality in retail loyalty programmes (LPSQual). International Journal of Retail and Distribution Management, 39 (10), 759–784. http://dx.doi.org/10.1108/09590551111162257 .

Pan, J., Wong, D. F. K., & Ye, S. (2013). Post-migration growth scale for Chinese international students: development and validation. Journal of Happiness Studies, 14 , 1639–1655. http://dx.doi.org/10.1007/s10902-012-9401-z .

Pasquali, L. (2010). Instrumentação psicológica: fundamentos e práticas . Porto Alegre: Artmed.

Patwardhan, H., & Balasubramanian, S. K. (2011). Brand romance: a complementary approach to explain emotional attachment toward brands. Journal of Product and Brand Management, 20 (4), 297–308. http://dx.doi.org/10.1108/10610421111148315 .

Pimentel, C. E., Gouveia, V. V., & Pessoa, V. S. (2007). Escala de Preferência Musical: construção e comprovação da sua estrutura fatorial. Psico-USF, 12 (2), 145–155.

Podsakoff, N. P., Podsakoff, P. M., MacKenzie, S. B., & Klinger, R. L. (2013). Are we really measuring what we say we’re measuring? Using video techniques to supplement traditional construct validation procedures. Journal of Applied Psychology, 98 (1), 99–113. http://dx.doi.org/10.1037/a0029570 .

Pommer, AM, Prins, L, van Ranst, D, Meijer, J, Hul, AV, Janssen, J, … Pop, VJM. (2013). Development and validity of the Patient-Centred COPD Questionnaire (PCQ). Journal of Psychosomatic Research, 75 , 563-571. doi: http://dx.doi.org/10.1016/j.jpsychores.2013.10.001

Prados, J. M. (2007). Development of a new scale of beliefs about the worry consequences. Annals of Psychology, 23 (2), 226–230.

Raykov, T. (2008). Alpha if item deleted: a note on loss of criterion validity in scale development if maximizing coefficient alpha. British Journal of Mathematical and Statistical Psychology, 61 , 275–285. http://dx.doi.org/10.1348/000711007X188520 .

Reed, L. L., Vidaver-Cohen, D., & Colwell, S. R. (2011). A new scale to measure executive servant leadership: development, analysis, and implications for research. Journal of Business Ethics, 101 , 415–434. http://dx.doi.org/10.1007/s10551-010-0729-1 .

Reise, S. P., Waller, N. G., & Comrey, A. L. (2000). Factor analysis and scale revision. Psychological Assessment, 12 (3), 287–297. http://dx.doi.org/10.1037//1040-3590.12.3.287 .

Rice, S. M., Fallon, B. J., Aucote, H. M., & Möller-Leimkühler, A. M. (2013). Development and preliminary validation of the male depression risk scale: Furthering the assessment of depression in men. Journal of Affective Disorders, 151 , 950–958. http://dx.doi.org/10.1016/j.jad.2013.08.013 .

Riedel, M., Spellmann, I., Schennach-Wolff, R., Obermeier, M., & Musil, R. (2011). The RSM-scale: a pilot study on a new specific scale for self- and observer-rated quality of life in patients with schizophrenia. Quality of Life Research, 20 , 263–272. http://dx.doi.org/10.1007/s11136-010-9744-z .

Roberson, R. B., III, Elliott, T. R., Chang, J. E., & Hill, J. N. (2014). Exploratory factor analysis in rehabilitation psychology: a content analysis. Rehabilitation Psychology, 59 (4), 429–438. http://dx.doi.org/10.1037/a0037899 .

Rodrigues, A. C. A., & Bastos, A. V. B. (2012). Organizational entrenchment: scale development and validation. Psicologia: Reflexão e Crítica, 25 (4), 688–700. http://dx.doi.org/10.1590/S0102-79722012000400008 .

Rodríguez, I., Kozusznik, M. W., & Peiró, J. M. (2013). Development and validation of the Valencia Eustress-Distress Appraisal Scale. International Journal of Stress Management, 20 (4), 279–308. http://dx.doi.org/10.1037/a0034330 .

Rosenthal, S. (2011). Measuring knowledge of indoor environmental hazards. Journal of Environmental Psychology, 31 , 137–146. http://dx.doi.org/10.1016/j.jenvp.2010.08.003 .

Saxena, S., Ayers, C. R., Dozier, M. E., & Maidment, K. M. (2015). The UCLA Hoarding Severity Scale: development and validation. Journal of Affective Disorders, 175 , 488–493. http://dx.doi.org/10.1016/j.jad.2015.01.030 .

Schafer, J. L., & Graham, J. W. (2002). Missing data: our view of the state of the Art. Psychological Methods, 7 (2), 147–177. http://dx.doi.org/10.1037//1082-989X.7.2.147 .

Schlosser, F. K., & McNaughton, R. B. (2009). Using the I-MARKOR scale to identify market-oriented individuals in the financial services sector. Journal of Services Marketing, 23 (4), 236–248. http://dx.doi.org/10.1108/08876040910965575 .

Sewitch, M. J., Abrahamowicz, M., Dobkin, P. L., & Tamblyn, R. (2003). Measuring differences between patients’ and physicians’ health perceptions: the patient–physician discordance scale. Journal of Behavioral Medicine, 26 (3), 245–263. http://dx.doi.org/10.1023/A:1023412604715 .

Sharma, P. (2010). Measuring personal cultural orientations: scale development and validation. Journal of the Academy of Marketing Science, 38 , 787–806. http://dx.doi.org/.1007/s11747-009-0184-7 .

Sharma, D., & Gassenheimer, J. B. (2009). Internet channel and perceived cannibalization. European Journal of Marketing, 43 (7/8), 1076–1091. http://dx.doi.org/10.1108/03090560910961524 .

Shawyer, F., Ratcliff, K., Mackinnon, A., Farhall, J., Hayes, S. C., & Copolov, D. (2007). The Voices Acceptance and Action Scale (VAAS): pilot data. Journal of Clinical Psychology, 63 (6), 593–606. http://dx.doi.org/10.1002/jclp.20366 .

Sin, L. Y. M., Tse, A. C. B., & Yim, F. H. K. (2005). CRM: conceptualization and scale development. European Journal of Marketing, 39 (11/12), 1264–1290. http://dx.doi.org/10.1108/03090560510623253 .

Sohn, D., & Choi, S. M. (2014). Measuring expected interactivity: scale development and validation. New Media and Society, 16 (5), 856–870. http://dx.doi.org/10.1177/1461444813495808 .

Song, J. H., Uhm, D., & Yoon, S. W. (2011). Organizational knowledge creation practice. Leadership and Organization Development Journal, 32 (3), 243–259. http://dx.doi.org/10.1108/01437731111123906 .

Staines, Z. (2013). Managing tacit investigative knowledge: measuring “investigative thinking styles”. Policing: An International Journal of Police Strategies and Management, 36 (3), 604–619. http://dx.doi.org/10.1108/PIJPSM-07-2012-0072 .

Sultan, P., & Wong, H. (2010). Performance-based service quality model: an empirical study on Japanese universities. Quality Assurance in Education, 18 (2), 126–143. http://dx.doi.org/10.1108/09684881011035349 .

Sveinbjornsdottir, S., & Thorsteinsson, E. B. (2008). Adolescent coping scales: a critical psychometric review. Scandinavian Journal of Psychology, 49 (6), 533–548. http://dx.doi.org/10.1111/j.1467-9450.2008.00669.x .

Swaid, S. I., & Wigand, R. T. (2009). Measuring the quality of E-Service: scale development and initial validation. Journal of Electronic Commerce Research, 10 (1), 13–28.

Tanimura, C., Morimoto, M., Hiramatsu, K., & Hagino, H. (2011). Difficulties in the daily life of patients with osteoarthritis of the knee: scale development and descriptive study. Journal of Clinical Nursing, 20 , 743–753. http://dx.doi.org/10.1111/j.1365-2702.2010.03536.x .

Taute, H. A., & Sierra, J. (2014). Brand tribalism: an anthropological perspective. Journal of Product and Brand Management, 23 (1), 2–15. http://dx.doi.org/10.1108/JPBM-06-2013-0340 .

Tombaugh, J. R., Mayfield, C., & Durand, R. (2011). Spiritual expression at work: exploring the active voice of workplace spirituality. International Journal of Organizational Analysis, 19 (2), 146–170. http://dx.doi.org/10.1108/19348831111135083 .

Turker, D. (2009). Measuring corporate social responsibility: a scale development study. Journal of Business Ethics, 85 , 411–427. http://dx.doi.org/10.1007/s10551-008-9780-6 .

Uzunboylu, H., & Ozdamli, F. (2011). Teacher perception for m-learning: scale development and teachers’ perceptions. Journal of Computer Assisted Learning, 27 , 544–556. http://dx.doi.org/10.1111/j.1365-2729.2011.00415.x .

Van der Gaag, M, Schütz, C, ten Napel, A, Landa, Y, Delespaul, P, Bak, M, … Hert, M. (2013). Development of the Davos Assessment of Cognitive Biases Scale (DACOBS). Schizophrenia Research, 144 , 63-71. doi: http://dx.doi.org/10.1016/j.schres.2012.12.010

Von Steinbüchel, N, Wilson, L, Gibbons, H, Hawthorne, G, Höfer, S, Schmidt, S, … Truelle, J. (2010). Journal of Neurotrauma, 27 , 1167-1185. doi: http://dx.doi.org/10.1089/neu.2009.1076

Voon, B. H., Abdullah, F., Lee, N., & Kueh, K. (2014). Developing a HospiSE scale for hospital service excellence. International Journal of Quality and Reliability Management, 31 (3), 261–280. http://dx.doi.org/10.1108/IJQRM-10-2012-0143 .

Walshe, M., Peach, R. K., & Miller, N. (2009). Dysarthria Impact Profile: development of a scale to measure psychosocial effects. International Journal of Language and Communication Disorders, 44 (5), 693–715. http://dx.doi.org/10.1080/13682820802317536 .

Wang, C. L., & Mowen, J. C. (1997). The separateness-connectedness self-schema: scale development and application to message construction. Psychology and Marketing, 14 (2), 185–207. http://dx.doi.org/10.1002/(SICI)1520-6793(199703)14:2<185::AID-MAR5>3.0.CO;2-9 .

Wepener, M., & Boshoff, C. (2015). An instrument to measure the customer-based corporate reputation of large service organizations. Journal of Services Marketing, 29 (3), 163–172. http://dx.doi.org/10.1108/JSM-01-2014-0026 .

Williams, Z., Ponder, N., & Autry, C. W. (2009). Supply chain security culture: measure development and validation. The International Journal of Logistics Management, 20 (2), 243–260. http://dx.doi.org/10.1108/09574090910981323 .

Wilson, N. L., & Holmvall, C. M. (2013). The development and validation of the incivility from customers scale. Journal of Occupational Health Psychology, 18 (3), 310–326. http://dx.doi.org/10.1037/a0032753 .

Yang, M., Weng, S., & Hsiao, P. (2014). Measuring blog service innovation in social media services. Internet Research, 24 (1), 110–128. http://dx.doi.org/10.1108/IntR-12-2012-0253 .

Zhang, X., & Hu, D. (2011). Farmer-buyer relationships in China: the effects of contracts, trust and market environment. China Agricultural Economic Review, 3 (1), 42–53. http://dx.doi.org/10.1108/17561371111103534 .

Zheng, J, You, L, Lou, T, Chen, N, Lai, D, Liang, Y, … Zhai, C. (2010). Development and psychometric evaluation of the dialysis patient-perceived exercise benefits and barriers scale. International Journal of Nursing Studies, 47 , 166-180. doi: http://dx.doi.org/10.1016/j.ijnurstu.2009.05.023

Download references

Authors’ contributions

FFRM is responsible for all parts of this manuscript, from its conception to the final writing. JFFM, CMN, ACSA and MECF participated in the data collection, analysis and interpretation of data and critical review of the manuscript. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interest.

Author information

Authors and affiliations.

Institute of Education, Universidade Federal Rural do Rio de Janeiro, BR-465, km 7, Seropédica, Rio de Janeiro, 23890-000, Brazil

Fabiane F. R. Morgado

Faculty of Psychology, Universidade Federal de Juiz de Fora, Rua José Lourenço Kelmer, s/n—Campus Universitário Bairro São Pedro, Juiz de Fora, Minas Gerais, 36036-900, Brazil

Juliana F. F. Meireles, Clara M. Neves & Maria E. C. Ferreira

Faculty of Physical Education of the Instituto Federal de Educação, Ciência e Tecnologia do Sudeste de Minas Gerais, Av. Luz Interior, n 360, Estrela Sul, Juiz de Fora, Minas Gerais, 36030-776, Brazil

Ana C. S. Amaral

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Fabiane F. R. Morgado .

Additional information

An erratum to this article is available at http://dx.doi.org/10.1186/s41155-017-0059-7 .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Morgado, F.F.R., Meireles, J.F.F., Neves, C.M. et al. Scale development: ten main limitations and recommendations to improve future research practices. Psicol. Refl. Crít. 30 , 3 (2018). https://doi.org/10.1186/s41155-016-0057-1

Download citation

Received : 03 August 2016

Accepted : 22 December 2016

Published : 25 January 2017

DOI : https://doi.org/10.1186/s41155-016-0057-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Measurement
• Psychometrics
• Reliability

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 02 April 2024

Impact factors are outdated, but new research assessments still fail scientists

• Kelly Cobey 0

Kelly Cobey is a scientist at the University of Ottawa Heart Institute, an associate professor in the University of Ottawa School of Epidemiology and Public Health and co-chair of the Declaration on Research Assessment.

You can also search for this author in PubMed   Google Scholar

You have full access to this article via your institution.

Last September, while completing a grant application, I faltered at a section labelled ‘summary of progress’. This section, written in a narrative style, was meant to tell reviewers about who I was and why I should be funded. Among other things, it needed to outline any family leave I’d taken; to spell out why my budget was reasonable, given my past funding; and to include any broad ‘activities, contributions and impacts’ that would support the application.

How could I sensibly combine an acknowledgement of two maternity leaves with a description of my engagement with open science and discuss why I was worthy of the funding I’d requested? There was no indication of the criteria reviewers would use to evaluate what I wrote. I was at a loss.

Bring PhD assessment into the twenty-first century

When my application was rejected in January, the reviewers didn’t comment on my narrative summary. Yet they did mention my publication record, part of the conventional academic CV that I was also required to submit. So I’m still none the wiser as to how the summary was judged — or if it was considered at all.

As co-chair of the Declaration On Research Assessment (DORA) — a global initiative that aims to improve how research is evaluated — I firmly believe in using narrative reflections for job applications, promotions and funding. Narratives make space for broad research impacts, from diversity, equity and inclusion efforts to educational outreach, which are hard to include in typical CVs. But I hear stories like mine time and again. The academic community is attempting, in good faith, to move away from narrow assessment metrics such as publications in high-impact journals. But institutes are struggling to create workable narrative assessments, and researchers struggling to write them .

The problem arises because new research assessment systems are not being planned and implemented properly. This must change. Researchers need explicit evaluation criteria that help them to write narratives by spelling out how different aspects of the text will be weighted and judged.

Research communities must be involved in designing these criteria. All too often, researchers tell me about assessment systems being imposed from the top down, with no consultation. This risks these new systems being no better than those they are replacing.

How to boost your research: take a sabbatical in policy

Assessments should be mission-driven and open to change over time. For example, if an institute wants to increase awareness and implementation of open science, its assessments of which researchers should be promoted could reward those who have undertaken relevant training or implemented practices such as data sharing. As open science becomes more mainstream, assessments could reduce the weight given to such practices.

The value of different research outputs will vary between fields, institutes and countries. Funding bodies in Canada, where I work, might favour grants that prioritize Indigenous engagement and perspectives in research — a key focus of diversity, equity and inclusion efforts in the Canadian scientific community. But the same will not apply in all countries.

Organizations must understand that reform can’t be done well on the cheap. They should invest in implementation scientists, who are trained to investigate the factors that stop new initiatives succeeding and find ways to overcome them. These experts can help to get input from the research community, and to bring broad perspectives together into a coherent assessment framework.

Some might argue that it would be better for cash-strapped research organizations to rework existing assessments to suit their needs rather than spend money on experts to develop a new one. Yes, sharing resources and experiences is often useful. But because each research community is unique, copying a template is unlikely to produce a useful assessment. DORA is creating tools to help. One is Reformscape (see go.nature.com/4ab8aky ) — an organized database of mini case studies that highlight progress in research reform, including policies and sample CVs that can be adapted for use in fresh settings. This will allow institutions to build on existing successes.

The postdoc experience is broken. Funders such as the NIH must help to reimagine it

Crucially, implementation scientists are also well placed to audit how a new system is doing, and to make iterative changes. No research evaluation system will work perfectly at first — organizations must commit sustained resources to monitoring and improving it.

The Luxembourg National Research Fund (FNR) shows the value of this iterative approach. In 2021, it began requesting a narrative CV for funding applications, rather than a CV made up of the usual list of affiliations and publications. Since then, it has been studying how well this system works. It has had mostly positive feedback, but researchers in some fields are less satisfied, and there is evidence that institutes aren’t providing all researchers with the guidance they need to complete the narrative CV. In response, the FNR is now investigating how to adapt the CV to better serve its communities.

Each institution has its own work to do, if academia is truly to reform research assessment. Those institutions that drag their feet are sending a message that they are prepared to continue supporting a flawed system that wastes research time and investment.

Nature 628 , 9 (2024)

doi: https://doi.org/10.1038/d41586-024-00899-8

Reprints and permissions

Competing Interests

K.C. is the co-chair of DORA (Declaration On Research Assessment) — this in an unpaid role.

Related Articles

• Scientific community

Three ways ChatGPT helps me in my academic writing

Career Column 08 APR 24

How we landed job interviews for professorships straight out of our PhD programmes

How two PhD students overcame the odds to snag tenure-track jobs

How can we make PhD training fit for the modern world? Broaden its philosophical foundations

Correspondence 02 APR 24

Ready or not, AI is coming to science education — and students have opinions

Career Feature 08 APR 24

Assistant Professor in Integrated Photonics

We offer you the chance to design a unique and autonomous research program, networking with specialists, students and entrepreneurs.

Gothenburg (Stad), Västra Götaland (SE)

Chalmers University of Technology

Postdoctoral Fellow (Aging, Metabolic stress, Lipid sensing, Brain Injury)

Seeking a Postdoctoral Fellow to apply advanced knowledge & skills to generate insights into aging, metabolic stress, lipid sensing, & brain Injury.

Dallas, Texas (US)

UT Southwestern Medical Center - Douglas Laboratory

High-Level Talents at the First Affiliated Hospital of Nanchang University

For clinical medicine and basic medicine; basic research of emerging inter-disciplines and medical big data.

Nanchang, Jiangxi, China

The First Affiliated Hospital of Nanchang University

POSTDOCTORAL Fellow -- DEPARTMENT OF Surgery – BIDMC, Harvard Medical School

The Division of Urologic Surgery in the Department of Surgery at Beth Israel Deaconess Medical Center and Harvard Medical School invites applicatio...

Boston, Massachusetts (US)

Director of Research

Applications are invited for the post of Director of Research at Cancer Institute (WIA), Chennai, India.

Chennai, Tamil Nadu (IN)

Cancer Institute (W.I.A)

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Reframing the Mental Load of Parenting

New research suggests that carrying this mental load may have some advantages..

Posted April 8, 2024 | Reviewed by Jessica Schrader

• A Parent's Role
• Find a family counsellor near me
• The mental load of parenting refers to the work of managing a family that no one sees.
• It is often discussed in negative terms, but there may be positive impacts.
• A recent study found that different types of invisible labor are associated with different outcomes.

Have you heard of the “mental load” or “invisible labor”? The mental load refers to the work associated with having children and managing a household and family that no one sees— remembering to schedule doctor’s appointments, planning family social activities and holidays, arranging appointments for house maintenance and repairs, booking travel plans, planning for meals and snacks, registering children for school and summer camp, and even reading posts like this one.

The mental load is often discussed in negative terms, focusing on the stress and unseen work that go into carrying this burden. And, of course, it is important that we recognize the immense burden of the mental load but in practice, how the mental load impacts us may be more complicated. New research even suggests that there may be some positive impacts associated with carrying the mental load. It goes without saying that we should all push for societal changes that will minimize the mental load that all parents carry and make sure neither parent carries an unequal share of this burden. But, in the meantime, recognizing some positive aspects of this load may ultimately help the burden to feel lighter.

This study was published in 2023 in the Journal of Business and Psychology and provided some important insights into how the mental load impacts well-being.

The researchers identified three types of tasks that contribute to the mental load:

• Managerial : planning, organizing, supervising, and scheduling.
• Emotional : the worry and concern that goes into setting goals for the family and maintaining everyone’s well-being, comforting your children when they are upset, tending to everyone’s emotional needs.
• Cognitive : anticipating the family’s needs and wants, making decisions for the family, researching different options, and remembering what needs to be done.

The Findings

Not surprisingly, the researchers found that women spend more time on all three types of invisible labor than men. This backs up previous research finding that about 88% of women report that they are mostly responsible for organizing the families’ schedules, about 78% of women indicate that they are mostly responsible for knowing their child’s teachers and administrators at school, about 76% are mostly responsible for maintaining order and routine in the home, and about 64% are mostly responsible for being aware of the children’s emotional needs.

The mental load matters because previous research has found that when women carry the bulk of the mental load they feel more overwhelmed as a parent and feel more exhausted. However, this new study helped to tease apart the different types of tasks contributing to mental load and the researchers found that different types of invisible labor are associated with different outcomes. The cognitive load was associated with greater family satisfaction (translation: being happier with your family) and improved job performance . The researchers also found that the managerial load was positively associated with work-family enrichment (translation: when your experience with your family enhances your work and your experience with work enhances your family life). However the emotional part of the mental load was associated with negative impacts, such as greater work-family conflict , exhaustion related both to job and family , sleep problems , and poorer job performance . This backs up previous research finding that ensuring a child’s well-being is particularly linked to distress and a feeling of emptiness in women.

Limitations

This study had some important limitations. First, the researchers developed a new measure to look at the mental load so further research is needed to know how valid this measure is. This study also relied on self-report and, of course, people may over- or under-estimate their contribution to the invisible load (maybe you even have a partner who does this). Because the load is invisible, it is inherently difficult to measure it objectively. Finally, this study is a correlation study, meaning that we don’t know whether invisible labor causes these positive and negative impacts or if it is simply associated with them. However, the researchers did control for personality characteristics to make sure that the type of person who takes on more of the invisible labor isn’t also more likely to experience these positive outcomes.

Overall Translation

I wanted to highlight this study because I think it helps to provide a different perspective on invisible labor—that it may not totally negative and may even positively contribute to parents’ lives in some ways. This may be because some aspects of the mental load help to give parents meaning and purpose.

This research (along with previous research) provides the following suggestions for the managing the mental load:

• Reframe the mental load: It can be easy to get angry and resentful about carrying the mental load for your family but think about the positive aspects of serving this role as well. Does it give you a sense of pride or accomplishment? Are there any parts of it that you enjoy? Does it allow you to make choices for your family that are important to you? Do you feel good about serving your family in this way?
• Seek help in being the default parent for the family’s emotional needs: This study suggests that managing the family’s emotional needs may be the most draining aspect of the mental load. This can include calming upset children, resolving sibling conflicts, and setting the emotional tone for the house. You shouldn’t have to do this alone— seek help for this role from your parenting partner or another person in your child’s life. You can also seek help from a mental health professional.
• Make invisible labor visible and work with your partner on sharing more of the mental load : Write out on a piece of paper every “invisible task” you do for your family. If you have a parenting partner, have them do the same. Make sure to include all mental and emotional tasks. Writing it out will help you to appreciate each other and work towards making the split more equitable.
• Evaluate each task of your mental load: How does it impact your life (including your happiness , life satisfaction, and work)? If a particular task only impacts you negatively, what changes can you make? Can your partner handle it or can you outsource it? Can you decide that it actually doesn’t need to be done?
• Examine your own beliefs that have caused you to take on more of the mental load : Do you believe it is your “job” because your partner makes more money? Do you believe that is what a “good mother” or “good father” does? Challenge these beliefs if they are irrational or don't make sense.

Cara Goodwin, Ph.D., is a licensed clinical psychologist who specializes in translating scientific research into information that is useful, accurate, and relevant for parents.

• Find a Therapist
• Find a Treatment Center
• Find a Support Group
• International
• New Zealand
• South Africa
• Switzerland
• Asperger's
• Bipolar Disorder
• Chronic Pain
• Eating Disorders
• Passive Aggression
• Personality
• Goal Setting
• Positive Psychology
• Stopping Smoking
• Low Sexual Desire
• Relationships
• Child Development
• Therapy Center NEW
• Diagnosis Dictionary
• Types of Therapy

Understanding what emotional intelligence looks like and the steps needed to improve it could light a path to a more emotionally adept world.

• Coronavirus Disease 2019
• Affective Forecasting
• Neuroscience