how to use references in a research paper

• Privacy Policy

Buy Me a Coffee

Home » References in Research – Types, Examples and Writing Guide

References in Research – Types, Examples and Writing Guide

Table of Contents

References in Research

Definition:

References in research are a list of sources that a researcher has consulted or cited while conducting their study. They are an essential component of any academic work, including research papers, theses, dissertations, and other scholarly publications.

Types of References

There are several types of references used in research, and the type of reference depends on the source of information being cited. The most common types of references include:

References to books typically include the author’s name, title of the book, publisher, publication date, and place of publication.

Example: Smith, J. (2018). The Art of Writing. Penguin Books.

Journal Articles

References to journal articles usually include the author’s name, title of the article, name of the journal, volume and issue number, page numbers, and publication date.

Example: Johnson, T. (2021). The Impact of Social Media on Mental Health. Journal of Psychology, 32(4), 87-94.

Web sources

References to web sources should include the author or organization responsible for the content, the title of the page, the URL, and the date accessed.

Example: World Health Organization. (2020). Coronavirus disease (COVID-19) advice for the public. Retrieved from https://www.who.int/emergencies/disease/novel-coronavirus-2019/advice-for-public

Conference Proceedings

References to conference proceedings should include the author’s name, title of the paper, name of the conference, location of the conference, date of the conference, and page numbers.

Example: Chen, S., & Li, J. (2019). The Future of AI in Education. Proceedings of the International Conference on Educational Technology, Beijing, China, July 15-17, pp. 67-78.

References to reports typically include the author or organization responsible for the report, title of the report, publication date, and publisher.

Example: United Nations. (2020). The Sustainable Development Goals Report. United Nations.

Formats of References

Some common Formates of References with their examples are as follows:

APA (American Psychological Association) Style

The APA (American Psychological Association) Style has specific guidelines for formatting references used in academic papers, articles, and books. Here are the different reference formats in APA style with examples:

Author, A. A. (Year of publication). Title of book. Publisher.

Example : Smith, J. K. (2005). The psychology of social interaction. Wiley-Blackwell.

Journal Article

Author, A. A., Author, B. B., & Author, C. C. (Year of publication). Title of article. Title of Journal, volume number(issue number), page numbers.

Example : Brown, L. M., Keating, J. G., & Jones, S. M. (2012). The role of social support in coping with stress among African American adolescents. Journal of Research on Adolescence, 22(1), 218-233.

Author, A. A. (Year of publication or last update). Title of page. Website name. URL.

Example : Centers for Disease Control and Prevention. (2020, December 11). COVID-19: How to protect yourself and others. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html

Magazine article

Author, A. A. (Year, Month Day of publication). Title of article. Title of Magazine, volume number(issue number), page numbers.

Example : Smith, M. (2019, March 11). The power of positive thinking. Psychology Today, 52(3), 60-65.

Newspaper article:

Author, A. A. (Year, Month Day of publication). Title of article. Title of Newspaper, page numbers.

Example: Johnson, B. (2021, February 15). New study shows benefits of exercise on mental health. The New York Times, A8.

Edited book

Editor, E. E. (Ed.). (Year of publication). Title of book. Publisher.

Example : Thompson, J. P. (Ed.). (2014). Social work in the 21st century. Sage Publications.

Chapter in an edited book:

Author, A. A. (Year of publication). Title of chapter. In E. E. Editor (Ed.), Title of book (pp. page numbers). Publisher.

Example : Johnson, K. S. (2018). The future of social work: Challenges and opportunities. In J. P. Thompson (Ed.), Social work in the 21st century (pp. 105-118). Sage Publications.

MLA (Modern Language Association) Style

The MLA (Modern Language Association) Style is a widely used style for writing academic papers and essays in the humanities. Here are the different reference formats in MLA style:

Author’s Last name, First name. Title of Book. Publisher, Publication year.

Example : Smith, John. The Psychology of Social Interaction. Wiley-Blackwell, 2005.

Journal article

Author’s Last name, First name. “Title of Article.” Title of Journal, volume number, issue number, Publication year, page numbers.

Example : Brown, Laura M., et al. “The Role of Social Support in Coping with Stress among African American Adolescents.” Journal of Research on Adolescence, vol. 22, no. 1, 2012, pp. 218-233.

Author’s Last name, First name. “Title of Webpage.” Website Name, Publication date, URL.

Example : Centers for Disease Control and Prevention. “COVID-19: How to Protect Yourself and Others.” CDC, 11 Dec. 2020, https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html.

Author’s Last name, First name. “Title of Article.” Title of Magazine, Publication date, page numbers.

Example : Smith, Mary. “The Power of Positive Thinking.” Psychology Today, Mar. 2019, pp. 60-65.

Newspaper article

Author’s Last name, First name. “Title of Article.” Title of Newspaper, Publication date, page numbers.

Example : Johnson, Bob. “New Study Shows Benefits of Exercise on Mental Health.” The New York Times, 15 Feb. 2021, p. A8.

Editor’s Last name, First name, editor. Title of Book. Publisher, Publication year.

Example : Thompson, John P., editor. Social Work in the 21st Century. Sage Publications, 2014.

Chapter in an edited book

Author’s Last name, First name. “Title of Chapter.” Title of Book, edited by Editor’s First Name Last name, Publisher, Publication year, page numbers.

Example : Johnson, Karen S. “The Future of Social Work: Challenges and Opportunities.” Social Work in the 21st Century, edited by John P. Thompson, Sage Publications, 2014, pp. 105-118.

Chicago Manual of Style

The Chicago Manual of Style is a widely used style for writing academic papers, dissertations, and books in the humanities and social sciences. Here are the different reference formats in Chicago style:

Example : Smith, John K. The Psychology of Social Interaction. Wiley-Blackwell, 2005.

Author’s Last name, First name. “Title of Article.” Title of Journal volume number, no. issue number (Publication year): page numbers.

Example : Brown, Laura M., John G. Keating, and Sarah M. Jones. “The Role of Social Support in Coping with Stress among African American Adolescents.” Journal of Research on Adolescence 22, no. 1 (2012): 218-233.

Author’s Last name, First name. “Title of Webpage.” Website Name. Publication date. URL.

Example : Centers for Disease Control and Prevention. “COVID-19: How to Protect Yourself and Others.” CDC. December 11, 2020. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html.

Author’s Last name, First name. “Title of Article.” Title of Magazine, Publication date.

Example : Smith, Mary. “The Power of Positive Thinking.” Psychology Today, March 2019.

Author’s Last name, First name. “Title of Article.” Title of Newspaper, Publication date.

Example : Johnson, Bob. “New Study Shows Benefits of Exercise on Mental Health.” The New York Times, February 15, 2021.

Example : Thompson, John P., ed. Social Work in the 21st Century. Sage Publications, 2014.

Author’s Last name, First name. “Title of Chapter.” In Title of Book, edited by Editor’s First Name Last Name, page numbers. Publisher, Publication year.

Example : Johnson, Karen S. “The Future of Social Work: Challenges and Opportunities.” In Social Work in the 21st Century, edited by John P. Thompson, 105-118. Sage Publications, 2014.

Harvard Style

The Harvard Style, also known as the Author-Date System, is a widely used style for writing academic papers and essays in the social sciences. Here are the different reference formats in Harvard Style:

Author’s Last name, First name. Year of publication. Title of Book. Place of publication: Publisher.

Example : Smith, John. 2005. The Psychology of Social Interaction. Oxford: Wiley-Blackwell.

Author’s Last name, First name. Year of publication. “Title of Article.” Title of Journal volume number (issue number): page numbers.

Example: Brown, Laura M., John G. Keating, and Sarah M. Jones. 2012. “The Role of Social Support in Coping with Stress among African American Adolescents.” Journal of Research on Adolescence 22 (1): 218-233.

Author’s Last name, First name. Year of publication. “Title of Webpage.” Website Name. URL. Accessed date.

Example : Centers for Disease Control and Prevention. 2020. “COVID-19: How to Protect Yourself and Others.” CDC. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html. Accessed April 1, 2023.

Author’s Last name, First name. Year of publication. “Title of Article.” Title of Magazine, month and date of publication.

Example : Smith, Mary. 2019. “The Power of Positive Thinking.” Psychology Today, March 2019.

Author’s Last name, First name. Year of publication. “Title of Article.” Title of Newspaper, month and date of publication.

Example : Johnson, Bob. 2021. “New Study Shows Benefits of Exercise on Mental Health.” The New York Times, February 15, 2021.

Editor’s Last name, First name, ed. Year of publication. Title of Book. Place of publication: Publisher.

Example : Thompson, John P., ed. 2014. Social Work in the 21st Century. Thousand Oaks, CA: Sage Publications.

Author’s Last name, First name. Year of publication. “Title of Chapter.” In Title of Book, edited by Editor’s First Name Last Name, page numbers. Place of publication: Publisher.

Example : Johnson, Karen S. 2014. “The Future of Social Work: Challenges and Opportunities.” In Social Work in the 21st Century, edited by John P. Thompson, 105-118. Thousand Oaks, CA: Sage Publications.

Vancouver Style

The Vancouver Style, also known as the Uniform Requirements for Manuscripts Submitted to Biomedical Journals, is a widely used style for writing academic papers in the biomedical sciences. Here are the different reference formats in Vancouver Style:

Author’s Last name, First name. Title of Book. Edition number. Place of publication: Publisher; Year of publication.

Example : Smith, John K. The Psychology of Social Interaction. 2nd ed. Oxford: Wiley-Blackwell; 2005.

Author’s Last name, First name. Title of Article. Abbreviated Journal Title. Year of publication; volume number(issue number):page numbers.

Example : Brown LM, Keating JG, Jones SM. The Role of Social Support in Coping with Stress among African American Adolescents. J Res Adolesc. 2012;22(1):218-233.

Author’s Last name, First name. Title of Webpage. Website Name [Internet]. Publication date. [cited date]. Available from: URL.

Example : Centers for Disease Control and Prevention. COVID-19: How to Protect Yourself and Others [Internet]. 2020 Dec 11. [cited 2023 Apr 1]. Available from: https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html.

Author’s Last name, First name. Title of Article. Title of Magazine. Year of publication; month and day of publication:page numbers.

Example : Smith M. The Power of Positive Thinking. Psychology Today. 2019 Mar 1:32-35.

Author’s Last name, First name. Title of Article. Title of Newspaper. Year of publication; month and day of publication:page numbers.

Example : Johnson B. New Study Shows Benefits of Exercise on Mental Health. The New York Times. 2021 Feb 15:A4.

Editor’s Last name, First name, editor. Title of Book. Edition number. Place of publication: Publisher; Year of publication.

Example: Thompson JP, editor. Social Work in the 21st Century. 1st ed. Thousand Oaks, CA: Sage Publications; 2014.

Author’s Last name, First name. Title of Chapter. In: Editor’s Last name, First name, editor. Title of Book. Edition number. Place of publication: Publisher; Year of publication. page numbers.

Example : Johnson KS. The Future of Social Work: Challenges and Opportunities. In: Thompson JP, editor. Social Work in the 21st Century. 1st ed. Thousand Oaks, CA: Sage Publications; 2014. p. 105-118.

Turabian Style

Turabian style is a variation of the Chicago style used in academic writing, particularly in the fields of history and humanities. Here are the different reference formats in Turabian style:

Author’s Last name, First name. Title of Book. Place of publication: Publisher, Year of publication.

Example : Smith, John K. The Psychology of Social Interaction. Oxford: Wiley-Blackwell, 2005.

Author’s Last name, First name. “Title of Article.” Title of Journal volume number, no. issue number (Year of publication): page numbers.

Example : Brown, LM, Keating, JG, Jones, SM. “The Role of Social Support in Coping with Stress among African American Adolescents.” J Res Adolesc 22, no. 1 (2012): 218-233.

Author’s Last name, First name. “Title of Webpage.” Name of Website. Publication date. Accessed date. URL.

Example : Centers for Disease Control and Prevention. “COVID-19: How to Protect Yourself and Others.” CDC. December 11, 2020. Accessed April 1, 2023. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html.

Author’s Last name, First name. “Title of Article.” Title of Magazine, Month Day, Year of publication, page numbers.

Example : Smith, M. “The Power of Positive Thinking.” Psychology Today, March 1, 2019, 32-35.

Author’s Last name, First name. “Title of Article.” Title of Newspaper, Month Day, Year of publication.

Example : Johnson, B. “New Study Shows Benefits of Exercise on Mental Health.” The New York Times, February 15, 2021.

Editor’s Last name, First name, ed. Title of Book. Place of publication: Publisher, Year of publication.

Example : Thompson, JP, ed. Social Work in the 21st Century. Thousand Oaks, CA: Sage Publications, 2014.

Author’s Last name, First name. “Title of Chapter.” In Title of Book, edited by Editor’s Last name, First name, page numbers. Place of publication: Publisher, Year of publication.

Example : Johnson, KS. “The Future of Social Work: Challenges and Opportunities.” In Social Work in the 21st Century, edited by Thompson, JP, 105-118. Thousand Oaks, CA: Sage Publications, 2014.

IEEE (Institute of Electrical and Electronics Engineers) Style

IEEE (Institute of Electrical and Electronics Engineers) style is commonly used in engineering, computer science, and other technical fields. Here are the different reference formats in IEEE style:

Author’s Last name, First name. Book Title. Place of Publication: Publisher, Year of publication.

Example : Oppenheim, A. V., & Schafer, R. W. Discrete-Time Signal Processing. Upper Saddle River, NJ: Prentice Hall, 2010.

Author’s Last name, First name. “Title of Article.” Abbreviated Journal Title, vol. number, no. issue number, pp. page numbers, Month year of publication.

Example: Shannon, C. E. “A Mathematical Theory of Communication.” Bell System Technical Journal, vol. 27, no. 3, pp. 379-423, July 1948.

Conference paper

Author’s Last name, First name. “Title of Paper.” In Title of Conference Proceedings, Place of Conference, Date of Conference, pp. page numbers, Year of publication.

Example: Gupta, S., & Kumar, P. “An Improved System of Linear Discriminant Analysis for Face Recognition.” In Proceedings of the 2011 International Conference on Computer Science and Network Technology, Harbin, China, Dec. 2011, pp. 144-147.

Author’s Last name, First name. “Title of Webpage.” Name of Website. Date of publication or last update. Accessed date. URL.

Example : National Aeronautics and Space Administration. “Apollo 11.” NASA. July 20, 1969. Accessed April 1, 2023. https://www.nasa.gov/mission_pages/apollo/apollo11.html.

Technical report

Author’s Last name, First name. “Title of Report.” Name of Institution or Organization, Report number, Year of publication.

Example : Smith, J. R. “Development of a New Solar Panel Technology.” National Renewable Energy Laboratory, NREL/TP-6A20-51645, 2011.

Author’s Last name, First name. “Title of Patent.” Patent number, Issue date.

Example : Suzuki, H. “Method of Producing Carbon Nanotubes.” US Patent 7,151,019, December 19, 2006.

Standard Title. Standard number, Publication date.

Example : IEEE Standard for Floating-Point Arithmetic. IEEE Std 754-2008, August 29, 2008

ACS (American Chemical Society) Style

ACS (American Chemical Society) style is commonly used in chemistry and related fields. Here are the different reference formats in ACS style:

Author’s Last name, First name; Author’s Last name, First name. Title of Article. Abbreviated Journal Title Year, Volume, Page Numbers.

Example : Wang, Y.; Zhao, X.; Cui, Y.; Ma, Y. Facile Preparation of Fe3O4/graphene Composites Using a Hydrothermal Method for High-Performance Lithium Ion Batteries. ACS Appl. Mater. Interfaces 2012, 4, 2715-2721.

Author’s Last name, First name. Book Title; Publisher: Place of Publication, Year of Publication.

Example : Carey, F. A. Organic Chemistry; McGraw-Hill: New York, 2008.

Author’s Last name, First name. Chapter Title. In Book Title; Editor’s Last name, First name, Ed.; Publisher: Place of Publication, Year of Publication; Volume number, Chapter number, Page Numbers.

Example : Grossman, R. B. Analytical Chemistry of Aerosols. In Aerosol Measurement: Principles, Techniques, and Applications; Baron, P. A.; Willeke, K., Eds.; Wiley-Interscience: New York, 2001; Chapter 10, pp 395-424.

Author’s Last name, First name. Title of Webpage. Website Name, URL (accessed date).

Example : National Institute of Standards and Technology. Atomic Spectra Database. https://www.nist.gov/pml/atomic-spectra-database (accessed April 1, 2023).

Author’s Last name, First name. Patent Number. Patent Date.

Example : Liu, Y.; Huang, H.; Chen, H.; Zhang, W. US Patent 9,999,999, December 31, 2022.

Author’s Last name, First name; Author’s Last name, First name. Title of Article. In Title of Conference Proceedings, Publisher: Place of Publication, Year of Publication; Volume Number, Page Numbers.

Example : Jia, H.; Xu, S.; Wu, Y.; Wu, Z.; Tang, Y.; Huang, X. Fast Adsorption of Organic Pollutants by Graphene Oxide. In Proceedings of the 15th International Conference on Environmental Science and Technology, American Chemical Society: Washington, DC, 2017; Volume 1, pp 223-228.

AMA (American Medical Association) Style

AMA (American Medical Association) style is commonly used in medical and scientific fields. Here are the different reference formats in AMA style:

Author’s Last name, First name. Article Title. Journal Abbreviation. Year; Volume(Issue):Page Numbers.

Example : Jones, R. A.; Smith, B. C. The Role of Vitamin D in Maintaining Bone Health. JAMA. 2019;321(17):1765-1773.

Author’s Last name, First name. Book Title. Edition number. Place of Publication: Publisher; Year.

Example : Guyton, A. C.; Hall, J. E. Textbook of Medical Physiology. 13th ed. Philadelphia, PA: Saunders; 2015.

Author’s Last name, First name. Chapter Title. In: Editor’s Last name, First name, ed. Book Title. Edition number. Place of Publication: Publisher; Year: Page Numbers.

Example: Rajakumar, K. Vitamin D and Bone Health. In: Holick, M. F., ed. Vitamin D: Physiology, Molecular Biology, and Clinical Applications. 2nd ed. New York, NY: Springer; 2010:211-222.

Author’s Last name, First name. Webpage Title. Website Name. URL. Published date. Updated date. Accessed date.

Example : National Cancer Institute. Breast Cancer Prevention (PDQ®)–Patient Version. National Cancer Institute. https://www.cancer.gov/types/breast/patient/breast-prevention-pdq. Published October 11, 2022. Accessed April 1, 2023.

Author’s Last name, First name. Conference presentation title. In: Conference Title; Conference Date; Place of Conference.

Example : Smith, J. R. Vitamin D and Bone Health: A Meta-Analysis. In: Proceedings of the Annual Meeting of the American Society for Bone and Mineral Research; September 20-23, 2022; San Diego, CA.

Thesis or dissertation

Author’s Last name, First name. Title of Thesis or Dissertation. Degree level [Doctoral dissertation or Master’s thesis]. University Name; Year.

Example : Wilson, S. A. The Effects of Vitamin D Supplementation on Bone Health in Postmenopausal Women [Doctoral dissertation]. University of California, Los Angeles; 2018.

ASCE (American Society of Civil Engineers) Style

The ASCE (American Society of Civil Engineers) style is commonly used in civil engineering fields. Here are the different reference formats in ASCE style:

Author’s Last name, First name. “Article Title.” Journal Title, volume number, issue number (year): page numbers. DOI or URL (if available).

Example : Smith, J. R. “Evaluation of the Effectiveness of Sustainable Drainage Systems in Urban Areas.” Journal of Environmental Engineering, vol. 146, no. 3 (2020): 04020010. https://doi.org/10.1061/(ASCE)EE.1943-7870.0001668.

Example : McCuen, R. H. Hydrologic Analysis and Design. 4th ed. Upper Saddle River, NJ: Pearson Education; 2013.

Author’s Last name, First name. “Chapter Title.” In: Editor’s Last name, First name, ed. Book Title. Edition number. Place of Publication: Publisher; Year: page numbers.

Example : Maidment, D. R. “Floodplain Management in the United States.” In: Shroder, J. F., ed. Treatise on Geomorphology. San Diego, CA: Academic Press; 2013: 447-460.

Author’s Last name, First name. “Paper Title.” In: Conference Title; Conference Date; Location. Place of Publication: Publisher; Year: page numbers.

Example: Smith, J. R. “Sustainable Drainage Systems for Urban Areas.” In: Proceedings of the ASCE International Conference on Sustainable Infrastructure; November 6-9, 2019; Los Angeles, CA. Reston, VA: American Society of Civil Engineers; 2019: 156-163.

Author’s Last name, First name. “Report Title.” Report number. Place of Publication: Publisher; Year.

Example : U.S. Army Corps of Engineers. “Hurricane Sandy Coastal Risk Reduction Program, New York and New Jersey.” Report No. P-15-001. Washington, DC: U.S. Army Corps of Engineers; 2015.

CSE (Council of Science Editors) Style

The CSE (Council of Science Editors) style is commonly used in the scientific and medical fields. Here are the different reference formats in CSE style:

Author’s Last name, First Initial. Middle Initial. “Article Title.” Journal Title. Year;Volume(Issue):Page numbers.

Example : Smith, J.R. “Evaluation of the Effectiveness of Sustainable Drainage Systems in Urban Areas.” Journal of Environmental Engineering. 2020;146(3):04020010.

Author’s Last name, First Initial. Middle Initial. Book Title. Edition number. Place of Publication: Publisher; Year.

Author’s Last name, First Initial. Middle Initial. “Chapter Title.” In: Editor’s Last name, First Initial. Middle Initial., ed. Book Title. Edition number. Place of Publication: Publisher; Year:Page numbers.

Author’s Last name, First Initial. Middle Initial. “Paper Title.” In: Conference Title; Conference Date; Location. Place of Publication: Publisher; Year.

Example : Smith, J.R. “Sustainable Drainage Systems for Urban Areas.” In: Proceedings of the ASCE International Conference on Sustainable Infrastructure; November 6-9, 2019; Los Angeles, CA. Reston, VA: American Society of Civil Engineers; 2019.

Author’s Last name, First Initial. Middle Initial. “Report Title.” Report number. Place of Publication: Publisher; Year.

Bluebook Style

The Bluebook style is commonly used in the legal field for citing legal documents and sources. Here are the different reference formats in Bluebook style:

Case citation

Case name, volume source page (Court year).

Example : Brown v. Board of Education, 347 U.S. 483 (1954).

Statute citation

Name of Act, volume source § section number (year).

Example : Clean Air Act, 42 U.S.C. § 7401 (1963).

Regulation citation

Name of regulation, volume source § section number (year).

Example: Clean Air Act, 40 C.F.R. § 52.01 (2019).

Book citation

Author’s Last name, First Initial. Middle Initial. Book Title. Edition number (if applicable). Place of Publication: Publisher; Year.

Example: Smith, J.R. Legal Writing and Analysis. 3rd ed. New York, NY: Aspen Publishers; 2015.

Journal article citation

Author’s Last name, First Initial. Middle Initial. “Article Title.” Journal Title. Volume number (year): first page-last page.

Example: Garcia, C. “The Right to Counsel: An International Comparison.” International Journal of Legal Information. 43 (2015): 63-94.

Website citation

Author’s Last name, First Initial. Middle Initial. “Page Title.” Website Title. URL (accessed month day, year).

Example : United Nations. “Universal Declaration of Human Rights.” United Nations. https://www.un.org/en/universal-declaration-human-rights/ (accessed January 3, 2023).

Oxford Style

The Oxford style, also known as the Oxford referencing system or the documentary-note citation system, is commonly used in the humanities, including literature, history, and philosophy. Here are the different reference formats in Oxford style:

Author’s Last name, First name. Book Title. Place of Publication: Publisher, Year of Publication.

Example : Smith, John. The Art of Writing. New York: Penguin, 2020.

Author’s Last name, First name. “Article Title.” Journal Title volume, no. issue (year): page range.

Example: Garcia, Carlos. “The Role of Ethics in Philosophy.” Philosophy Today 67, no. 3 (2019): 53-68.

Chapter in an edited book citation

Author’s Last name, First name. “Chapter Title.” In Book Title, edited by Editor’s Name, page range. Place of Publication: Publisher, Year of Publication.

Example : Lee, Mary. “Feminism in the 21st Century.” In The Oxford Handbook of Feminism, edited by Jane Smith, 51-69. Oxford: Oxford University Press, 2018.

Author’s Last name, First name. “Page Title.” Website Title. URL (accessed day month year).

Example : Jones, David. “The Importance of Learning Languages.” Oxford Language Center. https://www.oxfordlanguagecenter.com/importance-of-learning-languages/ (accessed 3 January 2023).

Dissertation or thesis citation

Author’s Last name, First name. “Title of Dissertation/Thesis.” PhD diss., University Name, Year of Publication.

Example : Brown, Susan. “The Art of Storytelling in American Literature.” PhD diss., University of Oxford, 2020.

Newspaper article citation

Author’s Last name, First name. “Article Title.” Newspaper Title, Month Day, Year.

Example : Robinson, Andrew. “New Developments in Climate Change Research.” The Guardian, September 15, 2022.

AAA (American Anthropological Association) Style

The American Anthropological Association (AAA) style is commonly used in anthropology research papers and journals. Here are the different reference formats in AAA style:

Author’s Last name, First name. Year of Publication. Book Title. Place of Publication: Publisher.

Example : Smith, John. 2019. The Anthropology of Food. New York: Routledge.

Author’s Last name, First name. Year of Publication. “Article Title.” Journal Title volume, no. issue: page range.

Example : Garcia, Carlos. 2021. “The Role of Ethics in Anthropology.” American Anthropologist 123, no. 2: 237-251.

Author’s Last name, First name. Year of Publication. “Chapter Title.” In Book Title, edited by Editor’s Name, page range. Place of Publication: Publisher.

Example: Lee, Mary. 2018. “Feminism in Anthropology.” In The Oxford Handbook of Feminism, edited by Jane Smith, 51-69. Oxford: Oxford University Press.

Author’s Last name, First name. Year of Publication. “Page Title.” Website Title. URL (accessed day month year).

Example : Jones, David. 2020. “The Importance of Learning Languages.” Oxford Language Center. https://www.oxfordlanguagecenter.com/importance-of-learning-languages/ (accessed January 3, 2023).

Author’s Last name, First name. Year of Publication. “Title of Dissertation/Thesis.” PhD diss., University Name.

Example : Brown, Susan. 2022. “The Art of Storytelling in Anthropology.” PhD diss., University of California, Berkeley.

Author’s Last name, First name. Year of Publication. “Article Title.” Newspaper Title, Month Day.

Example : Robinson, Andrew. 2021. “New Developments in Anthropology Research.” The Guardian, September 15.

AIP (American Institute of Physics) Style

The American Institute of Physics (AIP) style is commonly used in physics research papers and journals. Here are the different reference formats in AIP style:

Example : Johnson, S. D. 2021. “Quantum Computing and Information.” Journal of Applied Physics 129, no. 4: 043102.

Example : Feynman, Richard. 2018. The Feynman Lectures on Physics. New York: Basic Books.

Example : Jones, David. 2020. “The Future of Quantum Computing.” In The Handbook of Physics, edited by John Smith, 125-136. Oxford: Oxford University Press.

Conference proceedings citation

Author’s Last name, First name. Year of Publication. “Title of Paper.” Proceedings of Conference Name, date and location: page range. Place of Publication: Publisher.

Example : Chen, Wei. 2019. “The Applications of Nanotechnology in Solar Cells.” Proceedings of the 8th International Conference on Nanotechnology, July 15-17, Tokyo, Japan: 224-229. New York: AIP Publishing.

Example : American Institute of Physics. 2022. “About AIP Publishing.” AIP Publishing. https://publishing.aip.org/about-aip-publishing/ (accessed January 3, 2023).

Patent citation

Author’s Last name, First name. Year of Publication. Patent Number.

Example : Smith, John. 2018. US Patent 9,873,644.

References Writing Guide

Here are some general guidelines for writing references:

• Follow the citation style guidelines: Different disciplines and journals may require different citation styles (e.g., APA, MLA, Chicago). It is important to follow the specific guidelines for the citation style required.
• Include all necessary information : Each citation should include enough information for readers to locate the source. For example, a journal article citation should include the author(s), title of the article, journal title, volume number, issue number, page numbers, and publication year.
• Use proper formatting: Citation styles typically have specific formatting requirements for different types of sources. Make sure to follow the proper formatting for each citation.
• Order citations alphabetically: If listing multiple sources, they should be listed alphabetically by the author’s last name.
• Be consistent: Use the same citation style throughout the entire paper or project.
• Check for accuracy: Double-check all citations to ensure accuracy, including correct spelling of author names and publication information.
• Use reputable sources: When selecting sources to cite, choose reputable and authoritative sources. Avoid sources that are biased or unreliable.
• Include all sources: Make sure to include all sources used in the research, including those that were not directly quoted but still informed the work.
• Use online tools : There are online tools available (e.g., citation generators) that can help with formatting and organizing references.

Purpose of References in Research

References in research serve several purposes:

• To give credit to the original authors or sources of information used in the research. It is important to acknowledge the work of others and avoid plagiarism.
• To provide evidence for the claims made in the research. References can support the arguments, hypotheses, or conclusions presented in the research by citing relevant studies, data, or theories.
• To allow readers to find and verify the sources used in the research. References provide the necessary information for readers to locate and access the sources cited in the research, which allows them to evaluate the quality and reliability of the information presented.
• To situate the research within the broader context of the field. References can show how the research builds on or contributes to the existing body of knowledge, and can help readers to identify gaps in the literature that the research seeks to address.

Importance of References in Research

References play an important role in research for several reasons:

• Credibility : By citing authoritative sources, references lend credibility to the research and its claims. They provide evidence that the research is based on a sound foundation of knowledge and has been carefully researched.
• Avoidance of Plagiarism : References help researchers avoid plagiarism by giving credit to the original authors or sources of information. This is important for ethical reasons and also to avoid legal repercussions.
• Reproducibility : References allow others to reproduce the research by providing detailed information on the sources used. This is important for verification of the research and for others to build on the work.
• Context : References provide context for the research by situating it within the broader body of knowledge in the field. They help researchers to understand where their work fits in and how it builds on or contributes to existing knowledge.
• Evaluation : References provide a means for others to evaluate the research by allowing them to assess the quality and reliability of the sources used.

Advantages of References in Research

There are several advantages of including references in research:

• Acknowledgment of Sources: Including references gives credit to the authors or sources of information used in the research. This is important to acknowledge the original work and avoid plagiarism.
• Evidence and Support : References can provide evidence to support the arguments, hypotheses, or conclusions presented in the research. This can add credibility and strength to the research.
• Reproducibility : References provide the necessary information for others to reproduce the research. This is important for the verification of the research and for others to build on the work.
• Context : References can help to situate the research within the broader body of knowledge in the field. This helps researchers to understand where their work fits in and how it builds on or contributes to existing knowledge.
• Evaluation : Including references allows others to evaluate the research by providing a means to assess the quality and reliability of the sources used.
• Ongoing Conversation: References allow researchers to engage in ongoing conversations and debates within their fields. They can show how the research builds on or contributes to the existing body of knowledge.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Institutional Review Board – Application Sample...

Evaluating Research – Process, Examples and...

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Referencing

A Quick Guide to Harvard Referencing | Citation Examples

Published on 14 February 2020 by Jack Caulfield . Revised on 15 September 2023.

Referencing is an important part of academic writing. It tells your readers what sources you’ve used and how to find them.

Harvard is the most common referencing style used in UK universities. In Harvard style, the author and year are cited in-text, and full details of the source are given in a reference list .

Harvard Reference Generator

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Harvard in-text citation, creating a harvard reference list, harvard referencing examples, referencing sources with no author or date, frequently asked questions about harvard referencing.

A Harvard in-text citation appears in brackets beside any quotation or paraphrase of a source. It gives the last name of the author(s) and the year of publication, as well as a page number or range locating the passage referenced, if applicable:

Note that ‘p.’ is used for a single page, ‘pp.’ for multiple pages (e.g. ‘pp. 1–5’).

An in-text citation usually appears immediately after the quotation or paraphrase in question. It may also appear at the end of the relevant sentence, as long as it’s clear what it refers to.

When your sentence already mentions the name of the author, it should not be repeated in the citation:

Sources with multiple authors

When you cite a source with up to three authors, cite all authors’ names. For four or more authors, list only the first name, followed by ‘ et al. ’:

Sources with no page numbers

Some sources, such as websites , often don’t have page numbers. If the source is a short text, you can simply leave out the page number. With longer sources, you can use an alternate locator such as a subheading or paragraph number if you need to specify where to find the quote:

Multiple citations at the same point

When you need multiple citations to appear at the same point in your text – for example, when you refer to several sources with one phrase – you can present them in the same set of brackets, separated by semicolons. List them in order of publication date:

Multiple sources with the same author and date

If you cite multiple sources by the same author which were published in the same year, it’s important to distinguish between them in your citations. To do this, insert an ‘a’ after the year in the first one you reference, a ‘b’ in the second, and so on:

Prevent plagiarism, run a free check.

A bibliography or reference list appears at the end of your text. It lists all your sources in alphabetical order by the author’s last name, giving complete information so that the reader can look them up if necessary.

The reference entry starts with the author’s last name followed by initial(s). Only the first word of the title is capitalised (as well as any proper nouns).

Sources with multiple authors in the reference list

As with in-text citations, up to three authors should be listed; when there are four or more, list only the first author followed by ‘ et al. ’:

Reference list entries vary according to source type, since different information is relevant for different sources. Formats and examples for the most commonly used source types are given below.

• Entire book
• Book chapter
• Translated book
• Edition of a book

Journal articles

• Print journal
• Online-only journal with DOI
• Online-only journal with no DOI
• General web page
• Online article or blog
• Social media post

Sometimes you won’t have all the information you need for a reference. This section covers what to do when a source lacks a publication date or named author.

No publication date

When a source doesn’t have a clear publication date – for example, a constantly updated reference source like Wikipedia or an obscure historical document which can’t be accurately dated – you can replace it with the words ‘no date’:

Note that when you do this with an online source, you should still include an access date, as in the example.

When a source lacks a clearly identified author, there’s often an appropriate corporate source – the organisation responsible for the source – whom you can credit as author instead, as in the Google and Wikipedia examples above.

When that’s not the case, you can just replace it with the title of the source in both the in-text citation and the reference list:

Harvard referencing uses an author–date system. Sources are cited by the author’s last name and the publication year in brackets. Each Harvard in-text citation corresponds to an entry in the alphabetised reference list at the end of the paper.

Vancouver referencing uses a numerical system. Sources are cited by a number in parentheses or superscript. Each number corresponds to a full reference at the end of the paper.

A Harvard in-text citation should appear in brackets every time you quote, paraphrase, or refer to information from a source.

The citation can appear immediately after the quotation or paraphrase, or at the end of the sentence. If you’re quoting, place the citation outside of the quotation marks but before any other punctuation like a comma or full stop.

In Harvard referencing, up to three author names are included in an in-text citation or reference list entry. When there are four or more authors, include only the first, followed by ‘ et al. ’

Though the terms are sometimes used interchangeably, there is a difference in meaning:

• A reference list only includes sources cited in the text – every entry corresponds to an in-text citation .
• A bibliography also includes other sources which were consulted during the research but not cited.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2023, September 15). A Quick Guide to Harvard Referencing | Citation Examples. Scribbr. Retrieved 2 April 2024, from https://www.scribbr.co.uk/referencing/harvard-style/

Is this article helpful?

Jack Caulfield

Other students also liked, harvard in-text citation | a complete guide & examples, harvard style bibliography | format & examples, referencing books in harvard style | templates & examples, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

• Manuscript Preparation

How to write your references quickly and easily

• 3 minute read
• 228.5K views

Table of Contents

Every scientific paper builds on previous research – even if it’s in a new field, related studies will have preceded and informed it. In peer-reviewed articles, authors must give credit to this previous research, through citations and references. Not only does this show clearly where the current research came from, but it also helps readers understand the content of the paper better.

There is no optimum number of references for an academic article but depending on the subject you could be dealing with more than 100 different papers, conference reports, video articles, medical guidelines or any number of other resources.

That’s a lot of content to manage. Before submitting your manuscript, this needs to be checked, cross-references in the text and the list, organized and formatted.

The exact content and format of the citations and references in your paper will depend on the journal you aim to publish in, so the first step is to check the journal’s Guide for Authors before you submit.

There are two main points to pay attention to – consistency and accuracy. When you go through your manuscript to edit or proofread it, look closely at the citations within the text. Are they all the same? For example, if the journal prefers the citations to be in the format (name, year), make sure they’re all the same: (Smith, 2016).

Your citations must also be accurate and complete. Do they match your references list? Each citation should be included in the list, so cross-checking is important. It’s also common for journals to prefer that most, if not all, of the articles listed in your references be cited within the text – after all, these should be studies that contributed to the knowledge underpinning your work, not just your bedtime reading. So go through them carefully, noting any missing references or citations and filling the gaps.

Each journal has its own requirements when it comes to the content and format of references, as well as where and how you should include them in your submission, so double-check before you hit send!

In general, a reference will include authors’ names and initials, the title of the article, name of the journal, volume and issue, date, page numbers and DOI. On ScienceDirect, articles are linked to their original source (if also published on ScienceDirect) or to their Scopus record, so including the DOI can help link to the correct article.

A spotless reference list

Luckily, compiling and editing the references in your scientific manuscript can be easy – and it no longer has to be manual. Management tools like Mendeley can keep track of all your references, letting you share them with your collaborators. With the Word plugin, it’s possible to select the right citation style for the journal you’re submitting to and the tool will format your references automatically.

Like with any other part of your manuscript, it’s important to make sure your reference list has been checked and edited. Elsevier Author Services Language Editing can help, with professional manuscript editing that will help make sure your references don’t hold you back from publication.

Why it’s best to ask a professional when it comes to translation

• Research Process

How to Choose a Journal to Submit an Article

You may also like.

Make Hook, Line, and Sinker: The Art of Crafting Engaging Introductions

Can Describing Study Limitations Improve the Quality of Your Paper?

A Guide to Crafting Shorter, Impactful Sentences in Academic Writing

6 Steps to Write an Excellent Discussion in Your Manuscript

How to Write Clear and Crisp Civil Engineering Papers? Here are 5 Key Tips to Consider

The Clear Path to An Impactful Paper: ②

The Essentials of Writing to Communicate Research in Medicine

Changing Lines: Sentence Patterns in Academic Writing

Input your search keywords and press Enter.

• Walden University
• Faculty Portal

Reference List: Common Reference List Examples

Article (with doi).

Alvarez, E., & Tippins, S. (2019). Socialization agents that Puerto Rican college students use to make financial decisions. Journal of Social Change , 11 (1), 75–85. https://doi.org/10.5590/JOSC.2019.11.1.07

Laplante, J. P., & Nolin, C. (2014). Consultas and socially responsible investing in Guatemala: A case study examining Maya perspectives on the Indigenous right to free, prior, and informed consent. Society & Natural Resources , 27 , 231–248. https://doi.org/10.1080/08941920.2013.861554

Use the DOI number for the source whenever one is available. DOI stands for "digital object identifier," a number specific to the article that can help others locate the source. In APA 7, format the DOI as a web address. Active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list. Also see our Quick Answer FAQ, "Can I use the DOI format provided by library databases?"

Jerrentrup, A., Mueller, T., Glowalla, U., Herder, M., Henrichs, N., Neubauer, A., & Schaefer, J. R. (2018). Teaching medicine with the help of “Dr. House.” PLoS ONE , 13 (3), Article e0193972. https://doi.org/10.1371/journal.pone.0193972

For journal articles that are assigned article numbers rather than page ranges, include the article number in place of the page range.

For more on citing electronic resources, see  Electronic Sources References .

Article (Without DOI)

Found in a common academic research database or in print.

Casler , T. (2020). Improving the graduate nursing experience through support on a social media platform. MEDSURG Nursing , 29 (2), 83–87.

If an article does not have a DOI and you retrieved it from a common academic research database through the university library, there is no need to include any additional electronic retrieval information. The reference list entry looks like the entry for a print copy of the article. (This format differs from APA 6 guidelines that recommended including the URL of a journal's homepage when the DOI was not available.) Note that APA 7 has additional guidance on reference list entries for articles found only in specific databases or archives such as Cochrane Database of Systematic Reviews, UpToDate, ProQuest Dissertations and Theses Global, and university archives. See APA 7, Section 9.30 for more information.

Found on an Open Access Website

Eaton, T. V., & Akers, M. D. (2007). Whistleblowing and good governance. CPA Journal , 77 (6), 66–71. http://archives.cpajournal.com/2007/607/essentials/p58.htm

Provide the direct web address/URL to a journal article found on the open web, often on an open access journal's website. In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list.

Weinstein, J. A. (2010).  Social change  (3rd ed.). Rowman & Littlefield.

If the book has an edition number, include it in parentheses after the title of the book. If the book does not list any edition information, do not include an edition number. The edition number is not italicized.

American Nurses Association. (2015). Nursing: Scope and standards of practice (3rd ed.).

If the author and publisher are the same, only include the author in its regular place and omit the publisher.

Lencioni, P. (2012). The advantage: Why organizational health trumps everything else in business . Jossey-Bass. https://amzn.to/343XPSJ

As a change from APA 6 to APA 7, it is no longer necessary to include the ebook format in the title. However, if you listened to an audiobook and the content differs from the text version (e.g., abridged content) or your discussion highlights elements of the audiobook (e.g., narrator's performance), then note that it is an audiobook in the title element in brackets. For ebooks and online audiobooks, also include the DOI number (if available) or nondatabase URL but leave out the electronic retrieval element if the ebook was found in a common academic research database, as with journal articles. APA 7 allows for the shortening of long DOIs and URLs, as shown in this example. See APA 7, Section 9.36 for more information.

Chapter in an Edited Book

Poe, M. (2017). Reframing race in teaching writing across the curriculum. In F. Condon & V. A. Young (Eds.), Performing antiracist pedagogy in rhetoric, writing, and communication (pp. 87–105). University Press of Colorado.

Include the page numbers of the chapter in parentheses after the book title.

Christensen, L. (2001). For my people: Celebrating community through poetry. In B. Bigelow, B. Harvey, S. Karp, & L. Miller (Eds.), Rethinking our classrooms: Teaching for equity and justice (Vol. 2, pp. 16–17). Rethinking Schools.

Also include the volume number or edition number in the parenthetical information after the book title when relevant.

Freud, S. (1961). The ego and the id. In J. Strachey (Ed.),  The standard edition of the complete psychological works of Sigmund Freud  (Vol. 19, pp. 3-66). Hogarth Press. (Original work published 1923)

When a text has been republished as part of an anthology collection, after the author’s name include the date of the version that was read. At the end of the entry, place the date of the original publication inside parenthesis along with the note “original work published.” For in-text citations of republished work, use both dates in the parenthetical citation, original date first with a slash separating the years, as in this example: Freud (1923/1961). For more information on reprinted or republished works, see APA 7, Sections 9.40-9.41.

Classroom Resources

Citing classroom resources.

If you need to cite content found in your online classroom, use the author (if there is one listed), the year of publication (if available), the title of the document, and the main URL of Walden classrooms. For example, you are citing study notes titled "Health Effects of Exposure to Forest Fires," but you do not know the author's name, your reference entry will look like this:

Health effects of exposure to forest fires [Lecture notes]. (2005). Walden University Canvas. https://waldenu.instructure.com

If you do know the author of the document, your reference will look like this:

Smith, A. (2005). Health effects of exposure to forest fires [PowerPoint slides]. Walden University Canvas. https://waldenu.instructure.com  

A few notes on citing course materials:

• [Lecture notes]
• [Course handout]
• [Study notes]
• It can be difficult to determine authorship of classroom documents. If an author is listed on the document, use that. If the resource is clearly a product of Walden (such as the course-based videos), use Walden University as the author. If you are unsure or if no author is indicated, place the title in the author spot, as above.
• If you cannot determine a date of publication, you can use n.d. (for "no date") in place of the year.

Note:  The web location for Walden course materials is not directly retrievable without a password, and therefore, following APA guidelines, use the main URL for the class sites: https://class.waldenu.edu.

Citing Tempo Classroom Resources

Clear author: 

Smith, A. (2005). Health effects of exposure to forest fires [PowerPoint slides]. Walden University Brightspace. https://mytempo.waldenu.edu

Unclear author:

Health effects of exposure to forest fires [Lecture notes]. (2005). Walden University Brightspace. https://mytempo.waldenu.edu

Conference Sessions and Presentations

Feinman, Y. (2018, July 27). Alternative to proctoring in introductory statistics community college courses [Poster presentation]. Walden University Research Symposium, Minneapolis, MN, United States. https://scholarworks.waldenu.edu/symposium2018/23/

Torgerson, K., Parrill, J., & Haas, A. (2019, April 5-9). Tutoring strategies for online students [Conference session]. The Higher Learning Commission Annual Conference, Chicago, IL, United States. http://onlinewritingcenters.org/scholarship/torgerson-parrill-haas-2019/

Dictionary Entry

Merriam-Webster. (n.d.). Leadership. In Merriam-Webster.com dictionary . Retrieved May 28, 2020, from https://www.merriam-webster.com/dictionary/leadership

When constructing a reference for an entry in a dictionary or other reference work that has no byline (i.e., no named individual authors), use the name of the group—the institution, company, or organization—as author (e.g., Merriam Webster, American Psychological Association, etc.). The name of the entry goes in the title position, followed by "In" and the italicized name of the reference work (e.g., Merriam-Webster.com dictionary , APA dictionary of psychology ). In this instance, APA 7 recommends including a retrieval date as well for this online source since the contents of the page change over time. End the reference entry with the specific URL for the defined word.

Discussion Board Post

Osborne, C. S. (2010, June 29). Re: Environmental responsibility [Discussion post]. Walden University Canvas.  https://waldenu.instructure.com  

Dissertations or Theses

Retrieved From a Database

Nalumango, K. (2019). Perceptions about the asylum-seeking process in the United States after 9/11 (Publication No. 13879844) [Doctoral dissertation, Walden University]. ProQuest Dissertations and Theses.

Retrieved From an Institutional or Personal Website

Evener. J. (2018). Organizational learning in libraries at for-profit colleges and universities [Doctoral dissertation, Walden University]. ScholarWorks. https://scholarworks.waldenu.edu/cgi/viewcontent.cgi?article=6606&context=dissertations

Unpublished Dissertation or Thesis

Kirwan, J. G. (2005). An experimental study of the effects of small-group, face-to-face facilitated dialogues on the development of self-actualization levels: A movement towards fully functional persons [Unpublished doctoral dissertation]. Saybrook Graduate School and Research Center.

For further examples and information, see APA 7, Section 10.6.

Legal Material

For legal references, APA follows the recommendations of The Bluebook: A Uniform System of Citation , so if you have any questions beyond the examples provided in APA, seek out that resource as well.

Court Decisions

Reference format:

Name v. Name, Volume Reporter Page (Court Date). URL

Sample reference entry:

Brown v. Board of Education, 347 U.S. 483 (1954). https://www.oyez.org/cases/1940-1955/347us483

Sample citation:

In Brown v. Board of Education (1954), the Supreme Court ruled racial segregation in schools unconstitutional.

Note: Italicize the case name when it appears in the text of your paper.

Name of Act, Title Source § Section Number (Year). URL

Sample reference entry for a federal statute:

Individuals With Disabilities Education Act, 20 U.S.C. § 1400 et seq. (2004). https://www.congress.gov/108/plaws/publ446/PLAW-108publ446.pdf

Sample reference entry for a state statute:

Minnesota Nurse Practice Act, Minn. Stat. §§ 148.171 et seq. (2019). https://www.revisor.mn.gov/statutes/cite/148.171

Sample citation: Minnesota nurses must maintain current registration in order to practice (Minnesota Nurse Practice Act, 2010).

Note: The § symbol stands for "section." Use §§ for sections (plural). To find this symbol in Microsoft Word, go to "Insert" and click on Symbol." Look in the Latin 1-Supplement subset. Note: U.S.C. stands for "United States Code." Note: The Latin abbreviation " et seq. " means "and what follows" and is used when the act includes the cited section and ones that follow. Note: List the chapter first followed by the section or range of sections.

Unenacted Bills and Resolutions

(Those that did not pass and become law)

Title [if there is one], bill or resolution number, xxx Cong. (year). URL

Sample reference entry for Senate bill:

Anti-Phishing Act, S. 472, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/senate-bill/472

Sample reference entry for House of Representatives resolution:

Anti-Phishing Act, H.R. 1099, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/house-bill/1099

The Anti-Phishing Act (2005) proposed up to 5 years prison time for people running Internet scams.

These are the three legal areas you may be most apt to cite in your scholarly work. For more examples and explanation, see APA 7, Chapter 11.

Magazine Article

Clay, R. (2008, June). Science vs. ideology: Psychologists fight back about the misuse of research. Monitor on Psychology , 39 (6). https://www.apa.org/monitor/2008/06/ideology

Note that for citations, include only the year: Clay (2008). For magazine articles retrieved from a common academic research database, leave out the URL. For magazine articles from an online news website that is not an online version of a print magazine, follow the format for a webpage reference list entry.

Newspaper Article (Retrieved Online)

Baker, A. (2014, May 7). Connecticut students show gains in national tests. New York Times . http://www.nytimes.com/2014/05/08/nyregion/national-assessment-of-educational-progress-results-in-Connecticut-and-New-Jersey.html

Include the full date in the format Year, Month Day. Do not include a retrieval date for periodical sources found on websites. Note that for citations, include only the year: Baker (2014). For newspaper articles retrieved from a common academic research database, leave out the URL. For newspaper articles from an online news website that is not an online version of a print newspaper, follow the format for a webpage reference list entry.

Online Video/Webcast

Walden University. (2013).  An overview of learning  [Video]. Walden University Canvas.  https://waldenu.instructure.com  

Use this format for online videos such as Walden videos in classrooms. Most of our classroom videos are produced by Walden University, which will be listed as the author in your reference and citation. Note: Some examples of audiovisual materials in the APA manual show the word “Producer” in parentheses after the producer/author area. In consultation with the editors of the APA manual, we have determined that parenthetical is not necessary for the videos in our courses. The manual itself is unclear on the matter, however, so either approach should be accepted. Note that the speaker in the video does not appear in the reference list entry, but you may want to mention that person in your text. For instance, if you are viewing a video where Tobias Ball is the speaker, you might write the following: Tobias Ball stated that APA guidelines ensure a consistent presentation of information in student papers (Walden University, 2013). For more information on citing the speaker in a video, see our page on Common Citation Errors .

Taylor, R. [taylorphd07]. (2014, February 27). Scales of measurement [Video]. YouTube. https://www.youtube.com/watch?v=PDsMUlexaMY

Walden University Academic Skills Center. (2020, April 15). One-way ANCOVA: Introduction [Video]. YouTube. https://youtu.be/_XnNDQ5CNW8

For videos from streaming sites, use the person or organization who uploaded the video in the author space to ensure retrievability, whether or not that person is the speaker in the video. A username can be provided in square brackets. As a change from APA 6 to APA 7, include the publisher after the title, and do not use "Retrieved from" before the URL. See APA 7, Section 10.12 for more information and examples.

See also reference list entry formats for TED Talks .

Technical and Research Reports

Edwards, C. (2015). Lighting levels for isolated intersections: Leading to safety improvements (Report No. MnDOT 2015-05). Center for Transportation Studies. http://www.cts.umn.edu/Publications/ResearchReports/reportdetail.html?id=2402

Technical and research reports by governmental agencies and other research institutions usually follow a different publication process than scholarly, peer-reviewed journals. However, they present original research and are often useful for research papers. Sometimes, researchers refer to these types of reports as gray literature , and white papers are a type of this literature. See APA 7, Section 10.4 for more information.

Reference list entires for TED Talks follow the usual guidelines for multimedia content found online. There are two common places to find TED talks online, with slightly different reference list entry formats for each.

TED Talk on the TED website

If you find the TED Talk on the TED website, follow the format for an online video on an organizational website:

Owusu-Kesse, K. (2020, June). 5 needs that any COVID-19 response should meet [Video]. TED Conferences. https://www.ted.com/talks/kwame_owusu_kesse_5_needs_that_any_covid_19_response_should_meet

The speaker is the author in the reference list entry if the video is posted on the TED website. For citations, use the speaker's surname.

TED Talk on YouTube

If you find the TED Talk on YouTube or another streaming video website, follow the usual format for streaming video sites:

TED. (2021, February 5). The shadow pandemic of domestic violence during COVID-19 | Kemi DaSilvalbru [Video]. YouTube. https://www.youtube.com/watch?v=PGdID_ICFII

TED is the author in the reference list entry if the video is posted on YouTube since it is the channel on which the video is posted. For citations, use TED as the author.

Walden University Course Catalog

To include the Walden course catalog in your reference list, use this format:

Walden University. (2020). 2019-2020 Walden University catalog . https://catalog.waldenu.edu/index.php

If you cite from a specific portion of the catalog in your paper, indicate the appropriate section and paragraph number in your text:

...which reflects the commitment to social change expressed in Walden University's mission statement (Walden University, 2020, Vision, Mission, and Goals section, para. 2).

And in the reference list:

Walden University. (2020). Vision, mission, and goals. In 2019-2020 Walden University catalog. https://catalog.waldenu.edu/content.php?catoid=172&navoid=59420&hl=vision&returnto=search

Vartan, S. (2018, January 30). Why vacations matter for your health . CNN. https://www.cnn.com/travel/article/why-vacations-matter/index.html

For webpages on the open web, include the author, date, webpage title, organization/site name, and URL. (There is a slight variation for online versions of print newspapers or magazines. For those sources, follow the models in the previous sections of this page.)

American Federation of Teachers. (n.d.). Community schools . http://www.aft.org/issues/schoolreform/commschools/index.cfm

If there is no specified author, then use the organization’s name as the author. In such a case, there is no need to repeat the organization's name after the title.

In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list.

Related Resources

Knowledge Check: Common Reference List Examples

Didn't find what you need? Search our website or email us .

Read our website accessibility and accommodation statement .

• Previous Page: Reference List: Overview
• Next Page: Common Military Reference List Examples
• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Research and Doctoral Services
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

• PRO Courses Guides New Tech Help Pro Expert Videos About wikiHow Pro Upgrade Sign In
• EDIT Edit this Article
• EXPLORE Tech Help Pro About Us Random Article Quizzes Request a New Article Community Dashboard This Or That Game Popular Categories Arts and Entertainment Artwork Books Movies Computers and Electronics Computers Phone Skills Technology Hacks Health Men's Health Mental Health Women's Health Relationships Dating Love Relationship Issues Hobbies and Crafts Crafts Drawing Games Education & Communication Communication Skills Personal Development Studying Personal Care and Style Fashion Hair Care Personal Hygiene Youth Personal Care School Stuff Dating All Categories Arts and Entertainment Finance and Business Home and Garden Relationship Quizzes Cars & Other Vehicles Food and Entertaining Personal Care and Style Sports and Fitness Computers and Electronics Health Pets and Animals Travel Education & Communication Hobbies and Crafts Philosophy and Religion Work World Family Life Holidays and Traditions Relationships Youth
• Browse Articles
• Learn Something New
• Quizzes Hot
• This Or That Game New
• Train Your Brain
• Explore More
• Support wikiHow
• About wikiHow
• Log in / Sign up
• Education and Communications
• College University and Postgraduate
• Academic Writing
• Research Papers

How to Cite a Research Paper

Last Updated: March 29, 2024 Fact Checked

This article was reviewed by Gerald Posner and by wikiHow staff writer, Jennifer Mueller, JD . Gerald Posner is an Author & Journalist based in Miami, Florida. With over 35 years of experience, he specializes in investigative journalism, nonfiction books, and editorials. He holds a law degree from UC College of the Law, San Francisco, and a BA in Political Science from the University of California-Berkeley. He’s the author of thirteen books, including several New York Times bestsellers, the winner of the Florida Book Award for General Nonfiction, and has been a finalist for the Pulitzer Prize in History. He was also shortlisted for the Best Business Book of 2020 by the Society for Advancing Business Editing and Writing. There are 8 references cited in this article, which can be found at the bottom of the page. This article has been fact-checked, ensuring the accuracy of any cited facts and confirming the authority of its sources. This article has been viewed 411,633 times.

When writing a paper for a research project, you may need to cite a research paper you used as a reference. The basic information included in your citation will be the same across all styles. However, the format in which that information is presented is somewhat different depending on whether you're using American Psychological Association (APA), Modern Language Association (MLA), Chicago, or American Medical Association (AMA) style.

Referencing a Research Paper

• In APA style, cite the paper: Last Name, First Initial. (Year). Title. Publisher.
• In Chicago style, cite the paper: Last Name, First Name. “Title.” Publisher, Year.
• In MLA style, cite the paper: Last Name, First Name. “Title.” Publisher. Year.

Citation Help

• For example: "Kringle, K., & Frost, J."

• For example: "Kringle, K., & Frost, J. (2012)."
• If the date, or any other information, are not available, use the guide at https://blog.apastyle.org/apastyle/2012/05/missing-pieces.html .

• For example: "Kringle, K., & Frost, J. (2012). Red noses, warm hearts: The glowing phenomenon among North Pole reindeer."
• If you found the research paper in a database maintained by a university, corporation, or other organization, include any index number assigned to the paper in parentheses after the title. For example: "Kringle, K., & Frost, J. (2012). Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. (Report No. 1234)."

• For example: "Kringle, K., & Frost, J. (2012). Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. (Report No. 1234). Retrieved from Alaska University Library Archives, December 24, 2017."

• For example: "(Kringle & Frost, 2012)."
• If there was no date on the research paper, use the abbreviation n.d. : "(Kringle & Frost, n.d.)."

• For example: "Kringle, Kris, and Jack Frost."

• For example: "Kringle, Kris, and Jack Frost. "Red Noses, Warm Hearts: The Glowing Phenomenon among North Pole Reindeer." Master's thesis."

• For example: "Kringle, Kris, and Jack Frost. "Red Noses, Warm Hearts: The Glowing Phenomenon among North Pole Reindeer." Master's thesis, Alaska University, 2012."

• For example: "Kringle, Kris, and Jack Frost. "Red Noses, Warm Hearts: The Glowing Phenomenon among North Pole Reindeer." Master's thesis, Alaska University, 2012. Accessed at https://www.northpolemedical.com/raising_rudolf."

• Footnotes are essentially the same as the full citation, although the first and last names of the authors aren't inverted.
• For parenthetical citations, Chicago uses the Author-Date format. For example: "(Kringle and Frost 2012)."

• For example: "Kringle, Kris, and Frost, Jack."

• For example: "Kringle, Kris, and Frost, Jack. "Red Noses, Warm Hearts: The Glowing Phenomenon Among North Pole Reindeer.""

• For example, suppose you found the paper in a collection of paper housed in university archives. Your citation might be: "Kringle, Kris, and Frost, Jack. "Red Noses, Warm Hearts: The Glowing Phenomenon Among North Pole Reindeer." Master's Theses 2000-2010. University of Alaska Library Archives. Accessed December 24, 2017."

• For example: "(Kringle & Frost, p. 33)."

• For example: "Kringle K, Frost J."

• For example: "Kringle K, Frost J. Red noses, warm hearts: The glowing phenomenon among North Pole reindeer."

• For example: "Kringle K, Frost J. Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. Nat Med. 2012; 18(9): 1429-1433."

• For example, if you're citing a paper presented at a conference, you'd write: "Kringle K, Frost J. Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. Oral presentation at Arctic Health Association Annual Summit; December, 2017; Nome, Alaska."
• To cite a paper you read online, you'd write: "Kringle K, Frost J. Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. https://www.northpolemedical.com/raising_rudolf"

• For example: "According to Kringle and Frost, these red noses indicate a subspecies of reindeer native to Alaska and Canada that have migrated to the North Pole and mingled with North Pole reindeer. 1 "

Community Q&A

• If you used a manual as a source in your research paper, you'll need to learn how to cite the manual also. Thanks Helpful 0 Not Helpful 0
• If you use any figures in your research paper, you'll also need to know the proper way to cite them in MLA, APA, AMA, or Chicago. Thanks Helpful 0 Not Helpful 0

You Might Also Like

• ↑ https://askus.library.wwu.edu/faq/116659
• ↑ https://guides.libraries.psu.edu/apaquickguide/intext
• ↑ https://owl.purdue.edu/owl/research_and_citation/chicago_manual_17th_edition/cmos_formatting_and_style_guide/general_format.html
• ↑ https://libanswers.snhu.edu/faq/48009
• ↑ https://www.chicagomanualofstyle.org/tools_citationguide/citation-guide-2.html
• ↑ https://owl.purdue.edu/owl/research_and_citation/mla_style/mla_formatting_and_style_guide/mla_in_text_citations_the_basics.html
• ↑ https://morningside.libguides.com/MLA8/location
• ↑ https://owl.purdue.edu/owl/research_and_citation/ama_style/index.html

About This Article

To cite a paper APA style, start with the author's last name and first initial, and the year of publication. Then, list the title of the paper, where you found it, and the date that you accessed it. In a paper, use a parenthetical reference with the last name of the author and the publication year. For an MLA citation, list the author's last name and then first name and the title of the paper in quotations. Include where you accessed the paper and the date you retrieved it. In your paper, use a parenthetical reference with the author's last name and the page number. Keep reading for tips on Chicago and AMA citations and exceptions to the citation rules! Did this summary help you? Yes No

• Send fan mail to authors

Did this article help you?

Featured Articles

Trending Articles

Watch Articles

• Terms of Use
• Privacy Policy
• Do Not Sell or Share My Info
• Not Selling Info

Don’t miss out! Sign up for

wikiHow’s newsletter

University Library

Start your research.

• Research Process
• Find Background Info
• Find Sources through the Library
• Evaluate Your Info
• Cite Your Sources
• Evaluate, Write & Cite

• is the right thing to do  to give credit to those who had the idea
• shows that you have read and understand  what experts have had to say about your topic
• helps people find the sources  that you used in case they want to read more about the topic
• provides   evidence  for your arguments
• is professional and  standard practice   for students and scholars

What is a Citation?

A citation identifies for the reader the original source for an idea, information, or image that is referred to in a work.

• In the body of a paper, the  in-text citation  acknowledges the source of information used.
• At the end of a paper, the citations are compiled on a  References  or  Works Cited  list. A basic citation includes the author, title, and publication information of the source. 

From:  Lemieux  Library,  University  of Seattle 

Why Should You Cite?

Quoting Are you quoting two or more consecutive words from a source? Then the original source should be cited and the words or phrase placed in quotes. 

Paraphrasing If an idea or information comes from another source,  even if you put it in your own words , you still need to credit the source.  General vs. Unfamiliar Knowledge You do not need to cite material which is accepted common knowledge. If in doubt whether your information is common knowledge or not, cite it. Formats We usually think of books and articles. However, if you use material from web sites, films, music, graphs, tables, etc. you'll also need to cite these as well.

Plagiarism is presenting the words or ideas of someone else as your own without proper acknowledgment of the source. When you work on a research paper and use supporting material from works by others, it's okay to quote people and use their ideas, but you do need to correctly credit them. Even when you summarize or paraphrase information found in books, articles, or Web pages, you must acknowledge the original author.

Citation Style Help

Helpful links:

• MLA ,  Works Cited : A Quick Guide (a template of core elements)
• CSE  (Council of Science Editors)

For additional writing resources specific to styles listed here visit the  Purdue OWL Writing Lab

Citation and Bibliography Resources

• How to Write an Annotated Bibliography
• << Previous: Evaluate Your Info
• Next: Evaluate, Write & Cite >>

Creative Commons Attribution 3.0 License except where otherwise noted.

Land Acknowledgement

The land on which we gather is the unceded territory of the Awaswas-speaking Uypi Tribe. The Amah Mutsun Tribal Band, comprised of the descendants of indigenous people taken to missions Santa Cruz and San Juan Bautista during Spanish colonization of the Central Coast, is today working hard to restore traditional stewardship practices on these lands and heal from historical trauma.

The land acknowledgement used at UC Santa Cruz was developed in partnership with the Amah Mutsun Tribal Band Chairman and the Amah Mutsun Relearning Program at the UCSC Arboretum .

• Search This Site All UCSD Sites Faculty/Staff Search Term
• Contact & Directions
• Climate Statement
• Cognitive Behavioral Neuroscience
• Cognitive Psychology
• Developmental Psychology
• Social Psychology
• Adjunct Faculty
• Non-Senate Instructors
• Researchers
• Psychology Grads
• Affiliated Grads
• New and Prospective Students
• Honors Program
• Experiential Learning
• Programs & Events
• Psi Chi / Psychology Club
• Prospective PhD Students
• Current PhD Students
• Area Brown Bags
• Colloquium Series
• Anderson Distinguished Lecture Series
• Speaker Videos
• Undergraduate Program
• Academic and Writing Resources

Writing Research Papers

• What Types of References Are Appropriate?

When writing a research paper, there are many different types of sources that you might consider citing.  Which are appropriate?  Which are less appropriate?  Here we discuss the different types of sources that you may wish to use when working on a research paper.   

Please note that the following represents a general set of recommended guidelines that is not specific to any class and does not represent department policy.  The types of allowable sources may vary by course and instructor.

Highly appropriate: peer-reviewed journal articles

In general, you should primarily cite peer-reviewed journal articles in your research papers.  Peer-reviewed journal articles are research papers that have been accepted for publication after having undergone a rigorous editorial review process.  During that review process, the article was carefully evaluated by at least one journal editor and a group of reviewers (usually scientists that are experts in the field or topic under investigation).  Often the article underwent revisions before it was judged to be satisfactory for publication. 

Most articles submitted to high quality journals are not accepted for publication.  As such, research that is successfully published in a respected peer-reviewed journal is generally regarded as higher quality than research that is not published or is published elsewhere, such as in a book, magazine, or on a website.  However, just because a study was published in a peer-reviewed journal does not mean that it is free from error or that its conclusions are correct.  Accordingly, it is important to critically read and carefully evaluate all sources, including peer-reviewed journal articles.

Tips for finding and using peer-reviewed journal articles:

• Many databases, such as PsycINFO, can be set to only search for peer-reviewed journal articles. Other search engines, such as Google Scholar, typically include both peer-reviewed and not peer-reviewed articles in search results, and thus should be used with greater caution. 
• Even though a peer-reviewed journal article is, by definition, a source that has been carefully vetted through an editorial process, it should still be critically evaluated by the reader. 

Potentially appropriate: books, encyclopedias, and other scholarly works

Another potential source that you might use when writing a research paper is a book, encyclopedia, or an official online source (such as demographic data drawn from a government website).  When relying on such sources, it is important to carefully consider its accuracy and trustworthiness.  For example, books vary in quality; most have not undergone any form of review process other than basic copyediting.  In many cases, a book’s content is little more than the author’s informed or uninformed opinion. 

However, there are books that have been edited prior to publication, as is the case with many reputable encyclopedias; also, many books from academic publishers are comprised of multiple chapters, each written by one or more researchers, with the entire volume carefully reviewed by one or more editors.  In those cases, the book has undergone a form of peer review, albeit often not as rigorous as that for a peer-reviewed journal article.

Tips for using books, encyclopedias, and other scholarly works:

• When using books, encyclopedias, and other scholarly works (that is, works written or produced by researchers, official agencies, or corporations), it is important to very carefully evaluate the quality of that source.
• If the source is an edited volume (in which case in the editor(s) will be listed on the cover), is published by a reputable source (such as Academic Press, MIT Press, and others), or is written by a major expert in the field (such as a researcher with a track record of peer-reviewed journal articles on the subject), then it is more likely to be trustworthy.
• For online encyclopedias such as Wikipedia, an instructor may or may not consider that an acceptable source (by default, don’t assume that a non-peer reviewed source will be considered acceptable). It is best to ask the instructor for clarification. 1

Usually inappropriate: magazines, blogs, and websites  

Most research papers can be written using only peer-reviewed journal articles as sources.  However, for many topics it is possible to find a plethora of sources that have not been peer-reviewed but also discuss the topic.  These may include articles in popular magazines or postings in blogs, forums, and other websites.  In general, although these sources may be well-written and easy to understand, their scientific value is often not as high as that of peer-reviewed articles.  Exceptions include some magazine and newspaper articles that might be cited in a research paper to make a point about public awareness of a given topic, to illustrate beliefs and attitudes about a given topic among journalists, or to refer to a news event that is relevant to a given topic. 

Tips for using magazines, blogs, and websites:

• Avoid such references if possible. You should primarily focus on peer-reviewed journal articles as sources for your research paper.  High quality research papers typically do not rely on non-academic and not peer-reviewed sources.
• Refer to non-academic, not peer-reviewed sources sparingly, and if you do, be sure to carefully evaluate the accuracy and scientific merit of the source.

Downloadable Resources

• How to Write APA Style Research Papers (a comprehensive guide) [ PDF ]
• Tips for Writing APA Style Research Papers (a brief summary) [ PDF ]

Further Resources

How-To Videos     

• Writing Research Paper Videos

Databases and Search Engines (may require connection to UCSD network)

• Google Scholar
• PubMed (NIH/NLM)
• Web of Science  

UCSD Resources on Finding and Evaluating Sources

• UCSD Library Databases A-Z
• UCSD Library Psychology Research Guide: Start Page
• UCSD Library Psychology Research Guide : Finding Articles
• UCSD Library Psychology Research Guide : Evaluating Sources

External Resources

• Critically Reading Journal Articles from PSU/ Colby College
• How to Seriously Read a Journal Article from Science Magazine
• How to Read Journal Articles from Harvard University
• How to Read a Scientific Paper Infographic from Elsevier Publishing
• Tips for searching PsycINFO from UC Berkeley Library
• Tips for using PsycINFO effectively from the APA Student Science Council

1 Wikipedia articles vary in quality; the site has a peer review system and the very best articles ( Featured Articles ), which go through a multi-stage review process, rival those in traditional encyclopedias and are considered the highest quality articles on the site.

Prepared by s. c. pan for ucsd psychology, graphic adapted from  t-x-generic-apply.svg , a public domain creation by the tango desktop project..

Back to top

• Research Paper Structure
• Formatting Research Papers
• Using Databases and Finding References
• Evaluating References and Taking Notes
• Citing References
• Writing a Literature Review
• Writing Process and Revising
• Improving Scientific Writing
• Academic Integrity and Avoiding Plagiarism
• Writing Research Papers Videos

Referencing Generator

Powered by chegg.

• Select style:
• Archive material
• Chapter of an edited book
• Conference proceedings
• Dictionary entry
• Dissertation
• DVD, video, or film
• E-book or PDF
• Edited book
• Encyclopedia article
• Government publication
• Music or recording
• Online image or video
• Presentation
• Press release
• Religious text

What Is Cite This For Me’s Reference Generator?

Cite This For Me’s open-access generator is an automated citation machine that turns any of your sources into references in just a click. Using a reference generator helps students to integrate referencing into their research and writing routine; turning a time-consuming ordeal into a simple task.

A referencing generator accesses information from across the web, drawing the relevant information into a fully-formatted bibliography that clearly presents all of the sources that have contributed to your work.

If you don’t know how to reference a website correctly, or have a fast-approaching deadline, Cite This For Me’s accurate and intuitive reference generator will lend you the confidence to realise your full academic potential. In order to get a grade that reflects all your hard work, your references must be accurate and complete. Using a citation machine not only saves you time but also ensures that you don’t lose valuable marks on your assignment.

Not sure how to format your citations, what citations are, or just want to find out more about Cite This For Me’s reference generator? This guide outlines everything you need to know to equip yourself with the know-how and confidence to research and cite a wide range of diverse sources in your work.

Why Do I Need To Reference?

Simply put, when another source contributes to your work, you have to give the original owner the appropriate credit. After all, you wouldn’t steal someone else’s possessions so why would you steal their ideas?

Regardless of whether you are referencing a website, an article or a podcast, any factual material or ideas you take from another source must be acknowledged in a citation unless it is common knowledge (e.g. Winston Churchill was English). Failing to credit all of your sources, even when you’ve paraphrased or completely reworded the information, is plagiarism. Plagiarising will result in disciplinary action, which can range from losing precious marks on your assignment to expulsion from your university.

What’s more, attributing your research infuses credibility and authority into your work, both by supporting your own ideas and by demonstrating the breadth of your research. For many students, crediting sources can be a confusing and tedious process, but it’s a surefire way to improve the quality of your work so it’s essential to get it right. Luckily for you, using Cite This For Me’s reference generator makes creating accurate references easier than ever, leaving more time for you to excel in your studies.

In summary, the citing process serves three main functions:

• To validate the statements and conclusions in your work by providing directions to other sound sources that support and verify them.
• To help your readers locate, read and check your sources, as well as establishing their contribution to your work.
• To give credit to the original author and hence avoid committing intellectual property theft (known as ‘plagiarism’ in academia).

How Do I Cite My Sources With The Cite This For Me Referencing Generator?

Cite This For Me’s reference generator is the most accurate citation machine available, so whether you’re not sure how to format in-text references or are looking for a foolproof solution to automate a fully-formatted bibliography, this referencing generator will solve all of your citing needs.

Crediting your source material doesn’t just prevent you from losing valuable marks for plagiarism, it also provides all of the information to help your reader find for themselves the book, article, or other item you are citing. The accessible interface of the reference generator makes it easy for you to identify the source you have used – simply enter its unique identifier into the citation machine search bar. If this information is not available you can search for the title or author instead, and then select from the search results that appear below the reference generator.

Don’t know how to reference a website? The good news is that by using tools such as Cite This For Me’s reference generator, which help you work smarter, you don’t need to limit your research to sources that are traditional to cite. In fact, there are no limits to what you can cite, whether you are referencing a website, a YouTube video or a tweet.

To use the reference generator, simply:

• Select your style from Harvard, APA, OSCOLA and many more*
• Choose the type of source you would like to cite (e.g. website, book, journal, video)
• Enter the URL , DOI , ISBN , title, or other unique source information to find your source
• Click the ‘Cite’ button on the reference generator
• Copy your new citation straight from the referencing generator into your bibliography
• Repeat for each source that has contributed to your work.

*If you require another style for your paper, essay or other academic work, you can select from over 1,000 styles by creating a free Cite This For Me account.

Once you have created your Cite This For Me account you will be able to use the reference generator to create multiple references and save them into a project. Use Cite This For Me’s highly-rated iOS or Android apps to generate references in a flash with your smartphone camera, export your complete bibliography in one go, and much more.

What Will The Reference Generator Create For Me?

Cite This For Me’s reference generator will create your citation in two parts: an in-text citation and a full citation to be copied straight into your work.

The reference generator will auto-generate the correct formatting for your bibliography depending on your chosen style. For instance, if you select a parenthetical style the reference generator will generate an in-text citation in parentheses, along with a full citation to slot into your bibliography. Likewise, if the reference generator is set to a footnote style then it will create a fully-formatted citation for your reference list and bibliography, as well as a corresponding footnote to insert at the bottom of the page containing the relevant source.

Parenthetical style examples:

In-text example: A nation has been defined as an imagined community (Anderson, 2006).* Alternative format: Anderson (2006) defined a nation as an imagined community.

*The reference generator will create your references in the first style, but this should be edited if the author’s name already appears in the text.

Bibliography / Works Cited list example: Anderson, B. (2006). Imagined Communities. London: Verso.

What Are Citation Styles?

A citation style is a set of rules that you, as an academic writer, must follow to ensure the quality and relevance of your work. There are thousands of styles that are used in different academic institutions around the world, but in the UK the most common are Harvard, APA and Oscola.

The style you need to use will depend on the preference of your lecturer, discipline or academic institution – so if you’re unsure which style you should be using, consult your department and follow their guidelines exactly, as this is what you’ll be evaluated on when it comes to marking. You can also find your university’s style by logging into your Cite This For Me account and setting your institution in ‘My Profile’.

Citing isn’t just there to guard against plagiarism – presenting your research in a clear and consistent way eases the reader’s comprehension. Each style has a different set of rules for formatting both the page and your references. Be sure to adhere to formatting rules such as font type, font size and line spacing to ensure that your work is easily legible. Furthermore, if your work is published as part of an anthology or collected works, each entry will need to be presented in the same style to maintain uniformity throughout. It is important to make sure that you don’t jump from one style to another, so follow the rules carefully to ensure your reference list and bibliography are both accurate and complete.

If you need a hand with your citations then why not try Cite This For Me’s reference generator? It’s the quickest and easiest way to cite any source, in any style. The reference generator above will create your citations in the Harvard referencing style as standard, but it can generate fully-formatted references in over 1,000 styles – including university variations of each style. So, whether your lecturer has asked you to adopt APA referencing , or your subject requires you to use OSCOLA referencing , we’re sure to have the style you need. To access all of them, simply go to Cite This For Me’s website to create your free Cite This For Me account and search for your specific style such as MLA or Vancouver .

How Do I Format A Reference List Or Bibliography?

Drawing on a wide range of sources greatly enhances the quality of your work, and reading above and beyond your recommended reading list – and then using these sources to support your own thesis – is an excellent way to impress your reader. A clearly presented reference list or bibliography demonstrates the lengths you have gone to in researching your chosen topic.

Typically, a reference list starts on a new page at the end of the main body of text and includes a complete list of the sources you have actually cited in your paper. This list should contain all the information needed for the reader to locate the original source of the information, quote or statistic that directly contributed to your work. On the other hand, a bibliography is a comprehensive list of all the material you may have consulted throughout your research and writing process. Both provide the necessary information for readers to retrieve and check the sources cited in your work.

Each style’s guidelines will define the terminology of ‘reference list’ and ‘bibliography’, as well as providing formatting guidelines for font, line spacing and page indentations. In addition, it will instruct you on how to order each list – this will usually be either alphabetical or chronological (meaning the order that these sources appear in your work). Before submitting your work, be sure to check that you have formatted your whole paper according to your style’s formatting guidelines.

Sounds complicated? Citing has never been so easy; Cite This For Me’s reference generator will automatically generate fully-formatted citations for your reference list or bibliography in your chosen style. Sign in to your Cite This For Me account to save and export your bibliography.

How Do References Actually Work?

Although the reference generator will create your bibliography for you in record time, it is still useful to understand how this system works behind the scenes. As well as saving you time with its referencing generator, Cite This For Me provides the learning resources to help you fully understand the citing process and the benefits of adopting great citing standards.

The referencing process:

• Find a book, journal, website or other source that will contribute to your work
• Save the quote, image, data or other information that you will use in your work
• Save the source information that enables you to find it again (i.e. URL, ISBN, DOI etc.)
• Format the source information into a citation
• Copy and paste the citation into the body of the text
• Repeat for each source that contributes to your work.
• Export or copy and paste the fully-formatted citation into your bibliography.

Manage all your references in one place

Create projects, add notes, cite directly from the browser and scan books’ barcodes with a mobile app.

Sign up to Cite This For Me – the ultimate reference management tool.

How To Write A Research Paper

Find Sources For A Research Paper

How to Find Sources For a Research Paper | A Guide

10 min read

Published on: Mar 26, 2024

Last updated on: Mar 25, 2024

People also read

How to Write a Research Paper Step by Step

How to Write a Proposal For a Research Paper in 10 Steps

A Comprehensive Guide to Creating a Research Paper Outline

Types of Research - Methodologies and Characteristics

300+ Engaging Research Paper Topics to Get You Started

Interesting Psychology Research Topics & Ideas

Qualitative Research - Types, Methods & Examples

Understanding Quantitative Research - Definition, Types, Examples, And More

Research Paper Example - Examples for Different Formats

How To Start A Research Paper - Steps With Examples

How to Write an Abstract That Captivates Your Readers

How To Write a Literature Review for a Research Paper | Steps & Examples

Types of Qualitative Research Methods - An Overview

Understanding Qualitative vs. Quantitative Research - A Complete Guide

How to Cite a Research Paper in Different Citation Styles

Easy Sociology Research Topics for Your Next Project

200+ Outstanding History Research Paper Topics With Expert Tips

How To Write a Hypothesis in a Research Paper | Steps & Examples

How to Write an Introduction for a Research Paper - A Step-by-Step Guide

How to Write a Good Research Paper Title

How to Write a Conclusion for a Research Paper in 3 Simple Steps

How to Write an Abstract For a Research Paper with Examples

How To Write a Thesis For a Research Paper Step by Step

How to Write a Discussion For a Research Paper | Objectives, Steps & Examples

How to Write the Results Section of a Research Paper - Structure and Tips

How to Write a Problem Statement for a Research Paper in 6 Steps

How To Write The Methods Section of a Research Paper Step-by-Step

Share this article

Research papers are an essential part of academic life, but one of the most challenging aspects can be finding credible sources to support your arguments. 

With the vast amount of information available online, it's easy to feel overwhelmed. However, by following some simple steps, you can streamline the process of finding reliable sources for your research paper . 

In this guide, we'll break down the process into easy-to-follow steps to help you find the best sources for your paper.

On This Page On This Page -->

Step 1: Define Your Topic and Research Questions

Before you venture into your quest for sources, it's essential to have a clear understanding of your research topic and the specific questions you aim to address. Define the scope of your paper and identify keywords and key concepts that will guide your search for relevant sources.

Step 2: Utilize Academic Databases

Academic databases are treasure troves of scholarly articles, research papers, and academic journals covering a wide range of subjects. Institutions often provide access to these databases through their libraries. Some popular academic databases include:

• IEEE Xplore
• Google Scholar

These databases allow you to search for peer-reviewed articles and academic papers related to your topic. 

Use advanced search features to narrow down your results based on publication date, author, and keywords .

Academic Resources Classified by Discipline

Here's a breakdown of prominent databases categorized by academic discipline:

Step 3: Explore Library Catalogs

Your university or local library's catalog is another valuable resource for finding sources. Library catalogs contain books, periodicals, and other materials that may not be available online. 

Use the catalog's search function to locate relevant books, journals, and other materials that can contribute to your research.

Step 4: Consult Bibliographies and References

When you find a relevant source, take note of its bibliography or make a list of sources for the research paper. These lists often contain citations to other works that may be useful for your research. 

By exploring the references cited in a particular source, you can uncover additional resources and expand your understanding of the topic.

Step 5: Boolean Operators for Effective Searches

Boolean operators are words or symbols used to refine search queries by defining the relationships between search terms. The three primary operators include "AND," which narrows searches by requiring all terms to be present; "OR," which broadens searches by including either term or both; and "NOT," which excludes specific terms to refine results further. 

Most databases provide advanced search features for seamless application of Boolean logic.

Step 6: Consider Primary Sources 

Depending on your research topic, primary sources such as interviews, surveys, archival documents, and original data sets can provide valuable insights and support for your arguments. 

Primary sources offer firsthand accounts and original perspectives on historical events, social phenomena, and scientific discoveries.

Step 7: Evaluate the Credibility of Sources

Not all sources are created equal, and it's crucial to evaluate the credibility and reliability of the information you encounter. 

Consider the author's credentials, the publication venue, and whether the source is peer-reviewed. Look for evidence of bias or conflicts of interest that may undermine the source's credibility.

Step 8: Keep Track of Your Sources

As you gather sources for your research paper, maintain a systematic record of the materials you consult.  Keep track of bibliographic information, including author names, publication dates, titles, and page numbers . This information will be invaluable when citing your sources and creating a bibliography or works cited page.

Other Online Sources

In addition to academic databases and library catalogs, exploring popular online sources can provide valuable insights and perspectives on your research topic.  Here are some types of online sources you can consider:

Websites hosted by reputable organizations, institutions, and experts (such as the New York Times) can offer valuable information and analysis on a wide range of topics. Look for websites belonging to universities, research institutions, government agencies, and established non-profit organizations.

Crowdsourced Encyclopedias like Wikipedia

While Wikipedia can provide a broad overview of a topic and lead you to other sources, it's essential to verify the information found there with more authoritative sources. 

Use Wikipedia as a starting point for your research, but rely on peer-reviewed journal articles and academic sources for in-depth analysis and evidence.

Tips for Assessing the Credibility of Online Sources

When using online sources, it's important to exercise caution and critically evaluate the credibility and reliability of the information you find. Here are some tips for assessing the credibility of online sources:

• Check the Domain Extension: Look for websites with domain extensions that indicate credibility. URLs ending in .edu are educational resources, while URLs ending in .gov are government-related resources. These sites often provide reliable and authoritative information.
• Look for DOIs (Digital Object Identifiers): DOIs are unique alphanumeric strings assigned to scholarly articles and indicate that the article has been published in a peer-reviewed, scientific journal. Finding a DOI can help you assess the scholarly rigor of the source.
• Evaluate the Authorship and Credentials: Consider the qualifications and expertise of the author or organization behind the website or blog. Look for information about the author's credentials, affiliations, and expertise in the subject matter.
• Consider the Currency and Relevance: Assess how up-to-date the information is and whether it aligns with the scope and focus of your research. Look for recent publications and timely analyses that reflect current trends and developments in the field.

Wrapping it up!

Finding sources for your research paper may seem like a challenge, but by following these steps, you can locate credible sources to support your arguments and enhance the quality of your paper. 

By approaching the research process systematically and critically evaluating the information you encounter, you can produce a well-researched and compelling research paper.

If you are struggling with finding credible sources or have time constraints, do not hesitate to seek writing help for your research papers . CollegeEssay.org has professional writers ready to assist you. 

Connect with our essay writing service now and receive expert guidance and support to elevate your research paper to the next level.

Cathy A. (Law)

For more than five years now, Cathy has been one of our most hardworking authors on the platform. With a Masters degree in mass communication, she knows the ins and outs of professional writing. Clients often leave her glowing reviews for being an amazing writer who takes her work very seriously.

Paper Due? Why Suffer? That’s our Job!

Keep reading

Legal & Policies

• Privacy Policy
• Cookies Policy
• Terms of Use
• Refunds & Cancellations
• Our Writers
• Success Stories
• Our Guarantees
• Affiliate Program
• Referral Program
• AI Essay Writer

Disclaimer: All client orders are completed by our team of highly qualified human writers. The essays and papers provided by us are not to be used for submission but rather as learning models only.

Assessing Religious Leadership: A Scoping Review of Leadership Effectiveness Criteria and Current Trends in the Academic Literature

• Open access
• Published: 15 November 2023
• Volume 73 , pages 1–21, ( 2024 )

Cite this article

You have full access to this open access article

• Annemarie Foppen   ORCID: orcid.org/0000-0002-8527-8540 1 &
• Joke W. van Saane 2  

1315 Accesses

3 Altmetric

Explore all metrics

A clear understanding of what constitutes effective religious leadership in the context of faith communities is essential for religious practitioners, religious communities, and educational institutes. Twenty-five years after the latest review study by Nauss, an updated overview is needed to account for new insights, especially regarding the latest developments in leadership research and the changing religious landscape. A scoping review was conducted in four databases from 1997 to 2022 to examine the academic literature. The search identified 64 unique references that researched effective religious leadership using 27 different effectiveness criteria. These criteria were categorized based on their target of evaluation: (1) the religious leader, (2) followers/members, and (3) the congregation. Although most references assessed leadership effectiveness by evaluating the religious leader, the most used effectiveness criterion was ‘attendance or numerical growth.’ Over the last 25 years, the criteria for religious leadership effectiveness have become more diverse, and a growing number of references combine criteria from multiple targets to evaluate effectiveness. However, the focus on Christian leadership remains prevalent in the academic literature. The present overview is intended as a starting point for future research as it identifies the current trends and existing knowledge gaps. The study’s findings also invite religious practitioners and congregations to reflect on their methods of assessing leadership effectiveness.

Similar content being viewed by others

Epilogue: What Shall We Do Now?

A systematic review of leader credibility: its murky framework needs clarity

Ralph I. Williams Jr, Deana M. Raffo, … Leigh Anne Clark

Analyzing Leadership Attributes in Faith-Based Organizations: Idealism Versus Reality

Krystin Zigan, YingFei Héliot & Alan Le Grys

Avoid common mistakes on your manuscript.

Introduction

Assessing religious leadership is important for religious practitioners, religious communities, and educational and religious institutes. Insight into which competencies make religious leaders effective and how they affect people’s lives and their congregations can result in more vibrant religious communities (Boyatzis et al., 2011 ). However, identifying the criteria for religious leadership effectiveness has proven difficult (Malony, 2000 ; Nauss, 1994 ). The measurement of effectiveness among religious leaders is complicated and, in some contexts, debated or even resisted (McKenna & Eckard, 2009 ). Some consider religious leadership effectiveness untraceable due to its spiritual or transcendental dimension (Malony, 1984 ), making it not objectively measurable. For others, existing methods of assessment are perceived as uninspiring or unsuitable for their congregation (e.g., numerical growth; Dadswell & Ross, 2013 ; Strunk et al., 2017 ). The diverse contexts in which religious leaders operate provide another challenge. What is considered effective religious leadership can vary depending on the context, religion, denomination, congregational size, and geographical location (Nauss, 1996 ). Just as leadership has been defined in many ways, conceptions of leadership effectiveness vary among scholars; as Yukl and Gardner ( 2020 ) state, “The selection of [leadership effectiveness] criteria depends on the objectives and values of the person making the evaluation” (p. 30). As there does not seem to be a shared definition of leadership effectiveness in the literature nor in the domain of religion, and as most references studied were not explicit about their definition, the current study aims to provide an overview of the various effectiveness criteria used to assess religious leadership in the recent academic literature. Instead of proposing our own definition of effective religious leadership, we intend to identify and clarify the different ways effective religious leadership is being evaluated, sorted by the target of evaluation.

DeRue et al. ( 2011 ) propose a model of leader traits, behaviors, and effectiveness in which leadership effectiveness criteria are categorized along three dimensions to organize the various effectiveness criteria. The model differentiates between effectiveness criteria based on their (1) content (i.e., task performance, affective and relational criteria, or overall judgments of effectiveness), (2) the level of analysis (i.e., individual, dyadic, group, or organizational), and (3) the target of evaluation (e.g., leader effectiveness, group performance).

In his review study, Nauss ( 1996 ) utilizes the latter as he identifies two types of criteria used for evaluating clergy effectiveness based on their target of evaluation, namely, the minister and the congregation. The first type, referred to as primary criteria, includes descriptions of the behavior and characteristics of the minister, such as ‘preaching behavior,’ ‘ability to serve,’ ‘personal characteristics,’ and ‘traits.’ The second type, i.e., secondary criteria, focuses on the effects or consequences of the minister on the congregation and its members, such as ‘church growth,’ ‘commitment,’ ‘satisfaction,’ and ‘changes in faith.’ To the best of our knowledge, Nauss's review of the empirical literature remains, to this day, the most comprehensive overview of assessing effective religious leadership in the academic literature. However, the kind of leadership that is considered effective changes over time. Social, cultural, and technological changes have significantly impacted religious practice, causing changes in our perceptions of the behaviors and characteristics that contribute to religious leaders’ effectiveness (Dobrotka, 2021 ). Thus, after 25 years, the field requires an updated overview to account for new insights, especially concerning the latest developments in leadership research and the changing religious landscape.

A changing landscape

Global mobility has changed the religious landscape radically from the 20th to the 21st century as millions of migrants have brought their religions, values, and traditions to other parts of the world (Giordan, 2014 ). Consequently, many nations transformed from homogeneous religious societies into places with much religious diversity. The work of Nauss ( 1996 ) and the empirical studies he reviews, however, are limited to Christian leadership, in most cases Protestant ministers. The social scientific study of religion traditionally also has a narrow focus on Christianity and Judaism. However, other religions have gradually been included, such as Hinduism, Buddhism, Islam, and new religious movements (Rambo & Haar Farris, 2012 ). For the current article, we aimed to take into account the growing variety of religions and traditions by not focusing on a specific religion in our data search.

Additionally, almost every high-income country faces lower levels of religiosity, and many other countries have become less religious as well (Inglehart, 2021 ). This trend is reflected in the declining number of individuals involved in organized religion in Western societies and attending religious services (Brauer, 2018 ; European Social Survey, 2002 , 2016  as cited in De Hart & Van Houwelingen, 2018 ). At the same time, a rise in post-Christian or New Age spirituality is observed, characterized by a more individualized, experience-oriented, and anti-institutional spirituality (Marshall & Olson, 2018 ; Tromp et al., 2020 ). Along with these developments, confidence in and obedience to traditional religious leaders has declined in these societies (Hoffmann, 2013 ; Inglehart, 2020 ), giving rise to new types of authority and leadership. These changes in the religious landscape present new challenges for religious leaders in the West and will undoubtedly impact how effective religious leadership is viewed and evaluated.

Over the past decades, academic thinking about leadership has also changed (Zhu et al., 2019 ). In line with most of the earlier studies on ministerial effectiveness, Nauss focused on the religious leader and determined effective religious leadership in terms of ministerial functions or activities (McKenna & Eckard, 2009 ; Nauss, 1972 , 1996 ). This approach to leadership can be categorized as part of what Haslam et al. ( 2015 ) call “the classical leadership perspective.” New insights in leadership research, however, suggest that leadership is not solely about the person of the leader (e.g., character and competencies) but rather should be conceived of as a relational process between the leader, the followers, and the group (Haslam et al., 2020 ; Steffens et al., 2021 ). Leadership is now generally studied as a process of influencing other people to accomplish shared goals (Wilson, 2022 ). In line with this, most researchers evaluate leadership effectiveness “in terms of the consequences of influence on an individual, group, or organization” (Yukl & Gardner, 2020 , p. 28).

The current study

To account for these changes and developments in the religious landscape and leadership research, we conducted a scoping review in the recent academic literature to provide an overview of the various effectiveness criteria used to assess religious leadership over the last 25 years. A systematic approach was employed to search for studies on religious leadership that explicitly address the issue of effectiveness. A combination of search terms was used to account for the variety of religions (i.e., Abrahamic faiths, Hinduism, Buddhism, and spirituality), the various terms for effectiveness (e.g., success, flourishing, thriving, vitality, vibrancy), and the various indicators for religious leaders (e.g., clergy, minister, pastor, imam, priest, rabbi). However, to limit ourselves, we focused exclusively on what Chaves et al. ( 2022 ) call congregations’ religious leaders: “religious specialists who are leaders of organized religious communities.” As a result, this review did not examine religious or spiritual leaders in more diffuse and fluid, interstitial settings (Ammerman, 2020 ), such as spiritual guides or meditation teachers; nor religious or spiritual leadership in secular or organizational contexts, such as spiritual caregivers in health institutions (Timmins et al., 2018 ); or organizational leaders who incorporate spiritual values in the workplace (Oh & Wang, 2020 ). In addition, this review concentrated solely on the academic literature (i.e., peer-reviewed articles, books, and dissertations) as this literature could be accessed and searched systematically through bibliographic databases. Also, this approach ensures that only studies meeting the highest scholarly quality standards are included. Consequently, we did not evaluate the professional literature and the extensive work being done outside of academia, such as Duke’s Faith and Leadership initiative. We strongly encourage others to follow up on the current study by reviewing this body of literature as well.

The purpose of the current study is to contribute to the field by building on the work of Nauss ( 1996 ) and extending it in three ways: (1) by updating the data and mapping religious leadership effectiveness criteria that have been used over the last 25 years, (2) by charting recent trends and developments in the academic literature, and (3) by broadening the search from an emphasis on Protestant Christian ministers to reviewing effective religious leadership across different religions. The studies’ findings and conclusions are used to propose directions for future research on the topic.

The present study used a scoping review methodology to examine the academic literature on effective religious leadership. This scoping review aimed to survey the academic literature on effective religious leadership, identify the criteria used to evaluate effectiveness, and detect any knowledge gaps (see Munn et al., 2018 ). We adhered to the five steps of Arksey and O’Malley’s ( 2005 ) methodological framework for conducting a scoping study—(1) identifying the research question, (2) identifying relevant studies, (3) selecting the studies, (4) charting the data, (5) and compiling, summarizing, and reporting the results (Levac et al., 2010 ). The research question that guided this scoping review is: What leadership effectiveness criteria are used in the recent academic literature to assess religious leadership?

Identifying relevant studies

The review of Nauss ( 1996 ), which assessed relevant studies on effective religious leadership up to and including 1996, served as a reference point. A comprehensive search was performed in the bibliographic databases Atla Religion Database, PsycInfo, Eric, and Psychology and Behavioral Sciences Collection (via Ebsco) from January 1, 1997, to January 12, 2022. To conduct the search, we first combined various terms for ‘religion’ and ‘leaders(hip)’ to search for leadership studies across various religions. In addition, all studies with direct synonyms for religious leaders were searched for using terms such as ‘clergy,’ ‘pastor,’ ‘minister,’ ‘rabbi,’ and ‘imam.’ Next, a focus on effectiveness was added by combining the search string above with the term ‘effectiveness’ and closely related indicators such as ‘success,’ ‘flourishing,’ ‘thriving,’ ‘vital,’ ‘growing,’ ‘vibrant,’ and ‘efficacy.’ The full search strategy is presented in Table 1 . From these results, we included journal articles, dissertations, and books in the final search. The search was performed without language restrictions, and duplicates were excluded. The database search identified a total of 11,653 references.

Study selection

The two-stage review process was divided into a title-and-abstract screening and a full-text screening round. The 11,653 references were submitted to Rayyan (rayyan.qcri.org), an online review application, to facilitate the review process. References were included when they studied congregations’ religious leaders, defined in this study as “religious specialists who are leaders of organized religious communities,” as opposed to religious specialists who are private practitioners (Chaves et al., 2022 , p. 737). Next, to be included in this study, references had to explicitly address the issue of effectiveness either as a focus of empirical research or theoretical reflection. The first author screened the articles and discussed her findings midway with the second author to further refine the inclusion and exclusion criteria. In addition, the second author reviewed a random selection of articles to assess the reliability of the review process ( n  = 112), resulting in an interrater agreement of 71% regarding which studies to include or exclude.

A total of 11,541 references were excluded during the abstract-and-title screening because they did not meet the inclusion criteria above, because they did not meet the initial criteria (i.e., not being published as a journal article, dissertation, or book), because an abstract was not available ( n  = 17), or because an abstract was not available in English and the reference was written in a language other than English, Dutch, French, or German ( n  = 2). Next, a full-text screening was performed with the remaining 112 references. During this phase, it was decided to focus only on journal articles and dissertations as obtaining the full text of the seven remaining books was challenging. Another 40 references were excluded based on the inclusion criteria during the full-text evaluation. One reference was excluded because the authors of the current study were not proficient in Polish. See Fig.  1 for the flow diagram and the number of references included in this review.

Flow Diagram of the Screening Process

Charting the data

For each of the 64 references included in this study, an overview was created with general information about the study. This overview included the study’s author(s), year of publication, research question, theoretical embedding, research methodology, the method used to assess effective religious leadership, and the most important outcomes reported in the study’s abstract. Charting how effective religious leadership was defined across the different studies proved difficult as many references did not provide an explicit definition. After careful consideration, it was decided to focus on recording the specific criteria used to assess religious leadership effectiveness.

Twenty-six of the 64 references examining effective religious leadership that were included in this study were published in 21 different academic journals, while the other 38 studies were dissertations. Ten studies were theoretical, literature-based contributions, whereas the additional 54 were empirical (i.e., 22 quantitative studies, 17 qualitative studies, and 15 mixed-method studies). Although the studies varied with regard to their discipline (e.g., psychology, religious studies, theology) and motives (to describe, understand, predict, or prescribe), there was a strong focus on Christian leadership ( n  = 60), with only three studies investigating Muslim leadership and one study researching religious leadership in general. In the years after 2010, the number of references published each year was considerately higher, with an average of 2.3 before 2010 and 3.5 from 2011 onwards. The nationality of the authors varied only minimally. All the dissertations were submitted to American universities, and the first author was situated in the United States in 16 of the 26 journal articles. The other 10 references were written by scholars from Australia (2), Nigeria (2), Germany (1), Indonesia (1), Ireland (1), Malaysia (1), South Africa (1), and the United Kingdom (1). An overview of the descriptives can be found in Table 2 .

Religious leadership effectiveness criteria

Twenty-seven different criteria of religious leadership effectiveness were identified among the included references. The criterion used most often in the references to assess effective religious leadership was ‘attendance or numerical growth’ (20 out of the 64 references; 31%). Other criteria that were used in ten or more of the references were ‘personal spirituality/character’ (11 references; 17%), ‘leadership style’ (10 references; 16%), ‘leadership skill and qualities’ (10 references; 16%), ‘perceived effectiveness’ (10 references; 16%), and ‘organizational mission and goals’ (10 references, 16%). Although we observed some differences between the journal articles and the dissertations, the overall use of criteria was quite similar, with an overlap of 23 out of the 27 identified criteria (89%). The most significant difference was observed in the criteria of ‘personal(ity) traits’ and ‘numerical growth or attendance’ as they were used as a criterion in 8 and 14 dissertation studies, respectively, versus 1 and 6 journal articles.

Following the model by DeRue et al. ( 2011 ), we categorized the leadership effectiveness criteria by their target of evaluation, which refers to “whether the leader is the target of evaluation (e.g., leader effectiveness, satisfaction with leader) or another outcome that is within the domain of leadership effectiveness but not specific to the leader [such as] group performance” (p. 11). The criteria used to assess religious leadership effectiveness could be subdivided into three target groups of evaluation:

leader -focused criteria that assess characteristics and outcomes associated with the religious leader as the target of evaluation, such as leadership style, leadership skills, or perceived effectiveness;

follower/member -focused criteria that evaluate outcomes specific to the followers or members of the congregations, such as satisfaction or spiritual growth;

congregation -focused criteria of effectiveness that evaluate outcomes associated with the religious community, such as numerical growth or attendance and organizational mission and goals.

An overview of the criteria sorted by target group can be found in Table 3 .

Leader-focused criteria

References that used leader-focused criteria examined individual characteristics, traits, skills, and qualities of the religious leader to determine effectiveness. The most utilized effectiveness criteria were ‘personal spirituality/character’ and ‘leadership style.’ The leadership styles associated with effective leadership in these references center on the relationship between leader and followers, such as transformational leadership (Taylor, 2019 ; Wasberg, 2013 ), servant leadership (Agee, 2001 ; Graham-Brown, 2020 ), and relational leadership (Watt, 2014 ). Other references list specific leadership skills, qualities, or personal traits as characteristic of effective religious leadership. The Ministerial Effectiveness Inventory (MEI; Majovski, 1982 ) was used in three references to assess leaders’ ministerial competencies; it contains items such as ‘the minister enables the congregation to experience opportunities for personal growth and spiritual enrichment’ and ‘I have an approach to ministry that emphasizes evangelistic and mission goals.’

Follower/member-focused criteria

The effectiveness criteria in this target group center around the congregants as the primary focus of leadership evaluation. Some criteria that focus on followers or members include their satisfaction with leadership, worship, and sacraments; how well their needs are being met; and their sense of community and financial support to the congregation. The most common follower/member-focused criterion used to measure leadership effectiveness was ‘spiritual growth.’

Congregation-focused criteria

The congregation-focused criteria operationalized leadership effectiveness through its effect on the religious community. The criterion most frequently used with respect to the congregation was’numerical growth/attendance.’ Other numerical criteria in this category included the number of conversions, baptisms, church plants, and religious classes. Additionally, references focused on the achievement of ‘organizational mission and goals,’ the degree of ‘community service and outreach,’ and congregational health / well-being as indicators of leadership effectiveness in the congregation.

Ways of assessing effective religious leadership

We observed different approaches to assessing effective religious leadership in the included references based on the targets that were used for evaluation. Some studies assessed religious leadership solely by using leader-focused criteria; other references exclusively used follower/member-focused or congregation-focused criteria. A last group combined the three target groups in various ways to determine leadership effectiveness (i.e., multiple targets). Most references utilized leader-focused criteria to determine leader effectiveness (44%), but a significant number of references used a multiple-targets approach to assess effective religious leadership (34%). This approach was especially prevalent in the dissertation studies. These references tended to take a more comprehensive approach to assessing religious leadership (e.g., Boyatzis et al., 2011 ; McKenna & Eckard, 2009 ). Among the journal articles, the focus on leader indicators was particularly prevalent (see Table 4 for an overview).

Trends and developments in the field

Indicators of effectiveness: from a focus on the minister’s behavior to a more diverse pallet.

While the main focus in Nauss’s review ( 1996 ) was on the minister and their observable behavior, more recent studies also include followers and the congregation as a factor in their evaluation of leadership effectiveness, with ‘numerical growth or attendance’ being the most used criterion of effectiveness. However, the centrality of leader-focused criteria still appears in the recent literature, especially in journal articles. At the same time, the pallet of indicators has become more diverse. For instance, ‘leadership style’ and ‘personal spirituality/character’ were added to the list of indicators, as well as some follower/member- and congregation-focused criteria such as meeting ‘followers’ needs,’ ‘congregational health,’ and achieving ‘organizational mission and goals.’

Beyond the bifocal view: Interactions between leader and congregation

In 1996, Nauss observed a bifurcation in his review study on how ministerial effectiveness was defined. References either focused “upon the minister’s behavior or the effects of the minister’s work in the activities and behavior of the church and its members.” A significant number of the references included in the current study departed from this bifocal view on assessing effective religious leadership by adopting a more dynamic approach. Twenty-two references explicitly combined criteria from multiple target groups to determine leadership effectiveness (5 articles and 17 dissertations). Also, many references that assessed effectiveness by focusing on one specific target often took the interactions between the leader and the congregation into account. For example, in various studies, leader characteristics were used as a predictor of follower/member-focused or congregation-focused criteria of effectiveness (e.g., Adams, 2013 ; Al-Hilali, 2012 ; Cobb-Hayes, 2018 ; Oney, 2009 ; Rowold, 2008 ; Rumley, 2011 ; Soldo, 2017 ). In addition, the transformational leadership model, a follower-oriented approach, was a recurring theory used by about 20 references (30%), either in the study’s design or as part of the theoretical embedding.

The “ultimate criterion”: A value-based approach to assessing religious leadership persists

A value-based approach to assessing religious leadership was recognized across the references as dominant criteria of effectiveness included ‘personal spirituality/character,’ leadership styles such as servant leadership, ethical leadership, and spiritual leadership, ‘organizational mission and goals,’ and ‘spiritual growth.’ Spiritual indicators were already being used as empirical measures of effectiveness in the review by Nauss ( 1996 ), but only minimally.

Measures of effective religious leadership remain scarce

Based on the included references, it appears that hardly any new instruments have been developed to measure effective religious leadership over the past 25 years, and there seems to be a specific lack of validated measures. The Ministerial Effectiveness Inventory by Majovski ( 1982 ) was used three times to map ministerial competencies. Another nine references administered the Multifactor Leadership Questionnaire (Avolio & Bass, 2004 ), a psychological inventory that measures transformational, transactional, and laissez-faire leadership and three leadership outcomes (i.e., followers’ perception of leadership effectiveness, followers’ satisfaction with the leadership, and the leader’s ability to meet followers’ needs, inspire extra effort, and achieve organizational goals). In addition, studies that used congregation-focused criteria indicators to assess leadership effectiveness sometimes employed congregational models, such as the National Church Live Survey and the Natural Church Development characteristics (Powell et al., 2022 ; Schwarz, 2006 ).

The focus on Christian leadership remains

Only four references were included in this study that examined effective religious leadership outside the Christian context (6%). One study discussed religious leadership in general (Malony, 2000 ), and three studies examined Muslim leadership (Al-Hilali, 2012 ; Mahazan & Abdullah, 2013 ; Soldo, 2017 ). Given the few references that examined religious leadership outside the Christian context, there still appears to be a one-sided emphasis on Christian leadership in the existing literature. Although, at first glance, the ways of assessing religious leadership in these four studies do not appear to diverge from the main findings discussed above, we cannot generalize our findings on religious leadership effectiveness outside of the Christian context.

The present study conducted a scoping review to provide an up-to-date overview of how effective religious leadership has been assessed over the last 25 years in the academic literature and to chart recent trends and developments in the field. A literature search yielded 64 references that together used 27 different criteria to assess religious leadership effectiveness. The criteria were categorized based on their target group of evaluation, namely, leader, follower/member, or congregation. Most references used leader-focused criteria to assess religious leadership, but a significant number of studies also employed a multiple-targets approach to determine leadership effectiveness. Comparing our results to the review conducted by Nauss ( 1996 ), we did not observe drastic changes in how effective religious leadership is assessed. There were, however, some significant developments, and these are discussed below.

Next to the importance of leader-focused criteria, follower/member-focused and congregation-focused criteria are increasingly being included as valid measures of effectiveness. ‘Attendance or numerical growth’ was the criterion most frequently used in the included references to assess effectiveness. Although congregational growth has been debated as a criterion of vitality or effectiveness (e.g., Thiessen et al., 2019 ), success and status are often attributed to those leaders who can break with the downward trend and draw large numbers of visitors to their services. Linking congregational growth or decline to leadership efforts aligns with several studies showing that clergy contribute to church growth (e.g., Francis et al., 2015 ; Haskell et al., 2016 ; Wollschleger, 2018 ).

Recent references adopted a more dynamic and extensive approach to assessing effective religious leadership by combining multiple criteria and different target groups. The shift from a strong focus on the leader to a more follower- and group-oriented approach to religious leadership has taken longer than it has in the mainstream research on leadership (Haslam et al., 2020 , pp. 19–44) but appears to be prevalent in the current literature on religious leadership as well. Transformational leadership has a prominent place in the references in the current review study, as it does in most contemporary leadership research (Dinh et al., 2014 ; Zhu et al., 2019 ), which shows a continued interest in (neo)charismatic leadership topics such as inspirational leadership, visionary leadership, empowerment, and followers’ trust. Transformational leadership refers to a multidimensional leadership style in which leaders motivate followers to perform beyond expectations by transforming followers’ attitudes, beliefs, and values as opposed to simply gaining compliance (Bass, 1985 ; Rafferty & Griffin, 2004 ; Yukl, 1999 ). Behavior linked to transformational leadership has the following four components (Bass & Riggio, 2005 ): idealized influence (i.e., serving as a role model for followers), inspirational motivation (i.e., communicating a stimulating vision), intellectual stimulation (i.e., encouraging innovation and creativity), and individualized consideration (i.e., attention for followers’ development).

A value-based approach to assessing religious leadership remained prevalent in the reviewed body of research, with many references using indicators such as ‘personal spirituality/character,’ ‘servant leadership,’ and ‘spiritual growth’ to determine effectiveness. A normative approach to assessing religious leadership also appears inherent to researching leadership in the religious context (McKenna & Eckard, 2009 ; Wollschleger, 2018 ). The spiritual dimension entails a normative element to effective religious leadership, resulting in ultimate criteria of effectiveness based on specific beliefs or values that are so important that they are usually not open for debate or modification. Although such a prescriptive approach to leadership appears specific to religion, value-based approaches to leadership are also emerging in mainstream research on leadership (Dinh et al., 2014 ; Zhu et al., 2019 ). Recent studies demonstrate a growing interest in leadership behaviors that include value elements such as ethical leadership, authentic leadership, spiritual leadership, and servant leadership. Zhu et al. ( 2019 ) link this trend to a growing focus on corporate social responsibility.

Implications for research and practice

By identifying the various criteria of effectiveness and sorting the literature into three target groups for assessing religious leadership, this study offers clarity and coherence in a rather fragmented field of research. Consequently, this overview can function as a starting point for future research identifying the trends and developments in the field and current knowledge gaps. This will make it easier for researchers to determine their research focus and build on previous work. Some concrete recommendations for future research are given below. Additionally, the current findings show interesting leads for collaboration between researchers of religious leadership and mainstream leadership scholars. For example, the shared interest in transformational leadership would be an interesting starting point as this theory has some evident religious connotations (Spoelstra, 2017 ). Aside from its focus on charisma, which has historical roots in Christian writings, common elements include the central notion of transformation and the religious concept of conversion. The transformational leader is expected to induce change within their followers to transform (i.e., convert) them into more effective, engaged, or moral individuals, embodying higher values for the company’s good (Delaney & Spoelstra, 2022 ; Spoelstra, 2017 ). Another lead for collaboration could be the emergence of value-based leadership in mainstream leadership research (Zhu et al., 2019 ) as a value-based approach to leadership is central in the domain of religion as well (e.g., Agee, 2001 ; Graham-Brown, 2020 ; Taylor, 2019 ).

The current findings invite religious practitioners and congregations to reflect on their own ways of evaluating leadership effectiveness. What are current practices, and what would be appropriate ways to measure effectiveness based on the shared beliefs and values of their specific contexts? The overview of indicators of effectiveness used in recent research will hopefully inspire this process and encourage new and more comprehensive approaches to measure effectiveness in helping religious communities to flourish. Educational institutes can benefit from the current findings by reflecting on the trends and developments that have been outlined. The growing significance of follower-oriented approaches to leadership can, for example, lead to new areas of focus in educational programs and the screening of candidates for religious vocations.

Limitations

A limitation of the current study design is that the outcomes of this review study are confined to the references that were included based on the data search that was conducted. In our discussion of the literature, we stayed close to the included references and focused on the trends and insights that emerged from these specific references. As a result, we might have missed relevant works on the study of religious leadership effectiveness because they were outside of our search terms and inclusion criteria. For example, the reviewers have brought to our attention the work of Friedman ( 2007 , 2011 ). His work was widely influential in the 2000s and has been especially valued by mainline Protestant and Jewish leaders for its holistic approach to religious leadership based on Bowen’s family systems theory. Friedman’s work was most likely not identified during our literature search because it does not explicitly address religious leadership and partly falls outside the selected timeframe. The same will apply to the professional literature as our data search was limited to the academic literature. In addition, studies that did not research religious leadership in the context of faith communities or that did not explicitly discuss the issue of effectiveness also fell outside the focus of the current study.

Recommendations: An agenda for future research and practice

A theme that should be high on the research agenda is broadening the scope of research from focusing on Christian leadership to examining religious leadership effectiveness across different religions. The current data search only identified three references that researched religious leadership from a Muslim perspective and one reference that focused on religious leadership in general. This suggests that the findings of the current study may not be applicable beyond the Christian context as only limited academic research was included on the effectiveness of religious leadership in general. Therefore, future research should focus on conducting studies across various religious backgrounds to deepen our understanding of effective religious leadership. Additionally, the studies included in the current overview were conducted mainly by scholars from the United States examining religious leaders in an American context. We therefore also need more studies from underrepresented parts of the world to get a more comprehensive perspective on religious leadership effectiveness.

Another recommendation for future research concerns the coherence in the field and its theoretical foundation. The references included in this study do not strongly build on previous work in the field, and, as a consequence, research on religious leadership appears to be less incremental. Also, an overarching narrative of assessing effective religious leadership seems to be lacking as academic reflection on the topic stagnated after the work of Nauss and Malony (e.g., Malony, 1984 , 2000 ; Malony & Hunt, 1991 ; Nauss, 1972 , 1996 ). An updated theoretical foundation that does justice to the current landscape is essential for the field to progress. Future studies can build on such a foundation and create more sound evidence regarding religious leadership effectiveness by collecting data that supplements earlier findings and addressing current knowledge and research gaps. For example, a theoretical framework could combine the three target groups (i.e., leader, follower, and congregation) into a dynamic and comprehensive model (see Boyatzis et al., 2011 ). Additionally, transformational leadership could serve as an overarching theory given the popularity it already has in the field and the promising effects of this type of leadership style on the congregation (e.g., Carter, 2009 ; Rowold, 2008 ).

A concrete recommendation for practice that follows from the current overview is to use multiple criteria from multiple target groups to evaluate religious leadership in congregations (DeRue et al., 2011 ). As McKenna and Eckard ( 2009 ) point out, the criteria deemed appropriate will vary depending on the mission and overall purpose of the congregation. However, using a range of criteria can help overcome a one-sided focus on, for example, numerical growth or leadership competencies. By evaluating religious leadership using various criteria, we can better understand the dynamic and diverse leadership contexts and, hopefully, alleviate some of the discomfort people feel in assessing religious leadership effectiveness.

A theme for the research agenda that follows from the above is the need for reliable and validated scales to measure religious leadership effectiveness. Over the past 25 years, only a few new instruments have been developed. One possible reason for this could be the limited interest of social scientists in religious leadership. The only validated measure used by references in this study that was specifically designed for the assessment of religious leadership was the Ministerial Effectiveness Inventory by Majovski ( 1982 ), a shortened version of the Profiles of Ministry (Schuller et al., 1980 ) consisting of 59 items (Nauss, 1996 ). There are growing concerns regarding the content validity of this scale in current research because the qualitative data underlying the development of this scale (mid-1970s) has become less representative of effective leadership in the twenty-first century (Dobrotka, 2021 ). Additionally, the sample consisted exclusively of respondents from the 200 seminaries that were members of the Association of Theological Schools in the United States and Canada at that time. Consequently, the field could benefit from well-researched measures that can be used to evaluate religious leadership across different religious and cultural contexts. In order to develop such a scale, scholars should collaborate with practitioners to create evidence-informed scales that will benefit religious leaders and their communities as well. The work by McElroy et al. ( 2014 ), which describes the development of a measure of intellectual humility in the context of religious leadership, could be used for inspiration. Alternatively, researchers could build on and extend work in related fields, such as the development of the Authentic Leadership Inventory (Neider & Schriesheim, 2011 ), Spiritual Leadership Scale (Fry et al., 2005 ), Identity Leadership Inventory (Steffens et al., 2014 ), or Servant Leadership Survey (Van Dierendonck & Nuijten, 2011 ).

In short, the current scoping review provided an up-to-date overview of how religious leadership effectiveness has been assessed over the last 25 years in the academic literature. Although a focus on Christian leadership remained prevalent among the included references, recent references adopt a more dynamic and extensive approach to evaluating effective religious leadership by combining criteria from different target groups (i.e., leader, follower/member, and congregation) while using multiple measures and emphasizing the interactions between the leader and congregation.

Adams, A. W. (2013). Shepherd leadership of Church of God pastors and how this relates to ministerial effectiveness (Publication No. 3570373) [Doctoral dissertation, University of Phoenix]. ProQuest Dissertations Publishing.

Adeloye, G. O. (2011). Pastoral leadership: A sine-qua-non for effective church growth. Practical Theology: Baptist College of Theology, Lagos, 4 , 84–95.

Google Scholar  

Adetoyese, J. O. (2018). The marks of a successful pastor. The American Journal of Biblical Theology, 19 (7), 110–120.

Agee, B. R. (2001). Servant leadership as an effective approach to leadership in the church. Southwestern Journal of Theology, 43 (3), 7–19.

Al-Hilali, M. (2012). Transformational leadership and organizational effectiveness: A predictive study at American Muslim Organizations (Publication No. 3570201) [Doctoral dissertation, University of Phoenix]. ProQuest Dissertations Publishing.

Ammerman, N. T. (2020). Rethinking religion: Toward a practice approach. American Journal of Sociology, 126 (1), 6–51.

Article   Google Scholar  

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8 (1), 19–32. https://doi.org/10.1080/1364557032000119616

Avolio, B. J., & Bass, B. M. (2004). Multifactor Leadership Questionnaire: Manual and sampler set (3rd ed.) Mind Garden.

Banfill, F. (2015). Multiple case studies in effective Africa leadership: A study of the leadership behaviors of effective local church pastors in the Africa inland church Tanzania Mara and Ukerewe diocese (Publication No. 10263321) [Doctoral dissertation, Indiana Institute of Technology]. ProQuest Dissertations Publishing.

Bass, B. M. (1985). Leadership and performance beyond expectations . Free Press.

Bass, B. M., & Riggio, R. E. (2005). Transformational leadership. Psychology Press. https://doi.org/10.4324/9781410617095

Bird, D. E. (1997). Personality factors as related to successful church planters among Texas Baptists (Publication No. 9720741) [Doctoral dissertation, Fuller Theological Seminary]. ProQuest Dissertations Publishing.

Boyatzis, R., Brizz, T., & Godwin, L. (2011). The effect of religious leaders’ emotional and social competencies on improving parish vibrancy. Journal of Leadership & Organizational Studies, 18 (2), 192–206. https://doi.org/10.1177/1548051810369676

Brauer, S. (2018). The surprising predictable decline of religion in the United States. Journal for the Scientific Study of Religion, 57 (4), 654–675. https://doi.org/10.1111/jssr.12551

Carter, J. C. (2009). Transformational leadership and pastoral leader effectiveness. Pastoral Psychology, 58 (3), 261–271.

Chaves, M., Roso, J., & Holleman, A. (2022). The National Survey of Religious Leaders: Background, methods, and lessons learned in the research process. Journal for the Scientific Study of Religion, 61 (3–4), 737–749. https://doi.org/10.1111/jssr.12803

Cloyd, M. (2019). Unseen success in Christian leadership failures. Journal of Biblical Perspectives in Leadership, 9 (1), 213–225.

ADS   Google Scholar  

Cobb-Hayes, J. (2018). An investigation of the relationship between authentic leadership and pastoral leadership effectiveness (Publication No. 13808904) [Doctoral dissertation, Indiana Wesleyan University]. ProQuest Dissertations Publishing.

Counts, J. D. (2020). Discovering leadership models that produce fruit within the Mid-Atlantic Episcopal district (Publication No. 28384349) [Doctoral dissertation, Ashland Theological Seminary]. ProQuest Dissertations Publishing.

Dadswell, D., & Ross, C. (2013). Church Growth Research Project: Church planting. Oxford Centre for Ecclesiology and Practical Theology Ripon College, Cuddesdon. https://www.churchofengland.org/sites/default/files/2019-06/cgrp_church_planting_from_cgrd_website.pdf

De Hart, J., & Van Houwelingen, P. (2018). Christenen in Nederland: Kerkelijke deelname en christelijke gelovigheid. Den Haag: Sociaal en Cultureel Planbureau. https://www.scp.nl/publicaties/publicaties/2018/12/19/christenen-in-nederland

Delaney, H., & Spoelstra, S. (2022). Transformational leadership: Secularized theology? In B. Caroll, J. Ford, & S. Taylor (Eds.), Leadership: Contemporary critical perspectives (3rd ed., pp. 75–93). SAGE.

DeNeal, D. A. (2019). Level 5 leadership and pastoral effectiveness: A study of lead pastors’ level 5 leadership score to church growth indicators (Publication No. 27670550) [Doctoral dissertation, Johnson University]. ProQuest Dissertations Publishing.

DeRue, D. S., Nahrgang, J. D., Wellman, N. E., & Humphrey, S. E. (2011). Trait and behavioral theories of leadership: An integration and meta-analytic test of their relative validity. Personnel Psychology, 64 (1), 7–52. https://doi.org/10.1111/j.1744-6570.2010.01201.x

Dinh, J. E., Lord, R. G., Gardner, W. L., Meuser, J. D., Liden, R. C., & Hu, J. (2014). Leadership theory and research in the new millennium: Current theoretical trends and changing perspectives. Leadership Quarterly, 25 (1), 36–62. https://doi.org/10.1016/j.leaqua.2013.11.005

Dobrotka, S. P. (2021). Measuring clergy effectiveness: The development of the Clergy Effectiveness Scale. In M. C. Bocarnea, B. E. Winston, & D. Dean (Eds.), Handbook of research on advancements in organizational data collection and measurements: Strategies for addressing attitudes, beliefs, and behaviors (pp. 293–313). IGI Global.

Chapter   Google Scholar  

Dodds, M. L. (1997). Case studies of pastoral leadership in the church, No. 19 (Publication No. 99724362) [Doctoral dissertation, Dallas Theological Seminary]. ProQuest Dissertations Publishing.

Douglas, S. M. (2015). Pastoral skills and qualities for effective ministry in Southern Baptist churches. Journal for Baptist Theology & Ministry, 12 (2), 31–43.

Dreibelbis, J. (2010). From maintaining to building communities of faith. Anglican Theological Review, 92 (1), 147–155.

Dye, W. H. (1998). Case studies of pastoral leadership in the church, No. 20 (Publication No. 9823645) [Doctoral dissertation, Dallas Theological Seminary]. ProQuest Dissertations Publishing.

Feaster, W. B. (2015). The relationship between time management behaviors and ministerial effectiveness among ministers of education in the state of Texas (Publication No. 3716280) [Doctoral dissertation, Dallas Baptist University]. ProQuest Dissertations Publishing.

Francis, L. J., Ratter, H., & Longden, G. (2015). Linking clergy psychological type profile and church growth in the Diocese of Chester: A pilot study. Journal of Beliefs & Values, 36 (3), 354–358. https://doi.org/10.1080/13617672.2015.1095544

Friedman, E. H. (2007). A failure of nerve: Leadership in the age of the quick fix. Church Publishing.

Friedman, E. H. (2011). Generation to generation: Family process in church and synagogue . Guilford Press.

Fry, L. W., Vitucci, S., & Cedillo, M. (2005). Spiritual leadership and army transformation: Theory, measurement, and establishing a baseline. Leadership Quarterly, 16 (5), 835–862. https://doi.org/10.1016/j.leaqua.2005.07.012

Garber, K. A. (2021). Developing a research-based approach to discovering characteristics and creating action steps that enhance pastoral effectiveness (Publication No. 28493783) [Doctoral dissertation, Regent University]. ProQuest Dissertations Publishing.

Garner, B. (2020). The resilient pastor: How to remain effective and finish well (Publication No. 28264457) [Doctoral dissertation, Biola University]. ProQuest Dissertations Publishing.

Giordan, G. (2014). Introduction: Pluralism as legitimization of diversity. In G. Giordan, E. Pace (Eds.), Religious pluralism . Springer, Cham. https://doi.org/10.1007/978-3-319-06623-3_1

Glatt, R. J. (2019). An examination of the correlation between church conflict and pastoral leadership effectiveness (Publication No. 27665046) [Doctoral dissertation, Trevecca Nazarene University]. ProQuest Dissertations Publishing.

Graham-Brown, F. B. (2020). Examining the pastoral servant leadership behaviors of metro New York City Baptist organizations and its impact on church effectiveness: A grounded theory study (Publication No. 28152784) [Doctoral dissertation, Sullivan University]. ProQuest Dissertations Publishing.

Green, C. C. (2001). Leadership theory of a Protestant minister. Journal of Leadership Studies, 8 (1), 77–95.

Gunardi, A., Setiadi, B. N., Hidajat, L. L., & Tjahjadi, S. P. L. (2022). Measurement of the effectiveness of Catholic priests’ pastoral services: A psychometric approach. Pastoral Psychology , 1–15. https://doi.org/10.1007/s11089-021-00985-y

Gyuroka, T. C. (2016). What do pastors in German-speaking Europe perceive as important leadership competencies in order to be effective pastoral leaders (Publication No. 10162853) [Doctoral dissertation, Andrews University]. ProQuest Dissertations Publishing.

Haskell, D. M., Flatt, K. N., & Burgoyne, S. (2016). Theology matters: Comparing the traits of growing and declining mainline Protestant church attendees and clergy. Review of Religious Research, 58 (4), 515–541. https://doi.org/10.1007/s13644-016-0255-4

Haslam, S. A., Reicher, S. D., & Platow, M. J. (2015). Leadership: Theory and practice. In M. Mikulincer & P. R. Shaver (Eds.), APA handbook of personality and social psychology, Vol. 2: Group processes (pp. 67–94) . American Psychological Association.

Haslam, S. A., Reicher, S. D., & Platow, M. J. (2020). The new psychology of leadership: Identity, influence and power (2 nd ed). Routledge. https://doi.org/10.4324/9781351108232

Hoffmann, J. P. (2013). Declining religious authority? Confidence in the leaders of religious organizations, 1973–2010. Review of Religious Research, 55 , 1–25. https://doi.org/10.1007/s13644-012-0090-1

Hudson, J. S. (2017). A competency model for church revitalization in Southern Baptist convention churches: A mixed-methods study [Doctoral dissertation, The Southern Baptist Theological Seminary]. Open Access Dissertation and Theses.

Hussey, I. (2013). Leadership in effective and growing Australian congregations: A study of three cases. Journal of Religious Leadership, 12 (1), 63–88.

Inglehart, R. F. (2020). Giving up on God: The global decline of religion. Foreign Affairs, 99 (5), 110–118.

Inglehart, R. F. (2021). Religion’s sudden decline: What’s causing it, and what comes next? Oxford University Press.

Book   Google Scholar  

Jacobsen, L. S. (2004). The leadership factor in church planting projects in Norway from 1990 to 2000 (Publication No. 3175621) [Doctoral dissertation, Regent University]. ProQuest Dissertations Publishing.

Jules, P. (2020). Description of leadership traits that contribute success within church or religiously affiliated organizations (Publication No. 28261923) [Doctoral dissertation, St. Thomas University]. ProQuest Dissertations Publishing.

Kane, M. N., & Jacobs, R. J. (2015). Perceptions of success and failure among aging Catholic priests. Journal of Religion, Spirituality & Aging , 27 (4), 288–304. https://doi.org/10.1080/15528030.2015.1046630

Kisling, R. A. (2007). Character for leadership: The role of personal characteristics in effective leadership behaviors (Publication No. 3292253) [Doctoral dissertation, Regent University]. ProQuest Dissertations Publishing.

Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5 (1), 1–9. https://doi.org/10.1186/1748-5908-5-69

Mahazan, A. M., & Abdullah, A. G. (2013). A model of imam’s leadership and mosque performance in Malaysia. Global Journal Al-Thaqafah, 3 (2), 53–64. https://doi.org/10.7187/GJAT432013.03.02

Main, J. M. (1998). Case studies of pastoral leadership in the church, No. 21 (Publication No. 9823648) [Doctoral dissertation, Dallas Theological Seminary]. ProQuest Dissertations Publishing.

Majovski, L. (1982). The Role of Psychological Assessment in Ministerial Selection [Unpublished doctoral dissertation, Fuller Theological Seminary].

Malony, H. N. (1984). Ministerial effectiveness: A review of recent research. Pastoral Psychology, 33 (2), 96–104. https://doi.org/10.1007/BF01086370

Malony, H. N. (2000). The psychological evaluation of religious professionals. Professional Psychology: Research and Practice , 31 (5), 521. https://doi.org/10.1037/0735-7028.31.5.521

Malony, H. N., & Hunt, R. A. (1991). Ministerial effectiveness. In H. N. Malony & R. A. Hunt (Eds.), The psychology of clergy (pp. 109–120). Morehouse.

Manners, A. T. (2008). Influence of transformational, autocratic, democratic, and laissez-faire leadership principles on the effectiveness of religious leaders (Publication No. 3370948) [Doctoral dissertation, University of Phoenix]. ProQuest Dissertations Publishing.

Marshall, J., & Olson, D. V. A. (2018). Is ‘spiritual but not religious’ a replacement for religion or just one step on the path between religion and non-religion? Review of Religious Research, 60 , 503–518. https://doi.org/10.1007/s13644-018-0342-9

Martinez, F. (2020). Best leadership practices of Protestant ministers in the Presbyterian Church of USA (Publication No. 28091689) [Doctoral dissertation, Pepperdine University]. ProQuest Dissertations Publishing.

McElroy, S. E., Rice, K. G., Davis, D. E., Hook, J. N., Hill, P. C., Worthington, E. L., Jr., & Van Tongeren, D. R. (2014). Intellectual humility: Scale development and theoretical elaborations in the context of religious leadership. Journal of Psychology and Theology, 42 (1), 19–30.

McKenna, R. B., & Eckard, K. (2009). Evaluating pastoral effectiveness: To measure or not to measure. Pastoral Psychology, 58 (3), 303–313. https://doi.org/10.1007/s11089-008-0191-5

Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18 , 1–7. https://doi.org/10.1186/s12874-018-0611-x

Nauss, A. (1972). Problems in measuring ministerial effectiveness. Journal for the Scientific Study of Religion, 11 (2), 141–151. https://doi.org/10.2307/1384926

Nauss, A. (1994). Ministerial effectiveness in ten functions. Review of Religious Research , 36 (1), 58–69. https://doi.org/10.2307/3511652

Nauss, A. (1996). Assessing ministerial effectiveness: A review of measures and their use. Research in the Social Scientific Study of Religion, 7 , 221–251. https://doi.org/10.1163/9789004496231_014

Neider, L. L., & Schriesheim, C. A. (2011). The Authentic Leadership Inventory (ALI): Development and empirical tests. Leadership Quarterly, 22 (6), 1146–1164. https://doi.org/10.1016/j.leaqua.2011.09.008

O’Donnell, D. (1997). Can we predict successful priests? The Furrow, 48 (4), 229–234.

Oh, J., & Wang, J. (2020). Spiritual leadership: Current status and agenda for future research and practice. Journal of Management, Spirituality & Religion, 17 (3), 223–248. https://doi.org/10.1080/14766086.2020.1728568

Oney, R. M. (2009). Exploring the causal relationship of emotional intelligence to clergy leadership effectiveness (Publication No. 3392554) [Doctoral dissertation, Regent University]. ProQuest Dissertations Publishing.

Palmer-Atkins, C. B. (2018). An exploratory study of a nondenominational church and leadership behaviors, principle, strategies and practices (Publication No. 10684473) [Doctoral dissertation, Walden University]. ProQuest Dissertations Publishing.

Park, J. S. (2010). A study of effective lay small group leaders in Korean American churches: Fostering spiritual growth (Publication No. 3399094) [Doctoral dissertation, Trinity International University]. ProQuest Dissertations Publishing.

Park, Y. J. D. (2014). Women’s effective leadership in contemporary Taiwanese churches (Publication No. 3621907) [Doctoral dissertation, Biola University]. ProQuest Dissertations Publishing.

Parrott, R. (2003). Competency, spirituality, and core-identity in pastors. Ashland Theological Journal, 35 , 73–81.

Pearse, N. J. (2011). Effective strategic leadership: Balancing roles during church transitions. HTS Teologiese Studies / Theological Studies, 67 (2), 1–7. https://doi.org/10.4102/hts.v67i2.980

Pickens, D. M. (2000). Predicting ministerial effectiveness: A review of empirical research (Publication No. ED444072) [Doctoral dissertation, Biola University]. ERIC.

Powell, R., Ward, N., Bellamy, J., Sterland, S., Jacka, K., & Pepper, M. (2022). Enriching Church Life: A guide to results from National Church Life Surveys for local churches (3rd ed.). Sydney, NSW: NCLS Research.

Puls, T. R., Ludden, L. L., & Freemyer, J. (2014). Authentic leadership and its relationship to ministerial effectiveness. Journal of Applied Christian Leadership, 8 (1), 55–75.

Rafferty, A. E., & Griffin, M. A. (2004). Dimensions of transformational leadership: Conceptual and empirical extensions. Leadership Quarterly, 15 (3), 329–354. https://doi.org/10.1016/j.leaqua.2004.02.009

Rambo, L. R., & Haar Farris, M. S. (2012). Psychology of religion: Toward a multidisciplinary paradigm. Pastoral Psychology, 61 (5), 711–720. https://doi.org/10.1007/s11089-011-0372-5

Rotaru, L. G. (2019). Leaders that finish well: Characteristics and practices of successful ministry (Publication No. 22616661) [Doctoral dissertation, Asbury Theological Seminary]. ProQuest Dissertations Publishing.

Rowold, J. (2008). Effects of transactional and transformational leadership of pastors. Pastoral Psychology, 56 (4), 403–411. https://doi.org/10.1007/s11089-008-0121-6

Rumley, D. D. (2011). Perceptions of the senior pastors’ transformational leadership style and its relationship to the eight markers of natural church development (Publication No. 3477630) [Doctoral dissertation, Indiana Wesleyan University]. ProQuest Dissertations Publishing.

Scuderi, N. F. (2010). Servant leadership and transformational leadership in church organizations (Publication No. 3413541) [Doctoral dissertation, The George Washington University]. ProQuest Dissertations Publishing.

Small, K. J. (2011). Successful leadership in the early years of ministry: Reflections for leadership formation in theological education. Journal of Religious Leadership, 10 (1), 51–80.

Schwarz, C. (2006). Natural Church Development: A guide to eight essential qualities of healthy churches (7th ed.). St. Charles: Churchsmart Resources.

Schuller, D., Strommen, M., & Brekke, M. (1980). Ministry in America . San Francisco: Harper and Row.

Soldo, A. (2017). Examining the relationship between leadership style and leadership effectiveness among imams in U.S. mosques (Publication No. 10600624) [Doctoral dissertation, Colorado Technical University]. ProQuest Dissertations Publishing.

Spoelstra, S. (2017). Leadership and religion. In J. Storey, J. Hartley, J.-L. Denis, P. ‘t Hart, & D. Ulrich (Eds.), The Routledge companion to leadership (pp. 260–277). Routledge.

Starr, E. J. (2001). Simply complex: An examination of paradoxical leadership in faith-based organizations (Publication No. 3039623) [Doctoral dissertation, State University of New York at Albany]. ProQuest Dissertations Publishing.

Steen, P. W. (2008). Leadership of growing churches: The role of the senior pastor’s attributes and leadership practices (Publication No. 3350606) [Doctoral dissertation, Biola University]. ProQuest Dissertations Publishing.

Steffens, N. K., Haslam, S. A., Reicher, S. D., Platow, M. J., Fransen, K., Yang, J., Ryan, M. K., Jetten, J., Peters, K., & Boen, F. (2014). Leadership as social identity management: Introducing the Identity Leadership Inventory (ILI) to assess and validate a four-dimensional model. Leadership Quarterly, 25 (5), 1001–1024. https://doi.org/10.1016/j.leaqua.2014.05.002

Steffens, N. K., Munt, K. A., van Knippenberg, D., Platow, M. J., & Haslam, S. A. (2021). Advancing the social identity theory of leadership: A meta-analytic review of leader group prototypicality. Organizational Psychology Review, 11 (1), 35–72. https://doi.org/10.1177/2041386620962569

Strunk, J., Milacci, F., & Zabloski, J. (2017). The convergence of ministry, tenure, and efficacy: Beyond speculation toward a new theory of pastoral efficacy. Pastoral Psychology, 66 (4), 537–550. https://doi.org/10.1007/s11089-017-0760-6

Sundby, M., & Nienaber, S. (2009). The truth beneath the myths: What we can learn from a study of clergy competencies. Congregations, 36 (4), 18–21.

Swanson, H. P. (1999). Pastoral effectiveness: A study of differences among comparison groups of Seventh-day Adventist clergy (Publication No. 9937628) [Doctoral dissertation, Andrews University]. ProQuest Dissertations Publishing.

Taylor, J. T. (2019). A comparative analysis of leadership styles that identifies effective religious leaders (Publication No. 27744663) [Doctoral dissertation, Northcentral University]. ProQuest Dissertations Publishing.

Thiessen, J., Wong, A., McAlpine, B., & Walker, K. (2019). What is a flourishing congregation? Leader perceptions, definitions, and experiences. Review of Religious Research, 61 (1), 13–37. https://doi.org/10.1007/s13644-018-0356-3

Thornhill, J. (1999). Effective pastoral leadership: Forming disciples of the Lord. The Australasian Catholic Record, 76 (2), 144–151.

Timmins, F., Caldeira, S., Murphy, M., Pujol, N., Sheaf, G., Weathers, E., Whelan, J., & Flanagan, B. (2018). The role of the healthcare chaplain: A literature review. Journal of Health Care Chaplaincy, 24 (3), 87–106. https://doi.org/10.1080/08854726.2017.1338048

Article   PubMed   Google Scholar  

Tromp, P., Pless, A., & Houtman, D. (2020). ‘Believing without belonging’ in twenty European countries (1981–2008): De-institutionalization of Christianity or spiritualization of religion? Review of Religious Research, 62 (4), 509–531. https://doi.org/10.1007/s13644-020-00432-z

Van Dierendonck, D., & Nuijten, I. (2011). The Servant Leadership Survey: Development and validation of a multidimensional measure. Journal of Business and Psychology , 26 , 249–267. https://doi.org/10.1007/s10869-010-9194-1

Vardaman, D. L. (2013). Leading change: Exploring the relationship between transformational, transactional, and change-oriented leadership and their impact on leadership effectiveness among pastors in a Protestant denomination in the mid-western United States (Publication No. 3613541) [Doctoral dissertation, Indiana Wesleyan University]. ProQuest Dissertations Publishing.

Wasberg, G. D. (2013). Differentiation of self and leadership effectiveness in Christian clergy: A mixed methods study (Publication No. 3602885) [Doctoral dissertation, Capella University]. ProQuest Dissertations Publishing.

Washington, K. S. (2016). Spiritual leadership in religious organizations: A grounded theory study (Publication No. 10013592) [Doctoral dissertation, The University of the Rockies]. ProQuest Dissertations Publishing.

Watt, L., & Voas, D. (2015). Psychological types and self-assessed leadership skills of clergy in the Church of England. Mental Health, Religion & Culture, 18 (7), 544–555. https://doi.org/10.1080/13674676.2014.961250

Watt, W. M. (2014). Relational principles for effective church leadership. Journal of Leadership Education , 13 (2), 125–139. https://doi.org/10.12806/V13/I2/T1

Wilson, D. C. (2022). Defining leadership. Philosophy of Management , 1–30. https://doi.org/10.1007/s40926-022-00210-7

Wollschleger, J. (2018). Pastoral leadership and congregational vitality. Review of Religious Research, 60 (4), 575–585. https://doi.org/10.1007/s13644-018-0352-7

Yukl, G. (1999). An evaluation of conceptual weaknesses in transformational and charismatic leadership. Leadership Quarterly, 10 (2), 285–305. https://doi.org/10.1016/S1048-9843(99)00013-2

Yukl, G., & Gardner, W. (2020). Leadership in organizations (9 th ed.). Pearson.

Zhu, J., Song, L. J., Zhu, L., & Johnson, R. E. (2019). Visualizing the landscape and evolution of leadership research. Leadership Quarterly, 30 (2), 215–232. https://doi.org/10.1016/j.leaqua.2018.06.003

Download references

Acknowledgements

We thank Michiel van Elk and Stefan Paas for their valuable feedback and careful reading of the manuscript.

Author information

Authors and affiliations.

Faculty of Religion and Theology, Department of Beliefs and Practices, Vrije Universiteit Amsterdam, De Boelelaan 1105, 1081 HV, Amsterdam, The Netherlands

Annemarie Foppen

University of Humanistic Studies, Utrecht, The Netherlands

Joke W. van Saane

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Annemarie Foppen .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Foppen, A., van Saane, J.W. Assessing Religious Leadership: A Scoping Review of Leadership Effectiveness Criteria and Current Trends in the Academic Literature. Pastoral Psychol 73 , 1–21 (2024). https://doi.org/10.1007/s11089-023-01113-8

Download citation

Accepted : 15 October 2023

Published : 15 November 2023

Issue Date : February 2024

DOI : https://doi.org/10.1007/s11089-023-01113-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Religious leadership
• Scoping review
• Effectiveness criteria
• Find a journal
• Publish with us
• Track your research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Working with sources

How to Find Sources | Scholarly Articles, Books, Etc.

Published on June 13, 2022 by Eoghan Ryan . Revised on May 31, 2023.

It’s important to know how to find relevant sources when writing a  research paper , literature review , or systematic review .

The types of sources you need will depend on the stage you are at in the research process , but all sources that you use should be credible , up to date, and relevant to your research topic.

There are three main places to look for sources to use in your research:

Research databases

• Your institution’s library
• Other online resources

Table of contents

Library resources, other online sources, other interesting articles, frequently asked questions about finding sources.

You can search for scholarly sources online using databases and search engines like Google Scholar . These provide a range of search functions that can help you to find the most relevant sources.

If you are searching for a specific article or book, include the title or the author’s name. Alternatively, if you’re just looking for sources related to your research problem , you can search using keywords. In this case, it’s important to have a clear understanding of the scope of your project and of the most relevant keywords.

Databases can be general (interdisciplinary) or subject-specific.

• You can use subject-specific databases to ensure that the results are relevant to your field.
• When using a general database or search engine, you can still filter results by selecting specific subjects or disciplines.

Example: JSTOR discipline search filter

Check the table below to find a database that’s relevant to your research.

Google Scholar

To get started, you might also try Google Scholar , an academic search engine that can help you find relevant books and articles. Its “Cited by” function lets you see the number of times a source has been cited. This can tell you something about a source’s credibility and importance to the field.

Example: Google Scholar “Cited by” function

Boolean operators

Boolean operators can also help to narrow or expand your search.

Boolean operators are words and symbols like AND , OR , and NOT that you can use to include or exclude keywords to refine your results. For example, a search for “Nietzsche NOT nihilism” will provide results that include the word “Nietzsche” but exclude results that contain the word “nihilism.”

Many databases and search engines have an advanced search function that allows you to refine results in a similar way without typing the Boolean operators manually.

Example: Project Muse advanced search

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

You can find helpful print sources in your institution’s library. These include:

• Journal articles
• Encyclopedias
• Newspapers and magazines

Make sure that the sources you consult are appropriate to your research.

You can find these sources using your institution’s library database. This will allow you to explore the library’s catalog and to search relevant keywords. You can refine your results using Boolean operators .

Once you have found a relevant print source in the library:

• Consider what books are beside it. This can be a great way to find related sources, especially when you’ve found a secondary or tertiary source instead of a primary source .
• Consult the index and bibliography to find the bibliographic information of other relevant sources.

You can consult popular online sources to learn more about your topic. These include:

• Crowdsourced encyclopedias like Wikipedia

You can find these sources using search engines. To refine your search, use Boolean operators in combination with relevant keywords.

However, exercise caution when using online sources. Consider what kinds of sources are appropriate for your research and make sure the sites are credible .

Look for sites with trusted domain extensions:

• URLs that end with .edu are educational resources.
• URLs that end with .gov are government-related resources.
• DOIs often indicate that an article is published in a peer-reviewed , scientific article.

Other sites can still be used, but you should evaluate them carefully and consider alternatives.

If you want to know more about ChatGPT, AI tools , citation , and plagiarism , make sure to check out some of our other articles with explanations and examples.

• ChatGPT vs human editor
• ChatGPT citations
• Is ChatGPT trustworthy?
• Using ChatGPT for your studies
• What is ChatGPT?
• Chicago style
• Paraphrasing

 Plagiarism

• Types of plagiarism
• Self-plagiarism
• Avoiding plagiarism
• Academic integrity
• Consequences of plagiarism
• Common knowledge

You can find sources online using databases and search engines like Google Scholar . Use Boolean operators or advanced search functions to narrow or expand your search.

For print sources, you can use your institution’s library database. This will allow you to explore the library’s catalog and to search relevant keywords.

It is important to find credible sources and use those that you can be sure are sufficiently scholarly .

• Consult your institute’s library to find out what books, journals, research databases, and other types of sources they provide access to.
• Look for books published by respected academic publishing houses and university presses, as these are typically considered trustworthy sources.
• Look for journals that use a peer review process. This means that experts in the field assess the quality and credibility of an article before it is published.

When searching for sources in databases, think of specific keywords that are relevant to your topic , and consider variations on them or synonyms that might be relevant.

Once you have a clear idea of your research parameters and key terms, choose a database that is relevant to your research (e.g., Medline, JSTOR, Project MUSE).

Find out if the database has a “subject search” option. This can help to refine your search. Use Boolean operators to combine your keywords, exclude specific search terms, and search exact phrases to find the most relevant sources.

There are many types of sources commonly used in research. These include:

You’ll likely use a variety of these sources throughout the research process , and the kinds of sources you use will depend on your research topic and goals.

Scholarly sources are written by experts in their field and are typically subjected to peer review . They are intended for a scholarly audience, include a full bibliography, and use scholarly or technical language. For these reasons, they are typically considered credible sources .

Popular sources like magazines and news articles are typically written by journalists. These types of sources usually don’t include a bibliography and are written for a popular, rather than academic, audience. They are not always reliable and may be written from a biased or uninformed perspective, but they can still be cited in some contexts.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Ryan, E. (2023, May 31). How to Find Sources | Scholarly Articles, Books, Etc.. Scribbr. Retrieved April 2, 2024, from https://www.scribbr.com/working-with-sources/finding-sources/

Is this article helpful?

Eoghan Ryan

Other students also liked, types of sources explained | examples & tips, primary vs. secondary sources | difference & examples, boolean operators | quick guide, examples & tips, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 26 March 2024

Predicting and improving complex beer flavor through machine learning

• Michiel Schreurs   ORCID: orcid.org/0000-0002-9449-5619 1 , 2 , 3   na1 ,
• Supinya Piampongsant 1 , 2 , 3   na1 ,
• Miguel Roncoroni   ORCID: orcid.org/0000-0001-7461-1427 1 , 2 , 3   na1 ,
• Lloyd Cool   ORCID: orcid.org/0000-0001-9936-3124 1 , 2 , 3 , 4 ,
• Beatriz Herrera-Malaver   ORCID: orcid.org/0000-0002-5096-9974 1 , 2 , 3 ,
• Christophe Vanderaa   ORCID: orcid.org/0000-0001-7443-5427 4 ,
• Florian A. Theßeling 1 , 2 , 3 ,
• Łukasz Kreft   ORCID: orcid.org/0000-0001-7620-4657 5 ,
• Alexander Botzki   ORCID: orcid.org/0000-0001-6691-4233 5 ,
• Philippe Malcorps 6 ,
• Luk Daenen 6 ,
• Tom Wenseleers   ORCID: orcid.org/0000-0002-1434-861X 4 &
• Kevin J. Verstrepen   ORCID: orcid.org/0000-0002-3077-6219 1 , 2 , 3  

Nature Communications volume  15 , Article number:  2368 ( 2024 ) Cite this article

44k Accesses

797 Altmetric

Metrics details

• Chemical engineering
• Gas chromatography
• Machine learning
• Metabolomics
• Taste receptors

The perception and appreciation of food flavor depends on many interacting chemical compounds and external factors, and therefore proves challenging to understand and predict. Here, we combine extensive chemical and sensory analyses of 250 different beers to train machine learning models that allow predicting flavor and consumer appreciation. For each beer, we measure over 200 chemical properties, perform quantitative descriptive sensory analysis with a trained tasting panel and map data from over 180,000 consumer reviews to train 10 different machine learning models. The best-performing algorithm, Gradient Boosting, yields models that significantly outperform predictions based on conventional statistics and accurately predict complex food features and consumer appreciation from chemical profiles. Model dissection allows identifying specific and unexpected compounds as drivers of beer flavor and appreciation. Adding these compounds results in variants of commercial alcoholic and non-alcoholic beers with improved consumer appreciation. Together, our study reveals how big data and machine learning uncover complex links between food chemistry, flavor and consumer perception, and lays the foundation to develop novel, tailored foods with superior flavors.

Similar content being viewed by others

BitterSweet: Building machine learning models for predicting the bitter and sweet taste of small molecules

Rudraksh Tuwani, Somin Wadhwa & Ganesh Bagler

Sensory lexicon and aroma volatiles analysis of brewing malt

Xiaoxia Su, Miao Yu, … Tianyi Du

Predicting odor from molecular structure: a multi-label classification approach

Kushagra Saini & Venkatnarayan Ramanathan

Introduction

Predicting and understanding food perception and appreciation is one of the major challenges in food science. Accurate modeling of food flavor and appreciation could yield important opportunities for both producers and consumers, including quality control, product fingerprinting, counterfeit detection, spoilage detection, and the development of new products and product combinations (food pairing) 1 , 2 , 3 , 4 , 5 , 6 . Accurate models for flavor and consumer appreciation would contribute greatly to our scientific understanding of how humans perceive and appreciate flavor. Moreover, accurate predictive models would also facilitate and standardize existing food assessment methods and could supplement or replace assessments by trained and consumer tasting panels, which are variable, expensive and time-consuming 7 , 8 , 9 . Lastly, apart from providing objective, quantitative, accurate and contextual information that can help producers, models can also guide consumers in understanding their personal preferences 10 .

Despite the myriad of applications, predicting food flavor and appreciation from its chemical properties remains a largely elusive goal in sensory science, especially for complex food and beverages 11 , 12 . A key obstacle is the immense number of flavor-active chemicals underlying food flavor. Flavor compounds can vary widely in chemical structure and concentration, making them technically challenging and labor-intensive to quantify, even in the face of innovations in metabolomics, such as non-targeted metabolic fingerprinting 13 , 14 . Moreover, sensory analysis is perhaps even more complicated. Flavor perception is highly complex, resulting from hundreds of different molecules interacting at the physiochemical and sensorial level. Sensory perception is often non-linear, characterized by complex and concentration-dependent synergistic and antagonistic effects 15 , 16 , 17 , 18 , 19 , 20 , 21 that are further convoluted by the genetics, environment, culture and psychology of consumers 22 , 23 , 24 . Perceived flavor is therefore difficult to measure, with problems of sensitivity, accuracy, and reproducibility that can only be resolved by gathering sufficiently large datasets 25 . Trained tasting panels are considered the prime source of quality sensory data, but require meticulous training, are low throughput and high cost. Public databases containing consumer reviews of food products could provide a valuable alternative, especially for studying appreciation scores, which do not require formal training 25 . Public databases offer the advantage of amassing large amounts of data, increasing the statistical power to identify potential drivers of appreciation. However, public datasets suffer from biases, including a bias in the volunteers that contribute to the database, as well as confounding factors such as price, cult status and psychological conformity towards previous ratings of the product.

Classical multivariate statistics and machine learning methods have been used to predict flavor of specific compounds by, for example, linking structural properties of a compound to its potential biological activities or linking concentrations of specific compounds to sensory profiles 1 , 26 . Importantly, most previous studies focused on predicting organoleptic properties of single compounds (often based on their chemical structure) 27 , 28 , 29 , 30 , 31 , 32 , 33 , thus ignoring the fact that these compounds are present in a complex matrix in food or beverages and excluding complex interactions between compounds. Moreover, the classical statistics commonly used in sensory science 34 , 35 , 36 , 37 , 38 , 39 require a large sample size and sufficient variance amongst predictors to create accurate models. They are not fit for studying an extensive set of hundreds of interacting flavor compounds, since they are sensitive to outliers, have a high tendency to overfit and are less suited for non-linear and discontinuous relationships 40 .

In this study, we combine extensive chemical analyses and sensory data of a set of different commercial beers with machine learning approaches to develop models that predict taste, smell, mouthfeel and appreciation from compound concentrations. Beer is particularly suited to model the relationship between chemistry, flavor and appreciation. First, beer is a complex product, consisting of thousands of flavor compounds that partake in complex sensory interactions 41 , 42 , 43 . This chemical diversity arises from the raw materials (malt, yeast, hops, water and spices) and biochemical conversions during the brewing process (kilning, mashing, boiling, fermentation, maturation and aging) 44 , 45 . Second, the advent of the internet saw beer consumers embrace online review platforms, such as RateBeer (ZX Ventures, Anheuser-Busch InBev SA/NV) and BeerAdvocate (Next Glass, inc.). In this way, the beer community provides massive data sets of beer flavor and appreciation scores, creating extraordinarily large sensory databases to complement the analyses of our professional sensory panel. Specifically, we characterize over 200 chemical properties of 250 commercial beers, spread across 22 beer styles, and link these to the descriptive sensory profiling data of a 16-person in-house trained tasting panel and data acquired from over 180,000 public consumer reviews. These unique and extensive datasets enable us to train a suite of machine learning models to predict flavor and appreciation from a beer’s chemical profile. Dissection of the best-performing models allows us to pinpoint specific compounds as potential drivers of beer flavor and appreciation. Follow-up experiments confirm the importance of these compounds and ultimately allow us to significantly improve the flavor and appreciation of selected commercial beers. Together, our study represents a significant step towards understanding complex flavors and reinforces the value of machine learning to develop and refine complex foods. In this way, it represents a stepping stone for further computer-aided food engineering applications 46 .

To generate a comprehensive dataset on beer flavor, we selected 250 commercial Belgian beers across 22 different beer styles (Supplementary Fig.  S1 ). Beers with ≤ 4.2% alcohol by volume (ABV) were classified as non-alcoholic and low-alcoholic. Blonds and Tripels constitute a significant portion of the dataset (12.4% and 11.2%, respectively) reflecting their presence on the Belgian beer market and the heterogeneity of beers within these styles. By contrast, lager beers are less diverse and dominated by a handful of brands. Rare styles such as Brut or Faro make up only a small fraction of the dataset (2% and 1%, respectively) because fewer of these beers are produced and because they are dominated by distinct characteristics in terms of flavor and chemical composition.

Extensive analysis identifies relationships between chemical compounds in beer

For each beer, we measured 226 different chemical properties, including common brewing parameters such as alcohol content, iso-alpha acids, pH, sugar concentration 47 , and over 200 flavor compounds (Methods, Supplementary Table  S1 ). A large portion (37.2%) are terpenoids arising from hopping, responsible for herbal and fruity flavors 16 , 48 . A second major category are yeast metabolites, such as esters and alcohols, that result in fruity and solvent notes 48 , 49 , 50 . Other measured compounds are primarily derived from malt, or other microbes such as non- Saccharomyces yeasts and bacteria (‘wild flora’). Compounds that arise from spices or staling are labeled under ‘Others’. Five attributes (caloric value, total acids and total ester, hop aroma and sulfur compounds) are calculated from multiple individually measured compounds.

As a first step in identifying relationships between chemical properties, we determined correlations between the concentrations of the compounds (Fig.  1 , upper panel, Supplementary Data  1 and 2 , and Supplementary Fig.  S2 . For the sake of clarity, only a subset of the measured compounds is shown in Fig.  1 ). Compounds of the same origin typically show a positive correlation, while absence of correlation hints at parameters varying independently. For example, the hop aroma compounds citronellol, and alpha-terpineol show moderate correlations with each other (Spearman’s rho=0.39 and 0.57), but not with the bittering hop component iso-alpha acids (Spearman’s rho=0.16 and −0.07). This illustrates how brewers can independently modify hop aroma and bitterness by selecting hop varieties and dosage time. If hops are added early in the boiling phase, chemical conversions increase bitterness while aromas evaporate, conversely, late addition of hops preserves aroma but limits bitterness 51 . Similarly, hop-derived iso-alpha acids show a strong anti-correlation with lactic acid and acetic acid, likely reflecting growth inhibition of lactic acid and acetic acid bacteria, or the consequent use of fewer hops in sour beer styles, such as West Flanders ales and Fruit beers, that rely on these bacteria for their distinct flavors 52 . Finally, yeast-derived esters (ethyl acetate, ethyl decanoate, ethyl hexanoate, ethyl octanoate) and alcohols (ethanol, isoamyl alcohol, isobutanol, and glycerol), correlate with Spearman coefficients above 0.5, suggesting that these secondary metabolites are correlated with the yeast genetic background and/or fermentation parameters and may be difficult to influence individually, although the choice of yeast strain may offer some control 53 .

Spearman rank correlations are shown. Descriptors are grouped according to their origin (malt (blue), hops (green), yeast (red), wild flora (yellow), Others (black)), and sensory aspect (aroma, taste, palate, and overall appreciation). Please note that for the chemical compounds, for the sake of clarity, only a subset of the total number of measured compounds is shown, with an emphasis on the key compounds for each source. For more details, see the main text and Methods section. Chemical data can be found in Supplementary Data  1 , correlations between all chemical compounds are depicted in Supplementary Fig.  S2 and correlation values can be found in Supplementary Data  2 . See Supplementary Data  4 for sensory panel assessments and Supplementary Data  5 for correlation values between all sensory descriptors.

Interestingly, different beer styles show distinct patterns for some flavor compounds (Supplementary Fig.  S3 ). These observations agree with expectations for key beer styles, and serve as a control for our measurements. For instance, Stouts generally show high values for color (darker), while hoppy beers contain elevated levels of iso-alpha acids, compounds associated with bitter hop taste. Acetic and lactic acid are not prevalent in most beers, with notable exceptions such as Kriek, Lambic, Faro, West Flanders ales and Flanders Old Brown, which use acid-producing bacteria ( Lactobacillus and Pediococcus ) or unconventional yeast ( Brettanomyces ) 54 , 55 . Glycerol, ethanol and esters show similar distributions across all beer styles, reflecting their common origin as products of yeast metabolism during fermentation 45 , 53 . Finally, low/no-alcohol beers contain low concentrations of glycerol and esters. This is in line with the production process for most of the low/no-alcohol beers in our dataset, which are produced through limiting fermentation or by stripping away alcohol via evaporation or dialysis, with both methods having the unintended side-effect of reducing the amount of flavor compounds in the final beer 56 , 57 .

Besides expected associations, our data also reveals less trivial associations between beer styles and specific parameters. For example, geraniol and citronellol, two monoterpenoids responsible for citrus, floral and rose flavors and characteristic of Citra hops, are found in relatively high amounts in Christmas, Saison, and Brett/co-fermented beers, where they may originate from terpenoid-rich spices such as coriander seeds instead of hops 58 .

Tasting panel assessments reveal sensorial relationships in beer

To assess the sensory profile of each beer, a trained tasting panel evaluated each of the 250 beers for 50 sensory attributes, including different hop, malt and yeast flavors, off-flavors and spices. Panelists used a tasting sheet (Supplementary Data  3 ) to score the different attributes. Panel consistency was evaluated by repeating 12 samples across different sessions and performing ANOVA. In 95% of cases no significant difference was found across sessions ( p  > 0.05), indicating good panel consistency (Supplementary Table  S2 ).

Aroma and taste perception reported by the trained panel are often linked (Fig.  1 , bottom left panel and Supplementary Data  4 and 5 ), with high correlations between hops aroma and taste (Spearman’s rho=0.83). Bitter taste was found to correlate with hop aroma and taste in general (Spearman’s rho=0.80 and 0.69), and particularly with “grassy” noble hops (Spearman’s rho=0.75). Barnyard flavor, most often associated with sour beers, is identified together with stale hops (Spearman’s rho=0.97) that are used in these beers. Lactic and acetic acid, which often co-occur, are correlated (Spearman’s rho=0.66). Interestingly, sweetness and bitterness are anti-correlated (Spearman’s rho = −0.48), confirming the hypothesis that they mask each other 59 , 60 . Beer body is highly correlated with alcohol (Spearman’s rho = 0.79), and overall appreciation is found to correlate with multiple aspects that describe beer mouthfeel (alcohol, carbonation; Spearman’s rho= 0.32, 0.39), as well as with hop and ester aroma intensity (Spearman’s rho=0.39 and 0.35).

Similar to the chemical analyses, sensorial analyses confirmed typical features of specific beer styles (Supplementary Fig.  S4 ). For example, sour beers (Faro, Flanders Old Brown, Fruit beer, Kriek, Lambic, West Flanders ale) were rated acidic, with flavors of both acetic and lactic acid. Hoppy beers were found to be bitter and showed hop-associated aromas like citrus and tropical fruit. Malt taste is most detected among scotch, stout/porters, and strong ales, while low/no-alcohol beers, which often have a reputation for being ‘worty’ (reminiscent of unfermented, sweet malt extract) appear in the middle. Unsurprisingly, hop aromas are most strongly detected among hoppy beers. Like its chemical counterpart (Supplementary Fig.  S3 ), acidity shows a right-skewed distribution, with the most acidic beers being Krieks, Lambics, and West Flanders ales.

Tasting panel assessments of specific flavors correlate with chemical composition

We find that the concentrations of several chemical compounds strongly correlate with specific aroma or taste, as evaluated by the tasting panel (Fig.  2 , Supplementary Fig.  S5 , Supplementary Data  6 ). In some cases, these correlations confirm expectations and serve as a useful control for data quality. For example, iso-alpha acids, the bittering compounds in hops, strongly correlate with bitterness (Spearman’s rho=0.68), while ethanol and glycerol correlate with tasters’ perceptions of alcohol and body, the mouthfeel sensation of fullness (Spearman’s rho=0.82/0.62 and 0.72/0.57 respectively) and darker color from roasted malts is a good indication of malt perception (Spearman’s rho=0.54).

Heatmap colors indicate Spearman’s Rho. Axes are organized according to sensory categories (aroma, taste, mouthfeel, overall), chemical categories and chemical sources in beer (malt (blue), hops (green), yeast (red), wild flora (yellow), Others (black)). See Supplementary Data  6 for all correlation values.

Interestingly, for some relationships between chemical compounds and perceived flavor, correlations are weaker than expected. For example, the rose-smelling phenethyl acetate only weakly correlates with floral aroma. This hints at more complex relationships and interactions between compounds and suggests a need for a more complex model than simple correlations. Lastly, we uncovered unexpected correlations. For instance, the esters ethyl decanoate and ethyl octanoate appear to correlate slightly with hop perception and bitterness, possibly due to their fruity flavor. Iron is anti-correlated with hop aromas and bitterness, most likely because it is also anti-correlated with iso-alpha acids. This could be a sign of metal chelation of hop acids 61 , given that our analyses measure unbound hop acids and total iron content, or could result from the higher iron content in dark and Fruit beers, which typically have less hoppy and bitter flavors 62 .

Public consumer reviews complement expert panel data

To complement and expand the sensory data of our trained tasting panel, we collected 180,000 reviews of our 250 beers from the online consumer review platform RateBeer. This provided numerical scores for beer appearance, aroma, taste, palate, overall quality as well as the average overall score.

Public datasets are known to suffer from biases, such as price, cult status and psychological conformity towards previous ratings of a product. For example, prices correlate with appreciation scores for these online consumer reviews (rho=0.49, Supplementary Fig.  S6 ), but not for our trained tasting panel (rho=0.19). This suggests that prices affect consumer appreciation, which has been reported in wine 63 , while blind tastings are unaffected. Moreover, we observe that some beer styles, like lagers and non-alcoholic beers, generally receive lower scores, reflecting that online reviewers are mostly beer aficionados with a preference for specialty beers over lager beers. In general, we find a modest correlation between our trained panel’s overall appreciation score and the online consumer appreciation scores (Fig.  3 , rho=0.29). Apart from the aforementioned biases in the online datasets, serving temperature, sample freshness and surroundings, which are all tightly controlled during the tasting panel sessions, can vary tremendously across online consumers and can further contribute to (among others, appreciation) differences between the two categories of tasters. Importantly, in contrast to the overall appreciation scores, for many sensory aspects the results from the professional panel correlated well with results obtained from RateBeer reviews. Correlations were highest for features that are relatively easy to recognize even for untrained tasters, like bitterness, sweetness, alcohol and malt aroma (Fig.  3 and below).

RateBeer text mining results can be found in Supplementary Data  7 . Rho values shown are Spearman correlation values, with asterisks indicating significant correlations ( p  < 0.05, two-sided). All p values were smaller than 0.001, except for Esters aroma (0.0553), Esters taste (0.3275), Esters aroma—banana (0.0019), Coriander (0.0508) and Diacetyl (0.0134).

Besides collecting consumer appreciation from these online reviews, we developed automated text analysis tools to gather additional data from review texts (Supplementary Data  7 ). Processing review texts on the RateBeer database yielded comparable results to the scores given by the trained panel for many common sensory aspects, including acidity, bitterness, sweetness, alcohol, malt, and hop tastes (Fig.  3 ). This is in line with what would be expected, since these attributes require less training for accurate assessment and are less influenced by environmental factors such as temperature, serving glass and odors in the environment. Consumer reviews also correlate well with our trained panel for 4-vinyl guaiacol, a compound associated with a very characteristic aroma. By contrast, correlations for more specific aromas like ester, coriander or diacetyl are underrepresented in the online reviews, underscoring the importance of using a trained tasting panel and standardized tasting sheets with explicit factors to be scored for evaluating specific aspects of a beer. Taken together, our results suggest that public reviews are trustworthy for some, but not all, flavor features and can complement or substitute taste panel data for these sensory aspects.

Models can predict beer sensory profiles from chemical data

The rich datasets of chemical analyses, tasting panel assessments and public reviews gathered in the first part of this study provided us with a unique opportunity to develop predictive models that link chemical data to sensorial features. Given the complexity of beer flavor, basic statistical tools such as correlations or linear regression may not always be the most suitable for making accurate predictions. Instead, we applied different machine learning models that can model both simple linear and complex interactive relationships. Specifically, we constructed a set of regression models to predict (a) trained panel scores for beer flavor and quality and (b) public reviews’ appreciation scores from beer chemical profiles. We trained and tested 10 different models (Methods), 3 linear regression-based models (simple linear regression with first-order interactions (LR), lasso regression with first-order interactions (Lasso), partial least squares regressor (PLSR)), 5 decision tree models (AdaBoost regressor (ABR), extra trees (ET), gradient boosting regressor (GBR), random forest (RF) and XGBoost regressor (XGBR)), 1 support vector regression (SVR), and 1 artificial neural network (ANN) model.

To compare the performance of our machine learning models, the dataset was randomly split into a training and test set, stratified by beer style. After a model was trained on data in the training set, its performance was evaluated on its ability to predict the test dataset obtained from multi-output models (based on the coefficient of determination, see Methods). Additionally, individual-attribute models were ranked per descriptor and the average rank was calculated, as proposed by Korneva et al. 64 . Importantly, both ways of evaluating the models’ performance agreed in general. Performance of the different models varied (Table  1 ). It should be noted that all models perform better at predicting RateBeer results than results from our trained tasting panel. One reason could be that sensory data is inherently variable, and this variability is averaged out with the large number of public reviews from RateBeer. Additionally, all tree-based models perform better at predicting taste than aroma. Linear models (LR) performed particularly poorly, with negative R 2 values, due to severe overfitting (training set R 2  = 1). Overfitting is a common issue in linear models with many parameters and limited samples, especially with interaction terms further amplifying the number of parameters. L1 regularization (Lasso) successfully overcomes this overfitting, out-competing multiple tree-based models on the RateBeer dataset. Similarly, the dimensionality reduction of PLSR avoids overfitting and improves performance, to some extent. Still, tree-based models (ABR, ET, GBR, RF and XGBR) show the best performance, out-competing the linear models (LR, Lasso, PLSR) commonly used in sensory science 65 .

GBR models showed the best overall performance in predicting sensory responses from chemical information, with R 2 values up to 0.75 depending on the predicted sensory feature (Supplementary Table  S4 ). The GBR models predict consumer appreciation (RateBeer) better than our trained panel’s appreciation (R 2 value of 0.67 compared to R 2 value of 0.09) (Supplementary Table  S3 and Supplementary Table  S4 ). ANN models showed intermediate performance, likely because neural networks typically perform best with larger datasets 66 . The SVR shows intermediate performance, mostly due to the weak predictions of specific attributes that lower the overall performance (Supplementary Table  S4 ).

Model dissection identifies specific, unexpected compounds as drivers of consumer appreciation

Next, we leveraged our models to infer important contributors to sensory perception and consumer appreciation. Consumer preference is a crucial sensory aspects, because a product that shows low consumer appreciation scores often does not succeed commercially 25 . Additionally, the requirement for a large number of representative evaluators makes consumer trials one of the more costly and time-consuming aspects of product development. Hence, a model for predicting chemical drivers of overall appreciation would be a welcome addition to the available toolbox for food development and optimization.

Since GBR models on our RateBeer dataset showed the best overall performance, we focused on these models. Specifically, we used two approaches to identify important contributors. First, rankings of the most important predictors for each sensorial trait in the GBR models were obtained based on impurity-based feature importance (mean decrease in impurity). High-ranked parameters were hypothesized to be either the true causal chemical properties underlying the trait, to correlate with the actual causal properties, or to take part in sensory interactions affecting the trait 67 (Fig.  4A ). In a second approach, we used SHAP 68 to determine which parameters contributed most to the model for making predictions of consumer appreciation (Fig.  4B ). SHAP calculates parameter contributions to model predictions on a per-sample basis, which can be aggregated into an importance score.

A The impurity-based feature importance (mean deviance in impurity, MDI) calculated from the Gradient Boosting Regression (GBR) model predicting RateBeer appreciation scores. The top 15 highest ranked chemical properties are shown. B SHAP summary plot for the top 15 parameters contributing to our GBR model. Each point on the graph represents a sample from our dataset. The color represents the concentration of that parameter, with bluer colors representing low values and redder colors representing higher values. Greater absolute values on the horizontal axis indicate a higher impact of the parameter on the prediction of the model. C Spearman correlations between the 15 most important chemical properties and consumer overall appreciation. Numbers indicate the Spearman Rho correlation coefficient, and the rank of this correlation compared to all other correlations. The top 15 important compounds were determined using SHAP (panel B).

Both approaches identified ethyl acetate as the most predictive parameter for beer appreciation (Fig.  4 ). Ethyl acetate is the most abundant ester in beer with a typical ‘fruity’, ‘solvent’ and ‘alcoholic’ flavor, but is often considered less important than other esters like isoamyl acetate. The second most important parameter identified by SHAP is ethanol, the most abundant beer compound after water. Apart from directly contributing to beer flavor and mouthfeel, ethanol drastically influences the physical properties of beer, dictating how easily volatile compounds escape the beer matrix to contribute to beer aroma 69 . Importantly, it should also be noted that the importance of ethanol for appreciation is likely inflated by the very low appreciation scores of non-alcoholic beers (Supplementary Fig.  S4 ). Despite not often being considered a driver of beer appreciation, protein level also ranks highly in both approaches, possibly due to its effect on mouthfeel and body 70 . Lactic acid, which contributes to the tart taste of sour beers, is the fourth most important parameter identified by SHAP, possibly due to the generally high appreciation of sour beers in our dataset.

Interestingly, some of the most important predictive parameters for our model are not well-established as beer flavors or are even commonly regarded as being negative for beer quality. For example, our models identify methanethiol and ethyl phenyl acetate, an ester commonly linked to beer staling 71 , as a key factor contributing to beer appreciation. Although there is no doubt that high concentrations of these compounds are considered unpleasant, the positive effects of modest concentrations are not yet known 72 , 73 .

To compare our approach to conventional statistics, we evaluated how well the 15 most important SHAP-derived parameters correlate with consumer appreciation (Fig.  4C ). Interestingly, only 6 of the properties derived by SHAP rank amongst the top 15 most correlated parameters. For some chemical compounds, the correlations are so low that they would have likely been considered unimportant. For example, lactic acid, the fourth most important parameter, shows a bimodal distribution for appreciation, with sour beers forming a separate cluster, that is missed entirely by the Spearman correlation. Additionally, the correlation plots reveal outliers, emphasizing the need for robust analysis tools. Together, this highlights the need for alternative models, like the Gradient Boosting model, that better grasp the complexity of (beer) flavor.

Finally, to observe the relationships between these chemical properties and their predicted targets, partial dependence plots were constructed for the six most important predictors of consumer appreciation 74 , 75 , 76 (Supplementary Fig.  S7 ). One-way partial dependence plots show how a change in concentration affects the predicted appreciation. These plots reveal an important limitation of our models: appreciation predictions remain constant at ever-increasing concentrations. This implies that once a threshold concentration is reached, further increasing the concentration does not affect appreciation. This is false, as it is well-documented that certain compounds become unpleasant at high concentrations, including ethyl acetate (‘nail polish’) 77 and methanethiol (‘sulfury’ and ‘rotten cabbage’) 78 . The inability of our models to grasp that flavor compounds have optimal levels, above which they become negative, is a consequence of working with commercial beer brands where (off-)flavors are rarely too high to negatively impact the product. The two-way partial dependence plots show how changing the concentration of two compounds influences predicted appreciation, visualizing their interactions (Supplementary Fig.  S7 ). In our case, the top 5 parameters are dominated by additive or synergistic interactions, with high concentrations for both compounds resulting in the highest predicted appreciation.

To assess the robustness of our best-performing models and model predictions, we performed 100 iterations of the GBR, RF and ET models. In general, all iterations of the models yielded similar performance (Supplementary Fig.  S8 ). Moreover, the main predictors (including the top predictors ethanol and ethyl acetate) remained virtually the same, especially for GBR and RF. For the iterations of the ET model, we did observe more variation in the top predictors, which is likely a consequence of the model’s inherent random architecture in combination with co-correlations between certain predictors. However, even in this case, several of the top predictors (ethanol and ethyl acetate) remain unchanged, although their rank in importance changes (Supplementary Fig.  S8 ).

Next, we investigated if a combination of RateBeer and trained panel data into one consolidated dataset would lead to stronger models, under the hypothesis that such a model would suffer less from bias in the datasets. A GBR model was trained to predict appreciation on the combined dataset. This model underperformed compared to the RateBeer model, both in the native case and when including a dataset identifier (R 2  = 0.67, 0.26 and 0.42 respectively). For the latter, the dataset identifier is the most important feature (Supplementary Fig.  S9 ), while most of the feature importance remains unchanged, with ethyl acetate and ethanol ranking highest, like in the original model trained only on RateBeer data. It seems that the large variation in the panel dataset introduces noise, weakening the models’ performances and reliability. In addition, it seems reasonable to assume that both datasets are fundamentally different, with the panel dataset obtained by blind tastings by a trained professional panel.

Lastly, we evaluated whether beer style identifiers would further enhance the model’s performance. A GBR model was trained with parameters that explicitly encoded the styles of the samples. This did not improve model performance (R2 = 0.66 with style information vs R2 = 0.67). The most important chemical features are consistent with the model trained without style information (eg. ethanol and ethyl acetate), and with the exception of the most preferred (strong ale) and least preferred (low/no-alcohol) styles, none of the styles were among the most important features (Supplementary Fig.  S9 , Supplementary Table  S5 and S6 ). This is likely due to a combination of style-specific chemical signatures, such as iso-alpha acids and lactic acid, that implicitly convey style information to the original models, as well as the low number of samples belonging to some styles, making it difficult for the model to learn style-specific patterns. Moreover, beer styles are not rigorously defined, with some styles overlapping in features and some beers being misattributed to a specific style, all of which leads to more noise in models that use style parameters.

Model validation

To test if our predictive models give insight into beer appreciation, we set up experiments aimed at improving existing commercial beers. We specifically selected overall appreciation as the trait to be examined because of its complexity and commercial relevance. Beer flavor comprises a complex bouquet rather than single aromas and tastes 53 . Hence, adding a single compound to the extent that a difference is noticeable may lead to an unbalanced, artificial flavor. Therefore, we evaluated the effect of combinations of compounds. Because Blond beers represent the most extensive style in our dataset, we selected a beer from this style as the starting material for these experiments (Beer 64 in Supplementary Data  1 ).

In the first set of experiments, we adjusted the concentrations of compounds that made up the most important predictors of overall appreciation (ethyl acetate, ethanol, lactic acid, ethyl phenyl acetate) together with correlated compounds (ethyl hexanoate, isoamyl acetate, glycerol), bringing them up to 95 th percentile ethanol-normalized concentrations (Methods) within the Blond group (‘Spiked’ concentration in Fig.  5A ). Compared to controls, the spiked beers were found to have significantly improved overall appreciation among trained panelists, with panelist noting increased intensity of ester flavors, sweetness, alcohol, and body fullness (Fig.  5B ). To disentangle the contribution of ethanol to these results, a second experiment was performed without the addition of ethanol. This resulted in a similar outcome, including increased perception of alcohol and overall appreciation.

Adding the top chemical compounds, identified as best predictors of appreciation by our model, into poorly appreciated beers results in increased appreciation from our trained panel. Results of sensory tests between base beers and those spiked with compounds identified as the best predictors by the model. A Blond and Non/Low-alcohol (0.0% ABV) base beers were brought up to 95th-percentile ethanol-normalized concentrations within each style. B For each sensory attribute, tasters indicated the more intense sample and selected the sample they preferred. The numbers above the bars correspond to the p values that indicate significant changes in perceived flavor (two-sided binomial test: alpha 0.05, n  = 20 or 13).

In a last experiment, we tested whether using the model’s predictions can boost the appreciation of a non-alcoholic beer (beer 223 in Supplementary Data  1 ). Again, the addition of a mixture of predicted compounds (omitting ethanol, in this case) resulted in a significant increase in appreciation, body, ester flavor and sweetness.

Predicting flavor and consumer appreciation from chemical composition is one of the ultimate goals of sensory science. A reliable, systematic and unbiased way to link chemical profiles to flavor and food appreciation would be a significant asset to the food and beverage industry. Such tools would substantially aid in quality control and recipe development, offer an efficient and cost-effective alternative to pilot studies and consumer trials and would ultimately allow food manufacturers to produce superior, tailor-made products that better meet the demands of specific consumer groups more efficiently.

A limited set of studies have previously tried, to varying degrees of success, to predict beer flavor and beer popularity based on (a limited set of) chemical compounds and flavors 79 , 80 . Current sensitive, high-throughput technologies allow measuring an unprecedented number of chemical compounds and properties in a large set of samples, yielding a dataset that can train models that help close the gaps between chemistry and flavor, even for a complex natural product like beer. To our knowledge, no previous research gathered data at this scale (250 samples, 226 chemical parameters, 50 sensory attributes and 5 consumer scores) to disentangle and validate the chemical aspects driving beer preference using various machine-learning techniques. We find that modern machine learning models outperform conventional statistical tools, such as correlations and linear models, and can successfully predict flavor appreciation from chemical composition. This could be attributed to the natural incorporation of interactions and non-linear or discontinuous effects in machine learning models, which are not easily grasped by the linear model architecture. While linear models and partial least squares regression represent the most widespread statistical approaches in sensory science, in part because they allow interpretation 65 , 81 , 82 , modern machine learning methods allow for building better predictive models while preserving the possibility to dissect and exploit the underlying patterns. Of the 10 different models we trained, tree-based models, such as our best performing GBR, showed the best overall performance in predicting sensory responses from chemical information, outcompeting artificial neural networks. This agrees with previous reports for models trained on tabular data 83 . Our results are in line with the findings of Colantonio et al. who also identified the gradient boosting architecture as performing best at predicting appreciation and flavor (of tomatoes and blueberries, in their specific study) 26 . Importantly, besides our larger experimental scale, we were able to directly confirm our models’ predictions in vivo.

Our study confirms that flavor compound concentration does not always correlate with perception, suggesting complex interactions that are often missed by more conventional statistics and simple models. Specifically, we find that tree-based algorithms may perform best in developing models that link complex food chemistry with aroma. Furthermore, we show that massive datasets of untrained consumer reviews provide a valuable source of data, that can complement or even replace trained tasting panels, especially for appreciation and basic flavors, such as sweetness and bitterness. This holds despite biases that are known to occur in such datasets, such as price or conformity bias. Moreover, GBR models predict taste better than aroma. This is likely because taste (e.g. bitterness) often directly relates to the corresponding chemical measurements (e.g., iso-alpha acids), whereas such a link is less clear for aromas, which often result from the interplay between multiple volatile compounds. We also find that our models are best at predicting acidity and alcohol, likely because there is a direct relation between the measured chemical compounds (acids and ethanol) and the corresponding perceived sensorial attribute (acidity and alcohol), and because even untrained consumers are generally able to recognize these flavors and aromas.

The predictions of our final models, trained on review data, hold even for blind tastings with small groups of trained tasters, as demonstrated by our ability to validate specific compounds as drivers of beer flavor and appreciation. Since adding a single compound to the extent of a noticeable difference may result in an unbalanced flavor profile, we specifically tested our identified key drivers as a combination of compounds. While this approach does not allow us to validate if a particular single compound would affect flavor and/or appreciation, our experiments do show that this combination of compounds increases consumer appreciation.

It is important to stress that, while it represents an important step forward, our approach still has several major limitations. A key weakness of the GBR model architecture is that amongst co-correlating variables, the largest main effect is consistently preferred for model building. As a result, co-correlating variables often have artificially low importance scores, both for impurity and SHAP-based methods, like we observed in the comparison to the more randomized Extra Trees models. This implies that chemicals identified as key drivers of a specific sensory feature by GBR might not be the true causative compounds, but rather co-correlate with the actual causative chemical. For example, the high importance of ethyl acetate could be (partially) attributed to the total ester content, ethanol or ethyl hexanoate (rho=0.77, rho=0.72 and rho=0.68), while ethyl phenylacetate could hide the importance of prenyl isobutyrate and ethyl benzoate (rho=0.77 and rho=0.76). Expanding our GBR model to include beer style as a parameter did not yield additional power or insight. This is likely due to style-specific chemical signatures, such as iso-alpha acids and lactic acid, that implicitly convey style information to the original model, as well as the smaller sample size per style, limiting the power to uncover style-specific patterns. This can be partly attributed to the curse of dimensionality, where the high number of parameters results in the models mainly incorporating single parameter effects, rather than complex interactions such as style-dependent effects 67 . A larger number of samples may overcome some of these limitations and offer more insight into style-specific effects. On the other hand, beer style is not a rigid scientific classification, and beers within one style often differ a lot, which further complicates the analysis of style as a model factor.

Our study is limited to beers from Belgian breweries. Although these beers cover a large portion of the beer styles available globally, some beer styles and consumer patterns may be missing, while other features might be overrepresented. For example, many Belgian ales exhibit yeast-driven flavor profiles, which is reflected in the chemical drivers of appreciation discovered by this study. In future work, expanding the scope to include diverse markets and beer styles could lead to the identification of even more drivers of appreciation and better models for special niche products that were not present in our beer set.

In addition to inherent limitations of GBR models, there are also some limitations associated with studying food aroma. Even if our chemical analyses measured most of the known aroma compounds, the total number of flavor compounds in complex foods like beer is still larger than the subset we were able to measure in this study. For example, hop-derived thiols, that influence flavor at very low concentrations, are notoriously difficult to measure in a high-throughput experiment. Moreover, consumer perception remains subjective and prone to biases that are difficult to avoid. It is also important to stress that the models are still immature and that more extensive datasets will be crucial for developing more complete models in the future. Besides more samples and parameters, our dataset does not include any demographic information about the tasters. Including such data could lead to better models that grasp external factors like age and culture. Another limitation is that our set of beers consists of high-quality end-products and lacks beers that are unfit for sale, which limits the current model in accurately predicting products that are appreciated very badly. Finally, while models could be readily applied in quality control, their use in sensory science and product development is restrained by their inability to discern causal relationships. Given that the models cannot distinguish compounds that genuinely drive consumer perception from those that merely correlate, validation experiments are essential to identify true causative compounds.

Despite the inherent limitations, dissection of our models enabled us to pinpoint specific molecules as potential drivers of beer aroma and consumer appreciation, including compounds that were unexpected and would not have been identified using standard approaches. Important drivers of beer appreciation uncovered by our models include protein levels, ethyl acetate, ethyl phenyl acetate and lactic acid. Currently, many brewers already use lactic acid to acidify their brewing water and ensure optimal pH for enzymatic activity during the mashing process. Our results suggest that adding lactic acid can also improve beer appreciation, although its individual effect remains to be tested. Interestingly, ethanol appears to be unnecessary to improve beer appreciation, both for blond beer and alcohol-free beer. Given the growing consumer interest in alcohol-free beer, with a predicted annual market growth of >7% 84 , it is relevant for brewers to know what compounds can further increase consumer appreciation of these beers. Hence, our model may readily provide avenues to further improve the flavor and consumer appreciation of both alcoholic and non-alcoholic beers, which is generally considered one of the key challenges for future beer production.

Whereas we see a direct implementation of our results for the development of superior alcohol-free beverages and other food products, our study can also serve as a stepping stone for the development of novel alcohol-containing beverages. We want to echo the growing body of scientific evidence for the negative effects of alcohol consumption, both on the individual level by the mutagenic, teratogenic and carcinogenic effects of ethanol 85 , 86 , as well as the burden on society caused by alcohol abuse and addiction. We encourage the use of our results for the production of healthier, tastier products, including novel and improved beverages with lower alcohol contents. Furthermore, we strongly discourage the use of these technologies to improve the appreciation or addictive properties of harmful substances.

The present work demonstrates that despite some important remaining hurdles, combining the latest developments in chemical analyses, sensory analysis and modern machine learning methods offers exciting avenues for food chemistry and engineering. Soon, these tools may provide solutions in quality control and recipe development, as well as new approaches to sensory science and flavor research.

Beer selection

250 commercial Belgian beers were selected to cover the broad diversity of beer styles and corresponding diversity in chemical composition and aroma. See Supplementary Fig.  S1 .

Chemical dataset

Sample preparation.

Beers within their expiration date were purchased from commercial retailers. Samples were prepared in biological duplicates at room temperature, unless explicitly stated otherwise. Bottle pressure was measured with a manual pressure device (Steinfurth Mess-Systeme GmbH) and used to calculate CO 2 concentration. The beer was poured through two filter papers (Macherey-Nagel, 500713032 MN 713 ¼) to remove carbon dioxide and prevent spontaneous foaming. Samples were then prepared for measurements by targeted Headspace-Gas Chromatography-Flame Ionization Detector/Flame Photometric Detector (HS-GC-FID/FPD), Headspace-Solid Phase Microextraction-Gas Chromatography-Mass Spectrometry (HS-SPME-GC-MS), colorimetric analysis, enzymatic analysis, Near-Infrared (NIR) analysis, as described in the sections below. The mean values of biological duplicates are reported for each compound.

HS-GC-FID/FPD

HS-GC-FID/FPD (Shimadzu GC 2010 Plus) was used to measure higher alcohols, acetaldehyde, esters, 4-vinyl guaicol, and sulfur compounds. Each measurement comprised 5 ml of sample pipetted into a 20 ml glass vial containing 1.75 g NaCl (VWR, 27810.295). 100 µl of 2-heptanol (Sigma-Aldrich, H3003) (internal standard) solution in ethanol (Fisher Chemical, E/0650DF/C17) was added for a final concentration of 2.44 mg/L. Samples were flushed with nitrogen for 10 s, sealed with a silicone septum, stored at −80 °C and analyzed in batches of 20.

The GC was equipped with a DB-WAXetr column (length, 30 m; internal diameter, 0.32 mm; layer thickness, 0.50 µm; Agilent Technologies, Santa Clara, CA, USA) to the FID and an HP-5 column (length, 30 m; internal diameter, 0.25 mm; layer thickness, 0.25 µm; Agilent Technologies, Santa Clara, CA, USA) to the FPD. N 2 was used as the carrier gas. Samples were incubated for 20 min at 70 °C in the headspace autosampler (Flow rate, 35 cm/s; Injection volume, 1000 µL; Injection mode, split; Combi PAL autosampler, CTC analytics, Switzerland). The injector, FID and FPD temperatures were kept at 250 °C. The GC oven temperature was first held at 50 °C for 5 min and then allowed to rise to 80 °C at a rate of 5 °C/min, followed by a second ramp of 4 °C/min until 200 °C kept for 3 min and a final ramp of (4 °C/min) until 230 °C for 1 min. Results were analyzed with the GCSolution software version 2.4 (Shimadzu, Kyoto, Japan). The GC was calibrated with a 5% EtOH solution (VWR International) containing the volatiles under study (Supplementary Table  S7 ).

HS-SPME-GC-MS

HS-SPME-GC-MS (Shimadzu GCMS-QP-2010 Ultra) was used to measure additional volatile compounds, mainly comprising terpenoids and esters. Samples were analyzed by HS-SPME using a triphase DVB/Carboxen/PDMS 50/30 μm SPME fiber (Supelco Co., Bellefonte, PA, USA) followed by gas chromatography (Thermo Fisher Scientific Trace 1300 series, USA) coupled to a mass spectrometer (Thermo Fisher Scientific ISQ series MS) equipped with a TriPlus RSH autosampler. 5 ml of degassed beer sample was placed in 20 ml vials containing 1.75 g NaCl (VWR, 27810.295). 5 µl internal standard mix was added, containing 2-heptanol (1 g/L) (Sigma-Aldrich, H3003), 4-fluorobenzaldehyde (1 g/L) (Sigma-Aldrich, 128376), 2,3-hexanedione (1 g/L) (Sigma-Aldrich, 144169) and guaiacol (1 g/L) (Sigma-Aldrich, W253200) in ethanol (Fisher Chemical, E/0650DF/C17). Each sample was incubated at 60 °C in the autosampler oven with constant agitation. After 5 min equilibration, the SPME fiber was exposed to the sample headspace for 30 min. The compounds trapped on the fiber were thermally desorbed in the injection port of the chromatograph by heating the fiber for 15 min at 270 °C.

The GC-MS was equipped with a low polarity RXi-5Sil MS column (length, 20 m; internal diameter, 0.18 mm; layer thickness, 0.18 µm; Restek, Bellefonte, PA, USA). Injection was performed in splitless mode at 320 °C, a split flow of 9 ml/min, a purge flow of 5 ml/min and an open valve time of 3 min. To obtain a pulsed injection, a programmed gas flow was used whereby the helium gas flow was set at 2.7 mL/min for 0.1 min, followed by a decrease in flow of 20 ml/min to the normal 0.9 mL/min. The temperature was first held at 30 °C for 3 min and then allowed to rise to 80 °C at a rate of 7 °C/min, followed by a second ramp of 2 °C/min till 125 °C and a final ramp of 8 °C/min with a final temperature of 270 °C.

Mass acquisition range was 33 to 550 amu at a scan rate of 5 scans/s. Electron impact ionization energy was 70 eV. The interface and ion source were kept at 275 °C and 250 °C, respectively. A mix of linear n-alkanes (from C7 to C40, Supelco Co.) was injected into the GC-MS under identical conditions to serve as external retention index markers. Identification and quantification of the compounds were performed using an in-house developed R script as described in Goelen et al. and Reher et al. 87 , 88 (for package information, see Supplementary Table  S8 ). Briefly, chromatograms were analyzed using AMDIS (v2.71) 89 to separate overlapping peaks and obtain pure compound spectra. The NIST MS Search software (v2.0 g) in combination with the NIST2017, FFNSC3 and Adams4 libraries were used to manually identify the empirical spectra, taking into account the expected retention time. After background subtraction and correcting for retention time shifts between samples run on different days based on alkane ladders, compound elution profiles were extracted and integrated using a file with 284 target compounds of interest, which were either recovered in our identified AMDIS list of spectra or were known to occur in beer. Compound elution profiles were estimated for every peak in every chromatogram over a time-restricted window using weighted non-negative least square analysis after which peak areas were integrated 87 , 88 . Batch effect correction was performed by normalizing against the most stable internal standard compound, 4-fluorobenzaldehyde. Out of all 284 target compounds that were analyzed, 167 were visually judged to have reliable elution profiles and were used for final analysis.

Discrete photometric and enzymatic analysis

Discrete photometric and enzymatic analysis (Thermo Scientific TM Gallery TM Plus Beermaster Discrete Analyzer) was used to measure acetic acid, ammonia, beta-glucan, iso-alpha acids, color, sugars, glycerol, iron, pH, protein, and sulfite. 2 ml of sample volume was used for the analyses. Information regarding the reagents and standard solutions used for analyses and calibrations is included in Supplementary Table  S7 and Supplementary Table  S9 .

NIR analyses

NIR analysis (Anton Paar Alcolyzer Beer ME System) was used to measure ethanol. Measurements comprised 50 ml of sample, and a 10% EtOH solution was used for calibration.

Correlation calculations

Pairwise Spearman Rank correlations were calculated between all chemical properties.

Sensory dataset

Trained panel.

Our trained tasting panel consisted of volunteers who gave prior verbal informed consent. All compounds used for the validation experiment were of food-grade quality. The tasting sessions were approved by the Social and Societal Ethics Committee of the KU Leuven (G-2022-5677-R2(MAR)). All online reviewers agreed to the Terms and Conditions of the RateBeer website.

Sensory analysis was performed according to the American Society of Brewing Chemists (ASBC) Sensory Analysis Methods 90 . 30 volunteers were screened through a series of triangle tests. The sixteen most sensitive and consistent tasters were retained as taste panel members. The resulting panel was diverse in age [22–42, mean: 29], sex [56% male] and nationality [7 different countries]. The panel developed a consensus vocabulary to describe beer aroma, taste and mouthfeel. Panelists were trained to identify and score 50 different attributes, using a 7-point scale to rate attributes’ intensity. The scoring sheet is included as Supplementary Data  3 . Sensory assessments took place between 10–12 a.m. The beers were served in black-colored glasses. Per session, between 5 and 12 beers of the same style were tasted at 12 °C to 16 °C. Two reference beers were added to each set and indicated as ‘Reference 1 & 2’, allowing panel members to calibrate their ratings. Not all panelists were present at every tasting. Scores were scaled by standard deviation and mean-centered per taster. Values are represented as z-scores and clustered by Euclidean distance. Pairwise Spearman correlations were calculated between taste and aroma sensory attributes. Panel consistency was evaluated by repeating samples on different sessions and performing ANOVA to identify differences, using the ‘stats’ package (v4.2.2) in R (for package information, see Supplementary Table  S8 ).

Online reviews from a public database

The ‘scrapy’ package in Python (v3.6) (for package information, see Supplementary Table  S8 ). was used to collect 232,288 online reviews (mean=922, min=6, max=5343) from RateBeer, an online beer review database. Each review entry comprised 5 numerical scores (appearance, aroma, taste, palate and overall quality) and an optional review text. The total number of reviews per reviewer was collected separately. Numerical scores were scaled and centered per rater, and mean scores were calculated per beer.

For the review texts, the language was estimated using the packages ‘langdetect’ and ‘langid’ in Python. Reviews that were classified as English by both packages were kept. Reviewers with fewer than 100 entries overall were discarded. 181,025 reviews from >6000 reviewers from >40 countries remained. Text processing was done using the ‘nltk’ package in Python. Texts were corrected for slang and misspellings; proper nouns and rare words that are relevant to the beer context were specified and kept as-is (‘Chimay’,’Lambic’, etc.). A dictionary of semantically similar sensorial terms, for example ‘floral’ and ‘flower’, was created and collapsed together into one term. Words were stemmed and lemmatized to avoid identifying words such as ‘acid’ and ‘acidity’ as separate terms. Numbers and punctuation were removed.

Sentences from up to 50 randomly chosen reviews per beer were manually categorized according to the aspect of beer they describe (appearance, aroma, taste, palate, overall quality—not to be confused with the 5 numerical scores described above) or flagged as irrelevant if they contained no useful information. If a beer contained fewer than 50 reviews, all reviews were manually classified. This labeled data set was used to train a model that classified the rest of the sentences for all beers 91 . Sentences describing taste and aroma were extracted, and term frequency–inverse document frequency (TFIDF) was implemented to calculate enrichment scores for sensorial words per beer.

The sex of the tasting subject was not considered when building our sensory database. Instead, results from different panelists were averaged, both for our trained panel (56% male, 44% female) and the RateBeer reviews (70% male, 30% female for RateBeer as a whole).

Beer price collection and processing

Beer prices were collected from the following stores: Colruyt, Delhaize, Total Wine, BeerHawk, The Belgian Beer Shop, The Belgian Shop, and Beer of Belgium. Where applicable, prices were converted to Euros and normalized per liter. Spearman correlations were calculated between these prices and mean overall appreciation scores from RateBeer and the taste panel, respectively.

Pairwise Spearman Rank correlations were calculated between all sensory properties.

Machine learning models

Predictive modeling of sensory profiles from chemical data.

Regression models were constructed to predict (a) trained panel scores for beer flavors and quality from beer chemical profiles and (b) public reviews’ appreciation scores from beer chemical profiles. Z-scores were used to represent sensory attributes in both data sets. Chemical properties with log-normal distributions (Shapiro-Wilk test, p  <  0.05 ) were log-transformed. Missing chemical measurements (0.1% of all data) were replaced with mean values per attribute. Observations from 250 beers were randomly separated into a training set (70%, 175 beers) and a test set (30%, 75 beers), stratified per beer style. Chemical measurements (p = 231) were normalized based on the training set average and standard deviation. In total, three linear regression-based models: linear regression with first-order interaction terms (LR), lasso regression with first-order interaction terms (Lasso) and partial least squares regression (PLSR); five decision tree models, Adaboost regressor (ABR), Extra Trees (ET), Gradient Boosting regressor (GBR), Random Forest (RF) and XGBoost regressor (XGBR); one support vector machine model (SVR) and one artificial neural network model (ANN) were trained. The models were implemented using the ‘scikit-learn’ package (v1.2.2) and ‘xgboost’ package (v1.7.3) in Python (v3.9.16). Models were trained, and hyperparameters optimized, using five-fold cross-validated grid search with the coefficient of determination (R 2 ) as the evaluation metric. The ANN (scikit-learn’s MLPRegressor) was optimized using Bayesian Tree-Structured Parzen Estimator optimization with the ‘Optuna’ Python package (v3.2.0). Individual models were trained per attribute, and a multi-output model was trained on all attributes simultaneously.

Model dissection

GBR was found to outperform other methods, resulting in models with the highest average R 2 values in both trained panel and public review data sets. Impurity-based rankings of the most important predictors for each predicted sensorial trait were obtained using the ‘scikit-learn’ package. To observe the relationships between these chemical properties and their predicted targets, partial dependence plots (PDP) were constructed for the six most important predictors of consumer appreciation 74 , 75 .

The ‘SHAP’ package in Python (v0.41.0) was implemented to provide an alternative ranking of predictor importance and to visualize the predictors’ effects as a function of their concentration 68 .

Validation of causal chemical properties

To validate the effects of the most important model features on predicted sensory attributes, beers were spiked with the chemical compounds identified by the models and descriptive sensory analyses were carried out according to the American Society of Brewing Chemists (ASBC) protocol 90 .

Compound spiking was done 30 min before tasting. Compounds were spiked into fresh beer bottles, that were immediately resealed and inverted three times. Fresh bottles of beer were opened for the same duration, resealed, and inverted thrice, to serve as controls. Pairs of spiked samples and controls were served simultaneously, chilled and in dark glasses as outlined in the Trained panel section above. Tasters were instructed to select the glass with the higher flavor intensity for each attribute (directional difference test 92 ) and to select the glass they prefer.

The final concentration after spiking was equal to the within-style average, after normalizing by ethanol concentration. This was done to ensure balanced flavor profiles in the final spiked beer. The same methods were applied to improve a non-alcoholic beer. Compounds were the following: ethyl acetate (Merck KGaA, W241415), ethyl hexanoate (Merck KGaA, W243906), isoamyl acetate (Merck KGaA, W205508), phenethyl acetate (Merck KGaA, W285706), ethanol (96%, Colruyt), glycerol (Merck KGaA, W252506), lactic acid (Merck KGaA, 261106).

Significant differences in preference or perceived intensity were determined by performing the two-sided binomial test on each attribute.

Reporting summary

Further information on research design is available in the  Nature Portfolio Reporting Summary linked to this article.

Data availability

The data that support the findings of this work are available in the Supplementary Data files and have been deposited to Zenodo under accession code 10653704 93 . The RateBeer scores data are under restricted access, they are not publicly available as they are property of RateBeer (ZX Ventures, USA). Access can be obtained from the authors upon reasonable request and with permission of RateBeer (ZX Ventures, USA).  Source data are provided with this paper.

Code availability

The code for training the machine learning models, analyzing the models, and generating the figures has been deposited to Zenodo under accession code 10653704 93 .

Tieman, D. et al. A chemical genetic roadmap to improved tomato flavor. Science 355 , 391–394 (2017).

Article   ADS   CAS   PubMed   Google Scholar  

Plutowska, B. & Wardencki, W. Application of gas chromatography–olfactometry (GC–O) in analysis and quality assessment of alcoholic beverages – A review. Food Chem. 107 , 449–463 (2008).

Article   CAS   Google Scholar  

Legin, A., Rudnitskaya, A., Seleznev, B. & Vlasov, Y. Electronic tongue for quality assessment of ethanol, vodka and eau-de-vie. Anal. Chim. Acta 534 , 129–135 (2005).

Loutfi, A., Coradeschi, S., Mani, G. K., Shankar, P. & Rayappan, J. B. B. Electronic noses for food quality: A review. J. Food Eng. 144 , 103–111 (2015).

Ahn, Y.-Y., Ahnert, S. E., Bagrow, J. P. & Barabási, A.-L. Flavor network and the principles of food pairing. Sci. Rep. 1 , 196 (2011).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Bartoshuk, L. M. & Klee, H. J. Better fruits and vegetables through sensory analysis. Curr. Biol. 23 , R374–R378 (2013).

Article   CAS   PubMed   Google Scholar  

Piggott, J. R. Design questions in sensory and consumer science. Food Qual. Prefer. 3293 , 217–220 (1995).

Article   Google Scholar  

Kermit, M. & Lengard, V. Assessing the performance of a sensory panel-panellist monitoring and tracking. J. Chemom. 19 , 154–161 (2005).

Cook, D. J., Hollowood, T. A., Linforth, R. S. T. & Taylor, A. J. Correlating instrumental measurements of texture and flavour release with human perception. Int. J. Food Sci. Technol. 40 , 631–641 (2005).

Chinchanachokchai, S., Thontirawong, P. & Chinchanachokchai, P. A tale of two recommender systems: The moderating role of consumer expertise on artificial intelligence based product recommendations. J. Retail. Consum. Serv. 61 , 1–12 (2021).

Ross, C. F. Sensory science at the human-machine interface. Trends Food Sci. Technol. 20 , 63–72 (2009).

Chambers, E. IV & Koppel, K. Associations of volatile compounds with sensory aroma and flavor: The complex nature of flavor. Molecules 18 , 4887–4905 (2013).

Pinu, F. R. Metabolomics—The new frontier in food safety and quality research. Food Res. Int. 72 , 80–81 (2015).

Danezis, G. P., Tsagkaris, A. S., Brusic, V. & Georgiou, C. A. Food authentication: state of the art and prospects. Curr. Opin. Food Sci. 10 , 22–31 (2016).

Shepherd, G. M. Smell images and the flavour system in the human brain. Nature 444 , 316–321 (2006).

Meilgaard, M. C. Prediction of flavor differences between beers from their chemical composition. J. Agric. Food Chem. 30 , 1009–1017 (1982).

Xu, L. et al. Widespread receptor-driven modulation in peripheral olfactory coding. Science 368 , eaaz5390 (2020).

Kupferschmidt, K. Following the flavor. Science 340 , 808–809 (2013).

Billesbølle, C. B. et al. Structural basis of odorant recognition by a human odorant receptor. Nature 615 , 742–749 (2023).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Smith, B. Perspective: Complexities of flavour. Nature 486 , S6–S6 (2012).

Pfister, P. et al. Odorant receptor inhibition is fundamental to odor encoding. Curr. Biol. 30 , 2574–2587 (2020).

Moskowitz, H. W., Kumaraiah, V., Sharma, K. N., Jacobs, H. L. & Sharma, S. D. Cross-cultural differences in simple taste preferences. Science 190 , 1217–1218 (1975).

Eriksson, N. et al. A genetic variant near olfactory receptor genes influences cilantro preference. Flavour 1 , 22 (2012).

Ferdenzi, C. et al. Variability of affective responses to odors: Culture, gender, and olfactory knowledge. Chem. Senses 38 , 175–186 (2013).

Article   PubMed   Google Scholar  

Lawless, H. T. & Heymann, H. Sensory evaluation of food: Principles and practices. (Springer, New York, NY). https://doi.org/10.1007/978-1-4419-6488-5 (2010).

Colantonio, V. et al. Metabolomic selection for enhanced fruit flavor. Proc. Natl. Acad. Sci. 119 , e2115865119 (2022).

Fritz, F., Preissner, R. & Banerjee, P. VirtualTaste: a web server for the prediction of organoleptic properties of chemical compounds. Nucleic Acids Res 49 , W679–W684 (2021).

Tuwani, R., Wadhwa, S. & Bagler, G. BitterSweet: Building machine learning models for predicting the bitter and sweet taste of small molecules. Sci. Rep. 9 , 1–13 (2019).

Dagan-Wiener, A. et al. Bitter or not? BitterPredict, a tool for predicting taste from chemical structure. Sci. Rep. 7 , 1–13 (2017).

Pallante, L. et al. Toward a general and interpretable umami taste predictor using a multi-objective machine learning approach. Sci. Rep. 12 , 1–11 (2022).

Malavolta, M. et al. A survey on computational taste predictors. Eur. Food Res. Technol. 248 , 2215–2235 (2022).

Lee, B. K. et al. A principal odor map unifies diverse tasks in olfactory perception. Science 381 , 999–1006 (2023).

Mayhew, E. J. et al. Transport features predict if a molecule is odorous. Proc. Natl. Acad. Sci. 119 , e2116576119 (2022).

Niu, Y. et al. Sensory evaluation of the synergism among ester odorants in light aroma-type liquor by odor threshold, aroma intensity and flash GC electronic nose. Food Res. Int. 113 , 102–114 (2018).

Yu, P., Low, M. Y. & Zhou, W. Design of experiments and regression modelling in food flavour and sensory analysis: A review. Trends Food Sci. Technol. 71 , 202–215 (2018).

Oladokun, O. et al. The impact of hop bitter acid and polyphenol profiles on the perceived bitterness of beer. Food Chem. 205 , 212–220 (2016).

Linforth, R., Cabannes, M., Hewson, L., Yang, N. & Taylor, A. Effect of fat content on flavor delivery during consumption: An in vivo model. J. Agric. Food Chem. 58 , 6905–6911 (2010).

Guo, S., Na Jom, K. & Ge, Y. Influence of roasting condition on flavor profile of sunflower seeds: A flavoromics approach. Sci. Rep. 9 , 11295 (2019).

Ren, Q. et al. The changes of microbial community and flavor compound in the fermentation process of Chinese rice wine using Fagopyrum tataricum grain as feedstock. Sci. Rep. 9 , 3365 (2019).

Hastie, T., Friedman, J. & Tibshirani, R. The Elements of Statistical Learning. (Springer, New York, NY). https://doi.org/10.1007/978-0-387-21606-5 (2001).

Dietz, C., Cook, D., Huismann, M., Wilson, C. & Ford, R. The multisensory perception of hop essential oil: a review. J. Inst. Brew. 126 , 320–342 (2020).

CAS   Google Scholar  

Roncoroni, Miguel & Verstrepen, Kevin Joan. Belgian Beer: Tested and Tasted. (Lannoo, 2018).

Meilgaard, M. Flavor chemistry of beer: Part II: Flavor and threshold of 239 aroma volatiles. in (1975).

Bokulich, N. A. & Bamforth, C. W. The microbiology of malting and brewing. Microbiol. Mol. Biol. Rev. MMBR 77 , 157–172 (2013).

Dzialo, M. C., Park, R., Steensels, J., Lievens, B. & Verstrepen, K. J. Physiology, ecology and industrial applications of aroma formation in yeast. FEMS Microbiol. Rev. 41 , S95–S128 (2017).

Article   PubMed   PubMed Central   Google Scholar  

Datta, A. et al. Computer-aided food engineering. Nat. Food 3 , 894–904 (2022).

American Society of Brewing Chemists. Beer Methods. (American Society of Brewing Chemists, St. Paul, MN, U.S.A.).

Olaniran, A. O., Hiralal, L., Mokoena, M. P. & Pillay, B. Flavour-active volatile compounds in beer: production, regulation and control. J. Inst. Brew. 123 , 13–23 (2017).

Verstrepen, K. J. et al. Flavor-active esters: Adding fruitiness to beer. J. Biosci. Bioeng. 96 , 110–118 (2003).

Meilgaard, M. C. Flavour chemistry of beer. part I: flavour interaction between principal volatiles. Master Brew. Assoc. Am. Tech. Q 12 , 107–117 (1975).

Briggs, D. E., Boulton, C. A., Brookes, P. A. & Stevens, R. Brewing 227–254. (Woodhead Publishing). https://doi.org/10.1533/9781855739062.227 (2004).

Bossaert, S., Crauwels, S., De Rouck, G. & Lievens, B. The power of sour - A review: Old traditions, new opportunities. BrewingScience 72 , 78–88 (2019).

Google Scholar  

Verstrepen, K. J. et al. Flavor active esters: Adding fruitiness to beer. J. Biosci. Bioeng. 96 , 110–118 (2003).

Snauwaert, I. et al. Microbial diversity and metabolite composition of Belgian red-brown acidic ales. Int. J. Food Microbiol. 221 , 1–11 (2016).

Spitaels, F. et al. The microbial diversity of traditional spontaneously fermented lambic beer. PLoS ONE 9 , e95384 (2014).

Blanco, C. A., Andrés-Iglesias, C. & Montero, O. Low-alcohol Beers: Flavor Compounds, Defects, and Improvement Strategies. Crit. Rev. Food Sci. Nutr. 56 , 1379–1388 (2016).

Jackowski, M. & Trusek, A. Non-Alcohol. beer Prod. – Overv. 20 , 32–38 (2018).

Takoi, K. et al. The contribution of geraniol metabolism to the citrus flavour of beer: Synergy of geraniol and β-citronellol under coexistence with excess linalool. J. Inst. Brew. 116 , 251–260 (2010).

Kroeze, J. H. & Bartoshuk, L. M. Bitterness suppression as revealed by split-tongue taste stimulation in humans. Physiol. Behav. 35 , 779–783 (1985).

Mennella, J. A. et al. A spoonful of sugar helps the medicine go down”: Bitter masking bysucrose among children and adults. Chem. Senses 40 , 17–25 (2015).

Wietstock, P., Kunz, T., Perreira, F. & Methner, F.-J. Metal chelation behavior of hop acids in buffered model systems. BrewingScience 69 , 56–63 (2016).

Sancho, D., Blanco, C. A., Caballero, I. & Pascual, A. Free iron in pale, dark and alcohol-free commercial lager beers. J. Sci. Food Agric. 91 , 1142–1147 (2011).

Rodrigues, H. & Parr, W. V. Contribution of cross-cultural studies to understanding wine appreciation: A review. Food Res. Int. 115 , 251–258 (2019).

Korneva, E. & Blockeel, H. Towards better evaluation of multi-target regression models. in ECML PKDD 2020 Workshops (eds. Koprinska, I. et al.) 353–362 (Springer International Publishing, Cham, 2020). https://doi.org/10.1007/978-3-030-65965-3_23 .

Gastón Ares. Mathematical and Statistical Methods in Food Science and Technology. (Wiley, 2013).

Grinsztajn, L., Oyallon, E. & Varoquaux, G. Why do tree-based models still outperform deep learning on tabular data? Preprint at http://arxiv.org/abs/2207.08815 (2022).

Gries, S. T. Statistics for Linguistics with R: A Practical Introduction. in Statistics for Linguistics with R (De Gruyter Mouton, 2021). https://doi.org/10.1515/9783110718256 .

Lundberg, S. M. et al. From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell. 2 , 56–67 (2020).

Ickes, C. M. & Cadwallader, K. R. Effects of ethanol on flavor perception in alcoholic beverages. Chemosens. Percept. 10 , 119–134 (2017).

Kato, M. et al. Influence of high molecular weight polypeptides on the mouthfeel of commercial beer. J. Inst. Brew. 127 , 27–40 (2021).

Wauters, R. et al. Novel Saccharomyces cerevisiae variants slow down the accumulation of staling aldehydes and improve beer shelf-life. Food Chem. 398 , 1–11 (2023).

Li, H., Jia, S. & Zhang, W. Rapid determination of low-level sulfur compounds in beer by headspace gas chromatography with a pulsed flame photometric detector. J. Am. Soc. Brew. Chem. 66 , 188–191 (2008).

Dercksen, A., Laurens, J., Torline, P., Axcell, B. C. & Rohwer, E. Quantitative analysis of volatile sulfur compounds in beer using a membrane extraction interface. J. Am. Soc. Brew. Chem. 54 , 228–233 (1996).

Molnar, C. Interpretable Machine Learning: A Guide for Making Black-Box Models Interpretable. (2020).

Zhao, Q. & Hastie, T. Causal interpretations of black-box models. J. Bus. Econ. Stat. Publ. Am. Stat. Assoc. 39 , 272–281 (2019).

Article   MathSciNet   Google Scholar  

Hastie, T., Tibshirani, R. & Friedman, J. The Elements of Statistical Learning. (Springer, 2019).

Labrado, D. et al. Identification by NMR of key compounds present in beer distillates and residual phases after dealcoholization by vacuum distillation. J. Sci. Food Agric. 100 , 3971–3978 (2020).

Lusk, L. T., Kay, S. B., Porubcan, A. & Ryder, D. S. Key olfactory cues for beer oxidation. J. Am. Soc. Brew. Chem. 70 , 257–261 (2012).

Gonzalez Viejo, C., Torrico, D. D., Dunshea, F. R. & Fuentes, S. Development of artificial neural network models to assess beer acceptability based on sensory properties using a robotic pourer: A comparative model approach to achieve an artificial intelligence system. Beverages 5 , 33 (2019).

Gonzalez Viejo, C., Fuentes, S., Torrico, D. D., Godbole, A. & Dunshea, F. R. Chemical characterization of aromas in beer and their effect on consumers liking. Food Chem. 293 , 479–485 (2019).

Gilbert, J. L. et al. Identifying breeding priorities for blueberry flavor using biochemical, sensory, and genotype by environment analyses. PLOS ONE 10 , 1–21 (2015).

Goulet, C. et al. Role of an esterase in flavor volatile variation within the tomato clade. Proc. Natl. Acad. Sci. 109 , 19009–19014 (2012).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Borisov, V. et al. Deep Neural Networks and Tabular Data: A Survey. IEEE Trans. Neural Netw. Learn. Syst. 1–21 https://doi.org/10.1109/TNNLS.2022.3229161 (2022).

Statista. Statista Consumer Market Outlook: Beer - Worldwide.

Seitz, H. K. & Stickel, F. Molecular mechanisms of alcoholmediated carcinogenesis. Nat. Rev. Cancer 7 , 599–612 (2007).

Voordeckers, K. et al. Ethanol exposure increases mutation rate through error-prone polymerases. Nat. Commun. 11 , 3664 (2020).

Goelen, T. et al. Bacterial phylogeny predicts volatile organic compound composition and olfactory response of an aphid parasitoid. Oikos 129 , 1415–1428 (2020).

Article   ADS   Google Scholar  

Reher, T. et al. Evaluation of hop (Humulus lupulus) as a repellent for the management of Drosophila suzukii. Crop Prot. 124 , 104839 (2019).

Stein, S. E. An integrated method for spectrum extraction and compound identification from gas chromatography/mass spectrometry data. J. Am. Soc. Mass Spectrom. 10 , 770–781 (1999).

American Society of Brewing Chemists. Sensory Analysis Methods. (American Society of Brewing Chemists, St. Paul, MN, U.S.A., 1992).

McAuley, J., Leskovec, J. & Jurafsky, D. Learning Attitudes and Attributes from Multi-Aspect Reviews. Preprint at https://doi.org/10.48550/arXiv.1210.3926 (2012).

Meilgaard, M. C., Carr, B. T. & Carr, B. T. Sensory Evaluation Techniques. (CRC Press, Boca Raton). https://doi.org/10.1201/b16452 (2014).

Schreurs, M. et al. Data from: Predicting and improving complex beer flavor through machine learning. Zenodo https://doi.org/10.5281/zenodo.10653704 (2024).

Download references

Acknowledgements

We thank all lab members for their discussions and thank all tasting panel members for their contributions. Special thanks go out to Dr. Karin Voordeckers for her tremendous help in proofreading and improving the manuscript. M.S. was supported by a Baillet-Latour fellowship, L.C. acknowledges financial support from KU Leuven (C16/17/006), F.A.T. was supported by a PhD fellowship from FWO (1S08821N). Research in the lab of K.J.V. is supported by KU Leuven, FWO, VIB, VLAIO and the Brewing Science Serves Health Fund. Research in the lab of T.W. is supported by FWO (G.0A51.15) and KU Leuven (C16/17/006).

Author information

These authors contributed equally: Michiel Schreurs, Supinya Piampongsant, Miguel Roncoroni.

Authors and Affiliations

VIB—KU Leuven Center for Microbiology, Gaston Geenslaan 1, B-3001, Leuven, Belgium

Michiel Schreurs, Supinya Piampongsant, Miguel Roncoroni, Lloyd Cool, Beatriz Herrera-Malaver, Florian A. Theßeling & Kevin J. Verstrepen

CMPG Laboratory of Genetics and Genomics, KU Leuven, Gaston Geenslaan 1, B-3001, Leuven, Belgium

Leuven Institute for Beer Research (LIBR), Gaston Geenslaan 1, B-3001, Leuven, Belgium

Laboratory of Socioecology and Social Evolution, KU Leuven, Naamsestraat 59, B-3000, Leuven, Belgium

Lloyd Cool, Christophe Vanderaa & Tom Wenseleers

VIB Bioinformatics Core, VIB, Rijvisschestraat 120, B-9052, Ghent, Belgium

Łukasz Kreft & Alexander Botzki

AB InBev SA/NV, Brouwerijplein 1, B-3000, Leuven, Belgium

Philippe Malcorps & Luk Daenen

You can also search for this author in PubMed   Google Scholar

Contributions

S.P., M.S. and K.J.V. conceived the experiments. S.P., M.S. and K.J.V. designed the experiments. S.P., M.S., M.R., B.H. and F.A.T. performed the experiments. S.P., M.S., L.C., C.V., L.K., A.B., P.M., L.D., T.W. and K.J.V. contributed analysis ideas. S.P., M.S., L.C., C.V., T.W. and K.J.V. analyzed the data. All authors contributed to writing the manuscript.

Corresponding author

Correspondence to Kevin J. Verstrepen .

Ethics declarations

Competing interests.

K.J.V. is affiliated with bar.on. The other authors declare no competing interests.

Peer review

Peer review information.

Nature Communications thanks Florian Bauer, Andrew John Macintosh and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, description of additional supplementary files, supplementary data 1, supplementary data 2, supplementary data 3, supplementary data 4, supplementary data 5, supplementary data 6, supplementary data 7, reporting summary, source data, source data, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Schreurs, M., Piampongsant, S., Roncoroni, M. et al. Predicting and improving complex beer flavor through machine learning. Nat Commun 15 , 2368 (2024). https://doi.org/10.1038/s41467-024-46346-0

Download citation

Received : 30 October 2023

Accepted : 21 February 2024

Published : 26 March 2024

DOI : https://doi.org/10.1038/s41467-024-46346-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Research: Where to Begin

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

We live in an age overflowing with sources of information. With so many information sources at our fingertips, knowing where to start, sorting through it all and finding what we want can be overwhelming! This handout provides answers to the following research-related questions: Where do I begin? Where should I look for information? What types of sources are available?

Research isn't something that only scientists and professors do. Any time you use sources to investigate claims or reach new conclusions, you are performing research. Research happens in virtually all fields, so it’s vitally important to know how to conduct research and navigate through source material regardless of your professional or academic role.

Choosing and Narrowing Your Research Topic

Before beginning the process of looking for sources, it’s important to choose a research topic that is specific enough to explore in-depth. If your focus is too broad, it will be difficult to find sources that back up what you’re trying to say.

If your instructor gives you the flexibility to choose your own research topic, you might begin by brainstorming  a list of topics that interest you ( click here to visit an OWL page that can help you get started brainstorming or prewriting ). Once you find something that grabs your attention, the next step is to narrow your topic to a manageable scope. Some ways to narrow your focus are by sub-topic, demographic, or time period.

For example, suppose that you want to research cancer treatments. Cancer treatment is a fairly broad topic, so you would be wise to at least consider narrowing your scope. For example, you could focus on a sub-topic of cancer treatment, such as chemotherapy or radiation therapy. However, these are still broad topics, so you might also narrow your topic to a narrower sub-topic or even examine how these topics relate to a specific demographic or time period. In the end, you might decide to research how radiation therapy for women over fifty has changed in the past twenty years. In sum, having a specific idea of what you want to research helps you find a topic that feels more manageable.

Writing Your Research Question

Writing your research topic as a question helps you focus your topic in a clear and concise way. It ensure that your topic is arguable. While not all research papers have to offer an explicit argument, many do.

For the above example, you might phrase your research question like this: "How has radiation therapy changed in the past twenty years for women over fifty?" Of course, phrasing this topic as a question assumes that the research has, in fact, changed. Reading your sources (or, to begin with, at least summaries and abstracts of those sources) will help you formulate a research question that makes sense.

Knowing What Types of Sources You Need

Depending on the type of research you’re doing, you may need to use different types of sources. Research is usually divided into scholarly and popular, and primary and secondary. For more information on specific details about these types of sources, visit our "Where to Begin" page in our "Evaluating Sources" subsection.  This subsection contains additional pages that explore various kinds of sources (like, e.g., internet sources) in more detail.

Asking Productive Questions

Before you begin your research, you should ask yourself questions that help narrow your search parameters.

What kind of information are you looking for?

Different types of research will require different sources. It’s important to know what kinds of sources your research demands. Ask whether you need facts or opinions, news reports, research studies, statistics and data, personal reflections, archival research, etc. Restricting yourself to only the most relevant kinds of sources will make the research process seem less daunting.

Where do you need to look for your research?

Your research topic will also dictate where you find your sources. This extends beyond simply whether you use the internet or a print source. For example, if you are searching for information on a current event, a well-regarded newspaper like the  New York Times  or  Wall Street Journal  could  be a useful source. If you are searching for statistics on some aspect of the U.S. population, then you might want to start with government documents, such as census reports. While much high-level academic research relies mainly on the sorts of academic journal articles and scholarly books that can be found in university libraries, depending the nature of your research project, you may need to look elsewhere.

How much information do you need?

Different research projects require different numbers of sources. For example, if you need to address both sides of a controversial issue, you may need to find more sources than if you were pursuing a non-controversial topic. Be sure to speak with your instructor if you are unclear on how many sources you will be expected to use.

How timely does your research need to be?

Depending on your research topic, the timeliness of your source may or may not matter. For example, if you are looking into recent changes in a specific scientific field, you would want the most up-to-date research. However, if you were researching the War of 1812, you might benefit from finding primary sources written during that time period.

Help | Advanced Search

Computer Science > Human-Computer Interaction

Title: "that's not good science": an argument for the thoughtful use of formative situations in research through design.

Abstract: Most currently accepted approaches to evaluating Research through Design (RtD) presume that design prototypes are finalized and ready for robust testing in laboratory or in-the-wild settings. However, it is also valuable to assess designs at intermediate phases with mid-fidelity prototypes, not just to inform an ongoing design process, but also to glean knowledge of broader use to the research community. We propose 'formative situations' as a frame for examining mid-fidelity prototypes-in-process in this way. We articulate a set of criteria to help the community better assess the rigor of formative situations, in the service of opening conversation about establishing formative situations as a valuable contribution type within the RtD community.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Physical Review Research

• Collections
• Editorial Team
• Open Access

Reservoir engineering for classical nonlinear fields

Benedikt tissot, hugo ribeiro, and florian marquardt, phys. rev. research 6 , 023015 – published 3 april 2024.

• No Citing Articles
• INTRODUCTION
• SYSTEM-RESERVOIR COUPLING WITH PARTICLE…
• MICROSCOPIC MODEL: THE ANHARMONIC CHAIN
• CONCLUSIONS
• ACKNOWLEDGMENTS

Reservoir engineering has become a prominent tool to control quantum systems. Recently, there have been first experiments applying it to many-body systems, especially with a view to engineer particle-conserving dissipation for quantum simulations using bosons. In this paper, we explore the dissipative dynamics of these systems in the classical limit. We derive a general equation of motion capturing the effective nonlinear dissipation introduced by the bath and apply it to the special case of a Bose-Hubbard model, where it leads to an unconventional type of dissipative nonlinear Schrödinger equation. Building on that, we study the dynamics of one and two solitons in such a dissipative classical field theory.

• Received 16 November 2023
• Accepted 5 March 2024

DOI: https://doi.org/10.1103/PhysRevResearch.6.023015

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI. Open access publication funded by Max Planck Society.

Published by the American Physical Society

Physics Subject Headings (PhySH)

• Research Areas
• Physical Systems

Authors & Affiliations

• 1 Department of Physics, University of Konstanz, D-78457 Konstanz, Germany
• 2 Department of Physics and Applied Physics, University of Massachusetts Lowell, Lowell, Massachusetts 01854, USA
• 3 Max Planck Institute for the Science of Light, Staudtstr. 2, 91058 Erlangen, Germany
• 4 Department of Physics, Friedrich-Alexander-Universität Erlangen-Nürnberg, Staudstr. 7, 91058 Erlangen, Germany
• * [email protected]
• † [email protected]
• ‡ [email protected]

Article Text

Vol. 6, Iss. 2 — April - June 2024

Subject Areas

• Atomic and Molecular Physics
• Nonlinear Dynamics

Authorization Required

Other options.

• Buy Article »
• Find an Institution with the Article »

Download & Share

(a) The physical scenario of a bosonic system (in the classical limit) coupled to a driven reservoir, in a particle-conserving way. (b) Particular illustrative realization considered in the text, a Bose-Hubbard chain (blue b j ) with sites coupled to driven cavities (orange bosonic modes a j ) with coupling strength χ and to each other with hopping rate J . See main text for details. (c) Spectrum of a dissipative cavity, where the incoming drive can be upscattered in frequency through interaction with the bosonic many-body system, extracting energy. The detuning Δ and the dissipation rate κ determine the properties of this driven reservoir.

Effective parameters as defined in Eqs. ( 5 ) and ( 7 ) as functions of the detuning: The correction of the effective on-site interaction δ g (black solid) as well as the dissipation parameter γ (dashed gray). The dissipation parameter is positive in the red-detuned domain ( Δ < 0 ) and changes sign in the blue-detuned domain ( Δ > 0 ). Both the correction of the on-site interaction, as well as the dissipation parameter scale with the second order of the bath interaction but have different limiting behavior at large detuning Δ .

Comparison of the PCDNSE [Eq. ( 10 )] and the discrete equations of motion (EOM) including the reservoir dynamics, see Eqs. ( 2 ) and ( 3 ). In panel (a) we compare simulations for different array lengths to the PCDNSE and in (b) different cavity parameters leading to the same effective evolution. Only when the continuum limit or weak-coupling limit are not satisfied (small L or Δ = − 2 J , respectively) the discrete dynamics are in disagreement with the dynamics of the PCDNSE. Unless otherwise specified (see legend) we use L = 800 , κ = J , Δ = − 0.1 J , η = J and χ , α such that we get the effective parameters g = − 0.1 J , γ = 0.05 . The distributions are taken after an evolution for J t = 50 ( L / 400 ) 2 of an initial stable soliton [see inset of (b)] with height ψ ( 0 ) = 1 and velocity v ( 0 ) ≈ 0.48 . The PCDNSE uses a length of L = 400 . For the evolution according to the discrete EOM, we assume that the cavities are initially in their unperturbed steady state η / ( κ / 2 − i Δ ) .

Velocity damping rate as a function of g γ ψ 4 ( 0 ) . We see good agreement between the variational approach and numerical results of the particle-conserving dissipative nonlinear Schrödinger equation (PCDNSE). The deviation at the smallest value is due to the soliton breaking up in the simulation, underlined by the datapoint given by the star where the relative and absolute tolerances where decreased to 10 − 13 and 10 − 12 respectively from 10 − 8 for both. The inset shows that the deviation | δ ( J t = 4 ) | = | ψ 2 ( J t = 4 ) / ψ 2 ( 0 ) − 1 | (pentagons, diamonds) as well as d ( J t = 4 ) (triangles) from the stable soliton stays small for the evolution. The PCDNSE simulation is for a stable soliton with ψ ( 0 ) = 1 , x ( 0 ) = L / 8 , v ( 0 ) ≈ 0.49 , and L = 600 and for a duration J t = 4 .

Shape stabilization of ill prepared solitons. The maximal amplitude of the PCDNSE (dashed lines) is compared to the prediction of the collective coordinates (solid lines) in (a) and (b) for the different deviations δ = ψ ( 0 ) / ψ SS − 1 [see legend in (c)]. While for δ = − 0.1 , 0.01 both solutions oscillate around the steady-state amplitude, for δ = 0.3 the soliton shape breaks down and this oscillation is visible in neither solution. The breakout is also visible in the inset of (b), which displays | Ψ ( J t , x − x 0 ) / ψ S S | 2 according to the PCDNSE and where the color corresponds to 0 (darkest purple) to 1.7 (lightest yellow). Therefore, we investigate the dynamics for long times for the smaller δ = − 0.1 , 0.01 [(b)–(d)]. Panel (b) shows the continued oscillation around the steady state amplitude after a prolonged evolution time of J t = 2 × 10 6 and (c) displays the modulo square distribution after this evolution. To quantify the long time dynamics we compare the envelope of the deviation of the maximal amplitude ψ ( t ) deviation from the steady state amplitude ψ SS on a long timescale in these cases (d). To this end, we calculate the envelope as the maximal deviation within a time-window of length 5 × 10 4 . The relaxation process is sketched in the inset of (d). We use g = − 0.1 J , N = 1 , γ = 0.1 , L = 10 w SS , and only deviate from the stable soliton via δ ≠ 0 (i.e., d = 0 ).

Influence of the dissipative dynamics due to the interaction of two stable solitons. The figure depicts the energy of the two solitons compared to the single soliton solution as a function of time during the interaction for different γ (see legend). The inset shows | Ψ ( x , t ) | 2 for γ = 10 − 2 where the values range from 0 (dark) to ≈ 3.62 (light). We use g = − 0.1 J and the initial solitons have ψ ( 0 ) = 1 ( N ≈ 9 ), and | v ( 0 ) | ≈ 0.5 with opposite signs and are placed 5 w ( 0 ) apart on a space of size 20 w ( 0 ) . The figure shows that the interaction of the two solitons can enhance the energy dissipation.

Sign up to receive regular email alerts from Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

• Forgot your username/password?
• Create an account

Article Lookup

Paste a citation or doi, enter a citation.