• Search Menu
  • Advance Articles
  • Editor's Choice
  • CME Reviews
  • Best of 2021 collection
  • Abbreviated Breast MRI Virtual Collection
  • Contrast-enhanced Mammography Collection
  • Author Guidelines
  • Submission Site
  • Open Access
  • Self-Archiving Policy
  • Accepted Papers Resource Guide
  • About Journal of Breast Imaging
  • About the Society of Breast Imaging
  • Guidelines for Reviewers
  • Resources for Reviewers and Authors
  • Editorial Board
  • Advertising Disclaimer
  • Advertising and Corporate Services
  • Journals on Oxford Academic
  • Books on Oxford Academic

Society of Breast Imaging

  • < Previous

A Step-by-Step Guide to Writing a Scientific Review Article

  • Article contents
  • Figures & tables
  • Supplementary Data

Manisha Bahl, A Step-by-Step Guide to Writing a Scientific Review Article, Journal of Breast Imaging , Volume 5, Issue 4, July/August 2023, Pages 480–485, https://doi.org/10.1093/jbi/wbad028

  • Permissions Icon Permissions

Scientific review articles are comprehensive, focused reviews of the scientific literature written by subject matter experts. The task of writing a scientific review article can seem overwhelming; however, it can be managed by using an organized approach and devoting sufficient time to the process. The process involves selecting a topic about which the authors are knowledgeable and enthusiastic, conducting a literature search and critical analysis of the literature, and writing the article, which is composed of an abstract, introduction, body, and conclusion, with accompanying tables and figures. This article, which focuses on the narrative or traditional literature review, is intended to serve as a guide with practical steps for new writers. Tips for success are also discussed, including selecting a focused topic, maintaining objectivity and balance while writing, avoiding tedious data presentation in a laundry list format, moving from descriptions of the literature to critical analysis, avoiding simplistic conclusions, and budgeting time for the overall process.

  • narrative discourse

Email alerts

Citing articles via.

  • Recommend to your Librarian
  • Journals Career Network


  • Online ISSN 2631-6129
  • Print ISSN 2631-6110
  • Copyright © 2024 Society of Breast Imaging
  • About Oxford Academic
  • Publish journals with us
  • University press partners
  • What we publish
  • New features  
  • Open access
  • Institutional account management
  • Rights and permissions
  • Get help with access
  • Accessibility
  • Advertising
  • Media enquiries
  • Oxford University Press
  • Oxford Languages
  • University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

  • Copyright © 2024 Oxford University Press
  • Cookie settings
  • Cookie policy
  • Privacy policy
  • Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

OWL logo

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.


  • An introductory paragraph that explains what your working topic and thesis is
  • A forecast of key topics or texts that will appear in the review
  • Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
  • Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
  • Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
  • Critically Evaluate: Mention the strengths and weaknesses of your sources
  • Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.


  • Summarize the key findings you have taken from the literature and emphasize their significance
  • Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

  • Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
  • Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
  • Qualitative versus quantitative research
  • Empirical versus theoretical scholarship
  • Divide the research by sociological, historical, or cultural sources
  • Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

  • It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
  • Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
  • Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
  • Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Logo for University of Southern Queensland

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7 Writing a Literature Review

Hundreds of original investigation research articles on health science topics are published each year. It is becoming harder and harder to keep on top of all new findings in a topic area and – more importantly – to work out how they all fit together to determine our current understanding of a topic. This is where literature reviews come in.

In this chapter, we explain what a literature review is and outline the stages involved in writing one. We also provide practical tips on how to communicate the results of a review of current literature on a topic in the format of a literature review.

7.1 What is a literature review?

Screenshot of journal article

Literature reviews provide a synthesis and evaluation  of the existing literature on a particular topic with the aim of gaining a new, deeper understanding of the topic.

Published literature reviews are typically written by scientists who are experts in that particular area of science. Usually, they will be widely published as authors of their own original work, making them highly qualified to author a literature review.

However, literature reviews are still subject to peer review before being published. Literature reviews provide an important bridge between the expert scientific community and many other communities, such as science journalists, teachers, and medical and allied health professionals. When the most up-to-date knowledge reaches such audiences, it is more likely that this information will find its way to the general public. When this happens, – the ultimate good of science can be realised.

A literature review is structured differently from an original research article. It is developed based on themes, rather than stages of the scientific method.

In the article Ten simple rules for writing a literature review , Marco Pautasso explains the importance of literature reviews:

Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications. For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively. Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests. Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read. For such summaries to be useful, however, they need to be compiled in a professional way (Pautasso, 2013, para. 1).

An example of a literature review is shown in Figure 7.1.

Video 7.1: What is a literature review? [2 mins, 11 secs]

Watch this video created by Steely Library at Northern Kentucky Library called ‘ What is a literature review? Note: Closed captions are available by clicking on the CC button below.

Examples of published literature reviews

  • Strength training alone, exercise therapy alone, and exercise therapy with passive manual mobilisation each reduce pain and disability in people with knee osteoarthritis: a systematic review
  • Traveler’s diarrhea: a clinical review
  • Cultural concepts of distress and psychiatric disorders: literature review and research recommendations for global mental health epidemiology

7.2 Steps of writing a literature review

Writing a literature review is a very challenging task. Figure 7.2 summarises the steps of writing a literature review. Depending on why you are writing your literature review, you may be given a topic area, or may choose a topic that particularly interests you or is related to a research project that you wish to undertake.

Chapter 6 provides instructions on finding scientific literature that would form the basis for your literature review.

Once you have your topic and have accessed the literature, the next stages (analysis, synthesis and evaluation) are challenging. Next, we look at these important cognitive skills student scientists will need to develop and employ to successfully write a literature review, and provide some guidance for navigating these stages.

Steps of writing a ltierature review which include: research, synthesise, read abstracts, read papers, evaualte findings and write

Analysis, synthesis and evaluation

Analysis, synthesis and evaluation are three essential skills required by scientists  and you will need to develop these skills if you are to write a good literature review ( Figure 7.3 ). These important cognitive skills are discussed in more detail in Chapter 9.

Diagram with the words analysis, synthesis and evaluation. Under analysis it says taking a process or thing and breaking it down. Under synthesis it says combining elements of separate material and under evaluation it says critiquing a product or process

The first step in writing a literature review is to analyse the original investigation research papers that you have gathered related to your topic.

Analysis requires examining the papers methodically and in detail, so you can understand and interpret aspects of the study described in each research article.

An analysis grid is a simple tool you can use to help with the careful examination and breakdown of each paper. This tool will allow you to create a concise summary of each research paper; see Table 7.1 for an example of  an analysis grid. When filling in the grid, the aim is to draw out key aspects of each research paper. Use a different row for each paper, and a different column for each aspect of the paper ( Tables 7.2 and 7.3 show how completed analysis grid may look).

Before completing your own grid, look at these examples and note the types of information that have been included, as well as the level of detail. Completing an analysis grid with a sufficient level of detail will help you to complete the synthesis and evaluation stages effectively. This grid will allow you to more easily observe similarities and differences across the findings of the research papers and to identify possible explanations (e.g., differences in methodologies employed) for observed differences between the findings of different research papers.

Table 7.1: Example of an analysis grid

A tab;e split into columns with annotated comments

Table 7.3: Sample filled-in analysis grid for research article by Ping and colleagues

Source: Ping, WC, Keong, CC & Bandyopadhyay, A 2010, ‘Effects of acute supplementation of caffeine on cardiorespiratory responses during endurance running in a hot and humid climate’, Indian Journal of Medical Research, vol. 132, pp. 36–41. Used under a CC-BY-NC-SA licence.

Step two of writing a literature review is synthesis.

Synthesis describes combining separate components or elements to form a connected whole.

You will use the results of your analysis to find themes to build your literature review around. Each of the themes identified will become a subheading within the body of your literature review.

A good place to start when identifying themes is with the dependent variables (results/findings) that were investigated in the research studies.

Because all of the research articles you are incorporating into your literature review are related to your topic, it is likely that they have similar study designs and have measured similar dependent variables. Review the ‘Results’ column of your analysis grid. You may like to collate the common themes in a synthesis grid (see, for example Table 7.4 ).

Table showing themes of the article including running performance, rating of perceived exertion, heart rate and oxygen uptake

Step three of writing a literature review is evaluation, which can only be done after carefully analysing your research papers and synthesising the common themes (findings).

During the evaluation stage, you are making judgements on the themes presented in the research articles that you have read. This includes providing physiological explanations for the findings. It may be useful to refer to the discussion section of published original investigation research papers, or another literature review, where the authors may mention tested or hypothetical physiological mechanisms that may explain their findings.

When the findings of the investigations related to a particular theme are inconsistent (e.g., one study shows that caffeine effects performance and another study shows that caffeine had no effect on performance) you should attempt to provide explanations of why the results differ, including physiological explanations. A good place to start is by comparing the methodologies to determine if there are any differences that may explain the differences in the findings (see the ‘Experimental design’ column of your analysis grid). An example of evaluation is shown in the examples that follow in this section, under ‘Running performance’ and ‘RPE ratings’.

When the findings of the papers related to a particular theme are consistent (e.g., caffeine had no effect on oxygen uptake in both studies) an evaluation should include an explanation of why the results are similar. Once again, include physiological explanations. It is still a good idea to compare methodologies as a background to the evaluation. An example of evaluation is shown in the following under ‘Oxygen consumption’.

Annotated paragraphs on running performance with annotated notes such as physiological explanation provided; possible explanation for inconsistent results

7.3 Writing your literature review

Once you have completed the analysis, and synthesis grids and written your evaluation of the research papers , you can combine synthesis and evaluation information to create a paragraph for a literature review ( Figure 7.4 ).

Bubble daigram showing connection between synethesis, evaulation and writing a paragraph

The following paragraphs are an example of combining the outcome of the synthesis and evaluation stages to produce a paragraph for a literature review.

Note that this is an example using only two papers – most literature reviews would be presenting information on many more papers than this ( (e.g., 106 papers in the review article by Bain and colleagues discussed later in this chapter). However, the same principle applies regardless of the number of papers reviewed.

Introduction paragraph showing where evaluation occurs

The next part of this chapter looks at the each section of a literature review and explains how to write them by referring to a review article that was published in Frontiers in Physiology and shown in Figure 7.1. Each section from the published article is annotated to highlight important features of the format of the review article, and identifies the synthesis and evaluation information.

In the examination of each review article section we will point out examples of how the authors have presented certain information and where they display application of important cognitive processes; we will use the colour code shown below:

Colour legend

This should be one paragraph that accurately reflects the contents of the review article.

An annotated abstract divided into relevant background information, identification of the problem, summary of recent literature on topic, purpose of the review


The introduction should establish the context and importance of the review

An annotated introduction divided into relevant background information, identification of the issue and overview of points covered

Body of literature review

Annotated body of literature review with following comments annotated on the side: subheadings are included to separate body of review into themes; introductory sentences with general background information; identification of gap in current knowledge; relevant theoretical background information; syntheis of literature relating to the potential importance of cerebral metabolism; an evaluation; identification of gaps in knowledge; synthesis of findings related to human studies; author evaluation

The reference section provides a list of the references that you cited in the body of your review article. The format will depend on the journal of publication as each journal has their own specific referencing format.

It is important to accurately cite references in research papers to acknowledge your sources and ensure credit is appropriately given to authors of work you have referred to. An accurate and comprehensive reference list also shows your readers that you are well-read in your topic area and are aware of the key papers that provide the context to your research.

It is important to keep track of your resources and to reference them consistently in the format required by the publication in which your work will appear. Most scientists will use reference management software to store details of all of the journal articles (and other sources) they use while writing their review article. This software also automates the process of adding in-text references and creating a reference list. In the review article by Bain et al. (2014) used as an example in this chapter, the reference list contains 106 items, so you can imagine how much help referencing software would be. Chapter 5 shows you how to use EndNote, one example of reference management software.

Click the drop down below to review the terms learned from this chapter.

Copyright note:

  • The quotation from Pautasso, M 2013, ‘Ten simple rules for writing a literature review’, PLoS Computational Biology is use under a CC-BY licence. 
  • Content from the annotated article and tables are based on Schubert, MM, Astorino, TA & Azevedo, JJL 2013, ‘The effects of caffeinated ‘energy shots’ on time trial performance’, Nutrients, vol. 5, no. 6, pp. 2062–2075 (used under a CC-BY 3.0 licence ) and P ing, WC, Keong , CC & Bandyopadhyay, A 2010, ‘Effects of acute supplementation of caffeine on cardiorespiratory responses during endurance running in a hot and humid climate’, Indian Journal of Medical Research, vol. 132, pp. 36–41 (used under a CC-BY-NC-SA 4.0 licence ). 

Bain, A.R., Morrison, S.A., & Ainslie, P.N. (2014). Cerebral oxygenation and hyperthermia. Frontiers in Physiology, 5 , 92.

Pautasso, M. (2013). Ten simple rules for writing a literature review. PLoS Computational Biology, 9 (7), e1003149.

How To Do Science Copyright © 2022 by University of Southern Queensland is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Get science-backed answers as you write with Paperpal's Research feature

What is a Literature Review? How to Write It (with Examples)

literature review

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

  • What is the purpose of literature review? 
  • a. Habitat Loss and Species Extinction: 
  • b. Range Shifts and Phenological Changes: 
  • c. Ocean Acidification and Coral Reefs: 
  • d. Adaptive Strategies and Conservation Efforts: 
  • How to write a good literature review 
  • Choose a Topic and Define the Research Question: 
  • Decide on the Scope of Your Review: 
  • Select Databases for Searches: 
  • Conduct Searches and Keep Track: 
  • Review the Literature: 
  • Organize and Write Your Literature Review: 
  • Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

literature review on scientific paper

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

  • Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 
  • Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 
  • Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 
  • Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 
  • Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 
  • Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

literature review on scientific paper

How to write a good literature review

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

  • Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
  • Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
  • Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
  • Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
  • Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

  • Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
  • Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
  • Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
  • Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
  • Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

  • Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
  • Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
  • Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
  • Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
  • Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

what is a literature review

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

  • Select a topic that is relevant to your field of study. 
  • Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

  • Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
  • Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
  • Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

  • Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
  • Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

  • Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
  • Record and document your search strategy for transparency and replicability. 
  • Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

  • Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
  • Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
  • Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
  • Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
  • Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

  • Literature review outline should be based on themes, chronological order, or methodological approaches. 
  • Write a clear and coherent narrative that synthesizes the information gathered. 
  • Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
  • Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. 

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 


  • Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
  • Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
  • Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

  • Empirical Research: A Comprehensive Guide for Academics 
  • How to Write a Scientific Paper in 10 Steps 
  • Life Sciences Papers: 9 Tips for Authors Writing in Biological Sciences
  • What is an Argumentative Essay? How to Write It (With Examples)

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, 4 ways paperpal encourages responsible writing with ai, what are scholarly sources and where can you..., how to write a hypothesis types and examples , measuring academic success: definition & strategies for excellence, what is academic writing: tips for students, why traditional editorial process needs an upgrade, paperpal’s new ai research finder empowers authors to..., what is hedging in academic writing  , how to use ai to enhance your college..., ai + human expertise – a paradigm shift....

  • USC Libraries
  • Research Guides

Organizing Your Social Sciences Research Paper

  • 5. The Literature Review
  • Purpose of Guide
  • Design Flaws to Avoid
  • Independent and Dependent Variables
  • Glossary of Research Terms
  • Reading Research Effectively
  • Narrowing a Topic Idea
  • Broadening a Topic Idea
  • Extending the Timeliness of a Topic Idea
  • Academic Writing Style
  • Applying Critical Thinking
  • Choosing a Title
  • Making an Outline
  • Paragraph Development
  • Research Process Video Series
  • Executive Summary
  • The C.A.R.S. Model
  • Background Information
  • The Research Problem/Question
  • Theoretical Framework
  • Citation Tracking
  • Content Alert Services
  • Evaluating Sources
  • Primary Sources
  • Secondary Sources
  • Tiertiary Sources
  • Scholarly vs. Popular Publications
  • Qualitative Methods
  • Quantitative Methods
  • Insiderness
  • Using Non-Textual Elements
  • Limitations of the Study
  • Common Grammar Mistakes
  • Writing Concisely
  • Avoiding Plagiarism
  • Footnotes or Endnotes?
  • Further Readings
  • Generative AI and Writing
  • USC Libraries Tutorials and Other Guides
  • Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

  • Give a new interpretation of old material or combine new with old interpretations,
  • Trace the intellectual progression of the field, including major debates,
  • Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
  • Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

  • Place each work in the context of its contribution to understanding the research problem being studied.
  • Describe the relationship of each work to the others under consideration.
  • Identify new ways to interpret prior research.
  • Reveal any gaps that exist in the literature.
  • Resolve conflicts amongst seemingly contradictory previous studies.
  • Identify areas of prior scholarship to prevent duplication of effort.
  • Point the way in fulfilling a need for additional research.
  • Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

  • An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
  • Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
  • An explanation of how each work is similar to and how it varies from the others,
  • Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

  • Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
  • Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
  • Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
  • Persuasiveness -- which of the author's theses are most convincing or least convincing?
  • Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

  • Current Situation : Information necessary to understand the current topic or focus of the literature review.
  • Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
  • History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
  • Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
  • Standards : Description of the way in which you present your information.
  • Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

  • Sources in your literature review do not clearly relate to the research problem;
  • You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
  • Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
  • Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
  • Does not describe the search procedures that were used in identifying the literature to review;
  • Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
  • Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

  • How are they organizing their ideas?
  • What methods have they used to study the problem?
  • What theories have been used to explain, predict, or understand their research problem?
  • What sources have they cited to support their conclusions?
  • How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

  • Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
  • Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
  • Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

  • << Previous: Theoretical Framework
  • Next: Citation Tracking >>
  • Last Updated: May 2, 2024 4:39 PM
  • URL: https://libguides.usc.edu/writingguide

Research in the Biological and Life Sciences: A Guide for Cornell Researchers: Literature Reviews

  • Books and Dissertations
  • Databases and Journals
  • Locating Theses
  • Resource Not at Cornell?
  • Citing Sources
  • Staying Current
  • Measuring your research impact
  • Plagiarism and Copyright
  • Data Management
  • Literature Reviews
  • Evidence Synthesis and Systematic Reviews
  • Writing an Honors Thesis
  • Poster Making and Printing
  • Research Help

What is a Literature Review?

A literature review is a body of text that aims to review the critical points of current knowledge on a particular topic. Most often associated with science-oriented literature, such as a thesis, the literature review usually proceeds a research proposal, methodology and results section. Its ultimate goals is to bring the reader up to date with current literature on a topic and forms that basis for another goal, such as the justification for future research in the area. (retrieved from  http://en.wikipedia.org/wiki/Literature_review )

Writing a Literature Review

The literature review is the section of your paper in which you cite and briefly review the related research studies that have been conducted. In this space, you will describe the foundation on which  your  research will be/is built. You will:

  • discuss the work of others
  • evaluate their methods and findings
  • identify any gaps in their research
  • state how  your  research is different

The literature review should be selective and should group the cited studies in some logical fashion.

If you need some additional assistance writing your literature review, the Knight Institute for Writing in the Disciplines offers a  Graduate Writing Service .

Demystifying the Literature Review

For more information, visit our guide devoted to " Demystifying the Literature Review " which includes:

  • guide to conducting a literature review,
  • a recorded 1.5 hour workshop covering the steps of a literature review, a checklist for drafting your topic and search terms, citation management software for organizing your results, and database searching.

Online Resources

  • A Guide to Library Research at Cornell University
  • Literature Reviews: An Overview for Graduate Students North Carolina State University 
  • The Literature Review: A Few Tips on Conducting Written by Dena Taylor, Director, Health Sciences Writing Centre, and Margaret Procter, Coordinator, Writing Support, University of Toronto
  • How to Write a Literature Review University Library, University of California, Santa Cruz
  • Review of Literature The Writing Center, University of Wisconsin-Madison

Print Resources

literature review on scientific paper

  • << Previous: Writing
  • Next: Evidence Synthesis and Systematic Reviews >>
  • Last Updated: Oct 25, 2023 11:28 AM
  • URL: https://guides.library.cornell.edu/bio

Enago Academy

How to Write a Good Scientific Literature Review

' src=

Nowadays, there is a huge demand for scientific literature reviews as they are especially appreciated by scholars or researchers when designing their research proposals. While finding information is less of a problem to them, discerning which paper or publication has enough quality has become one of the biggest issues. Literature reviews narrow the current knowledge on a certain field and examine the latest publications’ strengths and weaknesses. This way, they are priceless tools not only for those who are starting their research, but also for all those interested in recent publications. To be useful, literature reviews must be written in a professional way with a clear structure. The amount of work needed to write a scientific literature review must be considered before starting one since the tasks required can overwhelm many if the working method is not the best.

Designing and Writing a Scientific Literature Review

Writing a scientific review implies both researching for relevant academic content and writing , however, writing without having a clear objective is a common mistake. Sometimes, studying the situation and defining the work’s system is so important and takes equally as much time as that required in writing the final result. Therefore, we suggest that you divide your path into three steps.

Define goals and a structure

Think about your target and narrow down your topic. If you don’t choose a well-defined topic, you can find yourself dealing with a wide subject and plenty of publications about it. Remember that researchers usually deal with really specific fields of study.

It is time to be a critic and locate only pertinent publications. While researching for content consider publications that were written 3 years ago at the most. Write notes and summarize the content of each paper as that will help you in the next step.

Time to write

Check some literature review examples to decide how to start writing a good literature review . When your goals and structure are defined, begin writing without forgetting your target at any moment.

Related: Conducting a literature survey? Wish to learn more about scientific misconduct? Check out this resourceful infographic.

Here you have a to-do list to help you write your review :

Review Article

  • A scientific literature review usually includes a title, abstract, index, introduction, corpus, bibliography, and appendices (if needed).
  • Present the problem clearly.
  • Mention the paper’s methodology, research methods, analysis, instruments, etc.
  • Present literature review examples that can help you express your ideas.
  • Remember to cite accurately.
  • Limit your bias
  • While summarizing also identify strengths and weaknesses as this is critical.

Scholars and researchers are usually the best candidates to write scientific literature reviews, not only because they are experts in a certain field, but also because they know the exigencies and needs that researchers have while writing research proposals or looking for information among thousands of academic papers. Therefore, considering your experience as a researcher can help you understand how to write a scientific literature review.

Have you faced challenges while drafting your first literature review? How do you think can these tips help you in acing your next literature review? Let us know in the comments section below! You can also visit our  Q&A forum  for frequently asked questions related to copyrights answered by our team that comprises eminent researchers and publication experts.

literature review on scientific paper

Thank you for your information. It adds knowledge on critical review being a first time to do it, it helps a lot.

yes. i would like to ndertake the course Bio ststistics

Rate this article Cancel Reply

Your email address will not be published.

literature review on scientific paper

Enago Academy's Most Popular Articles

Best AI-Based Literature Review Tools

  • Reporting Research

AI Assistance in Academia for Searching Credible Scholarly Sources

The journey of academia is a grand quest for knowledge, more specifically an adventure to…

Writing a Literature Review

  • Manuscripts & Grants

Writing a Research Literature Review? — Here are tips to guide you through!

Literature review is both a process and a product. It involves searching within a defined…

article summarizer

  • AI in Academia

How to Scan Through Millions of Articles and Still Cut Down on Your Reading Time — Why not do it with an AI-based article summarizer?

Researcher 1: “It’s flooding articles every time I switch on my laptop!” Researcher 2: “Why…

literature mapping

How to Master at Literature Mapping: 5 Most Recommended Tools to Use

This article is also available in: Turkish, Spanish, Russian, and Portuguese

literature review on scientific paper

  • Old Webinars
  • Webinar Mobile App

Improving Your Chances of Publication in International Peer-reviewed Journals

Types of literature reviews Tips for writing review articles Role of meta-analysis Reporting guidelines

How to Scan Through Millions of Articles and Still Cut Down on Your Reading Time —…

literature review on scientific paper

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

  • 2000+ blog articles
  • 50+ Webinars
  • 10+ Expert podcasts
  • 50+ Infographics
  • 10+ Checklists
  • Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

literature review on scientific paper

What should universities' stance be on AI tools in research and academic writing?

Scientific paper recommendation systems: a literature review of recent publications

  • Open access
  • Published: 05 October 2022
  • Volume 23 , pages 335–369, ( 2022 )

Cite this article

You have full access to this open access article

literature review on scientific paper

  • Christin Katharina Kreutz   ORCID: orcid.org/0000-0002-5075-7699 1 &
  • Ralf Schenkel   ORCID: orcid.org/0000-0001-5379-5191 2  

10k Accesses

15 Citations

7 Altmetric

Explore all metrics

Scientific writing builds upon already published papers. Manual identification of publications to read, cite or consider as related papers relies on a researcher’s ability to identify fitting keywords or initial papers from which a literature search can be started. The rapidly increasing amount of papers has called for automatic measures to find the desired relevant publications, so-called paper recommendation systems. As the number of publications increases so does the amount of paper recommendation systems. Former literature reviews focused on discussing the general landscape of approaches throughout the years and highlight the main directions. We refrain from this perspective, instead we only consider a comparatively small time frame but analyse it fully. In this literature review we discuss used methods, datasets, evaluations and open challenges encountered in all works first released between January 2019 and October 2021. The goal of this survey is to provide a comprehensive and complete overview of current paper recommendation systems.

Similar content being viewed by others

literature review on scientific paper

Scientific Paper Recommender Systems: A Review

literature review on scientific paper

Concept-Based Approach for Research Paper Recommendation

literature review on scientific paper

Scholarly recommendation systems: a literature survey

Avoid common mistakes on your manuscript.

1 Introduction

The rapidly increasing number of publications leads to a large quantity of possibly relevant papers [ 6 ] for more specific tasks such as finding related papers [ 28 ], finding ones to read [ 109 ] or literature search in general to inspire new directions and understand the state-of-the-art approaches [ 46 ]. Overall researchers typically spend a large amount of time on searching for relevant related work [ 7 ]. Keyword-based search options are insufficient to find relevant papers [ 9 , 52 , 109 ], they require some form of initial knowledge about a field. Oftentimes, users’ information needs are not explicitly specified [ 56 ] which impedes this task further.

To close this gap, a plethora of paper recommendation systems have been proposed recently [ 37 , 39 , 88 , 104 , 117 ]. These systems should fulfil different functions: for  junior researchers  systems  should  recommend a broad variety of papers, for senior ones the recommendations should align more with their already established interests [ 9 ] or help them discover relevant interdisciplinary research [ 100 ]. In general paper recommendation approaches positively affect researchers’ professional lives as they enable finding relevant literature more easily and faster [ 50 ].

As there are many different approaches, their objectives and assumptions are also diverse. A simple problem definition of a paper recommendation system could be the following: given one paper recommend a list of papers fitting the source paper [ 68 ]. This definition would not fit all approaches as some specifically do not require any initial paper to be specified but instead observe a user as input [ 37 ]. Some systems recommend sets of publications fitting the queried terms only if these papers are all observed together [ 60 , 61 ], most of the approaches suggest a number of single publications as their result [ 37 , 39 , 88 , 117 ], such that any single one of these papers satisfies the information need of a user fully. Most approaches assume that all required data to run a system is present already [ 37 , 117 ] but some works [ 39 , 88 ] explicitly crawl general publication information or even abstracts and keywords from the web.

In this literature review we observe papers recently published in the area of scientific paper recommendation between and including January 2019 and October 2021 Footnote 1 . We strive to give comprehensive overviews on their utilised methods as well as their datasets, evaluation measures and open challenges of current approaches. Our contribution is fourfold:

We propose a current multidimensional characterisation of current paper recommendation approaches.

We compile a list of recently used datasets in evaluations of paper recommendation approaches.

We compile a list of recently used evaluation measures for paper recommendation.

We analyse existing open challenges and identify current novel problems in paper recommendation which could be specifically helpful for future approaches to address.

In the following Sect.  2 we describe the general problem statement for paper recommendation systems before we dive into the literature review in Sect.  3 . Section  4 gives insight into datasets used in current work. In the following Sect.  5 different definitions of relevance, relevance assessment as well as evaluation measures are analysed. Open challenges and objectives are discussed in detail in Sect.  7 . Lastly Sect.  8 concludes this literature review.

2 Problem statement

Over  the  years  different  formulations  for  a  problem statement of a paper recommendation system have emerged. In general they should specify the input for the recommendation system, the type of recommendation results, the point in time when the recommendation will be made and which specific goal an approach tries to achieve. Additionally, the target audience should be specified.

As input we can either specify an initial paper [ 28 ], keywords [ 117 ], a user [ 37 ], a user and a paper [ 5 ] or more  complex  information  such  as  user-constructed knowledge graphs [ 109 ]. Users can be modelled as a combination  of  features  of  papers  they  interacted with [ 19 , 21 ], e.g. their clicked [ 26 ] or authored publications [ 22 ]. Papers can for example be represented by their textual content [ 88 ].

As types of recommendation we could either specify single (independent) papers [ 37 ] or a set of papers which is to be observed completely to satisfy the information need [ 61 ]. A study by Beierle et al. [ 18 ] found that existing digital libraries recommend between three and ten single papers, in their case the optimal number of suggestions to display to users was five to six.

As for the point in time , most work focuses on immediate recommendation of papers. Only a few approaches also consider delayed suggestion Footnote 2 via newsletter for example [ 56 ].

In general, recommended papers should be relevant in one way or another to achieve certain goals . The intended goal of authors of papers could, e.g. either be to recommend papers which should be read [ 109 ] by a user or recommend papers which are simply somehow related to an initial paper [ 28 ], by topic, citations or user interactions.

Different target audiences , for example junior or senior researcher, have different demands from paper recommendation systems [ 9 ]. Usually paper recommendation approaches target single users but there are also works which strive to recommend papers for sets of users [ 110 , 111 ].

3 Literature review

In this chapter we first clearly define the scope of our literature review (see Sect.  3.1 ) before we conduct a meta-analysis on the observed papers (see Sect.  3.2 ). Afterwards our categorisation or lack thereof is discussed in depth (see Sect.  3.3 ), before we give short overviews of all paper recommendation systems we found (see Sect.  3.5 ) and some other relevant related work (see Sect.  3.6 ).

To the best of our knowledge the literature reviews by Bai et al. [ 9 ], Li and Zou [ 58 ] and Shahid et al. [ 92 ] are the most recent ones targeting the domain of scientific paper recommendation systems. They were accepted for publication or published in 2019 so they only consider paper recommendation systems up until 2019 at most. We want to bridge the gap between papers published after their surveys were finalised and current work so we only focus on the discussion of publications which appeared between January 2019 and October 2021 when this literature search was conducted.

figure 1

PRISMA workflow of our literature review process

We conducted our literature search on the following digital libraries: ACM Footnote 3 , dblp Footnote 4 , GoogleScholar Footnote 5 and Springer Footnote 6 . Titles of considered publications had to contain either paper , article or publication as well as some form of recommend . Papers had to be written in English to be observed. We judged relevance of retrieved publications by observing titles and abstracts if the title alone did not suffice to assess their topical relevance. In addition to these papers found by systematically searching digital libraries, we also considered their referenced publications if they were from the specified time period and of topical fit. For all papers their date of first publication determines their publication year which decides if they lie in our time observed time frame or not. For example, for journal articles we consider the point in time when they were first published online instead of the date on which they were published in an issue, for conference articles we consider the date of the conference instead a later date when they were published online. Figure  1 depicts the PRISMA [ 79 ] workflow for this study.

We refrain from including works in our study which do not identify as scientific paper recommendation systems such as Wikipedia article recommendation [ 70 , 78 , 85 ] or general news article recommendation [ 33 , 43 , 103 ]. Citation recommendation systems [ 72 , 90 , 124 ] are also out of scope of this literature review. Even though citation and paper recommendation can be regarded as analogous [ 45 ], we argue the differing functions of citations [ 34 ] and tasks of these recommendation systems [ 67 ] should not be mixed with the problem of paper recommendation. Färber and Jatowt [ 32 ] also support this view by stating that both are disjunctive, with paper recommendation pursuing the goal of providing papers to read and investigate while incorporating user interaction data and citation recommendation supporting users with finding citations for given text passages. Footnote 7 We also consciously refrain from discussing the plethora of more area-independent recommender systems which could be adopted to the domain of scientific paper recommendation.

Our literature research resulted in 82 relevant papers. Of these, three were review articles. We found 14 manuscripts which do not present paper recommendation systems but are relevant works for the area nonetheless, they are discussed in Sect.  3.6 . This left 65 publications describing paper recommendation systems for us to analyse in the following.

3.2 Meta analysis

For papers within our scope, we consider their publication year as stated in the citation information for this meta-analysis. This could affect the publication year of papers compared to the former definition of which papers are included in this survey. For example, for journal articles we do not set the publication year as the point in time when they were first published online, instead for consistency (this data is present in the citation information of papers) for this analysis we use the year the issue was published in which the article is contained. Of the 65 relevant system papers, 21 were published in 2019, 23 were published in 2020 and 21 were published in 2021. On average each paper has 4.0462 authors (std. dev. = 1.6955) and 12.4154 pages (std. dev. = 9.2402). 35 (53.85%) of the papers appeared as conference papers, 27 (41.54%) papers were published in journals and there were two preprints (3.08%) which have not yet been published otherwise. There has been one master’s thesis (1.54%) within scope. The most common venues for publications were the ones depicted in Table  1 . Some papers [ 74 , 75 , 76 , 93 , 94 ] described the same approach without modification or extension of the actual paper recommendation methodology, e.g. by providing evaluations Footnote 8 . This left us with 62 different paper recommendation systems to discuss.

3.3 Categorisation

3.3.1 former categorisation.

The  already  mentioned  three  most  recent [ 9 , 58 , 92 ] and one older but highly influential [ 16 ] literature reviews in scientific paper recommendation utilise different categorisations to group approaches. Beel et al. [ 16 ] categorise observed papers by their underlying recommendation  principle  into  stereotyping,  content-based filtering, collaborative filtering, co-occurrence, graph-based, global relevance and hybrid models. Bai et al. [ 9 ] only utilise the classes content-based filtering, collaborative filtering, graph-based methods, hybrid methods and other models. Li and Zou [ 58 ] use the categories content-based recommendation, hybrid recommendation, graph-based recommendation and recommendation based on deep learning. Shahid et al. [ 92 ] label approaches by the criterion they identify relevant papers with: content, metadata, collaborative filtering and citations.

The four predominant categories thus are content-based filtering, collaborative filtering, graph-based and hybrid systems. Most of these categories are defined precisely but graph-based approaches are not always characterised concisely: Content-based filtering (CBF) methods are said to be ones where user interest is inferred by observing their historic interactions with papers [ 9 , 16 , 58 ]. Recommendations are composed by observing features of papers and users [ 5 ]. In collaborative filtering (CF) systems the preferences of users similar to a current one are observed to identify likely relevant publications [ 9 , 16 , 58 ]. Current users’ past interactions need to be similar to similar users’ past interactions [ 9 , 16 ]. Hybrid approaches are ones which combine multiple types of recommendations [ 9 , 16 , 58 ].

Graph-based methods can be characterised in multiple ways. A very narrow definition only encompasses ones which observe the recommendation task as a link prediction problem or utilise random walk [ 5 ]. Another less strict definition identifies these systems as ones which construct networks of papers and authors and then  apply  some  graph  algorithm  to  estimate relevance [ 9 ]. Another definition specifies this class as one using graph metrics such as random walk with restart, bibliographic coupling or co-citation inverse document frequency [ 106 ]. Li and Zhou [ 58 ] abstain from clearly characterising this type of systems directly but give examples which hint that in their understanding of graph-based methods somewhere in the recommendation process, some type of graph information, e.g. bibliographic coupling or co-citation strength, should be used. Beel et al. [ 16 ] as well as Bai et al. [ 9 ] follow a similar line, they characterise graph-based methods broadly as ones which build upon the existing connections in a scientific context to construct a graph network.

When trying to classify approaches by their recommendation type, we encountered some problems:

We have to refrain from only utilising the labels the works give themselves (see Table  2 for an overview of self-labels of works which do classify themselves). Works do not necessarily (clearly) state, which category they belong to [ 28 , 49 , 60 ]. Another problem with self-labelling is authors’ individual definitions of categories while disregarding all possible ones (as e.g. seen with Afsar et al. [ 1 ] or Ali et al. [ 5 ]). Mis-definition or omitting of categories could lead to an incorrect classification.

When considering the broadest definition of graph-based methods many recent paper recommendation systems tend to belong to the class of hybrid methods. Most of the approaches [ 5 , 46 , 48 , 49 , 57 , 88 , 105 , 117 ] utilise some type of graph structure information as part of the approach which would classify them as graph-based but as they also utilise historic user-interaction data or descriptions of paper features (see, e.g. Li et al. [ 57 ] who describe their approach as network-based while using a graph structure, textual components and user profiles) which would render them as either CF or CBF also.

Thus we argue the former categories do not suffice to classify the particularities of current approaches in a meaningful way. So instead, we introduce more dimensions by which systems could be grouped.

3.3.2 Current categorisation

Recent paper recommendation systems can be categorised in 20 different dimensions by general information on the approach (G), already existing data directly taken from the papers used (D) and methods which might create or (re-)structure data, which are part of the approach (M):

(G) Personalisation (person.): The approach produces personalised recommendations. The recommended items depend on the person using the approach, if personalisation is not considered, the recommendation solely depends on the input keywords or paper. This dimension is related to the existence of user profiles.

(G) Input: The approach requires some form of input, either a paper (p), keywords (k), user (u) or something else, e.g. an advanced type of input (o). Hybrid forms are also possible. In some cases the input is not clearly specified throughout the paper so it is unknown (?).

(D) Title: The approach utilises titles of papers.

(D) Abstract (abs.): The approach utilises abstracts of papers.

(D) Keyword (key.): The approach utilises keywords of papers. These keywords are usually explicitly defined by the authors of papers, contrasting key phrases.

(D) Text: The approach utilises some type of text of papers which is not clearly specified as titles, abstracts or keywords. In the evaluation this approach might utilise specified text fragments of publications.

(D) Citation (cit.): The approach utilises citation information, e.g. numbers of citations or co-references.

(D) Historic interaction (inter.): The approach uses some sort of historic user-interaction data, e.g. previously authored, cited or liked publications. An approach can only include historic user-interaction data if it also somehow contains user profiles.

(M) User profile (user): The approach constructs some sort of user profile or utilises profile information. Most approaches using personalisation also construct user profiles but some do not explicitly construct profiles but rather encode user information in the used structures.

(M) Popularity (popul.): The approach utilises some sort of popularity indication, e.g. CORE rank, numbers of citations Footnote 9 or number of likes.

(M) Key phrase (KP): The approach utilises key phrases. Key phrases are not explicitly provided by authors of papers but are usually computed from the titles and abstracts of papers to provide a descriptive summary, contrasting keywords of papers.

(M) Embedding (emb.): The approach utilises some sort  of  text  or  graph  embedding  technique,  e.g. BERT or Doc2Vec.

(M) Topic model (TM): The approach utilises some sort of topic model, e.g. LDA.

(M) Knowledge graph (KG): The approach utilises or builds some sort of knowledge graph. This dimension surpasses the mere incorporation of a graph which describes a network of nodes and edges of different types. A knowledge graph is a sub-category of a graph.

(M) Graph: The approach actively builds or directly uses a graph structure, e.g. a knowledge graph or scientific heterogeneous network. Utilisation of a neural network is not considered in this dimension.

(M) Meta-path (path): The approach utilises meta-paths. They usually are composed from paths in a network.

(M) Random Walk (with Restart) (RW): The approach utilises Random Walk or Random Walk with Restart.

(M) Advanced machine learning (AML): The approach utilises some sort of advanced machine learning component in its core such as a neural network. Utilisation of established embedding methods which themselves use neural networks (e.g. BERT) are not considered in this dimension. We do not consider traditional and simple ML techniques such as k means in this dimension but rather mention methods explicitly defining a loss function, using multi-layer perceptrons or GCNs.

(M) Crawling (crawl.): The approach conducts some sort of web crawling step.

(M) Cosine similarity (cosine): The approach utilises cosine similarity at some point.

Of the observed paper recommendation systems, six were general systems or methods which were only applied on the domain of paper recommendation [ 3 , 4 , 24 , 60 , 118 , 121 ]. Two were targeting explicit set-based recommendation of publications where only all papers in the set together satisfy users’ information needs [ 60 , 61 ], two recommend multiple papers [ 42 , 71 ] (e.g. on a path [ 42 ]), all the other approaches focused on recommendation of k single papers. Only two approaches focus on recommendation of papers to user groups instead of single users [ 110 , 111 ]. Only one paper [ 56 ] supports subscription-based recommendation of papers, all other approaches solely regarded a scenario in which papers were suggested straight away.

Table  3 classifies the observed approaches according to the afore discussed dimensions.

3.4 Comparison of paper recommendation systems in different categories

In this Section, we describe the scientific directions associated with the categories we presented in the previous section as the 65 relevant publications. We focus only on the methodological categories and describe how they are incorporated in the respective approaches.

3.4.1 User profile

32  approaches  construct  explicit  user  profiles.  They utilise different components to describe users. We differentiate  between  profiles  derived  from  user  interactions and ones derived from papers.

Most user profiles are constructed from users’ actual interactions : unspecified historical interaction [ 30 , 37 , 56 , 57 , 64 , 118 ], the mean of the representation of interacted with papers [ 19 ], time decayed interaction behaviour [ 62 ], liked papers [ 69 , 123 ], bookmarked papers [ 84 , 119 ], read papers [ 111 , 113 ], rated papers [ 3 , 4 , 110 ], clicked on papers [ 24 , 26 , 49 ], categories of clicked papers [ 1 ], features of clicked papers [ 104 ], tweets [ 74 , 75 , 76 ], social interactions [ 65 ] and explicitly defined topics of interest tags [ 119 ].

Some approaches derived user profiles from users’ written papers : authored papers [ 5 , 21 , 22 , 55 , 63 , 74 , 75 , 76 , 116 ], a partitioning of authored papers [ 27 ], research fields of authored papers [ 41 ] and referenced papers [ 116 ].

3.4.2 Popularity

We found 13 papers using some type of popularity measure. Those can be defined on authors, venues or papers.

For author-based popularity measures we found unspecified ones [ 65 ] such as authority [ 116 ] as well as ones regarding the citations an author received: citation count of papers [ 22 , 96 , 108 , 119 ], change in citation count [ 25 , 26 ], annual citation count [ 26 ], number of citations related to papers [ 59 ], h-index [ 26 ]. We found two definitions of author’s popularity using the graph structure of scholarly networks, namely the number of co-authors [ 41 ] and a person’s centrality [ 108 ].

For venue-based popularity measures, we found an unspecific reputation notion [ 116 ] as well as incorporation of the impact factor [ 26 , 117 ].

For paper-based popularity measures we encountered some citation-based definitions such as vitality [ 117 ], citation count of papers [ 22 ] and theirs centrality [ 96 ] in the citation network. Additionally, some approaches incorporated less formal interactions: number of downloads [ 56 ], social media mentions [ 119 ] and normalised number of bookmarks [ 84 ].

3.4.3 Key phrase

Only four papers use key phrases in some shape or form: Ahmad and Afzal [ 2 ] construct key terms from preprocessed titles and abstracts using tf-idf to represent papers. Collins and Beel [ 28 ] use the Distiller Framework [ 12 ] to extract uni-, bi- and tri-gram key phrase candidates from tokenised, part-of-speech tagged and stemmed titles and abstracts. Key phrase candidates were weighted and the top 20 represent candidate papers. Kang et al. [ 46 ] extract key phrases from CiteSeer to describe the diversity of recommended papers. Renuka et al. [ 86 ] apply rapid automatic keyword extraction.

In summary, different length key phrases usually get constructed from titles and abstracts with automatic methods such as tf-idf or the Distiller Framework to represent the most important content of publications.

3.4.4 Embedding

We found a lot of approaches utilising some form of embedding based on existing document representation methods. We distinguish by embedding of papers, users and papers and sophisticated embedding from the proposed approaches.

Among the most common methods was their application on papers : in an unspecified representation [ 30 , 119 ],  Word2Vec [ 19 , 37 , 44 , 45 , 55 , 104 , 113 ],  Word2Vec of LDA top words [ 24 , 107 ], Doc2vec [ 21 , 28 , 48 , 62 , 63 , 107 ], Doc2Vec of word pairs [ 109 ], BERT [ 123 ] and SBERT [ 5 , 19 ]. Most times these approaches do not mention which part of the paper to use as input but some specifically mention the following parts: titles [ 37 ], titles and abstracts [ 28 , 45 ], titles, abstracts and bodies [ 48 ], keywords and paper [ 119 ].

Few approaches observed user profiles and papers , here Word2Vec [ 21 ] and NPLM [ 29 ] embeddings were used.

Several approaches embed the information in their own model embedding: a heterogeneous information network [ 5 ], a two-layer NN [ 37 ], a scientific social reference network [ 41 ], the TransE model [ 56 ], node embeddings [ 63 ], paper, author and venue embedding [ 116 ], user and item embedding [ 118 ], a GRU and association rule mining model [ 71 ], a GCN embedding of users [ 104 ] and an LSTM model [ 113 ].

3.4.5 Topic model

Eight approaches use some topic modelling component. Most of them use LDA to represent papers’ content [ 3 , 5 , 24 , 27 , 107 , 117 ]. Only two of them do not follow this method: Subathra and Kumar [ 98 ] use LDA on papers to find their top n words, then they use LDA again on these words’ Wikipedia articles. Xie et al. [ 115 ] use a hierarchical LDA adoption on papers, which introduces a discipline classification.

3.4.6 Knowledge graph

Only six of the observed papers incorporate knowledge graphs. Only one uses a predefined one, the Watson for Genomics knowledge graph [ 95 ]. Most of the approaches build their own knowledge graphs, only one asks users to construct the graphs: Wang et al. [ 109 ] build two knowledge graphs, one in-domain and one cross-domain graph. The graphs are user-constructed and include representative papers for the different concepts.

All other approaches do not rely on users building the knowledge graph: Afsar et al. [ 1 ] utilise an expert-built knowledge base as a source for their categorisation of papers, which are then recommended to users. Li et al. [ 56 ] employ a knowledge graph-based embedding of authors, keywords and venues. Tang et al. [ 104 ] link words with high tf-idf weights from papers to LOD and then merge this knowledge graph with the user-paper graph. Wang et al. [ 113 ] construct a knowledge graph consisting of users and papers.

3.4.7 Graph

In terms of graphs, we found 33 approaches explicitly mentioning the graph structure they were utilising. We can describe which graph structure is used and which algorithms or methods are applied on the graphs.

Of the observed approaches, most specify some form of (heterogeneous) graph structure . Only a few of them are unspecific and mention an undefined heterogeneous graph [ 63 , 64 , 65 ] or a multi-layer [ 48 ] graph. Most works clearly define the type of graph they are using: author-paper-venue-label-topic graph [ 5 ], author-paper-venue-keyword graph [ 56 , 57 ], paper-author graph [ 19 , 29 , 55 , 104 ],   paper-topic   graph [ 29 ],   author-paper-venue graph [ 42 , 121 , 122 ],  author  graph [ 41 ],  paper-paper graph [ 42 , 49 ],  citation  graph [ 2 , 44 , 45 , 46 , 88 , 89 , 106 , 108 , 117 ] or undirected citation graph [ 60 , 61 ]. Some approaches specifically mention usage of co-citations [ 2 , 45 ], bibliographic coupling or both [ 88 , 89 , 96 , 108 ].

As for algorithms or methods used on these graphs , we encountered usage of centrality measures in different graph types [ 41 , 96 , 108 ], some use knowledge graphs (see Sect.  3.4.6 ), some using meta-paths (see Sect.  3.4.8 ), some using random walks e.g. in form of PageRank or hubs and authorities (see Sect.  3.4.9 ), construction of Steiner trees [ 61 ], usage of the graph as input for a GCN [ 104 ], BFS [ 113 ], clustering [ 117 ] or calculation of a closeness degree [ 117 ].

3.4.8 Meta-path

We found only four approaches incorporating meta-paths. Hua et al. [ 42 ] construct author-paper-author and author-paper-venue-paper-author paths by applying beam search. Papers on the most similar paths are recommended to users. Li et al. [ 57 ] construct meta-paths of a max length between users and papers and use random walk on these paths. Ma et al. [ 63 , 64 ] use meta-paths to measure the proximity between nodes in a graph.

3.4.9 Random walk (with restart)

We found twelve approaches using some form of random walk in their methodology. We differentiate between ones using random walk, random walk with restart and algorithms using a random walk component.

Some methods use random walk on heterogeneous graphs [ 29 , 65 ] and weighted multi-layer graphs [ 48 ]. A few approaches use random walk to identify [ 42 , 57 ] or determine the proximity between [ 64 ] meta-paths.

Three approaches explicitly utilise random walk with restart . They determine similarity between papers [ 106 ], identify papers to recommend [ 44 ] or find most relevant papers in clusters [ 117 ].

Some  approaches  use  algorithms  which  incorporate a random walk component : PageRank [ 107 ] and the identifications of hubs and authorities [ 122 ] with PageRank [ 121 ].

3.4.10 Advanced machine learning

29 approaches utilised some form of advanced machine learning. We encountered different methods being used and some papers specifically presenting novel machine learning models. All of these papers surpass mere usage of a topic model or typical pre-trained embedding method.

We found a multitude of machine learning methods being used, from multi armed bandits [ 1 ], LSTM [ 24 , 37 , 113 ], multi-layer perceptrons [ 62 , 96 , 104 ], (bi-)GRU [ 37 , 69 , 71 , 123 ], matrix factorisation [ 4 , 62 , 69 , 110 , 111 ], gradient ascent or descent [ 41 , 57 , 63 , 116 ], some form of simple neural network [ 30 , 37 , 56 ], some form of graph neural network [ 19 , 49 , 104 ], autoencoder [ 4 ], neural collaborative filtering [ 62 ], learning methods [ 30 , 123 ] to DTW [ 48 ]. Three approaches ranked the papers to recommend [ 56 , 57 , 118 ] with, e.g. Bayesian Personalized Ranking. Two of the observed papers proposed topic modelling approaches [ 3 , 115 ].

Several papers proposed models : a bipartite network embedding [ 5 ], heterogeneous graph embeddings [ 29 , 42 , 48 , 63 ], a scientific social reference network [ 41 ], a paper-author-venue embedding [ 116 ] and a relation prediction model [ 64 ].

3.4.11 Crawling

We found nine papers incorporating a crawling step as part of their approach. PDFs are oftentimes collected from CiteSeer [ 38 , 46 ] or CiteSeerX [ 2 , 93 , 94 ], in some cases [ 39 , 88 , 110 ] the sources are not explicitly mentioned. Fewer used data sources are Wikipedia for articles explaining the top words from papers [ 98 ] or papers from ACM, IEEE and EI [ 109 ]. Some approaches explicitly mention the extraction of citation information [ 2 , 38 , 39 , 46 , 88 , 93 , 94 ] e.g. to identify co-citations.

3.4.12 Cosine similarity

Some form of cosine similarity was encountered in most (31) paper recommendation approaches. It is often applied between papers, between users, between users and papers and in other forms.

For application between papers we encountered the possibility of using unspecified embeddings: unspecified word or vector representations of papers [ 30 , 48 , 107 , 110 ], papers’ key terms or top words [ 2 , 98 ] and key phrases [ 46 ]. We found some approaches using vector space model variants: unspecified [ 59 ], tf vectors [ 39 , 88 ], tf-idf vectors [ 42 , 95 , 111 ], dimensionality reduced tf-idf vectors [ 86 ] and lastly, tf-idf and entity embeddings [ 56 ]. Some approaches incorporated more advanced embedding techniques: SBERT embeddings [ 5 ], Doc2Vec embeddings [ 28 ], Doc2Vec embeddings with incorporation of their emotional score [ 109 ] and NPLM representations [ 29 ].

Cosine similarity was used between preferences or profiles of users and papers in the following ways: unspecified representations [ 63 , 84 , 113 , 115 ], Boolean representation of users and keywords [ 60 ], tf-idf vectors [ 21 , 74 , 75 , 76 ],  cf-idf  vectors [ 74 , 75 , 76 ]  and  hcf-idf vectors [ 74 , 75 , 76 ].

For between users application of cosine similarity, we found unspecified representations [ 41 ] and time-decayed Word2Vec embeddings of users’ papers’ keyword [ 55 ].

Other applications include the usage between input keywords and paper clusters [ 117 ] and between nodes in a graph represented by their neighbouring nodes [ 121 , 122 ].

3.5 Paper recommendation systems

The 65 relevant works identified in our literature search are described in this section. We deliberately refrain from trying to structure the section by classifying papers by an arbitrary dimension and instead point to Table  3 to identify those dimensions in which a reader is interested to navigate the following short descriptions. The works are ordered by the surname of the first author and ascending publication year. An exception to this rule are papers presenting extensions of previous approaches with different first authors. These papers are ordered to their preceding approaches.

Afsar et al. [ 1 ] propose KERS, a multi-armed bandit approach for patients to help with medical treatment decision making. It consists of two phases: first an exploration phase identifies categories users are implicitly interested in. This is supported by an expert-built knowledge base. Afterwards an exploitation phase takes place where articles from these categories are recommended until a user’s focus changes and another exploitation phase is initiated. The authors strive to minimise the exploration efforts while maximising users’ satisfaction.

Ahmedi et al. [ 3 ] propose a personalised approach which can also be applied to more general recommendation scenarios which include user profiles. They utilise Collaborative  Topic  Regression  to  mine  association rules from historic user interaction data.

Alfarhood and Cheng [ 4 ] introduce Collaborative Attentive Autoencoder, a deep learning-based model for general recommendation targeting the data sparsity problem. They apply probabilistic matrix factorisation while also utilising textual information to train a model which identifies latent factors in users and papers.

Ali et al. [ 5 ]  construct  PR-HNE,  a  personalised probabilistic paper recommendation model based on a joint representation of authors and publications. They utilise graph information such as citations as well as co-authorships, venue information and topical relevance to suggest papers. They apply SBERT and LDA to represent author embeddings and topic embeddings respectively.

Bereczki [ 19 ] models users and papers in a bipartite graph. Papers are represented by their contents’ Word2Vec or BERT embeddings, users’ vectors consist of representations of papers they interacted with. These vectors are then aggregated with simple graph convolution.

Bulut et al. [ 22 ] focus on current user interest in their approach which utilises k-Means and KNN. Users’ profiles are constructed from their authored papers. Recommended papers are the highest cited ones from the cluster most similar to a user. In a subsequent work they extended their research group to again work in the same domain. Bulut et al. [ 21 ] again focus on users’ features. They represent users as the sum of features of their papers. These representations are then compared with all papers’ vector representations to find the most similar ones. Papers can be represented by TF-IDF, Word2Vec or Doc2Vec vectors.

Chaudhuri et al. [ 25 ] use indirect features derived from direct features of papers in addition to direct ones in their paper recommendation approach: keyword diversification, text complexity and citation analysis. In an extended group Chaudhuri et al. [ 26 ] later propose usage of more indirect features such as quality in paper recommendation. Users’ profiles are composed of their clicked papers. Subsequently they again worked on an approach in the same area but in a slightly smaller group. Chaudhuri et al. [ 24 ] propose the general Hybrid Topic Model and apply it on paper recommendation. It learns users’ preferences and intentions by combining LDA and Word2Vec. They compute user’s interest from probability distributions of words of clicked papers and dominant topics in publications.

Chen and Ban [ 27 ] introduce CPM, a recommendation model based on topically clustered user interests mined from their published papers. They derive user need models from these clusters by using LDA and pattern equivalence class mining. Candidate papers are then ranked against the user need models to identify the best-fitting suggestions.

Collins and Beel [ 28 ] propose the usage of their paper recommendation system Mr. DLib as a recommender as-a-service. They compare representing papers via Doc2Vec with a key phrase-based recommender and TF-IDF vectors.

Du et al. [ 29 ] introduce HNPR, a heterogeneous network method using two different graphs. The approach incorporates citation information, co-author relations and research areas of publications. They apply random walk on the networks to generate vector representations of papers.

Du et al. [ 30 ] propose Polar++, a personalised active  one-shot  learning-based  paper  recommendation system where new users are presented articles to vote on before they obtain recommendations. The model trains a neural network by incorporating a matching score between a query article and the recommended articles as well as a personalisation score dependant on the user.

Guo et al. [ 37 ] recommend publications based on papers initially liked by a user. They learn semantics between titles and abstracts of papers on word- and sentence-level, e.g. with Word2Vec and LSTMs to represent user preferences.

Habib and Afzal [ 38 ] crawl full texts of papers from CiteSeer. They then apply bibliographic coupling between input papers and a clusters of candidate papers to identify the most relevant recommendations. In a subsequent work Afzal again used a similar technique. Ahmad and Afzal [ 2 ] crawled papers from CiteSeerX. Cosine similarity of TF-IDF representations of key terms from titles and abstracts is combined with co-citation strength of paper pairs. This combined score then ranks the most relevant papers the highest.

Haruna et al. [ 39 ] incorporate paper-citation relations combined with contents of titles and abstracts of papers to recommend the most fitting publications for an input query corresponding to a paper.

Hu et al. [ 41 ] present ADRCR, a paper recommendation  approach  incorporating  author-author  and author-paper citation relationships as well as authors’ and papers’ authoritativeness. A network is built which uses citation information as weights. Matrix decomposition helps learning the model.

Hua et al. [ 42 ] propose PAPR which recommends relevant paper sets as an ordered path. They strive to overcome recommendation merely based on similarity by observing topics in papers changing over time. They combine similarities of TF-IDF paper representations with random-walk on different scientific networks.

Jing and Yu [ 44 ] build a three-layer graph model which they traverse with random-walk with restart in an algorithm named PAFRWR. The graph model consists of one layer with citations between papers’ textual content represented via Word2Vec vectors, another layer modelling co-authorships between authors and the third layer encodes relationships between papers and topics contained in them.

Kanakia et al. [ 45 ] build their approach upon the MAG dataset and strive to overcome the common problems of scalability and cold-start. They combine TF-IDF and Word2Vec representations of the content with co-citations of papers to compute recommendations. Speedup is achieved by comparing papers to clusters of papers instead of all other single papers.

Kang et al. [ 46 ] crawl full texts of papers from CiteSeer and construct citation graphs to determine candidate papers. Then they compute a combination of section-based citation and key phrase similarity to rank recommendations.

Kong et al. [ 48 ] present VOPRec, a model combining textual components in form of Doc2vec and Paper2Vec paper representations with citation network information in form of Struc2vec. Those networks of papers connect the most similar publications based on text and structure. Random walk on these graphs contributes to the goal of learning vector representations.

L et al. [ 49 ] base their recommendation on lately accessed papers of users as they assume future accessed papers are similar to recently seen ones. They utilise a sliding window to generate sequences of papers, on those they construct a GNN to aggregate neighbouring papers to identify users’ interests.

Li et al. [ 56 ]  introduce  a  subscription-based  approach which learns a mapping between users’ browsing history and their clicks in the recommendation mails. They learn a re-ranking of paper recommendations by using its metadata, recency, word representations and entity representations by knowledge graphs as input for a neural network. Their defined target audience are new users.

Li et al. [ 55 ] present HNTA a paper recommendation method utilising heterogeneous networks and changing user interests. Paper similarities are calculated with Word2Vec representations of words recommended for each paper. Changing user interest is modelled with help of an exponential time decay function on word vectors.

Li et al. [ 57 ] utilise user profiles with a history of preferences to construct heterogeneous networks where they apply random walks on meta-paths to learn personalised weights. They strive to discover user preference patterns and model preferences of users as their recently cited papers.

Lin et al. [ 59 ] utilise authors’ citations and years they have been publishing papers in their recommendation approach. All candidate publications are matched against user-entered keywords, the two factors of authors of these candidate publications are combined to identify the overall top recommendations.

Liu et al. [ 60 ] explicitly do not require all recommended publications to fit the query of a user perfectly. Instead they state the set of recommended papers fulfils the information need only in the complete form. Here they treat paper recommendation as a link prediction problem incorporating publishing time, keywords and author influence. In a subsequent work, part of the previous research group again observes the same problem. In this work Liu et al. [ 61 ] propose an approach utilising numbers of citations (author popularity) and relationships between publications in an undirected citation graph. They compute Steiner trees to identify the sets of papers to recommend.

Lu et al. [ 62 ] propose TGMF-FMLP, a paper recommendation approach focusing on the changing preferences of users and novelty of papers. They combine category attributes (such as paper type, publisher or journal), a time-decay function, Doc2Vec representations of the papers’ content and a specialised matrix factorisation to compute recommendations.

Ma et al. [ 64 ] introduce HIPRec, a paper recommendation approach on heterogeneous networks of authors, papers, venues and topics specialised on new publications. They use the most interesting meta-paths to construct significant meta-paths. With these paths and features from these paths they train a model to identify new papers fitting users. Together with another researcher Ma further pursued this research direction. Ma and Wang [ 63 ] propose HGRec, a heterogeneous graph representation learning-based model working on the same network. They use meta-path-based features and Doc2Vec paper embeddings to learn the node embeddings in the network.

Manju et al. [ 65 ] attempt to solve the cold-start problem with their paper recommendation approach coding social interactions as well as topical relevance into a heterogeneous graph. They incorporate believe propagation into the network and compute recommendations by applying random walk.

Mohamed Hassan et al. [ 69 ] adopt an existing tag prediction model which relies on a hierarchical attention network to capture semantics of papers. Matrix factorisation then identifies the publications to recommend.

Nair et al. [ 71 ] propose C-SAR, a paper recommendation approach using a neural network. They input GloVe embeddings of paper titles into their Gated Recurrent Union model to compute probabilities of similarities of papers. The resulting adjacency matrix is input to an association rule mining a priori algorithm which generates the set of recommendations.

Nishioka et al. [ 74 , 75 ] state serendipity of recommendations as their main objective. They incorporate users’ tweets to construct profiles in hopes to model recent interests and developments which did not yet manifest in users’ papers. They strive to diversity the list of recommended papers. In more recent work Nishioka et al. [ 76 ] explained their evaluation more in depth.

Rahdari and Brusilovsky [ 84 ] observe paper recommendation  for  participants  of  scientific  conferences. Users’ profiles are composed of their past publications. Users control the impact of features such as publication similarity, popularity of papers and its authors to influence the ordering of their suggestions.

Renuka et al. [ 86 ] propose a paper recommendation approach utilising TF-IDF representations of automatically extracted keywords and key phrases. They then either use cosine similarity between vectors or a clustering method to identify the most similar papers for an input paper.

Sakib et al. [ 89 ] present a paper recommendation approach utilising second-level citation information and citation context. They strive to not rely on user profiles in the paper recommendation process. Instead they measure similarity of candidate papers to an input paper based on co-occurred or co-occurring papers. In a follow-up work with a bigger research group Sakib et al. [ 88 ] combine contents of titles, keywords and abstracts with their previously mentioned collaborative filtering approach. They again utilise second-level citation relationships between papers to find correlated publications.

Shahid et al. [ 94 ] utilise in-text citation frequencies and assume a reference is more important to a referencing paper the more often it occurs in the text. They crawl papers from CiteSeerX to retrieve the top 500 citing papers. In a follow-up work with a partially different research group Shahid et al. [ 93 ] evaluate the previously presented approach with a user study.

Sharma et al. [ 95 ] propose IBM PARSe, a paper recommendation system for the medical domain to reduce the number of papers to review for keeping an existing knowledge graph up-to-date. Classifiers identify new papers from target domains, named entity recognition finds relevant medical concepts before papers’ TF-IDF vectors are compared to ones in the knowledge graph. New publications most similar to already relevant ones with matching entities are recommended to be included in the knowledge base.

Subathra and Kumar [ 98 ] constructed an paper recommendation system which applies LDA on Wikipedia articles twice. Top related words are computed using pointwise mutual information before papers are recommended for these top words.

Tang et al. [ 104 ] introduce CGPrec, a content-based and knowledge graph-based paper recommendation system. They focus on users’ sparse interaction history with papers and strive to predict papers on which users are likely to click. They utilise Word2Vec and a Double Convolutional Neural Network to emulate users’ preferences directly from paper content as well as indirectly by using knowledge graphs.

Tanner et al. [ 106 ] consider relevance and strength of citation relations to weigh the citation network. They fetch citation information from the parsed full texts of papers. On the weighted citation networks they run either weighted co-citation inverse document frequency, weighted bibliographic coupling or random walk with restart to identify the highest scoring papers.

Tao et al. [ 107 ] use embeddings and topic modelling to compute paper recommendations. They combine LDA and Word2Vec to obtain topic embeddings. Then they calculate most similar topics for all papers using Doc2Vec vector representations and afterwards identify the most similar papers. With PageRank on the citation network they re-rank these candidate papers.

Waheed et al. [ 108 ] propose CNRN, a recommendation approach using a multilevel citation and authorship network to identify recommendation candidates. From these candidate papers ones to recommend are chosen by combining centrality measures and authors’ popularity. Highly correlated but unrelated Shi et al. [ 96 ] present AMHG, an approach utilising a multilayer perceptron. They also construct a multilevel citation network as described before with added author relations. Here they additionally utilise vector representations of publications and recency.

Wang et al. [ 113 ] introduce a knowledge-aware path recurrent network model. An LSTM mines path information from the knowledge graphs incorporating papers and users. Users are represented by their downloaded, collected and browsed papers, papers are represented by TF-IDF representations of their keywords.

Wang et al. [ 109 ] require users to construct knowledge graphs to specify the domain(s) and enter keywords for which recommended papers are suggested. From the keywords they compute initially selected papers. They apply Doc2Vec and emotion-weighted similarity between papers to identify recommendations.

Wang et al. [ 110 ] regard paper recommendation targeting a group of people instead of single users and introduce GPRAH_ER. They employ a two-step process which first individually predicts papers for users in the group before recommended papers are aggregated. Here users in the group are not considered equal, different importance and reliability weights are assigned such that important persons’ preferences are more decisive of the recommended papers. Together with a different research group two authors again pursued this definition of the paper recommendation problem. Wang et al. [ 111 ] recommend papers for groups of users in an approach called GPMF_ER. As with the previous approach they compute TF-IDF vectors of keywords of papers to calculate most similar publications for each user. Probabilistic matrix factorisation is used to integrate these similarities in a model such that predictive ratings of all users and papers can be obtained. In the aggregation phase the number of papers read by a user is determined to replace the importance component.

Xie et al. [ 116 ] propose JTIE, an approach incorporating contents, authors and venues of papers to learn paper embeddings. Further, directed citation relations are included into the model. Based on users’ authored and referenced papers personalised recommendations are computed. They consider explainability of recommendations.  In  a  subsequent  work  part  of  the  researchers again work on this topic. Xie et al. [ 115 ] specify on recommendation of papers from different areas for user-provided keywords or papers. They use hierarchical LDA to model evolving concepts of papers and citations as evidence of correlation in their approach.

Yang et al. [ 117 ] incorporate the age of papers and impact factors of venues as weights in their citation network-based approach named PubTeller. Papers are clustered by topic, the most popular ones from the clusters most similar to the query terms are recommendation candidates. In this approach, LDA and TF-IDF are used to represent publications.

Yu et al. [ 118 ] propose ICMN, a general collaborative memory network approach. User and item embeddings are composed by incorporating papers’ neighbourhoods and users’ implicit preferences.

Zavrel et al. [ 119 ] present the scientific literature recommendation  platform  Zeta  Alpha,  which  bases their recommended papers on examples tagged in user-defined categories. The approach includes these user-defined tags as well as paper content embeddings, social media mentions and citation information in their ensemble learning approach to recommend publications.

Zhang et al. [ 121 ] propose W-Rank, a general approach weighting edges in a heterogeneous author, paper and venue graph by incorporating citation relevance and author contribution. They apply their method on paper recommendation. Network- (via citations) and semantic-based (via AWD) similarity between papers is combined for weighting edges between papers, harmonic counting defines weights of edges between authors and papers. A HITS-inspired algorithm computes the final authority scores. In a subsequent work in a slightly smaller group they focus on a specialised approach  for  paper  recommendation.  Here  Zhang  et al. [ 122 ] strive to emulate a human expert recommending papers. They construct a heterogeneous network with authors, papers, venues and citations. Citation weights are determined by semantic- and network-level similarity  of  papers.  Lastly,  recommendation  candidates are re-ranked while combining the weighted heterogeneous network and recency of papers.

Zhao et al. [ 123 ] present a personalised approach focusing on diversity of results which consists of three parts. First LFM extracts latent factor vectors of papers and users from the users’ interactions history with papers. Then BERT vectors are constructed for each word of the papers, with those vectors as input and the latent factor vectors as label a BiGRU model is trained. Lastly, diversity and a user’s rating weights determine the ranking of recommended publications for the specific user.

3.6 Other relevant work

We now briefly discuss some papers which did not present novel paper recommendation approaches but are relevant in the scope of this literature review nonetheless.

3.6.1 Surrounding paper recommendation

Here we present two works which could be classified as ones to use on top of or in combination with existing paper recommendation systems: Lee et al. [ 51 ] introduce LIMEADE, a general approach for opaque recommendation systems which can for example be applied on any paper recommendation system. They produce explanations for recommendations as a list of weighted interpretable features such as influential paper terms.

Beierle  et  al. [ 18 ]  use  the  recommendation-as-a-service provider Mr. DLib to analyse choice overload in user evaluations. They report several click-based measures and discuss effects of different study parameters on engagement of users.

3.6.2 (R)Evaluations

The following four works can be grouped as ones which provide (r)evaluations of already existing approaches. Their results could be useful for the construction of novel systems: Ostendorff [ 77 ] suggests considering the context of paper similarity in background, methodology and findings sections instead of undifferentiated textual similarity for scientific paper recommendation.

Mohamed Hassan et al. [ 68 ] compare different text embedding methods such as BERT, ELMo, USE and InferSent to express semantics of papers. They perform paper recommendation and re-ranking of recommendation candidates based on cosine similarity of titles.

Le et al. [ 50 ] evaluate the already existing paper recommendation system Mendeley Suggest, which provides recommendations with different collaborative or content-based approaches. They observe different usage behaviours and state utilisation of paper recommendation systems does positively effect users’ professional lives.

Barolli et al. [ 11 ] compare similarities of paper pairs utilising n-grams, tf-idf and a transformer based on BERT. They model cosine similarities of these pairs into a paper connection graph and argue for the combination of content-based and graph-based methods in the context of COVID-19 paper recommendation systems.

3.6.3 Living labs

Living labs help researchers conduct meaningful evaluations by providing an environment, in which recommendations produced by experimental systems are shown to real users in realistic scenarios [ 14 ]. We found three relevant works for the area of scientific paper recommendation: Beel et al. [ 14 ] proposed a living lab for scholarly recommendation built on top of Mr. DLib, their recommender-as-a-service system. They log users’ actions such as clicks, downloads and purchases for related recommended papers. Additionally, they plan to extend their living lab to also incorporate research grant or research collaborator recommendation.

Gingstad et al. [ 36 ] propose ArXivDigest, an online living lab for explainable and personalised paper recommendations from arXiv. Users can either be suggested papers while browsing their website or via email as a subscription-type service. Different approaches can be hooked into ArXivDigest, the recommendations generated by them can then be evaluated by users. A simple text-based baseline compares user-input topics with articles. Target values of evaluations are users’ clicked and saved papers.

Schaer et al. [ 91 ] held the Living Labs for Academic Search (LiLAS) where they hosted two shared tasks: dataset recommendation for scientific papers and ad-hoc multi-lingual retrieval of most relevant publications regarding specific queries. To overcome the gap between real-world and lab-based evaluations they allowed integrating participants’ systems into real-world academic search systems, namely LIVIO and GESIS Search.

3.6.4 Multilingual/cross-lingual recommendation

The previous survey by Li and Zhou [ 58 ] identifies cross-language paper recommendation as a future research direction. The following two works could be useful for this aspect: Keller and Munz [ 47 ] present their results of participating on the CLEF LiLAS challenge where they tackled recommendation of multilingual papers based on queries. They utilised a pre-computed ranking approach, Solr and pseudo-relevance feedback to extend queries and identify fitting papers.

Safaryan et al. [ 87 ] compare different already existing techniques for cross-language recommendation of publications. They compare word by word translation, linear projection from a Russian to an English vector representation, VecMap alignment and MUSE word embeddings.

3.6.5 Related recommendation systems

Some recommendation approaches are slightly out of scope of pure paper recommendation systems but could still provide inspiration or relevant results: Ng [ 73 ] proposes CBRec, a children’s book recommendation system utilising matrix factorisation. His goal is to encourage good reading habits of children. The approach combines readability levels of users and books with TF-IDF representations of books to find ones which are similar to ones which a child may have already liked.

Patra et al. [ 80 ] recommend publications relevant for datasets to increase reusability. Those papers could describe the dataset, use it or be related literature. The authors represent datasets and articles as vectors and use cosine similarity to identify the best fitting papers. Re-ranking them with usage of Word2Vec embeddings results in the final recommendation.

As the discussed paper recommendation systems utilise different inputs or components of scientific publications and pursue slightly different objectives, datasets to experiment on are also of diverse nature. We do not consider datasets of approaches which do not contain an evaluation [ 60 , 119 ] or do not evaluate the actual paper recommendation [ 2 , 25 , 38 , 84 , 86 ] such as the cosine similarity between a recommended and an initial paper [ 2 , 86 ], the clustering quality on the constructed features [ 25 ] or the Jensen Shannon Divergence between probability distributions of words between an initial and recommended papers [ 38 ]. We also do not discuss datasets where only the data sources are mentioned but no remarks are made regarding the size or composition of the dataset [ 21 , 104 ] or ones where we were not able to identify actual numbers [ 65 ]. Table  4 gives an overview of datasets used in the evaluation of the considered discussed methods. Many of the datasets are unavailable only few years after publication of the approach. Most approaches utilise their own modified version of a public dataset which makes exact replication of experiments hard. In the following the main underlying data sources and publicly available datasets are discussed. Non-publicly available datasets are briefly described in Table  5 .

4.1 dblp-based datasets

The dblp computer science bibliography (dblp) is a digital library offering metadata on authors, papers and venues from the area of computer science and adjacent fields [ 54 ]. They provide publicly available short-time stored daily and longer-time stored monthly data dumps Footnote 10 .

The dblp + Citations v1 dataset [ 105 ] builds upon a dblp version from 2010 mapped on AMiner. It contains 1,632,442 publications with 2,327,450 citations.

The dblp + Citations v11 dataset Footnote 11 builds upon dblp. It contains 4,107,340 papers, 245,204 authors, 16,209 venues and 36,624,464 citations

These datasets do not contain supervised labels provided by human annotators even though the citation information could be used as interaction data.

4.2 SPRD-based datasets

The Scholarly Paper Recommendation Dataset (abbreviation: SPRD) Footnote 12 was constructed by collecting publications written by 50 researchers of different seniority from the area of computer science which are contained in dblp from 2000 to 2006 [ 58 , 101 , 102 ]. The dataset contains 100,351 candidate papers extracted from the ACM Digital Library as well as citations and references for papers. Relevance assessments of papers relevant to their current interests of the 50 researchers are also included.

A subset of SPRD, SPRD_Senior , which contains only the data of senior researchers can also be constructed [ 99 ].

These datasets specifically contain supervised labels provided by human annotators in the form of sets of papers, which researchers found relevant for themselves.

4.3 CiteULike-based datasets

CiteULike [ 20 ] was a social bookmarking site for scientific papers. It contained papers and their metadata. Users were able to include priorities, tags or comments for papers on their reading list. There were daily data dumps available from which datasets could be constructed.

Citeulike-a  [ 112 ] Footnote 13 contains 5,551 users, 16,980 papers with titles and abstracts from 2004 to 2006 and their 204,986 interactions between users and papers. Papers are represented by their title and abstract.

Citeulike-t  [ 112 ] Footnote 14 contains 7,947 users, 25,975 papers and 134,860 user-paper interactions. Papers are represented by their pre-processed title and abstract.

These datasets contain labelled data as they build upon CiteULike, which provides bookmarked papers of users.

4.4 ACM-based datasets

The ACM Digital Library (ACM) is a semi-open digital library offering information on scientific authors, papers, citations and venues from the area of computer science Footnote 15 . They offer an API to query for information. Datasets building upon this source do not contain supervised labels provided by annotators even though the citation information could be used as interaction data.

4.5 Scopus-based datasets

Scopus is a semi-open digital library containing metadata on authors, papers and affiliations in different scientific areas Footnote 16 . They offer an API to query for data. Datasets building upon this source usually do not contain labels provided by annotators.

4.6 AMiner-based datasets

ArnetMiner (AMiner) [ 105 ] is an open academic search system modelling the academic network consisting of authors, papers and venues from all areas Footnote 17 . They provide an API to query for information. Datasets building upon this source usually do not contain labelled user interaction data.

4.7 AAN-based datasets

The ACL Anthology Network (AAN) [ 81 , 82 , 83 ] is a networked database containing papers, authors and citations from the area of computational linguistics Footnote 18 . It consists of three networks representing paper-citation relations,  author-collaboration  relations  and  the  author-citation  relations.  The  original  dataset  contains 24,766 papers and 124,857 citations [ 71 ]. Datasets building  upon  this  source usually do  not  contain labelled user interaction data even though the paper-citation,  author-collaboration  or  author-citation relationships could be utilised to replace this data.

4.8 Sowiport-based datasets

Sowiport was an open digital library containing information on publications from the social sciences and adjacent fields [ 15 , 40 ]. The dataset linked papers by their attributes such as authors, publishers, keywords, journals, subjects and citation information. Via author names, keywords and venue titles the network could be traversed by triggering them to start a new search [ 40 ]. Sowiport co-operated with the recommendation-as-a-service system Mr. DLib [ 28 ]. Datasets building upon this  source  usually  contain  labelled  user  interaction data, the clicked papers of users.

4.9 CiteSeerX-based datasets

CiteSeerX [ 35 , 114 ] is a digital library focused on metadata and full-texts of open access literature Footnote 19 . It is the overhauled form of the former digital library CiteSeer. Datasets building upon this source usually do not inherently contain labelled user interaction data.

4.10 Patents-based datasets

The Patents dataset provides information on patents and trademarks granted by the United States Patent and Trademark Office Footnote 20 . Datasets building upon this source usually do not contain labelled user interaction data.

4.11 Hep-TH-based datasets

The original unaltered Hep-TH  [ 53 ] dataset Footnote 21 stems from the area of high energy physics theory. It contains papers in a graph which were published between 1993 and 2003. It was released as part of KDD Cup 2003. Datasets building upon this source usually do not contain labelled user interaction data.

4.12 MAG-based datasets

The Microsoft Academic Graph (MAG) [ 97 ] was an open scientific network containing metadata on academic communication activities Footnote 22 . Their heterogeneous graph consists of nodes representing fields of study, authors, affiliations, papers and venues. Datasets building upon this source usually do not contain labelled user interaction data besides citation information.

4.13 Others

The  following  datasets  have  no  common  underlying data source: The BBC Footnote 23 dataset contains 2,225 BBC news articles which stem from 5 topics. This dataset does not contain labelled user interaction data.

PRSDataset Footnote 24   contains  2,453  users,  21,940  items and 35,969 pairs of users and items. This dataset contains user-item interactions.

5 Evaluation

The performance of a paper recommendation system can be quantified by measuring how well a target value has been approximated by the recommended publications. Relevancy estimations of papers can come from different sources, such as human ratings or datasets. Different interactions derived from clicked or liked papers determine the target values which a recommendation system should approximate. The quality of the recommendation can be described by evaluation measures such as precision or MRR. For example, a dataset could provide information on clicked papers, that are then deemed relevant. The target value which should be approximated with the recommender system are those clicked papers, and the percentage of the recommendations which are contained in the clicked papers could then be reported as the system’s precision.

Due to the vast differences in approaches and datasets used to apply the methods, there is also a spectrum of used evaluation measures and objectives. In this section, first we observe different notions of relevance of recommended papers and individual assessment strategies for relevance. Afterwards we analyse commonly used evaluation measures and list ones which are only rarely encountered in evaluation of paper recommendation systems. Lastly we shed light on the different types of evaluation which authors conducted.

In this discussion we again only consider paper recommendation systems which also evaluate their actual approach. We disregard approaches which do evaluate other properties [ 2 , 25 , 38 , 84 , 86 , 122 ] or contain no evaluation [ 60 , 119 ]. Thus we observe 54 different approaches in this analysis.

5.1 Relevance and assessment

Relevance of recommended publications can be evaluated against multiple target values: clicked papers [ 24 , 56 , 104 ], references [ 44 , 115 ], references of recently authored papers [ 57 ], papers an author interacted with in the past [ 49 ], degree-of-relevancy which is determined by citation strength [ 94 ], a ranking based on future citation numbers [ 121 ] as well as papers accepted [ 26 ] or deemed relevant by authors [ 39 , 88 ].

Assessing the relevance of recommendations can also be conducted in different ways: the top n papers recommended by a system can be judged by either a referee team [ 109 ] or single persons [ 26 , 74 , 75 ]. Other options for relevance assessment are the usage of a dataset with user ratings [ 39 , 88 ] or emulation of users and their interests [ 1 , 57 ].

Table  6 holds information on utilised relevance indicators and target values which indicate relevance for the 54 discussed approaches. Relevancy describes the method that defines which of the recommended papers are relevant:

Human rating: The approach is evaluated using assessments of real users of results specific to the approach.

Dataset: The approach is evaluated using some type of assessment of a target value which is not specific to the approach but from a dataset. The assessment was either conducted for another approach and re-used or it was collected independent of an approach.

Papers: The approach is evaluated by some type of assessment of a target value which is directly generated from the papers contained in the dataset such as citations or their keywords.

The target values in Table  6 describe the entities which the approach tried to approximate:

Clicked: The approximated target value is derived from users’ clicks on papers.

Read: The approximated target value is derived from users’ read papers.

Cited: The approximated target value is derived from cited papers.

Liked: The approximated target value is derived from users’ liked papers.

Relevancy: The approximated target value is derived from users’ relevance assessment of papers.

Other user: The approximated target value is derived from other entities associated with a user input, e.g. acceptance of users, users’ interest and relevancy of the recommended papers’ topics.

Other automatic: The approximated target value is automatically derived from other entities, e.g. user profiles, papers with identical references, degree-of-relevancy, keywords extracted from papers, papers containing the query keywords in the optimal Steiner tree, neighbouring (cited and referencing) papers, included keywords, the classification tag, future citation numbers and an unknown measure derived from a dataset. We refrain from trying to introduce sub-categories for this broad field.

Only three approaches evaluate against multiple target values [ 21 , 30 , 104 ]. Six approaches (11.11%) utilise clicks of users, only one approach (1.85%) uses read papers as target value. Even though cited papers are not the main objective of paper recommendation systems but rather citation recommendation systems, this target was approximated by 13 (24.07%) of the observed systems. Ten approaches (18.52%) evaluated against liked papers, 15 (27.78%) against relevant papers and 13 (24.07%) against some other target value, either user input (three, 5.55%) or automatically derived (ten, 18.52%).

5.2 Evaluation measures

We differentiate between commonly used and rarely used evaluation measures for the task of scientific paper recommendation. They are described in the following sections. Table  6 holds indications of utilised evaluation measures for the 54 discussed approaches. Measures are the methods used to evaluate the approach’s ability to approximate the target value which can be of type precision, recall, f1 measure, nDCG, MRR, MAP or another one.

Out of the observed systems, twelve Footnote 25 approaches [ 1 , 28 , 30 , 49 , 59 , 64 , 69 , 71 , 74 , 75 , 76 , 107 , 115 , 116 ] (22.22%) only report one single measure, all others report at least two different ones.

5.2.1 Commonly used evaluation measures

Bai et al. [ 9 ] identify precision (P), recall (R), F1 , nDCG , MRR and MAP as evaluation features which have been used regularly in the area of paper recommendation systems. Table  7 gives usage percentages of each of these measures in observed related work.

Alfarhood and Cheng [ 4 ] argue against the use of precision when utilising implicit feedback. If a user gives no feedback for a paper it could either mean disinterest or that a user does not know of the existence of the specific publication.

5.2.2 Rarely used evaluation measures

We found a plethora of rarer used evaluation measures which have either been utilised only by the work they were introduced in or to evaluate few approaches. Our analysis in this aspect might be highly influenced by the narrow time frame we observe. Novel measures might require more time to be adopted by a broader audience. Thus we differentiate between novel rarely used evaluation measures and ones where authors do not explicitly claim they are novel. A list of rare but already defined evaluation measures can be found in Table  8 . In total 25 approaches (46.3%) did use an evaluation measure not considered common.

Novel rarely used Evaluation Measures. In our considered approaches we only encountered three novel evaluation measures: Recommendation quality as defined by Chaudhuri et al. [ 26 ] is the acceptance of recommendations by users rated on a Likert scale from 1 to 10.

TotNP_EU is a measure defined by Manju et al. [ 65 ] specifically introduced for measuring performance of approaches regarding the cold start problem. It indicates the number of new publications suggested to users with a prediction value above a certain threshold.

TotNP_AVG is another measure defined by Manju et al. [ 65 ] for measuring performance of approaches regarding the cold start problem. It indicates the average number of new publications suggested to users with a prediction value above a certain threshold.

5.3 Evaluation types

Evaluations can be classified into different categories. We follow the notion of Beel and Langer [ 17 ] who differentiate between user studies, online evaluations and offline evaluations. They define user studies as ones where users’ satisfaction with recommendation results is measured by collecting explicit ratings. Online evaluations are ones where users do not explicitly rate the recommendation results; relevancy is derived from e.g. clicks. In offline evaluations a ground truth is used to evaluate the approach.

From the 54 observed approaches we found four using multiple evaluation types [ 29 , 46 , 92 , 94 , 109 ]. Twelve (22.22%) were conducting user studies which describe the size and composition of the participant group. Footnote 26 Only two approaches [ 28 , 65 ] (3.7%) in the observed papers were evaluated with an online evaluation. We found 44 approaches (81.48%) providing an offline evaluation. Offline evaluations being the most common form of evaluation is unsurprising as this tendency has also been observed in an evaluation of general scientific recommender systems [ 23 ]. Offline evaluations are fast and do not require users [ 23 ]. Nevertheless the margin by which this form of evaluation is conducted could be rather surprising.

A distinction in lab-based vs. real world user studies can be conducted [ 16 , 17 ]. User studies where participants rate recommendations according to some criteria and are aware of the study are lab-based, all others are considered real-world studies. Living labs [ 14 , 36 , 91 ] for example enable real-world user studies. On average the lab-based user studies were conducted with 17.83 users. Table  9 holds information on the number of participants for all studies as well as the composition of groups in terms of seniority.

For offline evaluation, they can either be ones with an explicit ground truth given by a dataset containing user rankings, implicit ones by deriving user interactions such as liked or cited papers or expert ones with manually collected expert ratings [ 17 ]. We found 22 explicit offline evaluations (40.74%) corresponding to ones using datasets to estimate relevance (see Table  6 ) and 21 implicit offline evaluations (38.89%) corresponding to ones using paper information to identify relevant recommendations (see Table  6 ). We did not find any expert offline evaluations.

6 Changes compared to 2016

This chapter briefly summarises some of the changes in the set of papers we observed when compared to the study by Beel et al. [ 16 ]. Before we start the comparison, we want to point to the fact that we observed papers from two years in which the publication process could have been massively affected by the COVID-19 pandemic.

6.1 Number of papers per year and publication medium

Beel et al. [ 16 ] studied works between and including 1998 and 2013 while we observed works which appeared between January 2019 and October 2021. While the previous study did include all 185 papers (of which 96 were paper recommendation approaches) in their discussion of papers per year which were published in the area of the topic paper or citation recommendation but later on only studied 62 papers for an in-depth review, we generally only studied 65 publications which present novel paper recommendation approaches (see Sect.  3.5 ) in this aspect. Compared to the time frame observed in this previous literature review, we encountered fewer papers being published on the actual topic of scientific paper recommendation per year. In the former work, the published number of papers was rising and hitting 40 in 2013. We found this number being stuck on a constant level between 21 and 23 in the three years we observed. This could hint at differing interest in this topic over time, with a current demise or the trend to work in this area having surpassed its zenith.

While Beel et al. [ 16 ] found 59% of conference papers and 16% of journal articles, we found 54.85% of conference papers and 41.54% of journal articles. The shift to journal articles could stem from a general shift towards journal articles in computer science Footnote 27 .

6.2 Classification

While Beel et al. [ 16 ] found 55% of their studied 62 papers applying methods from content-based filtering, we found only found 7.69% (5) of our 65 papers identifying as content-based approaches. Beel et al. [ 16 ] report 18% of approaches applied collaborative filtering. We encountered 4.62% (three) having this component as part of their self-defined classification. As for graph-based recommendation approach, Beel et al. [ 16 ] found 16% while we only encountered 7.69% (five) of papers with this description. In terms of hybrid approaches, Beel et al. [ 16 ] encountered five (8.06%) truly hybrid ones. In our study, we found 18 approaches (27.69%) labelling themselves as hybrid recommendation systems. Footnote 28

6.3 Evaluation

Table  10 shows the comparison of the distributions of the different types of evaluations between our study observing 54 papers with evaluations and the one conducted by Beel et al. [ 16 ], which regards 75 papers for this aspect. The percentage of quantitative user studies (User quant) is comparable for both studies. A peculiar difference is the percentage of offline evaluations, which is much higher in our current time frame.

When observing the evaluation measures, we found some differences compared to the previous study. While 48.15% of papers with an evaluation report precision in our case, in Beel et al.’s [ 16 ] 72% of approaches with an evaluation report this value. As a contrast, we found 50% of papers reporting F1 while only 11% of papers reported this measure according to Beel et al. [ 16 ]. This might hint at a shift away from precision (which Beel et al. [ 16 ] did describe as a problematic measure) to focus more on also incorporating recall into the quality assessment of recommendation systems.

6.4 Discussion

In general, the two reviews regard different time frames. We encounter non-marginal differences in the three dimensions discussed in this Section. A more concise comparison could be made if a time slice would be regarded for both studies, such that the research output and shape could be observed from three years each. We cannot clearly identify emerging trends (as with the offline evaluation) as we do not know if it has been conducted in this percentage of papers since the 2010s or if it only just picked up to be a more wide-spread evaluation form.

7 Open challenges and objectives

All paper recommendation approaches which were considered in this survey could have been improved in some way or another. Some papers did not conduct evaluations which would satisfy a critical reader, others could be more convincing if they compared their methods to appropriate competitors. The possible problems we encountered within the papers can be summarised in different open challenges, which papers should strive to overcome. We separate our analysis and discussion of open challenges in those which have already been described by previous literature reviews (see Sect.  7.1 ) and ones we identify as new or emerging problems (see Sect.  7.2 ). Lastly we briefly discuss the presented challenges (see Sect.  7.3 ).

7.1 Challenges highlighted in previous works

In the following we will explain possible shortcomings which were already explicitly discussed in previous literature reviews [ 9 , 16 , 92 ]. We regard these challenges in light of current paper recommendation systems to identify problems which are nowadays still encountered.

7.1.1 Neglect of user modelling

Neglect of user modelling has been described by Beel et al. [ 16 ] as identification of target audiences’ information needs. They describe the trade-off between specifying keywords which brings recommendation systems closer to search engines and utilising user profiles as input.

Currently only some approaches consider users of systems to influence the recommendation outcome, as seen with Table  3 users are not always part of the input to systems. Instead many paper recommendation systems assume that users do not state their information needs explicitly but only enter keywords or a paper. With paper recommendation systems where users are not considered, the problem of neglecting user modelling still holds.

7.1.2 Focus on accuracy

Focus on accuracy as a problem is described by Beel et al. [ 16 ]. They state putting users’ satisfaction with recommendations on a level with accuracy of approaches does not depict reality. More factors should be considered.

Only over one fourth of current approaches do not only report precision or accuracy but also observe more diversity focused measures such as MMR. We also found usage of less widespread measures to capture different aspects such as popularity, serendipity or click-through-rate.

7.1.3 Translating research into practice

The missing translation of research into practice is described by Beel et al. [ 16 ]. They mention the small percentage of approaches which are available as prototype as well as the discrepancy between real world systems and methods described in scientific papers.

Only five of our observed approaches definitively must have been available online at any point in time [ 28 , 45 , 65 , 84 , 119 ]. We did not encounter any of the more complex approaches being used in widespread paper recommendation systems.

7.1.4 Persistence and authority

Beel et al. [ 16 ] describe the lack of persistence and authority in the field of paper recommendation systems as one of the main reasons why research is not adapted in practice.

The analysis of this possible shortcoming of current work could be highly affected by the short time period from which we observed works. We found several groups publishing multiple papers as seen in Table  11 which corresponds to 29.69% of approaches. The most papers a group published was three so this amount still cannot fully mark a research group as authority in the area.

7.1.5 Cooperation

Problems with cooperation are described by Beel et al. [ 16 ]. They state even though approaches have been proposed by multiple authors building upon prior work is rare. Corporations between different research groups are also only encountered sporadically.

Here again we want to point to the fact that our observed time frame of less than three years might be too short to make substantive claims regarding this aspect. Table  12 holds information on the different numbers of authors for papers and the percentage of papers out of the 64 observed ones which are authored by groups of this size. We only encountered little cooperation between different co-author groups (see Haruna et al. [ 39 ] and Sakib et al. [ 88 ] for an exception). There were several groups not extending their previous work [ 121 , 122 ]. We refrain from analysing citations of related previous approaches as our considered period of less than three years is too short for all publications to have been able to be recognised by the wider scientific community.

7.1.6 Information scarcity

Information scarcity is described by Beel et al. [ 16 ] as researchers’ tendency to only provide insufficient detail to re-implement their approaches. This leads to problems with reproducibility.

Many of the approaches we encountered did not provide sufficient information to make a re-implementation possible: with Afsar et al. [ 1 ] it is unclear how the knowledge graph and categories were formed, Collins and Beel [ 28 ] do not describe their Doc2Vec enough, Liu et al. [ 61 ] do not specify the extraction of keywords for papers in the graph and Tang et al. [ 104 ] do not clearly describe their utilisation of Word2Vec. In general oftentimes details are missing [ 3 , 4 , 60 , 117 ]. Exceptions to these observations are e.g. found with Bereczki [ 19 ], Nishioka et al. [ 74 , 75 , 76 ] and Sakib et al. [ 88 ].

We did not find a single paper’s code e.g. provided as a link to GitHub.

7.1.7 Cold start

Pure collaborative filtering systems encounter the cold start problem as described by Bai et al. [ 9 ] and Shahid et al. [ 92 ]. If new users are considered, no historical data is available, they cannot be compared to other users to find relevant recommendations.

While this problem still persists, most current approaches are no pure collaborative filtering based recommendation systems (see Sect.  3.3.1 ). Systems using deep learning could overcome this issue [ 58 ]. There are approaches specifically targeting this problem [ 59 , 96 ], some [ 59 ] also introduced specific evaluation measures (totNP_EU and avgNP_EU) to quantify systems’ ability to overcome the cold start problem.

7.1.8 Sparsity or reduce coverage

Bai et al. [ 9 ] state the user-paper-matrix being sparse for collaborative filtering based approaches. Shahid et al. [ 92 ] also mention this problem as the reduce coverage problem . This trait makes it hard for approaches to learn relevancy of infrequently rated papers.

Again, while this problem is still encountered, current approaches mostly are no longer pure collaborative filtering-based systems but instead utilise more information (see Sect.  3.3.1 ). Using deep learning in the recommendation process might reduce the impact of this problem [ 58 ].

7.1.9 Scalability

The problem of scalability was described by Bai et al. [ 9 ]. They state paper recommendation systems should be able to work in huge, ever expanding environments where new users and papers are added regularly.

A few approaches [ 38 , 46 , 88 , 109 ] contain a web crawling step which directly tackles challenges related to outdated or missing data. Some approaches [ 26 , 61 ] evaluate the time it takes to compute paper recommendations which also indicates their focus on this general problem. But most times scalability is not explicitly mentioned by current paper recommendation systems. There are several works [ 42 , 45 , 96 , 108 , 116 ] evaluating on bigger datasets with over 1 million papers and which thus are able to handle big amounts of data. Sizes of current relevant real-world data collections exceed this threshold many times over (see, e.g. PubMed with over 33 million papers Footnote 29 or SemanticScholar with over 203 million papers Footnote 30 ). Kanakia et al. [ 45 ] explicitly state scalability as a problem their approach is able to overcome. Instead of comparing each paper to all other papers they utilise clustering to reduce the number of required computations. They present the only approach running on several hundred million publications. Nair et al. [ 71 ] mention scalability issues they encountered even when only considering around 25,000 publications and their citation relations.

7.1.10 Privacy

The problem of privacy in personalised paper recommendation is described by Bai et al. [ 9 ]. Shahid et al. [ 92 ] also mention this as a problem occurring in collaborative filtering approaches. An issue is encountered when sensitive information such as habits or weaknesses that users might not want to disclose is used by a system. This leads to users’ having negative impressions of systems. Keeping sensitive information private should therefore be a main goal.

In the current approaches, we did not find a discussion of privacy concerns. Some approach even explicitly utilise likes [ 84 ] or association rules [ 3 ] of other users while failing to mention privacy altogether. In approaches not incorporating any user data, this issue does not arise at all.

7.1.11 Serendipity

Serendipity is described by Bai et al. [ 9 ] as an attribute often encountered in collaborative filtering [ 16 ]. Usually paper recommender systems focus on identification of relevant papers even though also including not obviously relevant ones might enhance the overall recommendation. Junior researchers could profit from stray recommendations to broaden their horizon, senior researchers might be able to gain knowledge to enhance their research. The ratio between clearly relevant and serendipitous papers is crucial to prevent users from losing trust in the recommender system.

A main objective of the works of Nishioka et al. [ 74 , 75 , 76 ] is serendipity. Other approaches do not mention this aspect.

7.1.12 Unified scholarly data standards

Different data formats of data collections is mentioned as a problem by Bai et al. [ 9 ]. They mention digital libraries containing relevant information which needs to be unified in order to use the data in a paper recommendation system. Additionally the combination of datasets could also lead to problems.

Many of the approaches we observe do not consider data collection or preparation as part of the approach, they often only mention the combination of different datasets as part of the evaluation (see e.g. Du et al. [ 29 ], Li et al. [ 56 ] or Xie et al. [ 115 ]). An exception to this general rule are systems which contain a web crawling step for data (see e.g. Ahmad and Afzal [ 2 ] or Sakib et al. [ 88 ]). Even with this type of approaches the combination of datasets and their diverse data formats is not identified as a problem.

7.1.13 Synonymy

Shahid et al. [ 92 ] describe the problem of synonymy encountered in collaborative filtering approaches. They define this problem as different words having the same meaning.

Even though there are still approaches (not necessarily CF ones) utilising basic TF-IDF representations of papers [ 2 , 42 , 86 , 95 ], nowadays this problem can be bypassed by using a text embedding method such as Doc2Vec or BERT.

7.1.14 Gray sheep

Gray sheep is a problem described by Shahid et al. [ 92 ] as an issue encountered in collaborative filtering approaches. They describe it as some users not consistently (dis)agreeing with any reference group.

We did not find any current approach mentioning this problem.

7.1.15 Black sheep

Black sheep is a problem described by Shahid et al. [ 92 ] as an issue encountered in collaborative filtering approaches. They describe it as some users not (dis)agree-ing with any reference group.

7.1.16 Shilling attack

Shilling attacks are described by Shahid et al. [ 92 ] as a problem encountered in collaborative filtering approaches. They define this problem as users being able to manually enhance visibility of their own research by rating authored papers as relevant while negatively rating any other recommendations.

Although we did not find any current approach mentioning this problem we assume maybe it is no longer highly relevant as most approaches are no longer pure collaborative filtering ones. Additionally from the considered collaborative filtering approaches no one explicitly stated to feed relevance ratings back into the system.

7.2 Emerging challenges

In addition to the open challenges discussed in former literature reviews by Bai et al. [ 9 ], Beel et al. [ 16 ] and Shahid et al. [ 92 ] we identified the following problems and derive desirable goals for future approaches from them.

7.2.1 User evaluation

Paper recommendation is always targeted at human users. But oftentimes an evaluation with real users to quantify users’ satisfaction with recommended publications is simply not conducted [ 84 ]. Conducting huge user studies is not feasible [ 38 ]. So sometimes user data to evaluate with is fetched from the presented datasets [ 39 , 88 ] or user behaviour is artificially emulated [ 1 , 19 , 57 ]. Noteworthy counter-examples Footnote 31 are the studies by Bulut et al. [ 22 ] who emailed 50 researchers to rate relevancy of recommended articles or Chaudhuri et al. [ 26 ] who asked 45 participants to rate their acceptance of recommended publications. Another option to overcome this issue is utilisation of living labs as seen with ArXivDigest [ 36 ], Mr. DLib’s living lab [ 14 ] or LiLAS for the related tasks of dataset recommendation for scientific publications and multi-lingual document retrieval [ 91 ].

Desirable goal Paper recommendation systems targeted at users should always contain a user evaluation with a description of the composition of participants.

7.2.2 Target audience

Current works mostly fail to clearly characterise the intended users of a system altogether and the varying interests of different types of users are not examined in their evaluations. There are some noteworthy counter-examples: Afsar et al. [ 1 ] mention cancer patients and their close relatives as intended target audience. Bereczki [ 19 ] identifies new users as a special group they want to recommend papers to. Hua et al. [ 42 ] consider users who start diving into a topic which they have not yet researched before. Sharma et al. [ 95 ] name subject matter experts incorporating articles into a medical knowledge base as their target audience. Shi et al. [ 96 ] clearly state use cases for their approach which always target users which are unaware of a topic but already have one interesting paper from the area. They strive to recommend more papers similar to the first one.

User characteristics such as registration status of users are already mentioned by Beel et al. [ 16 ] as a factor which is disregarded in evaluations. We want to extend on this point and highlight the oftentimes missing or inadequate descriptions of intended users of paper recommendation systems. Traits of users and their information needs are not only important for experiments but should also be regarded in the construction of an approach. The targeted audience of a paper recommendation system should influence its suggestions. Bai et al. [ 9 ] highlight different needs of junior researchers which should be recommended a broad variety of papers as they still have to figure out their direction. They state recommendations for senior researchers should be more in line with their already established interests. Sugiyama and Kan [ 100 ] describe the need to help discover interdisciplinary research for this experienced user group. Most works do not recognise possible different functions of paper recommendation systems for users depending on their level of seniority. If papers include an evaluation with real persons, they e.g. mix Master’s students with professors but do not address their different goals or expectations from paper recommendation [ 74 ]. Chaudhuri et al. [ 26 ] have junior, experienced and expert users as participants of their study and give individual ratings but do not calculate evaluation scores per user group. In some studies the exact composition of test users is not even mentioned (see Table  9 ).

Desirable goal Definition and consideration of a specific target audience for an approach and evaluation with members of this audience. If there is no specific person group a system should suit best, this should be discussed, executed and evaluated accordingly.

7.2.3 Recommendation scenario

Suggested papers from an approach should either be ones to read [ 44 , 109 ], to cite or fulfil another specified information need such as help patients in cancer treatment decision making [ 1 ]. Most work does not clearly state which is the case. Instead recommended papers are only said to be related [ 4 , 28 ], relevant [ 4 , 5 , 26 , 27 , 38 , 42 , 45 , 48 , 56 , 57 , 105 , 115 , 117 ], satisfactory [ 42 , 61 ], suitable [ 21 ], appropriate and useful [ 22 , 88 ] or a description which scenario is tackled is skipped altogether [ 3 , 37 , 39 , 84 ].

In rare cases if the recommendation scenario is mentioned there is the possibility of it not perfectly fitting the evaluated scenario. This can, e.g. be seen in the work of Jing and Yu [ 44 ] where they propose paper recommendation for papers to read but evaluate papers which were cited. Cited papers should always be ones which have been read beforehand but the decision to cite papers can be influenced by multiple aspects [ 34 ].

Desirable goal The clear description of the recommendation scenario is important for comparability of approaches as well as the validity of the evaluation.

7.2.4 Fairness/diversity

Anand et al. [ 8 ] define fairness as the balance between relevance and diversity of recommendation results. Only focusing on fit between the user or input paper and suggestions would lead to highly similar results which might not be vastly different from each other. Having diverse recommendation results can help cover multiple aspects of a user query instead of only satisfying the most prominent feature of the query [ 8 ]. In general more diverse recommendations provide greater utility for users [ 76 ]. Ekstrand et al. [ 31 ] give a detailed overview of current constructs for measuring algorithmic fairness in information access and describe possibly arising problems in this context.

Most of the current paper recommendation systems do not consider fairness but some approaches specifically mention diversity [ 26 , 74 , 75 , 76 ] while striving to recommend relevant publications. Thus these systems consider fairness.

Over one fourth of considered approaches with an evaluation report MMR as a measure of their system’s quality. This at least seems to show researchers’ awareness of the general problem of diverse recommendation results.

Desirable Goal Diversification of suggested papers to ensure fairness of the approach.

7.2.5 Complexity

Paper recommendation systems tend to become more complex, convoluted or composed of multiple parts. We observed this trend by regarding the classification of current systems compared to previous literature reviews (see Sect.  3.3.1 ). While systems’ complexity increases, users’ interaction with the systems should not become more complex. If an approach requires user interaction at all, it should be as simple as possible. Users should not be required to construct sophisticated knowledge graphs [ 109 ] or enter multiple rounds of keywords for an approach to learn their user profile [ 24 ].

Desirable Goal Maintain simplicity of usage even if approaches become more complex.

7.2.6 Explainability

Confidence in the recommendation system has already been mentioned by Beel et al. [ 16 ] as an example of what could enhance users’ satisfaction but what is overlooked in approaches in favour of accuracy. This aspect should be considered with more vigour as the general research area of explainable recommendation has gained immense traction [ 120 ]. Gingstad et al. [ 36 ] regard explainability as a core component of paper recommendation systems. Xie et al. [ 116 ] mention explainability as a key feature of their approach but do not state how they achieve it or if their explanations satisfy users. Suggestions of recommendation systems should be explainable to enhance their trustworthiness and make them more engaging [ 66 ]. Here, different explanation goals such as effectiveness, efficiency, transparency or trust and their influence on each other should be considered [ 10 ]. If an approach uses neural networks [ 24 , 37 , 49 , 56 ] it is oftentimes impossible to explain why the system learned, that a specific suggested paper might be relevant.

Lee et al. [ 51 ] introduce a general approach which could be applied to any paper recommendation system to generate explanations for recommendations. Even though this option seems to help solve the described problem it is not clear how valuable post-hoc explanations are compared to systems which construct them directly.

Desirable Goal The conceptualisation of recommendation systems which comprehensibly explain their users why a specific paper is suggested.

7.2.7 Public dataset

Current approaches utilise many different datasets (see Table  4 ). A large portion of them are built by the authors such that they are not publicly available for others to use as well [ 1 , 30 , 111 ]. Part of the approaches already use open datasets in their evaluation but a large portion still does not seem to regard this as a priority (see Table  5 ). Utilisation of already public data sources or construction of datasets which are also published and remain available thus should be a priority in order to support reproducibility of approaches.

Desirable Goal Utilisation of publicly available datasets in the evaluation of paper recommendation systems.

7.2.8 Comparability

From the approaches we observed, many identified themselves as paper recommendation ones but only evaluated against systems, which are more general recommendation systems or ones utilising some same methodologies but not from the sub-domain of paper recommendation (seen with e.g. Guo et al [ 37 ], Tanner et al. [ 106 ] or Yang et al. [ 117 ]). While some of the works might claim to only be applied on paper recommendation and be of more general applicability (see, e.g. the works by Ahmedi et al. [ 3 ] or Alfarhood and Cheng [ 4 ]) we state that they should still be compared to ones, which mainly identify as paper recommendation systems as seen in the work of Chaudhuri et al. [ 24 ]. Only if a more general approach is compared to a paper recommendation approach, its usefulness for the area of paper recommendation can be fully assessed.

Several times, the baselines to evaluate against are not even other works but artificially constructed ones [ 2 , 38 ] or no other approach at all [ 22 ].

Desirable Goal Evaluation of paper recommendation approaches, even those which are applicable in a wider context, should always be against at least one paper recommendation system to clearly report relevance of the proposed method in the claimed context.

7.3 Discussion and outlook

From the already existing problems, several of them are still encountered in current paper recommendation approaches. Users are not always part of the approaches so users are not always modelled but this also prevents privacy issues. Accuracy seems to still be the main focus of recommendation systems. Novel techniques proposed in papers are not available online or applied by existing paper recommendation systems. Approaches do not provide enough details to enable re-implementation. Providing the code online or in a living lab environment could help overcome many of these issues.

Other problems mainly encountered in pure collaborative filtering systems such as the cold start problem, sparsity, synonymy, gray sheep, black sheep and shilling attacks do not seem to be as relevant anymore. We observed a trend towards hybrid models, this recommendation system type can overcome these issues. These hybrid models should also be able to produce serendipitous recommendations.

Unifying data sources is conducted often but nowadays it does not seem to be regarded as a problem. With scalability we encountered the same. Approaches are oftentimes able to handle millions of papers, here they do not specifically mention scalability as a problem they overcome but they also mostly do not consider huge datasets with several hundreds of millions of publications.

Due to the limited scope of our survey we are not able to derive substantive claims regarding cooperation and persistence. We found around 30% of approaches published by groups which authored multiple papers and very few collaborations between different author groups.

As for the newly introduced problems, part of the observed approaches conducted evaluations with users, on publicly available datasets and against other paper recommendation systems. Many works considered a low complexity for users. Even though user evaluations are desirable, they come with high costs. Usage of evaluation datasets with real human annotations could help overcome this issue partially, another straightforward solution would be the incorporation in a living lab. The second option would also help with comparability of approaches. Usage of available datasets can become increasingly complicated if approaches use new data which is currently not contained in existing datasets. Footnote 32

Target audiences in general were rarely defined, the recommendation scenario was mostly not described. Diversity was considered by few. Overall the explainability of recommendations was dismissed. The first two of these issues are ones which could be comparatively easily fixed or addressed in the papers without changing the approach. As for diversity and explainability, the approaches would need to be modelled specifically such that these attributes could be satisfied.

To conclude, there are many challenges which are not constantly considered by current approaches. They define the requirements for future works in the area of paper recommendation systems.

8 Conclusion

This literature review of publications targeting paper recommendation between January 2019 and October 2021 provided comprehensive overviews of their methods, datasets and evaluation measures. We showed the need for a richer multi-dimensional characterisation of paper recommendation as former ones no longer seem sufficient in classifying the increasingly complex approaches. We also revisited known open challenges in the current time frame and highlighted possibly under-observed problems which future works could focus on.

Efforts should be made to standardise or better differentiate between the varying notions of relevancy and recommendation scenarios when it comes to paper recommendation. Future work could try revaluate already existing methods with real humans and against other paper recommendation systems. This could for example be realised in an extendable paper recommendation benchmarking system similar to the in a living lab environments ArXivDigest [ 36 ], Mr. DLib’s living lab [ 14 ] or LiLAS [ 91 ] but with the additional property that it also provides build-in offline evaluations. As fairness and explainability of current paper recommendation systems have not been tackled widely, those aspects should be further explored. Another direction could be the comparison of multiple rare evaluation measures on the same system to help identify those which should be focused on in the future. As we observed a vast variety in datasets utilised for evaluation of the approaches (see Table  4 ), construction of publicly available and widely reusable ones would be worthwhile.

The most recent surveys [ 9 , 58 , 92 ] focusing on scientific paper recommendation appeared in 2019 such that this time frame is not yet covered.

Non-immediate variants allow using methods which require more time to compute recommendations. Temporal patterns of user behaviour could be incorporated in the recommendation process to identify a fitting moment to present new recommendations to a user. The moment a recommendation is presented to a user influences their interest, as the delayed recommendation might no longer be relevant or does not fit the current task of a user.

https://dl.acm.org/ .

https://dblp.uni-trier.de/ .

https://scholar.google.com/ .

https://link.springer.com/ .

For a survey of current trends in citation recommendation refer to Färber and Jatowt [ 32 ].

These papers could either be a demo paper and a later published full paper or the conference and journal version of the same approach, which is then slightly extended by more experiments. These paper clusters are no exact duplicates or fraudulent publications.

The number of citations can be regarded both as an input data as well as a method to denote popularity.

https://dblp.uni-trier.de/xml/ .

https://www.aminer.org/citation .

(shortened) http://shorturl.at/cIQR1 .

https://github.com/js05212/citeulike-a .

https://github.com/js05212/citeulike-t .

https://www.scopus.com/home.uri .

https://www.aminer.org/ .

https://aan.how/download/ .

https://citeseerx.ist.psu.edu/index .

https://bulkdata.uspto.gov/ .

https://snap.stanford.edu/data/cit-HepTh.html .

(shortened) http://shorturl.at/orwXY .

http://mlg.ucd.ie/datasets/bbc.html .

https://sites.google.com/site/tinhuynhuit/dataset .

One approach is described in three papers.

Shi et al. [ 96 ] also conduct a user study but do not describe their participants.

Compare the 99.363 journal articles and 151.617 conference papers published in 2013 to the 187.263 journal articles and 157.460 conference articles in 2021 in dblp.

Note that not all approaches classified their type of paper recommendation and several papers did not classify themselves in the wide-spread categorisation (see Sect.  3.3.1 ).

https://pubmed.ncbi.nlm.nih.gov/ .

https://www.semanticscholar.org/product/api .

For a full list of approaches conducting user studies see Table  9 .

We did not encounter many papers utilising types of data as part of their approach, which is not typically included in existing datasets; one of the noteworthy exceptions could be the approach by Nishioka et al. [ 74 , 75 , 76 ], which utilised Tweets of users.

Afsar, M.M., Crump, T., Far, B.H.: An exploration on-demand article recommender system for cancer patients information provisioning. In: FLAIRS Conference’21 (2021). https://doi.org/10.32473/flairs.v34i1.128339

Ahmad, S., Afzal, M.T.: Combining metadata and co-citations for recommending related papers. Turkish J. Electr. Eng. Comput. Sci. 28 (3), 1519–1534 (2020). https://doi.org/10.3906/elk-1908-19

Article   Google Scholar  

Ahmedi, L., Rexhepi, E., Bytyçi, E.: Using association rule mining to enrich user profiles with research paper recommendation. Int. J. Comput. Dig Syst. (2021). https://doi.org/10.12785/ijcds/110192

Alfarhood, M., Cheng, J.: Collaborative attentive autoencoder for scientific article recommendation. In: ICMLA’19, pp. 168–174. IEEE (2019). https://doi.org/10.1109/ICMLA.2019.00034

Ali, Z., Qi, G., Muhammad, K., Ali, B., Abro, W.A.: Paper recommendation based on heterogeneous network embedding. Knowl. Based Syst. 210 , 106438 (2020). https://doi.org/10.1016/j.knosys.2020.106438

Alzoghbi, A., Ayala, V.A.A., Fischer, P.M., Lausen, G.: PubRec: Recommending Publications Based on Publicly Available Meta-Data. In: LWA’15, CEUR workshop proceedings. vol. 1458, pp. 11–18. CEUR-WS.org (2015). http://ceur-ws.org/Vol-1458/D01_CRC69_Alzoghbi.pdf

Amami, M., Faiz, R., Stella, F., Pasi, G.: A graph based approach to scientific paper recommendation. In: WI’17, pp. 777–782. ACM (2017). https://doi.org/10.1145/3106426.3106479

Anand, A., Chakraborty, T., Das, A.: FairScholar: balancing relevance and diversity for scientific paper recommendation. In: ECIR’17, LNCS. 10193 , 753–757 (2017). https://doi.org/10.1007/978-3-319-56608-5_76

Bai, X., Wang, M., Lee, I., Yang, Z., Kong, X., Xia, F.: Scientific paper recommendation: a survey. IEEE Access 7 , 9324–9339 (2019). https://doi.org/10.1109/ACCESS.2018.2890388

Balog, K., Radlinski, F.: Measuring recommendation explanation quality: the conflicting goals of explanations. In: SIGIR’20, pp. 329–338. ACM (2020). https://doi.org/10.1145/3397271.3401032

Barolli, L., Di Cicco, F., Fonisto, M.: An Investigation of Covid-19 Papers for a Content-Based Recommendation System. In: 3PGCIC’22, pp. 156–164. Springer (2022). https://doi.org/10.1007/978-3-030-89899-1_16

Basaldella, M., Nart, D.D., Tasso, C.: Introducing distiller: a unifying framework for knowledge extraction. In: IT@LIA@AI*IA’15, CEUR Workshop Proceedings, vol. 1509. CEUR-WS.org (2015). http://ceur-ws.org/Vol-1509/ITALIA2015_paper_4.pdf

Beel, J., Carevic, Z., Schaible, J., Neusch, G.: RARD: the related-article recommendation dataset. D Lib Mag. (2017). https://doi.org/10.1045/july2017-beel

Beel, J., Collins, A., Kopp, O., Dietz, L.W., Knoth, P.: Online Evaluations for Everyone: Mr. DLib’s Living Lab for Scholarly Recommendations. In: ECIR’19, LNCS, 11438 , 213–219. Springer (2019). https://doi.org/10.1007/978-3-030-15719-7_27

Beel, J., Dinesh, S., Mayr, P., Carevic, Z., Jain, R.: Stereotype and most-popular recommendations in the digital library Sowiport. In: ISI’17, Schriften zur Informationswissenschaft, 70 , 96–108. Verlag Werner Hülsbusch (2017). https://doi.org/10.18452/1441

Beel, J., Gipp, B., Langer, S., Breitinger, C.: Research-paper recommender systems: a literature survey. Int. J. Digit. Libr. 17 (4), 305–338 (2016). https://doi.org/10.1007/s00799-015-0156-0

Beel, J., Langer, S.: A Comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In: TPDL’15, LNCS, 9316 , 153–168. Springer (2015). https://doi.org/10.1007/978-3-319-24592-8_12

Beierle, F., Aizawa, A., Collins, A., Beel, J.: Choice overload and recommendation effectiveness in related-article recommendations. Int. J. Digit. Libr. 21 (3), 231–246 (2020). https://doi.org/10.1007/s00799-019-00270-7

Bereczki, M.: Graph neural networks for article recommendation based on implicit user feedback and content. Master’s thesis, KTH, School of Electrical Engineering and Computer Science (EECS) (2021)

Bogers, T., van den Bosch, A.: recommending scientific articles using citeulike. In: RecSys’08, pp. 287–290. ACM (2008). https://doi.org/10.1145/1454008.1454053

Bulut, B., Gündoğan, E., Kaya, B., Alhajj, R., Kaya, M.: User’s research interests based paper recommendation system: a deep learning approach. pp. 117–130. Springer (2020). https://doi.org/10.1007/978-3-030-33698-1_7

Bulut, B., Kaya, B., Kaya, M.: A paper recommendation system based on user interest and citations. In: UBMYK’19, pp. 1–5 (2019). https://doi.org/10.1109/UBMYK48245.2019.8965533

Champiri, Z.D., Asemi, A., Salim, S.S.B.: Meta-analysis of evaluation methods and metrics used in context-aware scholarly recommender systems. Knowl. Inf. Syst. 61 (2), 1147–1178 (2019). https://doi.org/10.1007/s10115-018-1324-5

Chaudhuri, A., Samanta, D., Sarma, M.: Modeling user behaviour in research paper recommendation system (2021). arXiv:2107.07831

Chaudhuri, A., Sarma, M., Samanta, D.: Advanced feature identification towards research article recommendation: a machine learning based approach. In: TENCON’19, pp. 7–12. IEEE (2019). https://doi.org/10.1109/TENCON.2019.8929386

Chaudhuri, A., Sinhababu, N., Sarma, M., Samanta, D.: Hidden features identification for designing an efficient research article recommendation system. Int. J. Digit. Libr. 22 (2), 233–249 (2021). https://doi.org/10.1007/s00799-021-00301-2

Chen, J., Ban, Z.: Academic Paper Recommendation Based on Clustering and Pattern Matching, pp. 171–182. Springer, Cham (2019). https://doi.org/10.1007/978-981-32-9298-7_14

Book   Google Scholar  

Collins, A., Beel, J.: Document embeddings vs. keyphrases vs. terms for recommender systems: a large-scale online evaluation. In: JCDL’19, pp. 130–133. IEEE (2019). https://doi.org/10.1109/JCDL.2019.00027

Du, N., Guo, J., Wu, C.Q., Hou, A., Zhao, Z., Gan, D.: Recommendation of academic papers based on heterogeneous information networks. In: AICCSA’20, pp. 1–6. IEEE (2020). https://doi.org/10.1109/AICCSA50499.2020.9316516

Du, Z., Tang, J., Ding, Y.: POLAR++: active one-shot personalized article recommendation. IEEE Trans. Knowl. Data Eng. 33 (6), 2709–2722 (2021). https://doi.org/10.1109/TKDE.2019.2953721

Ekstrand, M.D., Das, A., Burke, R., Diaz, F.: Fairness and discrimination in information access systems (2021). arXiv:2105.05779

Färber, M., Jatowt, A.: Citation recommendation: approaches and datasets. Int. J. Digit. Libr. 21 (4), 375–405 (2020). https://doi.org/10.1007/s00799-020-00288-2

Feng, S., Meng, J., Zhang, J.: News recommendation systems in the era of information overload. J. Web Eng. 20 (2), 459–470 (2021). https://doi.org/10.13052/jwe1540-9589.20210

Garfield, E.: Can citation indexing be automated? (1964)

Giles, C.L., Bollacker, K.D., Lawrence, S.: CiteSeer: an automatic citation indexing system. In: ACM DL’98, pp. 89–98. ACM (1998). https://doi.org/10.1145/276675.276685

Gingstad, K., Jekteberg, Ø., Balog, K.: ArXivDigest: a living lab for personalized scientific literature recommendation. In: CIKM’20, pp. 3393–3396. ACM (2020). https://doi.org/10.1145/3340531.3417417

Guo, G., Chen, B., Zhang, X., Liu, Z., Dong, Z., He, X.: Leveraging title-abstract attentive semantics for paper recommendation. In: IAAI’20, pp. 67–74. AAAI Press (2020). https://aaai.org/ojs/index.php/AAAI/article/view/5335

Habib, R., Afzal, M.T.: Sections-based bibliographic coupling for research paper recommendation. Scientometrics 119 (2), 643–656 (2019). https://doi.org/10.1007/s11192-019-03053-8

Haruna, K., Ismail, M.A., Qazi, A., Kakudi, H.A., Hassan, M., Muaz, S.A., Chiroma, H.: Research paper recommender system based on public contextual metadata. Scientometrics 125 (1), 101–114 (2020). https://doi.org/10.1007/s11192-020-03642-y

Hienert, D., Sawitzki, F., Mayr, P.: Digital library research in action: supporting information retrieval in sowiport. D Lib. Mag. (2015). https://doi.org/10.1045/march2015-hienert

Hu, D., Ma, H., Liu, Y., He, X.: Scientific paper recommendation using author’s dual role citation relationship. In: Intelligent information processing’20, IFIP advances in information and communication technology. 581 , 121–132. Springer (2020). https://doi.org/10.1007/978-3-030-46931-3_12

Hua, S., Chen, W., Li, Z., Zhao, P., Zhao, L.: Path-Based Academic Paper Recommendation. In: WISE’20, LNCS, 12343 , 343–356. Springer (2020). https://doi.org/10.1007/978-3-030-62008-0_24

Ji, Z., Wu, M., Yang, H., Armendáriz-Iñigo, J.E.: Temporal sensitive heterogeneous graph neural network for news recommendation. Future Gener. Comput. Syst. 125 , 324–333 (2021). https://doi.org/10.1016/j.future.2021.06.007

Jing, S., Yu, S.: Research of paper recommendation system based on citation network model. In: ML4CS’20, LNCS, 12488 , 237–247. Springer (2020). https://doi.org/10.1007/978-3-030-62463-7_22

Kanakia, A., Shen, Z., Eide, D., Wang, K.: A Scalable hybrid research paper recommender system for microsoft academic (2019). arXiv:1905.08880

Kang, Y., Hou, A., Zhao, Z., Gan, D.: A hybrid approach for paper recommendation. IEICE Trans. Inform. Syst. E104D (8), 1222–1231 (2021). https://doi.org/10.1587/transinf.2020BDP0008

Keller, J., Munz, L.P.M.: TEKMA at CLEF-2021: BM-25 based rankings for scientific publication retrieval and data set recommendation. In: CLEF’21, CEUR Workshop Proceedings. 2936 , 1700–1711. CEUR-WS.org (2021). http://ceur-ws.org/Vol-2936/paper-144.pdf

Kong, X., Mao, M., Wang, W., Liu, J., Xu, B.: VOPRec: vector representation learning of papers with text information and structural identity for recommendation. IEEE Trans. Emerg. Top. Comput. 9 (1), 226–237 (2021). https://doi.org/10.1109/TETC.2018.2830698

L, H., Liu, S., Pan, L.: Paper recommendation based on author-paper interest and graph structure. In: CSCWD’21, pp. 256–261. IEEE (2021). https://doi.org/10.1109/CSCWD49262.2021.9437743

Le, M., Kayal, S., Douglas, A.: The impact of recommenders on scientific article discovery: the case of mendeley suggest. In: ImpactRS@RecSys’19, CEUR Workshop Proceedings, vol. 2462. CEUR-WS.org (2019). http://ceur-ws.org/Vol-2462/paper5.pdf

Lee, B.C.G., Lo, K., Downey, D., Weld, D.S.: Explanation-based tuning of opaque machine learners with application to paper recommendation (2020). arXiv:2003.04315

Lee, J., Lee, K., Kim, J.G.: Personalized academic research paper recommendation system (2013). arXiv:1304.5457

Leskovec, J., Kleinberg, J.M., Faloutsos, C.: Graphs over time: densification laws, shrinking diameters and possible explanations. In: SIGKDD’05, pp. 177–187. ACM (2005). https://doi.org/10.1145/1081870.1081893

Ley, M.: DBLP: some lessons learned. Proc VLDB Endow. 2 (2), 1493–1500 (2009). https://doi.org/10.14778/1687553.1687577

Li, W., Chang, C., He, C., Wu, Z., Guo, J., Peng, B.: Academic paper recommendation method combining heterogeneous network and temporal attributes. In: Chinese CSCW’21, pp. 456–468. Springer, Singapore (2021). https://doi.org/10.1007/978-981-16-2540-4_33

Li, X., Chen, Y., Pettit, B., de Rijke, M.: Personalised reranking of paper recommendations using paper content and user behavior. ACM Trans. Inform. Syst. 37 (3), 1–23 (2019). https://doi.org/10.1145/3312528

Li, Y., Wang, R., Nan, G., Li, D., Li, M.: A personalized paper recommendation method considering diverse user preferences. Decis. Support Syst. 146 , 113546 (2021). https://doi.org/10.1016/j.dss.2021.113546

Li, Z., Zou, X.: A review on personalized academic paper recommendation. Comput. Inf. Sci. 12 (1), 33–43 (2019). https://doi.org/10.5539/cis.v12n1p33

Lin, S.j., Lee, G., Peng, S.L.: Academic article recommendation by considering the research field trajectory. pp. 447–454. Springer (2021). https://doi.org/10.1007/978-3-030-65407-8_39

Liu, H., Kou, H., Chi, X., Qi, L.: Combining time, keywords and authors information to construct papers correlation graph (S). In: SEKE’19, pp. 11–19. KSI Research Inc. and Knowledge Systems Institute Graduate School (2019). https://doi.org/10.18293/SEKE2019-161

Liu, H., Kou, H., Yan, C., Qi, L.: Keywords-driven and popularity-aware paper recommendation based on undirected paper citation graph. Complex (2020). https://doi.org/10.1155/2020/2085638

Lu, Y., He, Y., Cai, Y., Peng, Z., Tang, Y.: Time-aware neural collaborative filtering with multi-dimensional features on academic paper recommendation. In: CSCWD’21, pp. 1052–1057. IEEE (2021). https://doi.org/10.1109/CSCWD49262.2021.9437673

Ma, X., Wang, R.: Personalized scientific paper recommendation based on heterogeneous graph representation. IEEE Access 7 , 79887–79894 (2019). https://doi.org/10.1109/ACCESS.2019.2923293

Ma, X., Zhang, Y., Zeng, J.: Newly published scientific papers recommendation in heterogeneous information networks. Mob. Networks Appl. 24 (1), 69–79 (2019). https://doi.org/10.1007/s11036-018-1133-9

Manju, G., Abhinaya, P., Hemalatha, M.R., Manju, G.G., Manju, G.G.: Cold start problem alleviation in a research paper recommendation system using the random walk approach on a heterogeneous user-paper graph. Int. J. Intell. Inf. Technol. 16 (2), 24–48 (2020). https://doi.org/10.4018/IJIIT.2020040102

McInerney, J., Lacker, B., Hansen, S., Higley, K., Bouchard, H., Gruson, A., Mehrotra, R.: Explore, exploit, and explain: personalizing explainable recommendations with bandits. In: RecSys’18, pp. 31–39. ACM (2018). https://doi.org/10.1145/3240323.3240354

Medic, Z., Snajder, J.: A survey of citation recommendation tasks and methods. J. Comput. Inf. Technol. 28 (3), 183–205 (2020)

Google Scholar  

Mohamed Hassan, H.A., Sansonetti, G., Gasparetti, F., Micarelli, A., Beel, J.: BERT, ELMo, USE and InferSent sentence encoders: the panacea for research-paper recommendation? In: RecSys’19, CEUR Workshop Proceedings. 2431 , 6–10. CEUR-WS.org (2019). http://ceur-ws.org/Vol-2431/paper2.pdf

Mohamed Hassan, H.A., Sansonetti, G., Micarelli, A.: Tag-aware document representation for research paper recommendation (2020). https://www.researchgate.net/publication/343319230_Tag-Aware_Document_Representation_for_Research_Paper_Recommendation

Moskalenko, O., Sáez-Trumper, D., Parra, D.: Scalable recommendation of wikipedia articles to editors using representation learning. In: RecSys’20, CEUR workshop proceedings. 2697 . CEUR-WS.org (2020). http://ceur-ws.org/Vol-2697/paper1_complexrec.pdf

Nair, A.M., Benny, O., George, J.: Content based scientific article recommendation system using deep learning technique. In: Inventive systems and control, pp. 965–977. Springer, Singapore (2021). https://doi.org/10.1007/978-981-16-1395-1_70

Ng, Y.: Research paper recommendation based on content similarity, peer reviews, authority, and popularity. In: ICTAI’20, pp. 47–52. IEEE (2020). https://doi.org/10.1109/ICTAI50040.2020.00018

Ng, Y.K.: CBRec: a book recommendation system for children using the matrix factorisation and content-based filtering approaches. Int. J. Bus. Intell. Data Mining 16 (2), 129–149 (2020). https://doi.org/10.1504/IJBIDM.2020.104738

Article   MathSciNet   Google Scholar  

Nishioka, C., Hauke, J., Scherp, A.: Research paper recommender system with serendipity using tweets vs. diversification. In: ICADL’19, LNCS. 11853 , 63–70. Springer (2019). https://doi.org/10.1007/978-3-030-34058-2_7

Nishioka, C., Hauke, J., Scherp, A.: Towards serendipitous research paper recommender using tweets and diversification. In: TPDL’19, LNCS. 11799 , 339–343. Springer (2019). https://doi.org/10.1007/978-3-030-30760-8_29

Nishioka, C., Hauke, J., Scherp, A.: Influence of tweets and diversification on serendipitous research paper recommender systems. PeerJ Comput. Sci. 6 , e273 (2020). https://doi.org/10.7717/peerj-cs.273

Ostendorff, M.: Contextual document similarity for content-based literature recommender systems (2020). arXiv:2008.00202

Ostendorff, M., Breitinger, C., Gipp, B.: A qualitative evaluation of user preference for link-based vs. text-based recommendations of wikipedia articles (2021). arXiv:2109.07791

Page, M.J., McKenzie, J.E., Bossuyt, P.M., Boutron, I., Hoffmann, T.C., Mulrow, C.D., Shamseer, L., Tetzlaff, J.M., Akl, E.A., Brennan, S.E., Chou, R., Glanville, J., Grimshaw, J.M., Hróbjartsson, A., Lalu, M.M., Li, T., Loder, E.W., Mayo-Wilson, E., McDonald, S., McGuinness, L.A., Stewart, L.A., Thomas, J., Tricco, A.C., Welch, V.A., Whiting, P., Moher, D.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372 (2021). https://www.bmj.com/content/372/bmj.n71

Patra, B.G., Maroufy, V., Soltanalizadeh, B., Deng, N., Zheng, W.J., Roberts, K., Wu, H.: A content-based literature recommendation system for datasets to improve data reusability: a case study on gene expression omnibus (geo) datasets. J. Biomed. Inform. 104 , 103399 (2020). https://doi.org/10.1016/j.jbi.2020.103399

Radev, D.R., Joseph, M.T., Gibson, B., Muthukrishnan, P.: A bibliometric and network analysis of the field of computational linguistics. J. Am. Soc. Inform. Sci. Technol. (2009). https://doi.org/10.1002/asi.23394

Radev, D.R., Muthukrishnan, P., Qazvinian, V.: The ACL anthology network corpus. In: NLPIR4DL’09 (2009). https://doi.org/10.5555/1699750.1699759

Radev, D.R., Muthukrishnan, P., Qazvinian, V., Abu-Jbara, A.: The ACL anthology network corpus. Lang. Resour. Eval. (2013). https://doi.org/10.1007/s10579-012-9211-2

Rahdari, B., Brusilovsky, P.: User-controlled hybrid recommendation for academic papers. In: IUI Companion’19, pp. 99–100. ACM (2019). https://doi.org/10.1145/3308557.3308717

Rahdari, B., Brusilovsky, P., Thaker, K., Barria-Pineda, J.: Knowledge-driven wikipedia article recommendation for electronic textbooks. In: EC-TEL’20, LNCS. 12315 , 363–368. Springer (2020). https://doi.org/10.1007/978-3-030-57717-9_28

Renuka, S., Raj Kiran, G.S.S., Rohit, P.: An unsupervised content-based article recommendation system using natural language processing. Data Intell. Cogn. Inform. (2021). https://doi.org/10.1007/978-981-15-8530-2_13

Safaryan, A., Filchenkov, P., Yan, W., Kutuzov, A., Nikishina, I.: Semantic recommendation system for bilingual corpus of academic papers. In: AIST’20, Communications in Computer and Information Science. 1357 , 22–36. Springer (2020). https://doi.org/10.1007/978-3-030-71214-3_3

Sakib, N., Ahmad, R.B., Ahsan, M., Based, M.A., Haruna, K., Haider, J., Gurusamy, S.: A hybrid personalized scientific paper recommendation approach integrating public contextual metadata. IEEE Access 9 , 83080–83091 (2021). https://doi.org/10.1109/ACCESS.2021.3086964

Sakib, N., Ahmad, R.B., Haruna, K.: A collaborative approach toward scientific paper recommendation using citation context. IEEE Access 8 , 51246–51255 (2020). https://doi.org/10.1109/ACCESS.2020.2980589

Samad, A., Islam, M.A., Iqbal, M.A., Aleem, M.: Centrality-based paper citation recommender system. EAI Endorsed Trans. Ind. Networks Intell. Syst. 6 (19), e2 (2019). https://doi.org/10.4108/eai.13-6-2019.159121

Schaer, P., Breuer, T., Castro, L.J., Wolff, B., Schaible, J., Tavakolpoursaleh, N.: Overview of LiLAS 2021—living labs for academic search (extended overview). In: CLEF’21, CEUR Workshop Proceedings. 2936 , 1668–1699. CEUR-WS.org (2021). http://ceur-ws.org/Vol-2936/paper-143.pdf

Shahid, A., Afzal, M.T., Abdar, M., Basiri, M.E., Zhou, X., Yen, N.Y., Chang, J.: Insights into relevant knowledge extraction techniques: a comprehensive review. J. Supercomput. 76 (3), 1695–1733 (2020). https://doi.org/10.1007/s11227-019-03009-y

Shahid, A., Afzal, M.T., Alharbi, A., Aljuaid, H., Al-Otaibi, S.: In-text citation’s frequencies-based recommendations of relevant research papers. PeerJ Comput. Sci. 7 , e524 (2021). https://doi.org/10.7717/peerj-cs.524

Shahid, A., Afzal, M.T., Saleem, M.Q., Idrees, M.S.E., Omer, M.K.: Extension of direct citation model using in-text citations. Comput. Mater. Continua 66 (3), 3121–3138 (2021). https://doi.org/10.32604/cmc.2021.013809

Sharma, B., Willis, V.C., Huettner, C.S., Beaty, K., Snowdon, J.L., Xue, S., South, B.R., Jackson, G.P., Weeraratne, D., Michelini, V.: Predictive article recommendation using natural language processing and machine learning to support evidence updates in domain-specific knowledge graphs. JAMIA Open 3 (3), 332–337 (2020). https://doi.org/10.1093/jamiaopen/ooaa028

Shi, H., Ma, W., Zhang, X., Jiang, J., Liu, Y., Chen, S.: A hybrid paper recommendation method by using heterogeneous graph and metadata. In: IJCNN’20, pp. 1–8. IEEE (2020). https://doi.org/10.1109/IJCNN48605.2020.9206733

Sinha, A., Shen, Z., Song, Y., Ma, H., Eide, D., Hsu, B.P., Wang, K.: An overview of microsoft academic service (MAS) and applications. In: WWW’15, pp. 243–246. ACM (2015). https://doi.org/10.1145/2740908.2742839

Subathra, P., Kumar, P.N.: Recommending research article based on user queries using latent dirichlet allocation. in: soft computing and signal processing, pp. 163–175. Springer Singapore (2020). https://doi.org/10.1007/978-981-15-2475-2_15

Sugiyama, K., Kan, M.: Scholarly paper recommendation via user’s recent research interests. In: JCDL’10, pp. 29–38. ACM (2010). https://doi.org/10.1145/1816123.1816129

Sugiyama, K., Kan, M.: Serendipitous recommendation for scholarly papers considering relations among researchers. In: JCDL’11, pp. 307–310. ACM (2011). https://doi.org/10.1145/1998076.1998133

Sugiyama, K., Kan, M.: Exploiting potential citation papers in scholarly paper recommendation. In: JCDL’13, pp. 153–162. ACM (2013). https://doi.org/10.1145/2467696.2467701

Sugiyama, K., Kan, M.: A comprehensive evaluation of scholarly paper recommendation using potential citation papers. Int. J. Digit. Libr. 16 (2), 91–109 (2015). https://doi.org/10.1007/s00799-014-0122-2

Symeonidis, P., Kirjackaja, L., Zanker, M.: Session-based news recommendations using SimRank on multi-modal graphs. Expert Syst. Appl. 180 , 115028 (2021). https://doi.org/10.1016/j.eswa.2021.115028

Tang, H., Liu, B., Qian, J.: Content-based and knowledge graph-based paper recommendation: exploring user preferences with the knowledge graphs for scientific paper recommendation. Concurr. Comput. Pract. Exp. (2021). https://doi.org/10.1002/cpe.6227

Tang, J., Zhang, J., Yao, L., Li, J., Zhang, L., Su, Z.: ArnetMiner: extraction and mining of academic social networks. In: SIGKDD’08, pp. 990–998. ACM (2008). https://doi.org/10.1145/1401890.1402008

Tanner, W., Akbas, E., Hasan, M.: Paper recommendation based on citation relation. In: Big Data’19, pp. 3053–3059. IEEE (2019). https://doi.org/10.1109/BigData47090.2019.9006200

Tao, M., Yang, X., Gu, G., Li, B.: Paper recommend based on LDA and PageRank, pp. 571–584. Springer (2020). https://doi.org/10.1007/978-981-15-8101-4_51

Waheed, W., Imran, M., Raza, B., Malik, A.K., Khattak, H.A.: A hybrid approach toward research paper recommendation using centrality measures and author ranking. IEEE Access 7 , 33145–33158 (2019). https://doi.org/10.1109/ACCESS.2019.2900520

Wang, B., Weng, Z., Wang, Y.: A novel paper recommendation method empowered by knowledge graph: for research beginners (2021). arXiv:2103.08819

Wang, G., Wang, H., Yang, Y., Xu, D., Yang, J., Yue, F.: Group article recommendation based on ER rule in scientific social networks. Appl. Soft Comput. 110 , 107631 (2021). https://doi.org/10.1016/j.asoc.2021.107631

Wang, G., Zhang, X., Wang, H., Chu, Y., Shao, Z.: Group-oriented paper recommendation with probabilistic matrix factorization and evidential reasoning in scientific social network. IEEE Trans. Syst. Man Cybern. Syst. (2021). https://doi.org/10.1109/TSMC.2021.3072426

Wang, H., Chen, B., Li, W.: Collaborative topic regression with social regularization for tag recommendation. In: IJCAI’13, pp. 2719–2725. IJCAI/AAAI (2013). http://www.aaai.org/ocs/index.php/IJCAI/IJCAI13/paper/view/7006

Wang, X., Xu, H., Tan, W., Wang, Z., Xu, X.: Scholarly paper recommendation via related path analysis in knowledge graph. In: ICSS’20, pp. 36–43. IEEE (2020). https://doi.org/10.1109/ICSS50103.2020.00014

Wu, J., Kim, K., Giles, C.L.: CiteSeerX: 20 years of service to scholarly big data. In: AIDR’19, pp. 1:1–1:4. ACM (2019). https://doi.org/10.1145/3359115.3359119

Xie, Y., Sun, Y., Bertino, E.: Learning domain semantics and cross-domain correlations for paper recommendation. In: SIGIR’21, pp. 706–715. ACM (2021). https://doi.org/10.1145/3404835.3462975

Xie, Y., Wang, S., Pan, W., Tang, H., Sun, Y.: Embedding based personalized new paper recommendation. In: Chinese CSCW’21, pp. 558–570. Springer, Singapore (2021). https://doi.org/10.1007/978-981-16-2540-4_40

Yang, Q., Li, Z., Liu, A., Liu, G., Zhao, L., Zhang, X., Zhang, M., Zhou, X.: A novel hybrid publication recommendation system using compound information. World Wide Web 22 (6), 2499–2517 (2019). https://doi.org/10.1007/s11280-019-00687-9

Yu, M., Hu, Y., Li, X., Zhao, M., Xu, T., Liu, H., Xu, L., Yu, R.: Paper recommendation with item-level collaborative memory network. In: KSEM’19, LNCS. 11775 , 141–152. Springer (2019). https://doi.org/10.1007/978-3-030-29551-6_13

Zavrel, J., Grotov, A., Mitnik, J.: Building a platform for ensemble-based personalized research literature recommendations for AI and data science at zeta alpha. pp 536–537. Association for Computing Machinery (2021). https://doi.org/10.1145/3460231.3474619

Zhang, Y., Chen, X.: Explainable recommendation: a survey and new perspectives. Found. Trends Inf. Retr. 14 (1), 1–101 (2020). https://doi.org/10.1561/1500000066

Zhang, Y., Wang, M., Gottwalt, F., Saberi, M., Chang, E.: Ranking scientific articles based on bibliometric networks with a weighting scheme. J. Inform. 13 (2), 616–634 (2019). https://doi.org/10.1016/j.joi.2019.03.013

Zhang, Y., Wang, M., Saberi, M., Chang, E.: Towards expert preference on academic article recommendation using bibliometric networks. In: PAKDD’20, pp. 11–19. Springer (2020). https://doi.org/10.1007/978-3-030-60470-7_2

Zhao, X., Kang, H., Feng, T., Meng, C., Nie, Z.: A hybrid model based on LFM and BiGRU toward research paper recommendation. IEEE Access 8 , 188628–188640 (2020). https://doi.org/10.1109/ACCESS.2020.3031281

Zhu, Y., Lin, Q., Lu, H., Shi, K., Qiu, P., Niu, Z.: Recommending scientific paper via heterogeneous knowledge embedding based attentive recurrent neural networks. Knowl. Based Syst. 215 , 106744 (2021). https://doi.org/10.1016/j.knosys.2021.106744

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Cologne University of Applied Sciences, Cologne, Germany

Christin Katharina Kreutz

Trier University, Trier, Germany

Ralf Schenkel

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Christin Katharina Kreutz .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Kreutz, C.K., Schenkel, R. Scientific paper recommendation systems: a literature review of recent publications. Int J Digit Libr 23 , 335–369 (2022). https://doi.org/10.1007/s00799-022-00339-w

Download citation

Received : 23 December 2021

Revised : 07 September 2022

Accepted : 16 September 2022

Published : 05 October 2022

Issue Date : December 2022

DOI : https://doi.org/10.1007/s00799-022-00339-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Paper recommendation system
  • Publication suggestion
  • Literature review
  • Find a journal
  • Publish with us
  • Track your research

Help | Advanced Search

Computer Science > Data Structures and Algorithms

Title: teaching algorithm design: a literature review.

Abstract: Algorithm design is a vital skill developed in most undergraduate Computer Science (CS) programs, but few research studies focus on pedagogy related to algorithms coursework. To understand the work that has been done in the area, we present a systematic survey and literature review of CS Education studies. We search for research that is both related to algorithm design and evaluated on undergraduate-level students. Across all papers in the ACM Digital Library prior to August 2023, we only find 94 such papers. We first classify these papers by topic, evaluation metric, evaluation methods, and intervention target. Through our classification, we find a broad sparsity of papers which indicates that many open questions remain about teaching algorithm design, with each algorithm topic only being discussed in between 0 and 10 papers. We also note the need for papers using rigorous research methods, as only 38 out of 88 papers presenting quantitative data use statistical tests, and only 15 out of 45 papers presenting qualitative data use a coding scheme. Only 17 papers report controlled trials. We then synthesize the results of the existing literature to give insights into what the corpus reveals about how we should teach algorithms. Much of the literature explores implementing well-established practices, such as active learning or automated assessment, in the algorithms classroom. However, there are algorithms-specific results as well: a number of papers find that students may under-utilize certain algorithmic design techniques, and studies describe a variety of ways to select algorithms problems that increase student engagement and learning. The results we present, along with the publicly available set of papers collected, provide a detailed representation of the current corpus of CS Education work related to algorithm design and can orient further research in the area.

Submission history

Access paper:.

  • HTML (experimental)
  • Other Formats

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

BibSonomy logo

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

  • Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • 01 May 2024

Plagiarism in peer-review reports could be the ‘tip of the iceberg’

  • Jackson Ryan 0

Jackson Ryan is a freelance science journalist in Sydney, Australia.

You can also search for this author in PubMed   Google Scholar

Time pressures and a lack of confidence could be prompting reviewers to plagiarize text in their reports. Credit: Thomas Reimer/Zoonar via Alamy

Mikołaj Piniewski is a researcher to whom PhD students and collaborators turn when they need to revise or refine a manuscript. The hydrologist, at the Warsaw University of Life Sciences, has a keen eye for problems in text — a skill that came in handy last year when he encountered some suspicious writing in peer-review reports of his own paper.

Last May, when Piniewski was reading the peer-review feedback that he and his co-authors had received for a manuscript they’d submitted to an environmental-science journal, alarm bells started ringing in his head. Comments by two of the three reviewers were vague and lacked substance, so Piniewski decided to run a Google search, looking at specific phrases and quotes the reviewers had used.

To his surprise, he found the comments were identical to those that were already available on the Internet, in multiple open-access review reports from publishers such as MDPI and PLOS. “I was speechless,” says Piniewski. The revelation caused him to go back to another manuscript that he had submitted a few months earlier, and dig out the peer-review reports he received for that. He found more plagiarized text. After e-mailing several collaborators, he assembled a team to dig deeper.

literature review on scientific paper

Meet this super-spotter of duplicated images in science papers

The team published the results of its investigation in Scientometrics in February 1 , examining dozens of cases of apparent plagiarism in peer-review reports, identifying the use of identical phrases across reports prepared for 19 journals. The team discovered exact quotes duplicated across 50 publications, saying that the findings are just “the tip of the iceberg” when it comes to misconduct in the peer-review system.

Dorothy Bishop, a former neuroscientist at the University of Oxford, UK, who has turned her attention to investigating research misconduct, was “favourably impressed” by the team’s analysis. “I felt the way they approached it was quite useful and might be a guide for other people trying to pin this stuff down,” she says.

Peer review under review

Piniewski and his colleagues conducted three analyses. First, they uploaded five peer-review reports from the two manuscripts that his laboratory had submitted to a rudimentary online plagiarism-detection tool . The reports had 44–100% similarity to previously published online content. Links were provided to the sources in which duplications were found.

The researchers drilled down further. They broke one of the suspicious peer-review reports down to fragments of one to three sentences each and searched for them on Google. In seconds, the search engine returned a number of hits: the exact phrases appeared in 22 open peer-review reports, published between 2021 and 2023.

The final analysis provided the most worrying results. They took a single quote — 43 words long and featuring multiple language errors, including incorrect capitalization — and pasted it into Google. The search revealed that the quote, or variants of it, had been used in 50 peer-review reports.

Predominantly, these reports were from journals published by MDPI, PLOS and Elsevier, and the team found that the amount of duplication increased year-on-year between 2021 and 2023. Whether this is because of an increase in the number of open-access peer-review reports during this time or an indication of a growing problem is unclear — but Piniewski thinks that it could be a little bit of both.

Why would a peer reviewer use plagiarized text in their report? The team says that some might be attempting to save time , whereas others could be motivated by a lack of confidence in their writing ability, for example, if they aren’t fluent in English.

The team notes that there are instances that might not represent misconduct. “A tolerable rephrasing of your own words from a different review? I think that’s fine,” says Piniewski. “But I imagine that most of these cases we found are actually something else.”

The source of the problem

Duplication and manipulation of peer-review reports is not a new phenomenon. “I think it’s now increasingly recognized that the manipulation of the peer-review process, which was recognized around 2010, was probably an indication of paper mills operating at that point,” says Jennifer Byrne, director of biobanking at New South Wales Health in Sydney, Australia, who also studies research integrity in scientific literature.

Paper mills — organizations that churn out fake research papers and sell authorships to turn a profit — have been known to tamper with reviews to push manuscripts through to publication, says Byrne.

literature review on scientific paper

The fight against fake-paper factories that churn out sham science

However, when Bishop looked at Piniewski’s case, she could not find any overt evidence of paper-mill activity. Rather, she suspects that journal editors might be involved in cases of peer-review-report duplication and suggests studying the track records of those who’ve allowed inadequate or plagiarized reports to proliferate.

Piniewski’s team is also concerned about the rise of duplications as generative artificial intelligence (AI) becomes easier to access . Although his team didn’t look for signs of AI use, its ability to quickly ingest and rephrase large swathes of text is seen as an emerging issue.

A preprint posted in March 2 showed evidence of researchers using AI chatbots to assist with peer review, identifying specific adjectives that could be hallmarks of AI-written text in peer-review reports .

Bishop isn’t as concerned as Piniewski about AI-generated reports, saying that it’s easy to distinguish between AI-generated text and legitimate reviewer commentary. “The beautiful thing about peer review,” she says, is that it is “one thing you couldn’t do a credible job with AI”.

Preventing plagiarism

Publishers seem to be taking action. Bethany Baker, a media-relations manager at PLOS, who is based in Cambridge, UK, told Nature Index that the PLOS Publication Ethics team “is investigating the concerns raised in the Scientometrics article about potential plagiarism in peer reviews”.

literature review on scientific paper

How big is science’s fake-paper problem?

An Elsevier representative told Nature Index that the publisher “can confirm that this matter has been brought to our attention and we are conducting an investigation”.

In a statement, the MDPI Research Integrity and Publication Ethics Team said that it has been made aware of potential misconduct by reviewers in its journals and is “actively addressing and investigating this issue”. It did not confirm whether this was related to the Scientometrics article.

One proposed solution to the problem is ensuring that all submitted reviews are checked using plagiarism-detection software. In 2022, exploratory work by Adam Day, a data scientist at Sage Publications, based in Thousand Oaks, California, identified duplicated text in peer-review reports that might be suggestive of paper-mill activity. Day offered a similar solution of using anti-plagiarism software , such as Turnitin.

Piniewski expects the problem to get worse in the coming years, but he hasn’t received any unusual peer-review reports since those that originally sparked his research. Still, he says that he’s now even more vigilant. “If something unusual occurs, I will spot it.”

doi: https://doi.org/10.1038/d41586-024-01312-0

Piniewski, M., Jarić, I., Koutsoyiannis, D. & Kundzewicz, Z. W. Scientometrics https://doi.org/10.1007/s11192-024-04960-1 (2024).

Article   Google Scholar  

Liang, W. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2403.07183 (2024).

Download references

Related Articles

literature review on scientific paper

  • Peer review
  • Research management

Algorithm ranks peer reviewers by reputation — but critics warn of bias

Algorithm ranks peer reviewers by reputation — but critics warn of bias

Nature Index 25 APR 24

Researchers want a ‘nutrition label’ for academic-paper facts

Researchers want a ‘nutrition label’ for academic-paper facts

Nature Index 17 APR 24

Rwanda 30 years on: understanding the horror of genocide

Rwanda 30 years on: understanding the horror of genocide

Editorial 09 APR 24

Structure peer review to make it more robust

Structure peer review to make it more robust

World View 16 APR 24

Is ChatGPT corrupting peer review? Telltale words hint at AI use

Is ChatGPT corrupting peer review? Telltale words hint at AI use

News 10 APR 24

Mount Etna’s spectacular smoke rings and more — April’s best science images

Mount Etna’s spectacular smoke rings and more — April’s best science images

News 03 MAY 24

How reliable is this research? Tool flags papers discussed on PubPeer

How reliable is this research? Tool flags papers discussed on PubPeer

News 29 APR 24

Faculty Positions in Neurobiology, Westlake University

We seek exceptional candidates to lead vigorous independent research programs working in any area of neurobiology.

Hangzhou, Zhejiang, China

School of Life Sciences, Westlake University

literature review on scientific paper

Seeking Global Talents, the International School of Medicine, Zhejiang University

Welcome to apply for all levels of professors based at the International School of Medicine, Zhejiang University.

Yiwu, Zhejiang, China

International School of Medicine, Zhejiang University

literature review on scientific paper

Assistant, Associate, or Full Professor

Athens, Georgia

University of Georgia

literature review on scientific paper

Associate Professor - Synthetic Biology

Position Summary  We seek an Associate Professor in the department of Synthetic Biology (jcvi.org/research/synthetic-biology). We invite applicatio...

Rockville, Maryland

J. Craig Venter Institute

literature review on scientific paper

Associate or Senior Editor (microbial genetics, evolution, and epidemiology) Nature Communications

Job Title: Associate or Senior Editor (microbial genetics, evolution, and epidemiology), Nature Communications Locations: London, New York, Philade...

New York (US)

Springer Nature Ltd

literature review on scientific paper

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies


  1. Example of a Literature Review for a Research Paper by

    literature review on scientific paper

  2. How to Outline A Literature Review (Plus Examples You Can Use)

    literature review on scientific paper

  3. Literature Review Outline Template

    literature review on scientific paper

  4. 😂 Literature review introduction paragraph example. How to Write the

    literature review on scientific paper

  5. How To Make A Literature Review For A Research Paper

    literature review on scientific paper

  6. (PDF) Descriptive Review for Research Paper Format

    literature review on scientific paper


  1. Chapter two

  2. What is literature review?

  3. Systematic Literature Review: An Introduction [Urdu/Hindi]

  4. How to Do a Good Literature Review for Research Paper and Thesis

  5. all important Questions Business Research Methods |BBS

  6. How scientific papers are published


  1. Ten Simple Rules for Writing a Literature Review

    Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications .For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively .Given such mountains of papers, scientists cannot be expected to examine in detail every ...

  2. How to Write a Literature Review

    Examples of literature reviews. Step 1 - Search for relevant literature. Step 2 - Evaluate and select sources. Step 3 - Identify themes, debates, and gaps. Step 4 - Outline your literature review's structure. Step 5 - Write your literature review.

  3. A Step-by-Step Guide to Writing a Scientific Review Article

    Abstract. Scientific review articles are comprehensive, focused reviews of the scientific literature written by subject matter experts. The task of writing a scientific review article can seem overwhelming; however, it can be managed by using an organized approach and devoting sufficient time to the process.

  4. Writing a Literature Review

    Writing a Literature Review. A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels ...

  5. How to write a superb literature review

    The best proposals are timely and clearly explain why readers should pay attention to the proposed topic. It is not enough for a review to be a summary of the latest growth in the literature: the ...

  6. Writing a Literature Review

    When this happens, - the ultimate good of science can be realised. A literature review is structured differently from an original research article. It is developed based on themes, rather than stages of the scientific method. ... Note that this is an example using only two papers - most literature reviews would be presenting information on ...

  7. Writing a Literature Review Research Paper: A step-by-step approach

    A literature review is a surveys scholarly articles, books and other sources relevant to a particular. issue, area of research, or theory, and by so doing, providing a description, summary, and ...

  8. How to write a good scientific review article

    Literature reviews are valuable resources for the scientific community. With research accelerating at an unprecedented speed in recent years and more and more original papers being published, review articles have become increasingly important as a means to keep up-to-date with developments in a particular area of research.

  9. What is a Literature Review? How to Write It (with Examples)

    A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship ...

  10. How-to conduct a systematic literature review: A quick guide for

    A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following: • Clearly defined strategies to follow for a ...

  11. 5. The Literature Review

    A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories.A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that ...

  12. Literature review as a research methodology: An ...

    This paper discusses literature review as a methodology for conducting research and offers an overview of different types of reviews, as well as some guidelines to how to both conduct and evaluate a literature review paper. ... Provides guidelines for publishing review papers in the Journal of the Academy of Marketing Science. 2.1.1. Systematic ...

  13. PDF Conducting a Literature Review

    The Literature Research Workflow Web of Science The world's largest and highest quality ... Essential Science Indicators Reveals emerging science trends as well as influential individuals, institutions, papers, journals, and countries across 22 categories of research. ... Literature Review A literature review is a survey of scholarly sources ...

  14. Literature Reviews

    A literature review is a body of text that aims to review the critical points of current knowledge on a particular topic. Most often associated with science-oriented literature, such as a thesis, the literature review usually proceeds a research proposal, methodology and results section. ... The literature review is the section of your paper in ...

  15. How to Write a Good Scientific Literature Review

    Here you have a to-do list to help you write your review: A scientific literature review usually includes a title, abstract, index, introduction, corpus, bibliography, and appendices (if needed). Present the problem clearly. Mention the paper's methodology, research methods, analysis, instruments, etc. Present literature review examples that ...


    WHAT IS A LITERATURE REVIEW? PURPOSES OF A LITERATURE REVIEW. 1. orient your reader by defining key concepts (theoretical) and/or providing relevant background (empirical) 2. "motivate" your research, i.e. demonstrating the relevance of your project. ¡ contribute effectively to science, a collective knowledge-building enterprise.

  17. How to write a review paper

    a critical review of the relevant literature and then ensuring that their research design, methods, results, and conclusions follow logically from these objectives (Maier, 2013). There exist a number of papers devoted to instruction on how to write a good review paper. Among the most . useful for scientific reviews, in my estimation, are those by

  18. Scientific paper recommendation systems: a literature review of recent

    Scientific writing builds upon already published papers. Manual identification of publications to read, cite or consider as related papers relies on a researcher's ability to identify fitting keywords or initial papers from which a literature search can be started. The rapidly increasing amount of papers has called for automatic measures to find the desired relevant publications, so-called ...

  19. [2405.00832] Teaching Algorithm Design: A Literature Review

    Algorithm design is a vital skill developed in most undergraduate Computer Science (CS) programs, but few research studies focus on pedagogy related to algorithms coursework. To understand the work that has been done in the area, we present a systematic survey and literature review of CS Education studies. We search for research that is both related to algorithm design and evaluated on ...

  20. Ten simple rules for reading a scientific paper

    1. You are new to reading scientific papers. 1. For each panel of each figure, focus particularly on the questions outlined in Rule 3. 2. You are entering a new field and want to learn what is important in that field.

  21. Plagiarism in peer-review reports could be the 'tip of the iceberg'

    Peer review under review. ... who also studies research integrity in scientific literature. Paper mills — organizations that churn out fake research papers and sell authorships to turn a profit ...

  22. [PDF] The Effectiveness of Betel Leaf Water on ...

    Method: This literature review used the PRISMA flow chart 2020 to describe the efficacy of betel leaf water on perineal wound healing in post-partum mothers. Result: A review of eight articles showed that using betel leaf water significantly improved the perineal wound healing process in post-partum women with an average P-Value of <0.05.