literature review on recommender systems

• Survey paper
• Open access
• Published: 03 May 2022

A systematic review and research perspective on recommender systems

• Deepjyoti Roy   ORCID: orcid.org/0000-0002-8020-7145 1 &
• Mala Dutta 1  

Journal of Big Data volume  9 , Article number:  59 ( 2022 ) Cite this article

62k Accesses

94 Citations

6 Altmetric

Metrics details

Recommender systems are efficient tools for filtering online information, which is widespread owing to the changing habits of computer users, personalization trends, and emerging access to the internet. Even though the recent recommender systems are eminent in giving precise recommendations, they suffer from various limitations and challenges like scalability, cold-start, sparsity, etc. Due to the existence of various techniques, the selection of techniques becomes a complex work while building application-focused recommender systems. In addition, each technique comes with its own set of features, advantages and disadvantages which raises even more questions, which should be addressed. This paper aims to undergo a systematic review on various recent contributions in the domain of recommender systems, focusing on diverse applications like books, movies, products, etc. Initially, the various applications of each recommender system are analysed. Then, the algorithmic analysis on various recommender systems is performed and a taxonomy is framed that accounts for various components required for developing an effective recommender system. In addition, the datasets gathered, simulation platform, and performance metrics focused on each contribution are evaluated and noted. Finally, this review provides a much-needed overview of the current state of research in this field and points out the existing gaps and challenges to help posterity in developing an efficient recommender system.

Introduction

The recent advancements in technology along with the prevalence of online services has offered more abilities for accessing a huge amount of online information in a faster manner. Users can post reviews, comments, and ratings for various types of services and products available online. However, the recent advancements in pervasive computing have resulted in an online data overload problem. This data overload complicates the process of finding relevant and useful content over the internet. The recent establishment of several procedures having lower computational requirements can however guide users to the relevant content in a much easy and fast manner. Because of this, the development of recommender systems has recently gained significant attention. In general, recommender systems act as information filtering tools, offering users suitable and personalized content or information. Recommender systems primarily aim to reduce the user’s effort and time required for searching relevant information over the internet.

Nowadays, recommender systems are being increasingly used for a large number of applications such as web [ 1 , 67 , 70 ], books [ 2 ], e-learning [ 4 , 16 , 61 ], tourism [ 5 , 8 , 78 ], movies [ 66 ], music [ 79 ], e-commerce, news, specialized research resources [ 65 ], television programs [ 72 , 81 ], etc. It is therefore important to build high-quality and exclusive recommender systems for providing personalized recommendations to the users in various applications. Despite the various advances in recommender systems, the present generation of recommender systems requires further improvements to provide more efficient recommendations applicable to a broader range of applications. More investigation of the existing latest works on recommender systems is required which focus on diverse applications.

There is hardly any review paper that has categorically synthesized and reviewed the literature of all the classification fields and application domains of recommender systems. The few existing literature reviews in the field cover just a fraction of the articles or focus only on selected aspects such as system evaluation. Thus, they do not provide an overview of the application field, algorithmic categorization, or identify the most promising approaches. Also, review papers often neglect to analyze the dataset description and the simulation platforms used. This paper aims to fulfil this significant gap by reviewing and comparing existing articles on recommender systems based on a defined classification framework, their algorithmic categorization, simulation platforms used, applications focused, their features and challenges, dataset description and system performance. Finally, we provide researchers and practitioners with insight into the most promising directions for further investigation in the field of recommender systems under various applications.

In essence, recommender systems deal with two entities—users and items, where each user gives a rating (or preference value) to an item (or product). User ratings are generally collected by using implicit or explicit methods. Implicit ratings are collected indirectly from the user through the user’s interaction with the items. Explicit ratings, on the other hand, are given directly by the user by picking a value on some finite scale of points or labelled interval values. For example, a website may obtain implicit ratings for different items based on clickstream data or from the amount of time a user spends on a webpage and so on. Most recommender systems gather user ratings through both explicit and implicit methods. These feedbacks or ratings provided by the user are arranged in a user-item matrix called the utility matrix as presented in Table 1 .

The utility matrix often contains many missing values. The problem of recommender systems is mainly focused on finding the values which are missing in the utility matrix. This task is often difficult as the initial matrix is usually very sparse because users generally tend to rate only a small number of items. It may also be noted that we are interested in only the high user ratings because only such items would be suggested back to the users. The efficiency of a recommender system greatly depends on the type of algorithm used and the nature of the data source—which may be contextual, textual, visual etc.

Types of recommender systems

Recommender systems are broadly categorized into three different types viz. content-based recommender systems, collaborative recommender systems and hybrid recommender systems. A diagrammatic representation of the different types of recommender systems is given in Fig.  1 .

Content-based recommender system

In content-based recommender systems, all the data items are collected into different item profiles based on their description or features. For example, in the case of a book, the features will be author, publisher, etc. In the case of a movie, the features will be the movie director, actor, etc. When a user gives a positive rating to an item, then the other items present in that item profile are aggregated together to build a user profile. This user profile combines all the item profiles, whose items are rated positively by the user. Items present in this user profile are then recommended to the user, as shown in Fig.  2 .

One drawback of this approach is that it demands in-depth knowledge of the item features for an accurate recommendation. This knowledge or information may not be always available for all items. Also, this approach has limited capacity to expand on the users' existing choices or interests. However, this approach has many advantages. As user preferences tend to change with time, this approach has the quick capability of dynamically adapting itself to the changing user preferences. Since one user profile is specific only to that user, this algorithm does not require the profile details of any other users because they provide no influence in the recommendation process. This ensures the security and privacy of user data. If new items have sufficient description, content-based techniques can overcome the cold-start problem i.e., this technique can recommend an item even when that item has not been previously rated by any user. Content-based filtering approaches are more common in systems like personalized news recommender systems, publications, web pages recommender systems, etc.

Collaborative filtering-based recommender system

Collaborative approaches make use of the measure of similarity between users. This technique starts with finding a group or collection of user X whose preferences, likes, and dislikes are similar to that of user A. X is called the neighbourhood of A. The new items which are liked by most of the users in X are then recommended to user A. The efficiency of a collaborative algorithm depends on how accurately the algorithm can find the neighbourhood of the target user. Traditionally collaborative filtering-based systems suffer from the cold-start problem and privacy concerns as there is a need to share user data. However, collaborative filtering approaches do not require any knowledge of item features for generating a recommendation. Also, this approach can help to expand on the user’s existing interests by discovering new items. Collaborative approaches are again divided into two types: memory-based approaches and model-based approaches.

Memory-based collaborative approaches recommend new items by taking into consideration the preferences of its neighbourhood. They make use of the utility matrix directly for prediction. In this approach, the first step is to build a model. The model is equal to a function that takes the utility matrix as input.

Model = f (utility matrix)

Then recommendations are made based on a function that takes the model and user profile as input. Here we can make recommendations only to users whose user profile belongs to the utility matrix. Therefore, to make recommendations for a new user, the user profile must be added to the utility matrix, and the similarity matrix should be recomputed, which makes this technique computation heavy.

Recommendation = f (defined model, user profile) where user profile  ∈  utility matrix

Memory-based collaborative approaches are again sub-divided into two types: user-based collaborative filtering and item-based collaborative filtering. In the user-based approach, the user rating of a new item is calculated by finding other users from the user neighbourhood who has previously rated that same item. If a new item receives positive ratings from the user neighbourhood, the new item is recommended to the user. Figure  3 depicts the user-based filtering approach.

User-based collaborative filtering

In the item-based approach, an item-neighbourhood is built consisting of all similar items which the user has rated previously. Then that user’s rating for a different new item is predicted by calculating the weighted average of all ratings present in a similar item-neighbourhood as shown in Fig.  4 .

Item-based collaborative filtering

Model-based systems use various data mining and machine learning algorithms to develop a model for predicting the user’s rating for an unrated item. They do not rely on the complete dataset when recommendations are computed but extract features from the dataset to compute a model. Hence the name, model-based technique. These techniques also need two steps for prediction—the first step is to build the model, and the second step is to predict ratings using a function (f) which takes the model defined in the first step and the user profile as input.

Recommendation = f (defined model, user profile) where user profile  ∉  utility matrix

Model-based techniques do not require adding the user profile of a new user into the utility matrix before making predictions. We can make recommendations even to users that are not present in the model. Model-based systems are more efficient for group recommendations. They can quickly recommend a group of items by using the pre-trained model. The accuracy of this technique largely relies on the efficiency of the underlying learning algorithm used to create the model. Model-based techniques are capable of solving some traditional problems of recommender systems such as sparsity and scalability by employing dimensionality reduction techniques [ 86 ] and model learning techniques.

Hybrid filtering

A hybrid technique is an aggregation of two or more techniques employed together for addressing the limitations of individual recommender techniques. The incorporation of different techniques can be performed in various ways. A hybrid algorithm may incorporate the results achieved from separate techniques, or it can use content-based filtering in a collaborative method or use a collaborative filtering technique in a content-based method. This hybrid incorporation of different techniques generally results in increased performance and increased accuracy in many recommender applications. Some of the hybridization approaches are meta-level, feature-augmentation, feature-combination, mixed hybridization, cascade hybridization, switching hybridization and weighted hybridization [ 86 ]. Table 2 describes these approaches.

Recommender system challenges

This section briefly describes the various challenges present in current recommender systems and offers different solutions to overcome these challenges.

Cold start problem

The cold start problem appears when the recommender system cannot draw any inference from the existing data, which is insufficient. Cold start refers to a condition when the system cannot produce efficient recommendations for the cold (or new) users who have not rated any item or have rated a very few items. It generally arises when a new user enters the system or new items (or products) are inserted into the database. Some solutions to this problem are as follows: (a) Ask new users to explicitly mention their item preference. (b) Ask a new user to rate some items at the beginning. (c) Collect demographic information (or meta-data) from the user and recommend items accordingly.

Shilling attack problem

This problem arises when a malicious user fakes his identity and enters the system to give false item ratings [ 87 ]. Such a situation occurs when the malicious user wants to either increase or decrease some item’s popularity by causing a bias on selected target items. Shilling attacks greatly reduce the reliability of the system. One solution to this problem is to detect the attackers quickly and remove the fake ratings and fake user profiles from the system.

Synonymy problem

This problem arises when similar or related items have different entries or names, or when the same item is represented by two or more names in the system [ 78 ]. For example, babywear and baby cloth. Many recommender systems fail to distinguish these differences, hence reducing their recommendation accuracy. To alleviate this problem many methods are used such as demographic filtering, automatic term expansion and Singular Value Decomposition [ 76 ].

Latency problem

The latency problem is specific to collaborative filtering approaches and occurs when new items are frequently inserted into the database. This problem is characterized by the system’s failure to recommend new items. This happens because new items must be reviewed before they can be recommended in a collaborative filtering environment. Using content-based filtering may resolve this issue, but it may introduce overspecialization and decrease the computing time and system performance. To increase performance, the calculations can be done in an offline environment and clustering-based techniques can be used [ 76 ].

Sparsity problem

Data sparsity is a common problem in large scale data analysis, which arises when certain expected values are missing in the dataset. In the case of recommender systems, this situation occurs when the active users rate very few items. This reduces the recommendation accuracy. To alleviate this problem several techniques can be used such as demographic filtering, singular value decomposition and using model-based collaborative techniques.

Grey sheep problem

The grey sheep problem is specific to pure collaborative filtering approaches where the feedback given by one user do not match any user neighbourhood. In this situation, the system fails to accurately predict relevant items for that user. This problem can be resolved by using pure content-based approaches where predictions are made based on the user’s profile and item properties.

Scalability problem

Recommender systems, especially those employing collaborative filtering techniques, require large amounts of training data, which cause scalability problems. The scalability problem arises when the amount of data used as input to a recommender system increases quickly. In this era of big data, more and more items and users are rapidly getting added to the system and this problem is becoming common in recommender systems. Two common approaches used to solve the scalability problem is dimensionality reduction and using clustering-based techniques to find users in tiny clusters instead of the complete database.

Methodology

The purpose of this study is to understand the research trends in the field of recommender systems. The nature of research in recommender systems is such that it is difficult to confine each paper to a specific discipline. This can be further understood by the fact that research papers on recommender systems are scattered across various journals such as computer science, management, marketing, information technology and information science. Hence, this literature review is conducted over a wide range of electronic journals and research databases such as ACM Portal, IEEE/IEE Library, Google Scholars and Science Direct [ 88 ].

The search process of online research articles was performed based on 6 descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*”, “Hybrid Recommend*”. The following research papers described below were excluded from our research:

News articles.

Master’s dissertations.

Non-English papers.

Unpublished papers.

Research papers published before 2011.

We have screened a total of 350 articles based on their abstracts and content. However, only research papers that described how recommender systems can be applied were chosen. Finally, 60 papers were selected from top international journals indexed in Scopus or E-SCI in 2021. We now present the PRISMA flowchart of the inclusion and exclusion process in Fig.  5 .

PRISMA flowchart of the inclusion and exclusion process. Abstract and content not suitable to the study: * The use or application of the recommender system is not specified: **

Each paper was carefully reviewed and classified into 6 categories in the application fields and 3 categories in the techniques used to develop the system. The classification framework is presented in Fig.  6 .

Classification framework

The number of relevant articles come from Expert Systems with Applications (23%), followed by IEEE (17%), Knowledge-Based System (17%) and Others (43%). Table 3 depicts the article distribution by journal title and Table 4 depicts the sector-wise article distribution.

Both forward and backward searching techniques were implemented to establish that the review of 60 chosen articles can represent the domain literature. Hence, this paper can demonstrate its validity and reliability as a literature review.

Review on state-of-the-art recommender systems

This section presents a state-of-art literature review followed by a chronological review of the various existing recommender systems.

Literature review

In 2011, Castellano et al. [ 1 ] developed a “NEuro-fuzzy WEb Recommendation (NEWER)” system for exploiting the possibility of combining computational intelligence and user preference for suggesting interesting web pages to the user in a dynamic environment. It considered a set of fuzzy rules to express the correlations between user relevance and categories of pages. Crespo et al. [ 2 ] presented a recommender system for distance education over internet. It aims to recommend e-books to students using data from user interaction. The system was developed using a collaborative approach and focused on solving the data overload problem in big digital content. Lin et al. [ 3 ] have put forward a recommender system for automatic vending machines using Genetic algorithm (GA), k-means, Decision Tree (DT) and Bayesian Network (BN). It aimed at recommending localized products by developing a hybrid model combining statistical methods, classification methods, clustering methods, and meta-heuristic methods. Wang and Wu [ 4 ] have implemented a ubiquitous learning system for providing personalized learning assistance to the learners by combining the recommendation algorithm with a context-aware technique. It employed the Association Rule Mining (ARM) technique and aimed to increase the effectiveness of the learner’s learning. García-Crespo et al. [ 5 ] presented a “semantic hotel” recommender system by considering the experiences of consumers using a fuzzy logic approach. The system considered both hotel and customer characteristics. Dong et al. [ 6 ] proposed a structure for a service-concept recommender system using a semantic similarity model by integrating the techniques from the view of an ontology structure-oriented metric and a concept content-oriented metric. The system was able to deliver optimal performance when compared with similar recommender systems. Li et al. [ 7 ] developed a Fuzzy linguistic modelling-based recommender system for assisting users to find experts in knowledge management systems. The developed system was applied to the aircraft industry where it demonstrated efficient and feasible performance. Lorenzi et al. [ 8 ] presented an “assumption-based multiagent” system to make travel package recommendations using user preferences in the tourism industry. It performed different tasks like discovering, filtering, and integrating specific information for building a travel package following the user requirement. Huang et al. [ 9 ] proposed a context-aware recommender system through the extraction, evaluation and incorporation of contextual information gathered using the collaborative filtering and rough set model.

In 2012, Chen et al. [ 10 ] presented a diabetes medication recommender model by using “Semantic Web Rule Language (SWRL) and Java Expert System Shell (JESS)” for aggregating suitable prescriptions for the patients. It aimed at selecting the most suitable drugs from the list of specific drugs. Mohanraj et al. [ 11 ] developed the “Ontology-driven bee’s foraging approach (ODBFA)” to accurately predict the online navigations most likely to be visited by a user. The self-adaptive system is intended to capture the various requirements of the online user by using a scoring technique and by performing a similarity comparison. Hsu et al. [ 12 ] proposed a “personalized auxiliary material” recommender system by considering the specific course topics, individual learning styles, complexity of the auxiliary materials using an artificial bee colony algorithm. Gemmell et al. [ 13 ] demonstrated a solution for the problem of resource recommendation in social annotation systems. The model was developed using a linear-weighted hybrid method which was capable of providing recommendations under different constraints. Choi et al. [ 14 ] proposed one “Hybrid Online-Product rEcommendation (HOPE) system” by the integration of collaborative filtering through sequential pattern analysis-based recommendations and implicit ratings. Garibaldi et al. [ 15 ] put forward a technique for incorporating the variability in a fuzzy inference model by using non-stationary fuzzy sets for replicating the variabilities of a human. This model was applied to a decision problem for treatment recommendations of post-operative breast cancer.

In 2013, Salehi and Kmalabadi [ 16 ] proposed an e-learning material recommender system by “modelling of materials in a multidimensional space of material’s attribute”. It employed both content and collaborative filtering. Aher and Lobo [ 17 ] introduced a course recommender system using data mining techniques such as simple K-means clustering and Association Rule Mining (ARM) algorithm. The proposed e-learning system was successfully demonstrated for “MOOC (Massively Open Online Courses)”. Kardan and Ebrahimi [ 18 ] developed a hybrid recommender system for recommending posts in asynchronous discussion groups. The system was built combining both collaborative filtering and content-based filtering. It considered implicit user data to compute the user similarity with various groups, for recommending suitable posts and contents to its users. Chang et al. [ 19 ] adopted a cloud computing technology for building a TV program recommender system. The system designed for digital TV programs was implemented using Hadoop Fair Scheduler (HFC), K-means clustering and k-nearest neighbour (KNN) algorithms. It was successful in processing huge amounts of real-time user data. Lucas et al. [ 20 ] implemented a recommender model for assisting a tourism application by using associative classification and fuzzy logic to predict the context. Niu et al. [ 21 ] introduced “Affivir: An Affect-based Internet Video Recommendation System” which was developed by calculating user preferences and by using spectral clustering. This model recommended videos with similar effects, which was processed to get optimal results with dynamic adjustments of recommendation constraints.

In 2014, Liu et al. [ 22 ] implemented a new route recommendation model for offering personalized and real-time route recommendations for self-driven tourists to minimize the queuing time and traffic jams infamous tourist places. Recommendations were carried out by considering the preferences of users. Bakshi et al. [ 23 ] proposed an unsupervised learning-based recommender model for solving the scalability problem of recommender systems. The algorithm used transitive similarities along with Particle Swarm Optimization (PSO) technique for discovering the global neighbours. Kim and Shim [ 24 ] proposed a recommender system based on “latent Dirichlet allocation using probabilistic modelling for Twitter” that could recommend the top-K tweets for a user to read, and the top-K users to follow. The model parameters were learned from an inference technique by using the differential Expectation–Maximization (EM) algorithm. Wang et al. [ 25 ] developed a hybrid-movie recommender model by aggregating a genetic algorithm (GA) with improved K-means and Principal Component Analysis (PCA) technique. It was able to offer intelligent movie recommendations with personalized suggestions. Kolomvatsos et al. [ 26 ] proposed a recommender system by considering an optimal stopping theory for delivering books or music recommendations to the users. Gottschlich et al. [ 27 ] proposed a decision support system for stock investment recommendations. It computed the output by considering the overall crowd’s recommendations. Torshizi et al. [ 28 ] have introduced a hybrid recommender system to determine the severity level of a medical condition. It could recommend suitable therapies for patients suffering from Benign Prostatic Hyperplasia.

In 2015, Zahálka et al. [ 29 ] proposed a venue recommender: “City Melange”. It was an interactive content-based model which used the convolutional deep-net features of the visual domain and the linear Support Vector Machine (SVM) model to capture the semantic information and extract latent topics. Sankar et al. [ 30 ] have proposed a stock recommender system based on the stock holding portfolio of trusted mutual funds. The system employed the collaborative filtering approach along with social network analysis for offering a decision support system to build a trust-based recommendation model. Chen et al. [ 31 ] have put forward a novel movie recommender system by applying the “artificial immune network to collaborative filtering” technique. It computed the affinity of an antigen and the affinity between an antibody and antigen. Based on this computation a similarity estimation formula was introduced which was used for the movie recommendation process. Wu et al. [ 32 ] have examined the technique of data fusion for increasing the efficiency of item recommender systems. It employed a hybrid linear combination model and used a collaborative tagging system. Yeh and Cheng [ 33 ] have proposed a recommender system for tourist attractions by constructing the “elicitation mechanism using the Delphi panel method and matrix construction mechanism using the repertory grids”, which was developed by considering the user preference and expert knowledge.

In 2016, Liao et al. [ 34 ] proposed a recommender model for online customers using a rough set association rule. The model computed the probable behavioural variations of online consumers and provided product category recommendations for e-commerce platforms. Li et al. [ 35 ] have suggested a movie recommender system based on user feedback collected from microblogs and social networks. It employed the sentiment-aware association rule mining algorithm for recommendations using the prior information of frequent program patterns, program metadata similarity and program view logs. Wu et al. [ 36 ] have developed a recommender system for social media platforms by aggregating the technique of Social Matrix Factorization (SMF) and Collaborative Topic Regression (CTR). The model was able to compute the ratings of users to items for making recommendations. For improving the recommendation quality, it gathered information from multiple sources such as item properties, social networks, feedback, etc. Adeniyi et al. [ 37 ] put forward a study of automated web-usage data mining and developed a recommender system that was tested in both real-time and online for identifying the visitor’s or client’s clickstream data.

In 2017, Rawat and Kankanhalli [ 38 ] have proposed a viewpoint recommender system called “ClickSmart” for assisting mobile users to capture high-quality photographs at famous tourist places. Yang et al. [ 39 ] proposed a gradient boosting-based job recommendation system for satisfying the cost-sensitive requirements of the users. The hybrid algorithm aimed to reduce the rate of unnecessary job recommendations. Lee et al. [ 40 ] proposed a music streaming recommender system based on smartphone activity usage. The proposed system benefitted by using feature selection approaches with machine learning techniques such as Naive Bayes (NB), Support Vector Machine (SVM), Multi-layer Perception (MLP), Instance-based k -Nearest Neighbour (IBK), and Random Forest (RF) for performing the activity detection from the mobile signals. Wei et al. [ 41 ] have proposed a new stacked denoising autoencoder (SDAE) based recommender system for cold items. The algorithm employed deep learning and collaborative filtering method to predict the unknown ratings.

In 2018, Li et al. [ 42 ] have developed a recommendation algorithm using Weighted Linear Regression Models (WLRRS). The proposed system was put to experiment using the MovieLens dataset and it presented better classification and predictive accuracy. Mezei and Nikou [ 43 ] presented a mobile health and wellness recommender system based on fuzzy optimization. It could recommend a collection of actions to be taken by the user to improve the user’s health condition. Recommendations were made considering the user’s physical activities and preferences. Ayata et al. [ 44 ] proposed a music recommendation model based on the user emotions captured through wearable physiological sensors. The emotion detection algorithm employed different machine learning algorithms like SVM, RF, KNN and decision tree (DT) algorithms to predict the emotions from the changing electrical signals gathered from the wearable sensors. Zhao et al. [ 45 ] developed a multimodal learning-based, social-aware movie recommender system. The model was able to successfully resolve the sparsity problem of recommender systems. The algorithm developed a heterogeneous network by exploiting the movie-poster image and textual description of each movie based on the social relationships and user ratings.

In 2019, Hammou et al. [ 46 ] proposed a Big Data recommendation algorithm capable of handling large scale data. The system employed random forest and matrix factorization through a data partitioning scheme. It was then used for generating recommendations based on user rating and preference for each item. The proposed system outperformed existing systems in terms of accuracy and speed. Zhao et al. [ 47 ] have put forward a hybrid initialization method for social network recommender systems. The algorithm employed denoising autoencoder (DAE) neural network-based initialization method (ANNInit) and attribute mapping. Bhaskaran and Santhi [ 48 ] have developed a hybrid, trust-based e-learning recommender system using cloud computing. The proposed algorithm was capable of learning online user activities by using the Firefly Algorithm (FA) and K-means clustering. Afolabi and Toivanen [ 59 ] have suggested an integrated recommender model based on collaborative filtering. The proposed model “Connected Health for Effective Management of Chronic Diseases”, aimed for integrating recommender systems for better decision-making in the process of disease management. He et al. [ 60 ] proposed a movie recommender system called “HI2Rec” which explored the usage of collaborative filtering and heterogeneous information for making movie recommendations. The model used the knowledge representation learning approach to embed movie-related information gathered from different sources.

In 2020, Han et al. [ 49 ] have proposed one Internet of Things (IoT)-based cancer rehabilitation recommendation system using the Beetle Antennae Search (BAS) algorithm. It presented the patients with a solution for the problem of optimal nutrition program by considering the objective function as the recurrence time. Kang et al. [ 50 ] have presented a recommender system for personalized advertisements in Online Broadcasting based on a tree model. Recommendations were generated in real-time by considering the user preferences to minimize the overhead of preference prediction and using a HashMap along with the tree characteristics. Ullah et al. [ 51 ] have implemented an image-based service recommendation model for online shopping based random forest and Convolutional Neural Networks (CNN). The model used JPEG coefficients to achieve an accurate prediction rate. Cai et al. [ 52 ] proposed a new hybrid recommender model using a many-objective evolutionary algorithm (MaOEA). The proposed algorithm was successful in optimizing the novelty, diversity, and accuracy of recommendations. Esteban et al. [ 53 ] have implemented a hybrid multi-criteria recommendation system concerned with students’ academic performance, personal interests, and course selection. The system was developed using a Genetic Algorithm (GA) and aimed at helping university students. It combined both course information and student information for increasing system performance and the reliability of the recommendations. Mondal et al. [ 54 ] have built a multilayer, graph data model-based doctor recommendation system by exploiting the trust concept between a patient-doctor relationship. The proposed system showed good results in practical applications.

In 2021, Dhelim et al. [ 55 ] have developed a personality-based product recommending model using the techniques of meta path discovery and user interest mining. This model showed better results when compared to session-based and deep learning models. Bhalse et al. [ 56 ] proposed a web-based movie recommendation system based on collaborative filtering using Singular Value Decomposition (SVD), collaborative filtering and cosine similarity (CS) for addressing the sparsity problem of recommender systems. It suggested a recommendation list by considering the content information of movies. Similarly, to solve both sparsity and cold-start problems Ke et al. [ 57 ] proposed a dynamic goods recommendation system based on reinforcement learning. The proposed system was capable of learning from the reduced entropy loss error on real-time applications. Chen et al. [ 58 ] have presented a movie recommender model combining various techniques like user interest with category-level representation, neighbour-assisted representation, user interest with latent representation and item-level representation using Feed-forward Neural Network (FNN).

Comparative chronological review

A comparative chronological review to compare the total contributions on various recommender systems in the past 10 years is given in Fig.  7 .

Comparative chronological review of recommender systems under diverse applications

This review puts forward a comparison of the number of research works proposed in the domain of recommender systems from the year 2011 to 2021 using various deep learning and machine learning-based approaches. Research articles are categorized based on the recommender system classification framework as shown in Table 5 . The articles are ordered according to their year of publication. There are two key concepts: Application fields and techniques used. The application fields of recommender systems are divided into six different fields, viz. entertainment, health, tourism, web/e-commerce, education and social media/others.

Algorithmic categorization, simulation platforms and applications considered for various recommender systems

This section analyses different methods like deep learning, machine learning, clustering and meta-heuristic-based-approaches used in the development of recommender systems. The algorithmic categorization of different recommender systems is given in Fig.  8 .

Algorithmic categorization of different recommender systems

Categorization is done based on content-based, collaborative filtering-based, and optimization-based approaches. In [ 8 ], a content-based filtering technique was employed for increasing the ability to trust other agents and for improving the exchange of information by trust degree. In [ 16 ], it was applied to enhance the quality of recommendations using the account attributes of the material. It achieved better performance concerning with F1-score, recall and precision. In [ 18 ], this technique was able to capture the implicit user feedback, increasing the overall accuracy of the proposed model. The content-based filtering in [ 30 ] was able to increase the accuracy and performance of a stock recommender system by using the “trust factor” for making decisions.

Different collaborative filtering approaches are utilized in recent studies, which are categorized as follows:

Model-based techniques

Neuro-Fuzzy [ 1 ] based technique helps in discovering the association between user categories and item relevance. It is also simple to understand. K-Means Clustering [ 2 , 19 , 25 , 48 ] is efficient for large scale datasets. It is simple to implement and gives a fast convergence rate. It also offers automatic recovery from failures. The decision tree [ 2 , 44 ] technique is easy to interpret. It can be used for solving the classic regression and classification problems in recommender systems. Bayesian Network [ 3 ] is a probabilistic technique used to solve classification challenges. It is based on the theory of Bayes theorem and conditional probability. Association Rule Mining (ARM) techniques [ 4 , 17 , 35 ] extract rules for projecting the occurrence of an item by considering the existence of other items in a transaction. This method uses the association rules to create a more suitable representation of data and helps in increasing the model performance and storage efficiency. Fuzzy Logic [ 5 , 7 , 15 , 20 , 28 , 43 ] techniques use a set of flexible rules. It focuses on solving complex real-time problems having an inaccurate spectrum of data. This technique provides scalability and helps in increasing the overall model performance for recommender systems. The semantic similarity [ 6 ] technique is used for describing a topological similarity to define the distance among the concepts and terms through ontologies. It measures the similarity information for increasing the efficiency of recommender systems. Rough set [ 9 , 34 ] techniques use probability distributions for solving the challenges of existing recommender models. Semantic web rule language [ 10 ] can efficiently extract the dataset features and increase the model efficiency. Linear programming-based approaches [ 13 , 42 ] are employed for achieving quality decision making in recommender models. Sequential pattern analysis [ 14 ] is applied to find suitable patterns among data items. This helps in increasing model efficiency. The probabilistic model [ 24 ] is a famous tool to handle uncertainty in risk computations and performance assessment. It offers better decision-making capabilities. K-nearest neighbours (KNN) [ 19 , 37 , 44 ] technique provides faster computation time, simplicity and ease of interpretation. They are good for classification and regression-based problems and offers more accuracy. Spectral clustering [ 21 ] is also called graph clustering or similarity-based clustering, which mainly focuses on reducing the space dimensionality in identifying the dataset items. Stochastic learning algorithm [ 26 ] solves the real-time challenges of recommender systems. Linear SVM [ 29 , 44 ] efficiently solves the high dimensional problems related to recommender systems. It is a memory-efficient method and works well with a large number of samples having relative separation among the classes. This method has been shown to perform well even when new or unfamiliar data is added. Relational Functional Gradient Boosting [ 39 ] technique efficiently works on the relational dependency of data, which is useful for statical relational learning for collaborative-based recommender systems. Ensemble learning [ 40 ] combines the forecast of two or more models and aims to achieve better performance than any of the single contributing models. It also helps in reducing overfitting problems, which are common in recommender systems.

SDAE [ 41 ] is used for learning the non-linear transformations with different filters for finding suitable data. This aids in increasing the performance of recommender models. Multimodal network learning [ 45 ] is efficient for multi-modal data, representing a combined representation of diverse modalities. Random forest [ 46 , 51 ] is a commonly used approach in comparison with other classifiers. It has been shown to increase accuracy when handling big data. This technique is a collection of decision trees to minimize variance through training on diverse data samples. ANNInit [ 47 ] is a type of artificial neural network-based technique that has the capability of self-learning and generating efficient results. It is independent of the data type and can learn data patterns automatically. HashMap [ 50 ] gives faster access to elements owing to the hashing methodology, which decreases the data processing time and increases the performance of the system. CNN [ 51 ] technique can automatically fetch the significant features of a dataset without any supervision. It is a computationally efficient method and provides accurate recommendations. This technique is also simple and fast for implementation. Multilayer graph data model [ 54 ] is efficient for real-time applications and minimizes the access time through mapping the correlation as edges among nodes and provides superior performance. Singular Value Decomposition [ 56 ] can simplify the input data and increase the efficiency of recommendations by eliminating the noise present in data. Reinforcement learning [ 57 ] is efficient for practical scenarios of recommender systems having large data sizes. It is capable of boosting the model performance by increasing the model accuracy even for large scale datasets. FNN [ 58 ] is one of the artificial neural network techniques which can learn non-linear and complex relationships between items. It has demonstrated a good performance increase when employed in different recommender systems. Knowledge representation learning [ 60 ] systems aim to simplify the model development process by increasing the acquisition efficiency, inferential efficiency, inferential adequacy and representation adequacy. User-based approaches [ 2 , 55 , 59 ] specialize in detecting user-related meta-data which is employed to increase the overall model performance. This technique is more suitable for real-time applications where it can capture user feedback and use it to increase the user experience.

Optimization-based techniques

The Foraging Bees [ 11 ] technique enables both functional and combinational optimization for random searching in recommender models. Artificial bee colony [ 12 ] is a swarm-based meta-heuristic technique that provides features like faster convergence rate, the ability to handle the objective with stochastic nature, ease for incorporating with other algorithms, usage of fewer control parameters, strong robustness, high flexibility and simplicity. Particle Swarm Optimization [ 23 ] is a computation optimization technique that offers better computational efficiency, robustness in control parameters, and is easy and simple to implement in recommender systems. Portfolio optimization algorithm [ 27 ] is a subclass of optimization algorithms that find its application in stock investment recommender systems. It works well in real-time and helps in the diversification of the portfolio for maximum profit. The artificial immune system [ 31 ]a is computationally intelligent machine learning technique. This technique can learn new patterns in the data and optimize the overall system parameters. Expectation maximization (EM) [ 32 , 36 , 38 ] is an iterative algorithm that guarantees the likelihood of finding the maximum parameters when the input variables are unknown. Delphi panel and repertory grid [ 33 ] offers efficient decision making by solving the dimensionality problem and data sparsity issues of recommender systems. The Firefly algorithm (FA) [ 48 ] provides fast results and increases recommendation efficiency. It is capable of reducing the number of iterations required to solve specific recommender problems. It also provides both local and global sets of solutions. Beetle Antennae Search (BAS) [ 49 ] offers superior search accuracy and maintains less time complexity that promotes the performance of recommendations. Many-objective evolutionary algorithm (MaOEA) [ 52 ] is applicable for real-time, multi-objective, search-related recommender systems. The introduction of a local search operator increases the convergence rate and gets suitable results. Genetic Algorithm (GA) [ 2 , 22 , 25 , 53 ] based techniques are used to solve the multi-objective optimization problems of recommender systems. They employ probabilistic transition rules and have a simpler operation that provides better recommender performance.

Features and challenges

The features and challenges of the existing recommender models are given in Table 6 .

Simulation platforms

The various simulation platforms used for developing different recommender systems with different applications are given in Fig.  9 .

Simulation platforms used for developing different recommender systems

Here, the Java platform is used in 20% of the contributions, MATLAB is implemented in 7% of the contributions, different fold cross-validation are used in 8% of the contributions, 7% of the contributions are utilized by the python platform, 3% of the contributions employ R-programming and 1% of the contributions are developed by Tensorflow, Weka and Android environments respectively. Other simulation platforms like Facebook, web UI (User Interface), real-time environments, etc. are used in 50% of the contributions. Table 7 describes some simulation platforms commonly used for developing recommender systems.

Application focused and dataset description

This section provides an analysis of the different applications focused on a set of recent recommender systems and their dataset details.

Recent recommender systems were analysed and found that 11% of the contributions are focused on the domain of healthcare, 10% of the contributions are on movie recommender systems, 5% of the contributions come from music recommender systems, 6% of the contributions are focused on e-learning recommender systems, 8% of the contributions are used for online product recommender systems, 3% of the contributions are focused on book recommendations and 1% of the contributions are focused on Job and knowledge management recommender systems. 5% of the contributions concentrated on social network recommender systems, 10% of the contributions are focused on tourist and hotels recommender systems, 6% of the contributions are employed for stock recommender systems, and 3% of the contributions contributed for video recommender systems. The remaining 12% of contributions are miscellaneous recommender systems like Twitter, venue-based recommender systems, etc. Similarly, different datasets are gathered for recommender systems based on their application types. A detailed description is provided in Table 8 .

Performance analysis of state-of-art recommender systems

The performance evaluation metrics used for the analysis of different recommender systems is depicted in Table 9 . From the set of research works, 35% of the works use recall measure, 16% of the works employ Mean Absolute Error (MAE), 11% of the works take Root Mean Square Error (RMSE), 41% of the papers consider precision, 30% of the contributions analyse F1-measure, 31% of the works apply accuracy and 6% of the works employ coverage measure to validate the performance of the recommender systems. Moreover, some additional measures are also considered for validating the performance in a few applications.

Research gaps and challenges

In the recent decade, recommender systems have performed well in solving the problem of information overload and has become the more appropriate tool for multiple areas such as psychology, mathematics, computer science, etc. [ 80 ]. However, current recommender systems face a variety of challenges which are stated as follows, and discussed below:

Deployment challenges such as cold start, scalability, sparsity, etc. are already discussed in Sect. 3.

Challenges faced when employing different recommender algorithms for different applications.

Challenges in collecting implicit user data

Challenges in handling real-time user feedback.

Challenges faced in choosing the correct implementation techniques.

Challenges faced in measuring system performance.

Challenges in implementing recommender system for diverse applications.

Numerous recommender algorithms have been proposed on novel emerging dimensions which focus on addressing the existing limitations of recommender systems. A good recommender system must increase the recommendation quality based on user preferences. However, a specific recommender algorithm is not always guaranteed to perform equally for different applications. This encourages the possibility of employing different recommender algorithms for different applications, which brings along a lot of challenges. There is a need for more research to alleviate these challenges. Also, there is a large scope of research in recommender applications that incorporate information from different interactive online sites like Facebook, Twitter, shopping sites, etc. Some other areas for emerging research may be in the fields of knowledge-based recommender systems, methods for seamlessly processing implicit user data and handling real-time user feedback to recommend items in a dynamic environment.

Some of the other research areas like deep learning-based recommender systems, demographic filtering, group recommenders, cross-domain techniques for recommender systems, and dimensionality reduction techniques are also further required to be studied [ 83 ]. Deep learning-based recommender systems have recently gained much popularity. Future research areas in this field can integrate the well-performing deep learning models with new variants of hybrid meta-heuristic approaches.

During this review, it was observed that even though recent recommender systems have demonstrated good performance, there is no single standardized criteria or method which could be used to evaluate the performance of all recommender systems. System performance is generally measured by different evaluation matrices which makes it difficult to compare. The application of recommender systems in real-time applications is growing. User satisfaction and personalization play a very important role in the success of such recommender systems. There is a need for some new evaluation criteria which can evaluate the level of user satisfaction in real-time. New research should focus on capturing real-time user feedback and use the information to change the recommendation process accordingly. This will aid in increasing the quality of recommendations.

Conclusion and future scope

Recommender systems have attracted the attention of researchers and academicians. In this paper, we have identified and prudently reviewed research papers on recommender systems focusing on diverse applications, which were published between 2011 and 2021. This review has gathered diverse details like different application fields, techniques used, simulation tools used, diverse applications focused, performance metrics, datasets used, system features, and challenges of different recommender systems. Further, the research gaps and challenges were put forward to explore the future research perspective on recommender systems. Overall, this paper provides a comprehensive understanding of the trend of recommender systems-related research and to provides researchers with insight and future direction on recommender systems. The results of this study have several practical and significant implications:

Based on the recent-past publication rates, we feel that the research of recommender systems will significantly grow in the future.

A large number of research papers were identified in movie recommendations, whereas health, tourism and education-related recommender systems were identified in very few numbers. This is due to the availability of movie datasets in the public domain. Therefore, it is necessary to develop datasets in other fields also.

There is no standard measure to compute the performance of recommender systems. Among 60 papers, 21 used recall, 10 used MAE, 25 used precision, 18 used F1-measure, 19 used accuracy and only 7 used RMSE to calculate system performance. Very few systems were found to excel in two or more matrices.

Java and Python (with a combined contribution of 27%) are the most common programming languages used to develop recommender systems. This is due to the availability of a large number of standard java and python libraries which aid in the development process.

Recently a large number of hybrid and optimizations techniques are being proposed for recommender systems. The performance of a recommender system can be greatly improved by applying optimization techniques.

There is a large scope of research in using neural networks and deep learning-based methods for developing recommender systems. Systems developed using these methods are found to achieve high-performance accuracy.

This research will provide a guideline for future research in the domain of recommender systems. However, this research has some limitations. Firstly, due to the limited amount of manpower and time, we have only reviewed papers published in journals focusing on computer science, management and medicine. Secondly, we have reviewed only English papers. New research may extend this study to cover other journals and non-English papers. Finally, this review was conducted based on a search on only six descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*” and “Hybrid Recommend*”. Research papers that did not include these keywords were not considered. Future research can include adding some additional descriptors and keywords for searching. This will allow extending the research to cover more diverse articles on recommender systems.

Availability of data and materials

Not applicable.

Castellano G, Fanelli AM, Torsello MA. NEWER: A system for neuro-fuzzy web recommendation. Appl Soft Comput. 2011;11:793–806.

Article   Google Scholar  

Crespo RG, Martínez OS, Lovelle JMC, García-Bustelo BCP, Gayo JEL, Pablos PO. Recommendation system based on user interaction data applied to intelligent electronic books. Computers Hum Behavior. 2011;27:1445–9.

Lin FC, Yu HW, Hsu CH, Weng TC. Recommendation system for localized products in vending machines. Expert Syst Appl. 2011;38:9129–38.

Wang SL, Wu CY. Application of context-aware and personalized recommendation to implement an adaptive ubiquitous learning system. Expert Syst Appl. 2011;38:10831–8.

García-Crespo Á, López-Cuadrado JL, Colomo-Palacios R, González-Carrasco I, Ruiz-Mezcua B. Sem-Fit: A semantic based expert system to provide recommendations in the tourism domain. Expert Syst Appl. 2011;38:13310–9.

Dong H, Hussain FK, Chang E. A service concept recommendation system for enhancing the dependability of semantic service matchmakers in the service ecosystem environment. J Netw Comput Appl. 2011;34:619–31.

Li M, Liu L, Li CB. An approach to expert recommendation based on fuzzy linguistic method and fuzzy text classification in knowledge management systems. Expert Syst Appl. 2011;38:8586–96.

Lorenzi F, Bazzan ALC, Abel M, Ricci F. Improving recommendations through an assumption-based multiagent approach: An application in the tourism domain. Expert Syst Appl. 2011;38:14703–14.

Huang Z, Lu X, Duan H. Context-aware recommendation using rough set model and collaborative filtering. Artif Intell Rev. 2011;35:85–99.

Chen RC, Huang YH, Bau CT, Chen SM. A recommendation system based on domain ontology and SWRL for anti-diabetic drugs selection. Expert Syst Appl. 2012;39:3995–4006.

Mohanraj V, Chandrasekaran M, Senthilkumar J, Arumugam S, Suresh Y. Ontology driven bee’s foraging approach based self-adaptive online recommendation system. J Syst Softw. 2012;85:2439–50.

Hsu CC, Chen HC, Huang KK, Huang YM. A personalized auxiliary material recommendation system based on learning style on facebook applying an artificial bee colony algorithm. Comput Math Appl. 2012;64:1506–13.

Gemmell J, Schimoler T, Mobasher B, Burke R. Resource recommendation in social annotation systems: A linear-weighted hybrid approach. J Comput Syst Sci. 2012;78:1160–74.

Article   MathSciNet   Google Scholar  

Choi K, Yoo D, Kim G, Suh Y. A hybrid online-product recommendation system: Combining implicit rating-based collaborative filtering and sequential pattern analysis. Electron Commer Res Appl. 2012;11:309–17.

Garibaldi JM, Zhou SM, Wang XY, John RI, Ellis IO. Incorporation of expert variability into breast cancer treatment recommendation in designing clinical protocol guided fuzzy rule system models. J Biomed Inform. 2012;45:447–59.

Salehi M, Kmalabadi IN. A hybrid attribute–based recommender system for e–learning material recommendation. IERI Procedia. 2012;2:565–70.

Aher SB, Lobo LMRJ. Combination of machine learning algorithms for recommendation of courses in e-learning System based on historical data. Knowl-Based Syst. 2013;51:1–14.

Kardan AA, Ebrahimi M. A novel approach to hybrid recommendation systems based on association rules mining for content recommendation in asynchronous discussion groups. Inf Sci. 2013;219:93–110.

Chang JH, Lai CF, Wang MS, Wu TY. A cloud-based intelligent TV program recommendation system. Comput Electr Eng. 2013;39:2379–99.

Lucas JP, Luz N, Moreno MN, Anacleto R, Figueiredo AA, Martins C. A hybrid recommendation approach for a tourism system. Expert Syst Appl. 2013;40:3532–50.

Niu J, Zhu L, Zhao X, Li H. Affivir: An affect-based Internet video recommendation system. Neurocomputing. 2013;120:422–33.

Liu L, Xu J, Liao SS, Chen H. A real-time personalized route recommendation system for self-drive tourists based on vehicle to vehicle communication. Expert Syst Appl. 2014;41:3409–17.

Bakshi S, Jagadev AK, Dehuri S, Wang GN. Enhancing scalability and accuracy of recommendation systems using unsupervised learning and particle swarm optimization. Appl Soft Comput. 2014;15:21–9.

Kim Y, Shim K. TWILITE: A recommendation system for twitter using a probabilistic model based on latent Dirichlet allocation. Inf Syst. 2014;42:59–77.

Wang Z, Yu X, Feng N, Wang Z. An improved collaborative movie recommendation system using computational intelligence. J Vis Lang Comput. 2014;25:667–75.

Kolomvatsos K, Anagnostopoulos C, Hadjiefthymiades S. An efficient recommendation system based on the optimal stopping theory. Expert Syst Appl. 2014;41:6796–806.

Gottschlich J, Hinz O. A decision support system for stock investment recommendations using collective wisdom. Decis Support Syst. 2014;59:52–62.

Torshizi AD, Zarandi MHF, Torshizi GD, Eghbali K. A hybrid fuzzy-ontology based intelligent system to determine level of severity and treatment recommendation for benign prostatic hyperplasia. Comput Methods Programs Biomed. 2014;113:301–13.

Zahálka J, Rudinac S, Worring M. Interactive multimodal learning for venue recommendation. IEEE Trans Multimedia. 2015;17:2235–44.

Sankar CP, Vidyaraj R, Kumar KS. Trust based stock recommendation system – a social network analysis approach. Procedia Computer Sci. 2015;46:299–305.

Chen MH, Teng CH, Chang PC. Applying artificial immune systems to collaborative filtering for movie recommendation. Adv Eng Inform. 2015;29:830–9.

Wu H, Pei Y, Li B, Kang Z, Liu X, Li H. Item recommendation in collaborative tagging systems via heuristic data fusion. Knowl-Based Syst. 2015;75:124–40.

Yeh DY, Cheng CH. Recommendation system for popular tourist attractions in Taiwan using delphi panel and repertory grid techniques. Tour Manage. 2015;46:164–76.

Liao SH, Chang HK. A rough set-based association rule approach for a recommendation system for online consumers. Inf Process Manage. 2016;52:1142–60.

Li H, Cui J, Shen B, Ma J. An intelligent movie recommendation system through group-level sentiment analysis in microblogs. Neurocomputing. 2016;210:164–73.

Wu H, Yue K, Pei Y, Li B, Zhao Y, Dong F. Collaborative topic regression with social trust ensemble for recommendation in social media systems. Knowl-Based Syst. 2016;97:111–22.

Adeniyi DA, Wei Z, Yongquan Y. Automated web usage data mining and recommendation system using K-Nearest Neighbor (KNN) classification method. Appl Computing Inform. 2016;12:90–108.

Rawat YS, Kankanhalli MS. ClickSmart: A context-aware viewpoint recommendation system for mobile photography. IEEE Trans Circuits Syst Video Technol. 2017;27:149–58.

Yang S, Korayem M, Aljadda K, Grainger T, Natarajan S. Combining content-based and collaborative filtering for job recommendation system: A cost-sensitive Statistical Relational Learning approach. Knowl-Based Syst. 2017;136:37–45.

Lee WP, Chen CT, Huang JY, Liang JY. A smartphone-based activity-aware system for music streaming recommendation. Knowl-Based Syst. 2017;131:70–82.

Wei J, He J, Chen K, Zhou Y, Tang Z. Collaborative filtering and deep learning based recommendation system for cold start items. Expert Syst Appl. 2017;69:29–39.

Li C, Wang Z, Cao S, He L. WLRRS: A new recommendation system based on weighted linear regression models. Comput Electr Eng. 2018;66:40–7.

Mezei J, Nikou S. Fuzzy optimization to improve mobile health and wellness recommendation systems. Knowl-Based Syst. 2018;142:108–16.

Ayata D, Yaslan Y, Kamasak ME. Emotion based music recommendation system using wearable physiological sensors. IEEE Trans Consum Electron. 2018;64:196–203.

Zhao Z, Yang Q, Lu H, Weninger T. Social-aware movie recommendation via multimodal network learning. IEEE Trans Multimedia. 2018;20:430–40.

Hammou BA, Lahcen AA, Mouline S. An effective distributed predictive model with matrix factorization and random forest for big data recommendation systems. Expert Syst Appl. 2019;137:253–65.

Zhao J, Geng X, Zhou J, Sun Q, Xiao Y, Zhang Z, Fu Z. Attribute mapping and autoencoder neural network based matrix factorization initialization for recommendation systems. Knowl-Based Syst. 2019;166:132–9.

Bhaskaran S, Santhi B. An efficient personalized trust based hybrid recommendation (TBHR) strategy for e-learning system in cloud computing. Clust Comput. 2019;22:1137–49.

Han Y, Han Z, Wu J, Yu Y, Gao S, Hua D, Yang A. Artificial intelligence recommendation system of cancer rehabilitation scheme based on IoT technology. IEEE Access. 2020;8:44924–35.

Kang S, Jeong C, Chung K. Tree-based real-time advertisement recommendation system in online broadcasting. IEEE Access. 2020;8:192693–702.

Ullah F, Zhang B, Khan RU. Image-based service recommendation system: A JPEG-coefficient RFs approach. IEEE Access. 2020;8:3308–18.

Cai X, Hu Z, Zhao P, Zhang W, Chen J. A hybrid recommendation system with many-objective evolutionary algorithm. Expert Syst Appl. 2020. https://doi.org/10.1016/j.eswa.2020.113648 .

Esteban A, Zafra A, Romero C. Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization. Knowledge-Based Syst. 2020;194:105385.

Mondal S, Basu A, Mukherjee N. Building a trust-based doctor recommendation system on top of multilayer graph database. J Biomed Inform. 2020;110:103549.

Dhelim S, Ning H, Aung N, Huang R, Ma J. Personality-aware product recommendation system based on user interests mining and metapath discovery. IEEE Trans Comput Soc Syst. 2021;8:86–98.

Bhalse N, Thakur R. Algorithm for movie recommendation system using collaborative filtering. Materials Today: Proceedings. 2021. https://doi.org/10.1016/j.matpr.2021.01.235 .

Ke G, Du HL, Chen YC. Cross-platform dynamic goods recommendation system based on reinforcement learning and social networks. Appl Soft Computing. 2021;104:107213.

Chen X, Liu D, Xiong Z, Zha ZJ. Learning and fusing multiple user interest representations for micro-video and movie recommendations. IEEE Trans Multimedia. 2021;23:484–96.

Afolabi AO, Toivanen P. Integration of recommendation systems into connected health for effective management of chronic diseases. IEEE Access. 2019;7:49201–11.

He M, Wang B, Du X. HI2Rec: Exploring knowledge in heterogeneous information for movie recommendation. IEEE Access. 2019;7:30276–84.

Bobadilla J, Serradilla F, Hernando A. Collaborative filtering adapted to recommender systems of e-learning. Knowl-Based Syst. 2009;22:261–5.

Russell S, Yoon V. Applications of wavelet data reduction in a recommender system. Expert Syst Appl. 2008;34:2316–25.

Campos LM, Fernández-Luna JM, Huete JF. A collaborative recommender system based on probabilistic inference from fuzzy observations. Fuzzy Sets Syst. 2008;159:1554–76.

Funk M, Rozinat A, Karapanos E, Medeiros AKA, Koca A. In situ evaluation of recommender systems: Framework and instrumentation. Int J Hum Comput Stud. 2010;68:525–47.

Porcel C, Moreno JM, Herrera-Viedma E. A multi-disciplinar recommender system to advice research resources in University Digital Libraries. Expert Syst Appl. 2009;36:12520–8.

Bobadilla J, Serradilla F, Bernal J. A new collaborative filtering metric that improves the behavior of recommender systems. Knowl-Based Syst. 2010;23:520–8.

Ochi P, Rao S, Takayama L, Nass C. Predictors of user perceptions of web recommender systems: How the basis for generating experience and search product recommendations affects user responses. Int J Hum Comput Stud. 2010;68:472–82.

Olmo FH, Gaudioso E. Evaluation of recommender systems: A new approach. Expert Syst Appl. 2008;35:790–804.

Zhen L, Huang GQ, Jiang Z. An inner-enterprise knowledge recommender system. Expert Syst Appl. 2010;37:1703–12.

Göksedef M, Gündüz-Öğüdücü S. Combination of web page recommender systems. Expert Syst Appl. 2010;37(4):2911–22.

Shao B, Wang D, Li T, Ogihara M. Music recommendation based on acoustic features and user access patterns. IEEE Trans Audio Speech Lang Process. 2009;17:1602–11.

Shin C, Woo W. Socially aware tv program recommender for multiple viewers. IEEE Trans Consum Electron. 2009;55:927–32.

Lopez-Carmona MA, Marsa-Maestre I, Perez JRV, Alcazar BA. Anegsys: An automated negotiation based recommender system for local e-marketplaces. IEEE Lat Am Trans. 2007;5:409–16.

Yap G, Tan A, Pang H. Discovering and exploiting causal dependencies for robust mobile context-aware recommenders. IEEE Trans Knowl Data Eng. 2007;19:977–92.

Meo PD, Quattrone G, Terracina G, Ursino D. An XML-based multiagent system for supporting online recruitment services. IEEE Trans Syst Man Cybern. 2007;37:464–80.

Khusro S, Ali Z, Ullah I. Recommender systems: Issues, challenges, and research opportunities. Inform Sci Appl. 2016. https://doi.org/10.1007/978-981-10-0557-2_112 .

Blanco-Fernandez Y, Pazos-Arias JJ, Gil-Solla A, Ramos-Cabrer M, Lopez-Nores M. Providing entertainment by content-based filtering and semantic reasoning in intelligent recommender systems. IEEE Trans Consum Electron. 2008;54:727–35.

Isinkaye FO, Folajimi YO, Ojokoh BA. Recommendation systems: Principles, methods and evaluation. Egyptian Inform J. 2015;16:261–73.

Yoshii K, Goto M, Komatani K, Ogata T, Okuno HG. An efficient hybrid music recommender system using an incrementally trainable probabilistic generative model. IEEE Trans Audio Speech Lang Process. 2008;16:435–47.

Wei YZ, Moreau L, Jennings NR. Learning users’ interests by quality classification in market-based recommender systems. IEEE Trans Knowl Data Eng. 2005;17:1678–88.

Bjelica M. Towards TV recommender system: experiments with user modeling. IEEE Trans Consum Electron. 2010;56:1763–9.

Setten MV, Veenstra M, Nijholt A, Dijk BV. Goal-based structuring in recommender systems. Interact Comput. 2006;18:432–56.

Adomavicius G, Tuzhilin A. Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng. 2005;17:734–49.

Symeonidis P, Nanopoulos A, Manolopoulos Y. Providing justifications in recommender systems. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans. 2009;38:1262–72.

Zhan J, Hsieh C, Wang I, Hsu T, Liau C, Wang D. Privacy preserving collaborative recommender systems. IEEE Trans Syst Man Cybernet. 2010;40:472–6.

Burke R. Hybrid recommender systems: survey and experiments. User Model User-Adap Inter. 2002;12:331–70.

Article   MATH   Google Scholar  

Gunes I, Kaleli C, Bilge A, Polat H. Shilling attacks against recommender systems: a comprehensive survey. Artif Intell Rev. 2012;42:767–99.

Park DH, Kim HK, Choi IY, Kim JK. A literature review and classification of recommender systems research. Expert Syst Appl. 2012;39:10059–72.

Download references

Acknowledgements

We thank our colleagues from Assam Down Town University who provided insight and expertise that greatly assisted this research, although they may not agree with all the interpretations and conclusions of this paper.

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and affiliations.

Department of Computer Science & Engineering, Assam Down Town University, Panikhaiti, Guwahati, 781026, Assam, India

Deepjyoti Roy & Mala Dutta

You can also search for this author in PubMed   Google Scholar

Contributions

DR carried out the review study and analysis of the existing algorithms in the literature. MD has been involved in drafting the manuscript or revising it critically for important intellectual content. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Deepjyoti Roy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Roy, D., Dutta, M. A systematic review and research perspective on recommender systems. J Big Data 9 , 59 (2022). https://doi.org/10.1186/s40537-022-00592-5

Download citation

Received : 04 October 2021

Accepted : 28 March 2022

Published : 03 May 2022

DOI : https://doi.org/10.1186/s40537-022-00592-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Recommender system
• Machine learning
• Content-based filtering
• Collaborative filtering
• Deep learning

A systematic literature review of multicriteria recommender systems

• Published: 09 June 2020
• Volume 54 , pages 427–468, ( 2021 )

Cite this article

• Diego Monti   ORCID: orcid.org/0000-0002-3821-5379 1 ,
• Giuseppe Rizzo 2 &
• Maurizio Morisio 1  

2153 Accesses

29 Citations

8 Altmetric

Explore all metrics

Since the first years of the 90s, recommender systems have emerged as effective tools for automatically selecting items according to user preferences. Traditional recommenders rely on the relevance assessments that users express using a single rating for each item. However, some authors started to suggest that this approach could be limited, as we naturally tend to formulate different judgments according to multiple criteria. During the last decade, several studies introduced novel recommender systems capable of exploiting user preferences expressed over multiple criteria. This work proposes a systematic literature review in the field of multicriteria recommender systems. Following a replicable protocol, we selected a total number of 93 studies dealing with this topic. We subsequently analyzed them to provide an answer to five different research questions. We considered what are the most common research problems, recommendation approaches, data mining and machine learning algorithms mentioned in these studies. Furthermore, we investigated the domains of application, the exploited evaluation protocols, metrics and datasets, and the most promising suggestions for future works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Efficiency Evaluation of Recommender Systems: Study of Existing Problems and Possible Extensions

Operations Research and Recommender Systems

Recommender Systems: Sources of Knowledge and Evaluation Metrics

https://www.qsrinternational.com/nvivo .

Please note that the results for the year 2019 are not available, as the selection was performed in January 2019.

https://webscope.sandbox.yahoo.com .

https://www.cs.cmu.edu/~jiweil/html/hotel-review.html .

http://www.cs.virginia.edu/~hw5x/Data/LARA/TripAdvisor .

https://www.tripadvisor.com .

https://www.bcnrestaurantes.com .

Adomavicius G, Kwon Y (2015) Multi-criteria recommender systems. In: Recommender systems handbook. Springer, pp 847–880. https://doi.org/10.1007/978-1-4899-7637-6_25

Adomavicius G, Tuzhilin A (2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749. https://doi.org/10.1109/tkde.2005.99

Article   Google Scholar  

Adomavicius G, Tuzhilin A (2015) Context-aware recommender systems. In: Recommender systems handbook. Springer, pp 191–226 https://doi.org/10.1007/978-1-4899-7637-6_6

Balabanović M, Shoham Y (1997) Fab: content-based, collaborative recommendation. Commun ACM 40(3):66–72. https://doi.org/10.1145/245108.245124

Burke R (2007) Hybrid web recommender systems. In: The adaptive web. Springer, Berlin, pp 377–408. https://doi.org/10.1007/978-3-540-72079-9_12

Çano E, Morisio M (2017) Hybrid recommender systems: a systematic literature review. Intell Data Anal 21(6):1487–1524. https://doi.org/10.3233/IDA-163209

Cruzes DS, Dyba T (2011) Recommended steps for thematic synthesis in software engineering. In: 2011 international symposium on empirical software engineering and measurement. IEEE. https://doi.org/10.1109/esem.2011.36

de Gemmis M, Lops P, Musto C, Narducci F, Semeraro G (2015) Semantics-aware content-based recommender systems. In: Recommender systems handbook. Springer, pp 119–159. https://doi.org/10.1007/978-1-4899-7637-6_4

Figueroa C, Vagliano I, Rocha OR, Morisio M (2015) A systematic literature review of linked data-based recommender systems. Concurr Comput Pract Exp 27(17):4659–4684. https://doi.org/10.1002/cpe.3449

Goldberg D, Nichols D, Oki BM, Terry D (1992) Using collaborative filtering to weave an information tapestry. Commun ACM 35(12):61–70. https://doi.org/10.1145/138859.138867

Gunawardana A, Shani G (2015) Evaluating recommender systems. In: Recommender systems handbook, 2 edn, chap. 8. Springer, pp 265–308. https://doi.org/10.1007/978-1-4899-7637-6_8

Guy I (2015) Social recommender systems. In: Recommender systems handbook. Springer, pp 511–543. https://doi.org/10.1007/978-1-4899-7637-6_15

Herlocker JL, Konstan JA, Riedl J (2000). Explaining collaborative filtering recommendations. In: Proceedings of the 2000 ACM conference on computer supported cooperative work. ACM https://doi.org/10.1145/358916.358995

Herlocker JL, Konstan JA, Terveen LG, Riedl JT (2004) Evaluating collaborative filtering recommender systems. ACM Trans Inf Syst 22(1):5–53. https://doi.org/10.1145/963770.963772

Hong J, Suh E, Kim SJ (2009) Context-aware systems: a literature review and classification. Expert Syst Appl 36(4):8509–8522. https://doi.org/10.1016/j.eswa.2008.10.071

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf

Manouselis N, Costopoulou C (2007) Analysis and classification of multi-criteria recommender systems. World Wide Web 10(4):415–441. https://doi.org/10.1007/s11280-007-0019-8

Park DH, Kim HK, Choi IY, Kim JK (2012) A literature review and classification of recommender systems research. Expert Syst Appl 39(11):10059–10072. https://doi.org/10.1016/j.eswa.2012.02.038

Portugal I, Alencar P, Cowan D (2018) The use of machine learning algorithms in recommender systems: a systematic review. Expert Syst Appl 97:205–227. https://doi.org/10.1016/j.eswa.2017.12.020

Quadrana M, Cremonesi P, Jannach D (2018) Sequence-aware recommender systems. ACM Comput Surv 51(4):1–36. https://doi.org/10.1145/3190616

Ricci F, Rokach L, Shapira B (2015). Recommender systems: Introduction and challenges. In: Recommender systems handbook. Springer, pp 1–34. https://doi.org/10.1007/978-1-4899-7637-6_1

Said A, Bellogín A (2014) Comparative recommender system evaluation. In: Proceedings of the 8th ACM conference on recommender systems. ACM, pp 129–136. https://doi.org/10.1145/2645710.2645746

Wang H, Lu Y, Zhai C (2010) Latent aspect rating analysis on review text data. In: Proceedings of the 16th ACM SIGKDD international conference on knowledge discovery and data mining. ACM. https://doi.org/10.1145/1835804.1835903

Zhang S, Yao L, Sun A, Tay Y (2019) Deep learning based recommender system. ACM Comput Surv 52(1):1–38. https://doi.org/10.1145/3285029

Download references

Author information

Authors and affiliations.

Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129, Turin, Italy

Diego Monti & Maurizio Morisio

LINKS Foundation, Via Pier Carlo Boggio 61, 10138, Turin, Italy

Giuseppe Rizzo

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Diego Monti .

Ethics declarations

Conflict of interest.

The authors declare that they have no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: Selected studies

Rights and permissions.

Reprints and permissions

About this article

Monti, D., Rizzo, G. & Morisio, M. A systematic literature review of multicriteria recommender systems. Artif Intell Rev 54 , 427–468 (2021). https://doi.org/10.1007/s10462-020-09851-4

Download citation

Published : 09 June 2020

Issue Date : January 2021

DOI : https://doi.org/10.1007/s10462-020-09851-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Recommender system
• Multicriteria recommendation
• Systematic literature review
• Find a journal
• Publish with us
• Track your research

A Literature Review of Recommendation Systems

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

To read this content please select one of the options below:

Please note you do not have access to teaching notes, recommender systems: a systematic review of the state of the art literature and suggestions for future research.

ISSN : 0368-492X

Article publication date: 15 March 2018

Issue publication date: 2 May 2018

This paper aims to identify, evaluate and integrate the findings of all relevant and high-quality individual studies addressing one or more research questions about recommender systems and performing a comprehensive study of empirical research on recommender systems that have been divided into five main categories. To achieve this aim, the authors use systematic literature review (SLR) as a powerful method to collect and critically analyze the research papers. Also, the authors discuss the selected recommender systems and its main techniques, as well as their benefits and drawbacks in general.

Design/methodology/approach

In this paper, the SLR method is utilized with the aim of identifying, evaluating and integrating the findings of all relevant and high-quality individual studies addressing one or more research questions about recommender systems and performing a comprehensive study of empirical research on recommender systems that have been divided into five main categories. Also, the authors discussed recommender system and its techniques in general without a specific domain.

The major developments in categories of recommender systems are reviewed, and new challenges are outlined. Furthermore, insights on the identification of open issues and guidelines for future research are provided. Also, this paper presents the systematical analysis of the recommender system literature from 2005. The authors identified 536 papers, which were reduced to 51 primary studies through the paper selection process.

Originality/value

This survey will directly support academics and practical professionals in their understanding of developments in recommender systems and its techniques.

• Recommender system
• Collaborative filtering
• Demographic filtering

Alyari, F. and Jafari Navimipour, N. (2018), "Recommender systems: A systematic review of the state of the art literature and suggestions for future research", Kybernetes , Vol. 47 No. 5, pp. 985-1017. https://doi.org/10.1108/K-06-2017-0196

Emerald Publishing Limited

Copyright © 2018, Emerald Publishing Limited

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

A systematic literature review on educational recommender systems for teaching and learning: research trends, limitations and opportunities

Affiliations.

• 1 Centro de Estudos Interdisciplinares em Novas Tecnologias da Educação, Universidade Federal do Rio Grande do Sul, Porto Alegre, Rio Grande do Sul Brazil.
• 2 Departamento de Ciências Exatas e Sociais Aplicadas, Universidade Federal de Ciências da Saúde de Porto Alegre, Porto Alegre, Rio Grande do Sul Brazil.
• PMID: 36124004
• PMCID: PMC9472736
• DOI: 10.1007/s10639-022-11341-9

Recommender systems have become one of the main tools for personalized content filtering in the educational domain. Those who support teaching and learning activities, particularly, have gained increasing attention in the past years. This growing interest has motivated the emergence of new approaches and models in the field, in spite of it, there is a gap in literature about the current trends on how recommendations have been produced, how recommenders have been evaluated as well as what are the research limitations and opportunities for advancement in the field. In this regard, this paper reports the main findings of a systematic literature review covering these four dimensions. The study is based on the analysis of a set of primary studies ( N = 16 out of 756, published from 2015 to 2020) included according to defined criteria. Results indicate that the hybrid approach has been the leading strategy for recommendation production. Concerning the purpose of the evaluation, the recommenders were evaluated mainly regarding the quality of accuracy and a reduced number of studies were found that investigated their pedagogical effectiveness. This evidence points to a potential research opportunity for the development of multidimensional evaluation frameworks that effectively support the verification of the impact of recommendations on the teaching and learning process. Also, we identify and discuss main limitations to clarify current difficulties that demand attention for future research.

Supplementary information: The online version contains supplementary material available at 10.1007/s10639-022-11341-9.

Keywords: Computer-mediated teaching and learning; E-learning recommendation; Educational recommender systems; Systematic literature review.

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022, Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sensors (Basel)

A Systematic Review of Recommender Systems and Their Applications in Cybersecurity

Aleksandra pawlicka.

1 ITTI Sp. z o.o., Rubież 46, 61-612 Poznań, Poland; lp.moc.itti@ikcilwapm (M.P.); lp.moc.itti@kizokr (R.K.)

Marek Pawlicki

Rafał kozik, ryszard s. choraś.

2 Institute of Telecommunications and Computer Sciences, UTP University of Science and Technology, 85-796 Bydgoszcz, Poland; [email protected]

Associated Data

Not applicable.

This paper discusses the valuable role recommender systems may play in cybersecurity. First, a comprehensive presentation of recommender system types is presented, as well as their advantages and disadvantages, possible applications and security concerns. Then, the paper collects and presents the state of the art concerning the use of recommender systems in cybersecurity; both the existing solutions and future ideas are presented. The contribution of this paper is two-fold: to date, to the best of our knowledge, there has been no work collecting the applications of recommenders for cybersecurity. Moreover, this paper attempts to complete a comprehensive survey of recommender types, after noticing that other works usually mention two–three types at once and neglect the others.

1. Introduction

The digital revolution gave birth to cybercrime, and brought about the concerns about the security of data, privacy and other digital assets of citizens. Cybersecurity has become even more crucial at the outbreak of the Coronavirus (COVID-19) pandemic, when millions of people were forced to turn online almost overnight, without prior knowledge or experience. Never have so many people been so vulnerable to cyberattacks and online mischief [ 1 ].

In the cases when one must make choices without sufficient knowledge, or experience of alternatives, one used to rely on advice and recommendations from other people. It came in several forms, from face-to-face conversations to film and book reviews in magazines, and from letters of recommendation to printed book guides. It is a natural, social process that the so-called recommender systems try to augment and assist in, by giving personalized suggestions and preventing being overwhelmed with the amount of information [ 2 ].

Recommender systems have proven useful in innumerable applications, and each year, new ways of employing the techniques are proposed. In their survey on the development of recommender system applications, the authors of [ 3 ] grouped the possible implementations into eight “e-“ categories, e-government, e-business, e-commerce/e-shopping, e-library, e-tourism, e-resource services, and e-group activities. The application domains have been shown in Figure 1 .

The application domains of recommender systems, based on the survey by [ 3 ].

However, they do not mention the possible applications of recommender systems in cybersecurity.

In cybersecurity, the Security Operations Center (SOC) personnel faces vast amounts of data coming from a variety of sources, accumulated at rapid speeds. The constant flow of information often overwhelms the analysts, making timely and adequate response and mitigation unsustainable [ 4 , 5 , 6 ]. The so-called data triage automation is a well-known problem of SOCs [ 7 ], with the intensity of the domain and the characteristics of incident detection signals strongly degrading human performance [ 8 ]. However, the complexity of the task does not allow full automation at this point in time, which causes an ongoing discourse between the need for keeping human operators in the loop despite their physical limitations and the current state of possible automation [ 4 , 9 , 10 ].

On top of all those issues, cybersecurity is not only the SOCs concern—small businesses, especially in the domain of e-commerce [ 11 ] are prone to falling victim to cyberattacks, as they do not have the budget and the necessary skillset to protect their and their users’ assets [ 12 , 13 ].

One way of bridging the gap between the lack of resources allocated to cybersecurity, inadequate education, the limit of human capabilities and the technical state of automation in the domain is by building a system capable of recommending suitable cybersecurity response and mitigation measures. This solution would deload the human operator, and possibly make up for the deficiencies in education.

However, there have not been many cases of applying recommender systems in cybersecurity. Additionally, to the authors’ best knowledge, there has not been a survey gathering the cases where recommender systems were applied to aid cyberdefenders.

This is why a systematic review [ 14 ] has been conducted to identify the sources describing the applications or recommender systems in cybersecurity. The following work presents the results of the study.

The paper is structured as follows. In Section 2 , the review process has been described in detail and the Research Questions have been raised. Following this, the concept of recommender systems is discussed in detail, and the authors gather the list of recommender types. It is then followed by an outlook on their drawbacks and advantages, usual applications, and concerns about their security. Then, against this background, the results of the literature review are presented, i.e., the existing implementations of recommenders in cybersecurity are surveyed, along with some proposals thereof, followed by the final conclusions.

2. The Conduct of the Study

The desire to gather and describe the applications of recommender systems in cybersecurity, has been the primary motivation for conducting the study. The fact the works on the applications of recommender systems tend not mention this possible use has also contributed to it.

The pipeline of the study course has been presented in Figure 2 .

The pipeline of the study course.

At the beginning, the following research question was defined.

• RQ1: What is the current state of the art regarding the application of recommender systems for cybersecurity?

Additionally, during the study, it turned out that the analyzed works often did not agree about the system of dividing recommender system types. Conversely, some of them mentioned only the basic types (e.g., in [ 15 , 16 ]), or even fewer of them (like in [ 17 ]), while others mentioned other types in various combinations ([ 3 , 18 , 19 ], etc.). Thus, the authors of the study wished to aggregate as many types of recommender systems and another research question was formulated:

• RQ 2: What is the actual, up-to-date and the most comprehensive division of the recommender system types?

The study contained in this paper gathers the applications of recommender systems in cybersecurity found in scientific literature. Before disclosing the cases, the context is set by completing a comprehensive survey of recommender systems, augmenting and unifying the taxonomies found across numerous previous surveys, also including the information found in method descriptions brought up in the investigated research pieces.

The study took place from February 2021 to May 2021. The literature items were gathered, analyzed and selected with a mix of bibliographic methodologies, including the snowballing method [ 20 ], pearl-growing [ 21 ] and citation searching and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), all adjusted to fit the criteria of the PRISMA statement [ 22 ]. Pearl-Growing is a methodology of citation and subject search which involves finding a highly relevant piece of research and using it to find more relevant sources by extracting both the bibliography and relevant keywords, which can then in turn be used in another wave of searches. It is akin to the Snowballing method, which relies on consulting the bibliography of numerous research pieces to find other relevant papers, and then consulting the bibliographies of those to find even more relevant items. The weak point of those methodologies is that they focus on going backwards in time. To offset this circumstance, the authors employed Citation Searching. Citation Search uses the ‘Citation’ or ‘Cited in’ tab made available by the major publishers to find research papers which cite the investigated piece, a procedure which yields more recent items.

First, the keywords used for searching the items were determined. They were the combinations of the words “recommender systems” and “cybersecurity”, “threat intelligence”, or “attack mitigation”. Additionally, this set was expanded to include the alternative spellings: “recommendation systems” and “cyber security”.

Then, the publication databases were determined; the sources were thus searched in journal databases: Institute of Electrical and Electronics Engineers Xplore (IEEEXplore), SpringerLink, arXiv, Elsevier, Association for Computing Machinery (ACM) Digital Library and ScienceDirect. The more general sources such as Google Scholar were not taken into account as they mostly index the works from the aforementioned databases. The keywords were inserted in the search fields of the databases selected as sources. The initial, collective number of all the search engine hits was 1,700,067. The huge number of hits was the result of the fact that part of the search engines returned an immense number of false positives; as they were gradually less and less relevant, the browsing of the results stopped when the researchers decided the items became irrelevant to the study. The breakdown of the initial hits is presented in Table 1 .

Search engine hit breakdown.

At this stage, the titles, keywords and abstract (if available) of the papers were analyzed.

The inclusion criteria for the papers were as follows: the papers had to describe the use of recommender systems in cybersecurity, be published in conferences or journals, and be written in English. During the study, several theses were found which appeared relevant, so they were also subjected to further analyses. The exclusion criteria were as follows: papers did not address the relation between recommender systems and cybersecurity, or were the duplicates of the previously identified items.

Altogether, as a result of the aforementioned bibliographic methods, 393 research items were marked as potentially relevant and further analyzed. Out of them, 86 works were selected for the survey due to their quality and appropriateness to this survey topics, among them 12 other surveys [ 3 , 15 , 16 , 17 , 18 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ]. The papers most relevant to the application of recommender systems for attack mitigation are discussed in the Section 4 .

The following section presents the concept of recommender systems and their types; this will serve as the background for the analysis’ results which will be introduced in Section 4 .

3. What Are Recommender Systems?

The simplest definition of recommender systems is that they are programs attempting to recommend the best items to particular users, with the user’s interest in the item being predicted using the data on the items, the users, and the relations between them. Another definition emphasizes the fact that recommender systems assist and augment the social process of using recommendations of others, to make choices, when an individual lacks sufficient personal knowledge or experience of the possible alternatives [ 2 ].

The recommended items are usually particular products or services, while the users may be both individuals and businesses [ 3 ]. Recommender systems are built to effectively analyze the data and find out only the most relevant information from enormous volumes of data, thus avoiding the information overload and making the service as personalized as possible. An effective recommender system can “guess”, or “predict” the particular user’s interest or preference, based on the analysis of their behavior, or the behaviors of other users. This kind of systems have become an independent research topic in the mid-1990s [ 3 ].

The application of recommendation systems offers several significant advantages. In the online shopping environment, they enable easier finding of items, thus cutting the transaction costs, and ultimately, as they contribute to selling more products, they help yield more revenue [ 15 ]. However, although recommender systems are mostly known for the e-commerce applications, they have been found useful in multiple other domains. For example, in scientific online catalogues, they may enhance the experience, by recommending publications which are beyond the searches. The recommender systems which have entered collective consciousness, i.e., the ones that people are most often aware of are, e.g., the systems used by Netflix [ 30 ], YouTube [ 31 ], Amazon [ 32 ], or Hulu [ 33 ]. In general, recommendation systems have been widely accepted tools for boosting the decision-making processes in various domains [ 15 ]. The following benefits of the application of recommender systems have been listed by [ 23 ]: increased revenue, boosted client satisfaction, better-fit personalization, fulfilling the people’s need for discovery, and scrupulous reporting.

3.1. Basic Terms

Although there exist plenty of algorithms and techniques which fall under the umbrella of recommender systems, they all have several elements in common. According to [ 34 ], each recommender system contains the following three elements:

• transactions.

By items one understands the entity that is recommended by means of the system, i.e., its output [ 35 ]. Items have the attributes users are interested in. They also possess other structures, used to rate/value them. Then, the users are the ones, whose information is used by recommenders to make new item recommendations. Users have various goals and characteristics. Finally, a transaction means the potential interaction between a user and the system. The item rating is based on the set of transactions. In recommender systems, the information gathered from some transactions is used to make a new recommendation [ 34 ].

At the beginning of recommender systems, the features that researchers concentrated on were of two-dimensional, user x item nature. This was caused by the scarcer computational resources as well as by insufficient knowledge. Presently, additional data are gathered and used to make the predictions more accurate, such as demographic, temporal or social network data [ 36 ].

3.2. Creating a Recommender System: The Principles

When creating a recommender system, ref. [ 34 ] proposes for the process to take into account the following dimensions of the recommendation problem:

• “Users: who are the users of the system? What are their goals?
• Data: What are the characteristics of the data the recommendations are based on?
• Application: what is the application the recommender is part of? [ 34 ]”

3.3. Filtering Techniques

There are several techniques used for building recommendation systems. The most popular ones are: Collaborative Filtering (CF) [ 37 ], Content-Based (CB) [ 38 ] and Knowledge-Based (KB) [ 24 ]; they can also be combined to form various hybrids. The abovementioned types are the ones which most researchers agree on. However, there are other, less common methods that are mentioned by much fewer papers, or the researchers have different opinions on which group the methods should be classified into. This paper aims at mentioning at least most of the less-known and -used techniques, and lists them separately from the main three types. Figure 3 shows the division applied for the sake of this work.

Recommender system types by filtering technique; the division devised by the authors for the sake of this work.

3.3.1. Collaborative Filtering (CF)

Collaborative filtering was called “the most mature and the most commonly implemented” technique [ 15 ]. It mostly been applied in the analysis of customer preferences and purchase patterns, and other content which is unable to be described by metadata, such as music or movies. The basic task of collaborative filtering is to find users with similar preferences or tastes based on their opinions, the so-called nearest neighbors; in other words, to search for the items that the user may like, based on the reactions of the users who have a similar taste. Simply put, collaborative filtering analyses big groups of people and aim at finding much smaller sets of users who share their preferences with the user of interest. The items which were liked by the people from the set are the basis for building a ranked list of recommendations; the similarity of users may be calculated in several ways and so may be the recommendations, based on this data [ 39 ]. Figure 4 presents the types of CF presented in this paper.

The kinds of collaborative filtering discussed in this work.

In collaborative filtering, the results may either be predictions or recommendations. The former come in the form of a numerical value Rij , meaning the prediction for the score of the item j , while i expresses a particular user. Recommendations of the latter type come as a list of N items which will most likely be selected by a user [ 15 ].

As this kind of filtering is based on the actions of the existing users, i.e., the recommendation is produced depending on the observations of the actions a new user takes and comparing them with the actions and ratings of the existing users, in most CF techniques, the generated preferences are based on a user–item matrix [ 19 ]. Such a matrix is built of a set of items and a set of users who reacted to some items. As [ 39 ] explains, this reaction may be either explicit (numeric rating, or expressing “likes” or “dislikes”) or implicit (i.e., the user’s behavior: clicking the link, the fact of viewing the item, adding it to a wishlist, or spending a certain amount of time on the item). A sample of a matrix of this kind has been presented in Figure 5 .

A sample user–item matrix.

In this matrix, rows represent the ratings users gave, while columns mean the ratings that a particular item received. Therefore, the third user has rated the third item and gave it a rating of 1, etc. As it is hardly possible for every user to rate every possible item, and realistically, a person usually rates a few items, most matrix cells will remain empty. If most cells of the matrix are empty, it is called sparse; if most cells are filled, then it is said to be dense [ 39 ].

Based on the data, the nearest neighbors are uncovered. The collaborative recommender systems assume that those strongly correlated users will have an interest in similar items. Such users are grouped into the so-called neighborhoods. Consequently, this kind of systems will recommend buying the items which one of the similar users bought and the other did not, with the prediction value for that item being drawn from the proximity of this neighbor to the average rating of this user; in other words, a user is suggested to choose the items that other users in their neighborhood found favorable [ 15 ].

To calculate the similarity between users or items, several methods may be employed. One of the most widely used ones is the Pearson’s Correlation Coefficient (PCC); it has also been proven to perform better than other metrics by several researchers (e.g., [ 40 ]). It is used to determine the strength of the linear relationship between two variables. It is expressed by Equation ( 1 ) [ 36 ]:

where I u v is the set of the items that both users u and v have rated, while r u i and r v i are the ratings that both users gave to the item. In this metric, the similarity is measured on a scale from −1 to 1, where −1 is a perfect negative correlation, 0 means no correlation, while 1 represents a strong positive correlation. The users who display high positive correlation values have bought very similar items.

Another function which is commonly used in similarity calculations is the cosine similarity, shown in Equation ( 2 ) [ 36 ].

Cosine similarity calculates the similarity by measuring the cosine angle between two vectors of an inner product space [ 41 ]. The input parameter u is a user, while v is an item. If the item matches the user exactly, the angle between two vectors would be 0 degrees, the cosine of which would be 1. On the contrary, the angle between the most “dissimilar” vectors would be 90 degrees, leading to the score of −1. All the values between −1 and 1 represent intermediate (dis)similarity [ 35 ].

Other algorithms used for calculating similarity are, for example, the Euclidean distance, Manhattan distance, Spearman correlation, entropy-based uncertainty, men-square difference, Minkowski distance, etc. [ 35 , 41 ].

Collaborative filtering approaches fall into two main categories, memory-based and model-based recommendations. The memory-based techniques can be further divided into item-based and user-based ones; this paper also discusses additional, less-known ones. In turn, by the model-based techniques one usually means matrix factorization algorithms and deep learning [ 42 ].

Memory-Based Collaborative Filtering

In the memory-based collaborative filtering, the recommendations are made based on the whole collection of the rated items. List of recommended items is built based on the items chosen and rated frequently by the users belonging to the same group. These techniques are composed of the following steps: data pre-processing, selecting the set of K users/items that show the greatest similarity to the particular user/the items they have already rated (a.k.a., selecting the neighborhood) and computing the recommendations, i.e., generating predictions and listing the top-N of them [ 41 ]. The similarity between users or items can be measured using several metrics, usually the Pearson correlation coefficient and cosine similarity.

By this approach, the users are matched by the recommender engine based on their taste in the product in question [ 43 ]. Simply put, in the user-based collaborative filtering, the user U and the set of users similar to them are selected. Then, the rating for an item is searched for; the user has not rated the item. By choosing N of the similar users who did rate the item, the rating is then calculated.

This type of recommendation is based on the concept that customers tend to choose items similar to the ones they expressed an interest in, and at the same time will not buy the items they are not interested in. In this kind of system, the user–item matrices are used as an input for finding the relations among various items. The analysis of how the items interact is the basis for generating a personalized recommendation.

The item-based algorithms tend to perform better than the user-based ones, as the latter ones are known to have scalability issues, i.e., when the user–item matrix is of substantial size, the computational time becomes very considerable. As the relations between items are more stable than those among users, item-based algorithms usually need less computational time to make correct predictions, or the computations may be performed offline. Then, the rating may be calculated from the Pearson’s correlation coefficient and the N nearest neighbor.

Very seldom, the researchers classify other methods as memory-based collaborative filtering, such as Predictability Paths, cluster-based smoothing and trust inferences in [ 41 ], and so on.

Collaborative filtering faces several challenges, one of them being the so-called sparsity. It results from datasets lacking big amounts of data, which makes it hardly possible to make accurate recommendations. Another issue is called a cold start. It arises after a new user has rated too few items for the matrix to make accurate predictions based on the available ratings. It can also happen when the ratio of items to users is very high. In such a case, it may be impossible for the user to rate enough items for the recommender to make a prediction. Both sparsity and cold start result from the fact that it is hard to match users with a low number of ratings to similar neighbors. The possible solution to these problems is, e.g., to apply the spreading activation technique. It consists of turning the data from the matrix into a graph and finding the relations between users and items. In the graph in which distance is the number of edges between the item and the user, the recommendations are built depending on how close the item is from a user [ 44 ]. Another possible solution makes recommendations based on similar user ratings and probability. The accuracy of suggestions improves using the information which does exist for the users [ 45 ]. Finally, the accuracy may be improved by using a hybrid system, i.e., exploring other data sources, such as item attributes or demographic data [ 19 ].

Another issue which can directly impact the recommendation process are the so-called grey-sheep customers, i.e., the customers whose tastes are either unique or exotic. Thus, the recommendations made for them could be of poor quality, and, conversely, the grey-sheep customers negatively affect the recommendations made for all the other customers [ 46 , 47 ].

Model-Based Collaborative Filtering

Some of the drawbacks of the item-based and user-based models have been addressed by the model-based ones. The Netflix competition in 2006–2009 was said to spark interest in this type of technique [ 48 ]. The advantage of this approach lies in the fact that the data are first processed in an offline manner, subsequently leading to building a model. This makes it possible to avoid real-time computations. Additionally, by this approach, machine learning methods are used to find user ratings of unrated items. Thus, the model-based recommenders, after training, can make very accurate recommendations [ 25 ]. However, it must be noted that without sufficient training, the models may be less accurate than the memory-based methods [ 41 ]. Generally speaking, the model-based approach involves reducing or compressing a user–item matrix which is substantial in size, but sparse. The technique is called dimensionality reduction. It relates to the cases where just a small part of the available items has user ratings. Memory-based techniques are not able to generate accurate predictions in such cases; on the other hand, model-based algorithms are designed to find latent factors (i.e., implicit, product-specific features) that help predict the lacking ratings [ 49 ]. As the user–item matrix consists of two dimensions—the number of users and the number of items—the dimensions which are mostly empty contributes to boosted performance of the algorithm. The dimensionality reduction is usually performed by means of matrix factorization. It can also be done with autoencoders, clustering-based algorithms, etc.

It consists of breaking a large matrix down into a product of smaller ones [ 16 , 41 ]. The algorithms employed for factorizing matrices are Singular Value Decomposition (SVD), Principal Component Analysis (PCA), Non-negative Matrix Factorization (NMF), and so on [ 41 ]. Using the algorithms, the features may be extracted for every product that has been rated. Then, a comparison is made between them and the items which do not have any ratings and finally, based on this, the rating is predicted [ 35 , 36 ].

Usually, by this type of algorithms, one means the k-Nearest Neighbors (kNN), a Machine Learning (ML) technique. The aim of this algorithm is to search for clusters of similar users, the similarity being based on the users’ past behavior (like their ratings, the items they had already bought, etc.). Although the user-based collaborative filtering is based on the same concept, with the kNN the similarities are found based on an unsupervised machine learning model. Additionally, the number of similar users is limited to k [ 42 ]. It is worth noting that some researchers argue this technique does not belong to the model-based recommenders, but rather, it should be classified as a memory-based one, as though it is a machine learning technique, it is of non-parametric nature [ 42 ].

3.3.2. Content-Based Filtering

The content-recommendation systems are said to be most effective for recommending text-based items, such as documents, news items or web pages [ 15 ].

In the content-based recommendation systems, the predictions are made based on the data on the items and past actions of users, and not on the other users’ choices [ 15 ]. The concept behind this kind of filtering is that a user tends to buy future items which in some ways are similar to the ones they previously bought. The similarity of items is calculated based on several their features and/or attributes. The main challenge of this approach consists of the collection of the data about items. Lack thereof results in sparsity, as in the case of collaborative filtering. Although in content-based recommender systems there is no need for a vast number of users or item ratings (like in collaborative filtering), they require a proper amount of information to make accurate predictions. The features/attributes used in this type of filtering include the metadata or even the actual contents of documents [ 19 ]. Generally speaking, in CB recommender systems, two main techniques have been applied for making recommendations. One of them, the heuristic one, uses traditional tools such as cosine similarity measures. The other approach employs machine learning and statistical methods, using the past users’ data for learning the models which are then able to predict users’ interests [ 50 ]. With this method, when making a recommendation, a vector is built, in which 1 means a word is present within a document, and 0 indicates the document does not contain it. Following that, the vector is compared with other documents by the recommendation system. One of the challenges to this method is the fact that the vector is in favor of longer documents, and that the frequency of a word in the document is not taken into account [ 51 ]. To solve this problem, the analysis of documents should be made by means of the technique called Term Frequency-Inverse Document Frequency (TF-IDF). TF takes into consideration the number of appearances of a word within a document. In turn, IDF attributes greater weight to the words which are only present in one document, thus helping to emphasize the difference between it and other documents [ 52 ]. Other means of modeling the relations between the items could be probabilistic models, such as the Naïve Bayes classifier [ 53 ], decision trees [ 54 ] or neural networks [ 55 ]. In these techniques, the model is learned using the machine learning or statistical analysis techniques.

The recommendation process begins with gathering the information on and the ratings of the items previously bought by a user. Then, the system searches for similar items. The similarity is determined as follows: using a similarity calculation, the items are collected into neighborhoods; in other words, neighborhoods are built by making a comparison between new items and the items already in the inventory. If a user bought an item, it counts as a vote for the neighborhood associated with the item. An item ought to be recommended to a user if they rated highly k of the nearest neighbors. This method is said to require relatively little data to make accurate predictions, and to be easily adaptable [ 51 ], i.e., if there are changes to the user’s profile, the recommender can swiftly adjust the recommendations [ 15 ].

It is helpful to keep long-term profiles of users, as it makes the recommendations more accurate, although short-term profiles are useful as well, e.g., when users tend to grow new interests in items. Apart from the problems mentioned above, the content-based filtering may face other challenges. The type and quality of the data associated with the items is of vital importance to CB recommenders. If the item is not text-based, extracting its features may prove to be a daunting task. In turn, with the text-based items, the systems consider words only, while other, more subjective attributes are neglected. A possible solution could be entering the attributes in a manual way; however, this solution is not entirely realistic due to the fact how time- and resource-consuming it is. The CB recommendation systems may experience sparsity as well, when there are not enough item attributes. There is a concern to be had if the content-based system begins to make suggestions which are too similar. For example, it may recommend a user the same item they bought previously, but from a different company. To avoid this kind of problem, the system may either be supplied with some diversity, or the items which are too similar must be excluded, by means of a filter [ 56 ]. It is also possible that a content-based recommender system experiences the cold start issue; however, as this type of recommender system needs just a few ratings or pieces of information on past actions, the problem is not as severe as with the collaborative filtering recommenders [ 51 ]. Lastly, content-based filtering may sometimes suffer from overspecialization, i.e., the situation in which the system starts recommending items which are very similar to one another, without suggesting novel items [ 50 ].

3.3.3. Knowledge-Based Filtering

As [ 57 ] put it, knowledge-based recommenders are those that use different knowledge sources from the ones used by collaborative filtering and content-based recommender systems. Unlike the recommendations described before, the knowledge-based ones do not rely on the user–item data. Their predictions are based on explicit rules about the problem domain, as well as on the attributes of items. They do not track the actions of users and do not collect ratings; instead, this type of system gathers specific requirements from the users. Therefore, there is no problem with sparsity, even in the case of the seldom-bought items [ 57 ].

Generally speaking, knowledge-based recommenders may be divided into two approaches, constraint-based and case-based ones. Some researchers, such as [ 58 ], classify utility-based as belonging to the knowledge-based recommenders. For the sake of this paper, utility-based recommendation systems have been discussed separately.

The constraint-based recommenders are called this way, as they in fact compare the attributes of an item within the constraints, i.e., the requirements that the users give, or the constraints from the product domain. In other words, in this method, the recommendation equals satisfying the constraints, with the products which fulfil the constraints being the good recommendation [ 58 ]. This type of recommendation may also be made using a conjunctive query over the product database. In this approach, the user requirements for attributes are connected and a conjunctive query is created. Following this, a database query is made, and the items meeting the constraints are returned.

There are also case-based recommendations systems. They make predictions based on the similarities between items and the requirements. According to [ 59 ], the distance similarity of an item is dependent on the sum of all the similarities of attributes weighed by the requirements. Consequently, the distance between an attribute and a requirement is what the other items of interest depend on. In other words, local similarity is used for finding similar items. It is found by dividing the distance of the attribute of the item from the desired attribute by the total range of the attribute. Other case-based recommendations are made using a query-based paradigm, which is effective as long as users have specific requirements. The abovementioned case-based recommenders in fact depend on suggesting items that are close to the requirements, and do not need to fulfil all of them.

It is important to mention that although, following [ 60 ], this work classifies case-based recommenders as the knowledge-based ones, there are researchers such as [ 61 ], who classify them as content-based recommenders instead, or think of them as being a bigger, more general group, encompassing content-based and knowledge-based recommenders [ 62 ].

The limitations of the knowledge-based recommenders result from the inability to meet all the requirements the user has given. This may result in giving a null response. To avoid this, the system should be designed to suggest the users to relax the requirements, or even do it automatically. This way, the system is eventually able to suggest an item which is as close as possible to fulfilling the original requirements. Another way of making this type of system to give better predictions is to use a divide-and-conquer algorithm QuickXPlain [ 63 ]. The algorithm identifies the conflict between the user requirements and the potential items, i.e., constraints that the system is not capable of fulfilling.

3.3.4. The Comparison of the Three Main Filtering Approaches

Each of the three aforementioned methods has its own advantages and flaws. The main, most prominent advantages and disadvantages, as well as the applications the techniques are the most suitable for, have been shown in Table 2 .

The comparison of collaborative filtering, content-based and knowledge-based recommender systems; compiled by the authors based on the subject literature.

3.3.5. Hybrid Recommender Systems

As [ 3 ] remarks, the three main types of filtering play a dominant role in the majority of applications today; however, as the issues characteristic of the three aforementioned filtering techniques influence the recommendation’s quality in a negative way, hybrid filtering has been proposed [ 67 , 68 , 69 ].

The so-called hybrid recommender systems are usually based on the three main types of recommender systems. They can make more accurate predictions, as they combine different approaches to gather information. The final results depend heavily both on the used algorithms and the method of hybridization, i.e., the way and order in which the outcomes of an algorithm relate to the other ones. The recommenders that suffer from sparsity, i.e., CF and CB, are better for solving the issues where there is an abundance of data. On the other hand, knowledge-based recommenders cannot find associations between users and items. CF and CB systems can adjust to the changing needs of users, but the former systems outperform the latter ones in the cases when there is a lack of item attributes. Conversely, content-based ones can work better even without having a great number of user–item ratings to analyze.

To exploit the strengths of the systems, and not rely on their weaker points, hybrid recommenders are constructed to use various techniques on the same dataset. Afterwards, the resulting data are combined to make the final recommendations. For the combinations to give valid results, they must be given static weights. The weights may be influenced and changed, e.g., to reflect the users’ feedback.

One such hybrid might be a system which bases the used technique of computing recommendations on a given context/situation. Such a system is called a switching hybrid. If there were sparsity, it would first use a knowledge-based system and then, after the users’ rating, it would switch to collaborative filtering, and so on. The decision if and when switch the technique may be made based on the fact if the default configuration is able to give a valid result, or not [ 19 , 70 ].

A comprehensive list of the various hybridization techniques has been presented in Table 3 .

The hybridization methods, based on [ 43 ].

Besides the techniques mentioned in the table above, there are also other manners of combining the filtering methods. As [ 71 ] remarks, it is possible to, e.g., make a unified recommendation system, while treating user rating prediction as the issue of machine learning, with probabilistic latent semantic analysis, or combine the similarities in a unified kernel space, where the predictions are made based on support vector learning, etc.

Another approach to combining recommendation methods is by applying graphs. In a database of this kind of a recommender system, data are contained in nodes; their edges are linked together. The links representing the relations may be either weighted or unweighted. Thus, the relationships between nodes are easy to retrieve, especially if the entities in the system are strongly connected [ 72 ]. Although it may take slightly more time to compute [ 73 ], this method is intuitive and available, and helps overcome the issues such as data sparsity [ 71 ]. The graph-based recommendation systems have already been tested in various applications, such as in a digital library [ 74 ], collaborative ranking [ 75 ], and making recommendations of drugs [ 76 ], books [ 73 ], and movies [ 77 ], and so on. Most of the aforementioned solutions are based on the Neo4j graph data platform; it has also been deemed the best choice among graph databases by [ 78 ].

3.3.6. Other Types of Recommender Systems

There are several other, more specialized and thus less widespread recommender system types. It must be noted that there is no common agreement as far as the ontology is concerned, i.e., some of the types mentioned below are classified as belonging to the abovementioned groups by some researchers. This may be due to the dynamically changing domain and state of the art, or the fact that several methods show features which may belong to more than one group/type. In addition, several filtering types have been distinguished according to the technology applied and not the features taken into account when making recommendations; thus, some overlapping occurs, or the same technique is classified as more than one category.

Sometimes called CIRS, the computational intelligence recommender systems are the ones which include Artificial Neural Networks (ANN), Bayesian techniques, clustering techniques, genetic algorithms, fuzzy set techniques, etc., in their recommendation models. Bayesian classifiers solve classification problems based on probabilistics. They often are part of model-based recommenders, or help create a model for the content-based recommenders. With a Bayesian network being used for recommendations, the nodes correspond to items, while the states correspond to all the vote values possible. Thus, each item in the network will have a set of parent items—they will be its best predictors [ 3 ].

Artificial neural networks have also been used as part of recommendation engines. For example, ref. [ 79 ] have applied one in a personalized TV recommendation system. They trained an ANN of three layers with the back-propagation method. A hybrid movie recommender was presented by [ 80 ]. The trained ANN representing the preferences of individual users was responsible for content filtering.

To make the computational cost of finding k-nearest neighbors lower, clustering may be applied. Clustering consists of assigning items to groups. This way, the items within groups are more similar than the ones in other groups. With recommender systems, this may result in, e.g., smoothing the unrated data for users, by predicting the unrated items from a group of related items. Additionally, with the assumption that the nearest neighbor is within the Top-N most similar clusters to the active user, there is only the need for selecting the nearest neighbors in the Top-N clusters. This results in greater scalability of the system [ 3 , 81 ]. Furthermore, the technique can help tackle the cold start issue, by grouping items [ 82 ].

Genetic Algorithms (GA), i.e., stochastic search techniques, have mainly been applied in K-means clustering, for improved online shopping market segmentation, such as in [ 83 ]. Similarly, ref. [ 84 ] have used a GA method for obtaining optimal similarity function. Finally, several techniques based on the fuzzy set theory have been used to handle the non-stochastic uncertainty, e.g., the information being imprecise, or the classes of objects not being sharp enough [ 3 ].

The rapid increase in the social networking tools has directly resulted in social network analysis becoming an important part of recommender systems. Recommender systems offer the possibility for the users to make social interactions among one another, such as comments, adding to friendlist, etc. Based on these interactions, recommendations can be made. The social network recommendations rely heavily on the concept of “trust”. In human interactions, a person’s decision (to buy something) is more likely to be influenced by friends’ opinions than by an advertisement. Trust, i.e., the level of how one user trusts others concerning a product, is helpful in making predictions where the data on similar neighbors would be too sparse otherwise. Indeed, a positive correlation between trust and user similarity has been found scientifically [ 85 ]. In addition, the authors of [ 3 ] discuss other social interactions and relations which are used for making recommendations, namely social bookmarks, physical context, social tag, “co-authorship” relations, “co-citations”, and more.

In recommender systems, context is understood as any kind of information which may characterize a situation or an entity, such as a person, place or an object that is relevant to the user–item interaction [ 86 ]. Context may thus mean time or the company of other people. Applying context in recommendation process makes the results more personalized and appropriate. As [ 87 ] claim, the rating function is no longer two-dimensional, i.e., (R: User × Item → Rating); instead, it has become multi-dimensional (R: User × Item × Context → Rating).

Group recommendations are a method of making group suggestions “when group members are unable to gather for face-to-face negotiation, or their preferences are not clears despite meeting each other [ 3 ]”. They are used for recommending films, music, websites, evens or travels. The process of clustering people into a group may follow several strategies, based on the research of decision-making or social choice theory, such as the theory of average, least misery, most pleasure, and so on [ 44 ], as well as the strategies of sum or approval voting.

Some researchers, such as [ 23 , 88 ], describe the demographic filtering as a separate filtering technique. By this method, the system gathers the information such as age, gender, education level, place of residence, as well as users’ opinions on items. Then, the similarities are found between the users’ ratings; finally, the data are filtered by users’ age or the area they live in. According to [ 18 ], these methods form similar correlations to the ones present in collaborative filtering, but unlike the collaborative and content-based techniques, they may not need a history of user ratings. However, they may raise some security issues, due to the nature of data they gather [ 23 ].

Lastly, there are utility-based recommenders. In them, the utility of an item for a user is calculated, with gathered the users’ interest level in that attribute. As with the knowledge-based recommenders, the utility-based systems are not based on building long-term generalizations concerning the users. Rather, the recommendation is made based on the assessed match between the set of available options and the users’ needs. Specifically, the utility-based recommenders calculate the utility of each object to a user and then make recommendations based on that. The weight of the attribute may also be calculated by the system, lowering the load on users. To do so, the total utility must be determined. It is the sum of all the item values, i.e., the weight multiplied by the similarity function. The system returns a list of items ranked according to their similarity level to the user requirements [ 59 ]. There are various approaches to what makes utility and how to compute it, but the general idea is that the utility function should be based on item ratings that the users offered to describe their preferences [ 37 ]. One of the main advantages of this filtering technique is that the utility computation can be influenced by some non-product attributes (e.g., product availability). This way, for a user who needs to receive an item as soon as possible, such a system could enable trading off price against delivery schedule [ 18 ]. As mentioned before, utility-based recommender systems are either seen as separate method of filtering [ 18 ], or as being part of knowledge-based recommenders [ 19 ].

4. The Result of the Study—The State of the Art of Recommender Systems for Cybersecurity

This chapter aims at presenting the results of the literature study, by gathering the instances when recommender systems were employed in cybersecurity. This will allow formulating the answer to the Research Question 1.

In their work, Polatidis et al. remark that recommender systems had not been used for attack prediction before [ 89 ]. Thus, they have proposed a method which combines collaborative filtering recommendation systems with the methods of discovering attacks paths. The attack graphs were built based on data sourced from maritime supply chain infrastructure. Their tool was validated and evaluated experimentally, proving its effectiveness. It uses a “parameterized version of multi-level collaborative filtering method (…) although other methods could be applied according to the scenario and the available data [ 89 ]”. Their method first uses CF, and then the k-nearest neighbors are rearranged, by the similarity value and the number of co-rated items. More specifically, the tool first finds all the paths that could be used to perform an attack. Then, a recommender system is applied to predict what attacks might take place within this network. The method the authors used was a parametrized version of multi-level collaborative filtering, although they make a remark that other algorithms could be applied as well. By this method, after CF has been applied, the k-nearest neighbors list is rearranged, to reflect the value of similarity and how many items had been co-rated. For attack classification, the characteristics from the abovementioned method was used. The authors first employed classical CF using the Pearson Correlation Coefficient, as shown in Equation ( 3 ) [ 89 ].

There, Sim ( a , b ) relates to the similarity of users a and b , r a , p is the rating of user a for product p , r b , p is the rating of user b for product p and r a , r represent user’s average ratings, while P stands for the set of all products [ 89 ]. Then, the authors analyzed the similarity values and the co-rated vulnerabilities. Based on this, attacks were classified, on a scale from very high to very low. The last step consisted of checking whether there were any attack paths between the assets. The authors mentioned that they did not just use classical CF without additional parameters, as it would be less effective than the method they had proposed.

Lyons [ 19 ] also proposes using a recommender system in cyberdefence domain. The main goal of this effort was to make the Observe => Orient => Decide => Act (OODA) loop of cybersecurity defenders faster, chiefly by speeding up the decision part. They propose implementing existing, effective Intrusion Detection System (IDS) solutions with a recommender system to provide the best steps to take in a given situation, so that the IDS provides the information to the recommender system to predict the likelihood of certain events for given nodes of the network. The system returns a list of actions—the actions are ranked and the highest one is supposed to mitigate the most events that the system predicted are likely to happen. The input is provided by the IDS—Network Modeler proposed in [ 90 ]. It consists of three main parts: sensors, database, and modeler algorithm. The sensors are mostly Commercial-Off-The-Shelf (COTS) software besides the custom-written host monitoring (java-based program sending updates to the sensor machine). For observing the network, Snort and Nmap were used. The database is a combination of the various data gathered by the system, i.e., Snort, Nmap, vulnerability scores, host monitoring software, etc. The host monitoring software sends updates to the base on the sensor machine; the data such as CPU and memory usage, etc., it also controls the antivirus software for every host. The task of the modeler is collecting and classifying the data about the health of the network, i.e., determining if there is any threat to the network, by analyzing the data from all the sensors, as well as generating a list of possible actions which may be taken on client machines/firewall machines to mitigate the threat. (e.g., update operating system, upgrade application, create/delete user account, disable port, block source IP address, etc.). Then, the output XML file describing the network state is created so that every host receives data about the attacks, infections, user accounts, etc., as well as the actions which could be undertaken for that specific host.

The recommender system used in this model is of collaborative type; it uses the user-based nearest neighbor recommendation algorithm; the users and products being replaced with nodes and their attributes. The similarity between the nodes and the attributes for each node is calculated using the Pearson correlation coefficient. The values range from −1 = strong negative correlation to 1 = strong correlation. Then, based on the similarity values, the prediction is calculated. If the nodes are similar (i.e., have similar vulnerabilities, thus one may suppose they are prone to the same kinds of attacks), then the events occurring on them will result in higher predictions. As the author explains, “the predicted rating for nodes are the values used to determine vulnerabilities that the knowledge-based recommender system needs to consider when generating defensive actions”. Then, a knowledge-based recommender system is applied for making recommendations of defensive actions, with the selected paradigm being called a constraint-based recommender (The system depends on the previously set knowledge of the actions which are used to mitigate/counter certain cyberattacks). At the start of the recommender system, all the actions it knows are loaded from the XML file; the file is loaded only once, making the first computation time greater than the other ones. Various exploits may be mitigated by means of one action; this is why the proposed action ought to be able to counter the greatest possible number of the predicted attacks. Thus, the presented list of suggested actions is sorted according to the number of the mitigated attacks. However, in this solution, the final choice of the action is left to the cyberdefender. In this recommender system, all the possible actions which may be taken to mitigate the threat are considered.

After calculating all the action values, the system sorts them and presents as recommendations, in descending order. The author remarks that to find out which recommendation algorithm proves the most effective, it is necessary to observe multiple kinds of them. The so-called hybrid recommender systems comprise of the collaborative, content-based and knowledge-based; they may be combined using various techniques. The author believes that the nature of cyber threats requires one to use a knowledge-based recommender, as the collaborative and content-based ones will be affected by the issue of sparsity. Thus, in their system, the author has combined a knowledge-based recommender with a collaborative technique.

In their paper, Sula et al. [ 35 ] have proposed applying a recommender system to support the mitigation of the Distributed Denial of Service (DDoS) attacks, by helping in the decision-making process of the defender and suggesting the most appropriate cybersecurity solution. The system, called ProtecDdoS, takes into account the requirements of a customer, i.e., the service type (proactive/reactive), the type of a DDoS attack, the coverage region (city, country or even continent), deployment time (minutes, hours, days, weeks), leasing period, and the client’s budget. The three last parameters may also be given priority, e.g., if budget’s priority has been set to high, it will show the cheapest solutions first, while the remaining characteristics will be close to the selected ones. The recommendation engine uses the following similarity algorithms: cosine similarity, Euclidean distance, Manhattan distance, Minkowski distance and the Pearson correlation.

The client-side of the solution has an Application Programming Interface (API) implemented using React. All the abovementioned parameters can be set there. The attack types may be entered from the drop-down field, but can also be imported from attack log files. Once the user’s profile has been set, the recommendation is given, i.e., a list of suitable services is presented, along with their description. The recommendations are supported with visual representations of the results, in the form of plots, to provide clients with better grasp of how the recommender works. Moreover, to provide the maximum security, it is possible to check the service’s hash, to see whether it was manipulated or not and can be trusted. The users can add more services to the database, too.

The server-side of the solution was implemented using Flask 1.0.2 The recommendation process bases on two components: Service Helper and Recommendation Engine. The helper component is responsible for filtering irrelevant data from the dataset, according to the user’s preferred features. It also calculates the index of characteristics, by assigning an integer value to each characteristic taking variables into consideration. Then, the Recommendation Engine calculates the customer index and service index. Then, similarity score is calculated using various algorithms and a resulting list of services is returned, sorted by the similarity index.

Soldo et al. [ 91 ] have proposed a method of predicting future attacks/malicious activity from past behaviors, calling it the “predictive blacklisting”; blacklists being understood as logs of past attacks, attack sources, etc. The recommendation system part was inspired by the one used by Netflix. As the authors said, they “framed the problem as an implicit recommendation system, which paves the way to the application of powerful machine learning methods. (…). Given a set of attackers and a set of victims, a number r is associated with every (attack source, victim destination, time) triplet according to the logs: r can indicate, for example, the number of times an attacker has been reported to attack a victim over a time period. More generally, we interpret r as the rating, or preference, assigned by an attacker to a victim [ 91 ]”.

The architecture of their model consists of three algorithms blended in a linear way; a time series model, responsible for accounting for the temporal dynamics, followed by two neighborhood-based models. The first of the two latter models is a modified kNN model, it predicts the attack while concentrating on finding similarities between the victims who were attacked by the same sources, ideally at the same time. The other algorithm belongs to the co-clustering type; its goal is to discover a group of actors that attack a group of victims at the same time, and it does it automatically [ 91 ]. The method differs from a traditional recommender system in several aspects. First, the authors addressed the issue of the rating matrix varying over time due to the changes in attack intensity. In traditional recommender systems, the matrix remains static. Another significant difference is that in recommender systems, the users provide the ratings themselves. In the case of this tool, as the rating is made based on the attacks reported in the logs, it is implicit.

The tools were tested on real-life data from the dshield.org website, which captures hundreds of millions security logs from a great number of websites. The authors claim that their solution, when compared to the state-of-the-art methods, proves significantly more accurate and more robust against poisoning the dataset.

In their paper, Franco et al. [ 92 ] (as well as [ 40 ]) have presented the MENTOR system, which they describe as a “support tool for cybersecurity”. It was designed to be able to recommend the most suitable protection measures (such as specific anti-malware tools, firewalls, etc.), as the market for such services is booming and end-users may not be able to find the best solution themselves, especially when under cyberattack, needing an ad-hock response and thus in a hurry. The system can recommend both the most suitable prevention and mitigation measures and does it according to various demands, i.e., taking into account the profile of a customer, their budget, as well as the properties of an attack. The architecture is structured as follows: first, the Service Requestor gathers the data concerning the attacked infrastructure as well as the attack itself from the monitor logs. The relevant data are also stored in a database for future analysis. Then, the information is sent to the Extractor and the recommendation process begins. It comprises of several steps; first, the data are analyzed and the correlations with the attack type are found using the Classifier component, by comparing the data to previously identified and known attacks. This allows determining the best mitigation measure, by means of Service Aggregator, which contacts various service providers and gathers the information on available services and their features, such as the region they cover, their price, etc. Based on this information, a list of protection services that could be used is created; then, the Aggregator is queried by Retriever. It can address the clients’ demands fully or partially, providing the most desirable price, performance or technological solutions. Then, the data from the Retriever is sent to the Recommendation Engine. At this step, based on the requirements derived from the Retriever, it uses several different algorithms to find the most suitable solution. Lastly, the customer input is used by the recommendation engine to find out which solution from the list is the best recommendation.

The system’s recommendation process was assessed by means of four commonly used measures of similarity, capable of quantifying the similarity of two items: (1) Euclidean distance, (2) Manhattan distance, (3) cosine similarity and (4) Pearson correlation. Measuring the similarity in a geometric way is possible, as the available protection services and customers’ requirements are mapped as vectors in space. In other words, the similarity may be evaluated when their magnitude and attributes are presented as directions in space.

The recommendation process starts when an integer array is created by indexing the parameters required by the client, and each service. Then, the properties of each service are indexed. Next, the profile of a customer is mapped as the Y vector, while the protection services as the X vector. This constitutes the input for the algorithms assessing similarity. Lastly, a similarity dictionary is built of the ratings. Service ID serves as a key here; owing to this, it can be used for the purposes of exporting or plotting the similarity.

MENTOR’s prototype consisted of a web-based user interface and the Recommendation Engine. A customer may set the requirements and prioritize their demands (high => low) using a dashboard. The prioritizing affects the final recommendation, by neglecting the remaining, less prioritized demands. The Recommendation Engine is also capable of running without the dashboards; in such a case the system acquires necessary input through the MENTOR’s API. Additionally, the choice of the recommendation algorithm may also be made by the end-user. To make the choice easier, MENTOR gives the user various kind of information on the classification results, e.g., in the form of graphs plots, comparison of the vectors representing the profile vs. the vectors of each service, etc. It may be a significant decision, as the results can vary greatly, depending on the methods used. This is so, as the features of a service are shown as a vector in space, some properties (especially the variables of high-magnitude, such as price) may drastically change the vector’s direction in space and influence the rating. This in turn may result in the services specified in the client’s profile, and thus favored in the recommendation process, may turn out to be worse than other possible choices. This is true for the algorithms based on distance (Euclidean, cosine, Manhattan), therefore the authors suggest that the Pearson correlation may be more accurate here, as it shows invariance to the elements’ magnitude. Another suggested way of overcoming the problem is to group the vectors of the services for every attribute and then consequently make a comparison between the service attributes and client-specific attributes in a 1-1 way. Thus, it is the average rating of the service’s attributes which constitutes the final rating. However, the authors notice that the attributes of the services which perform better than those specified by the customer will receive worse ratings, so their suggestion is to rearrange the input attributes to reflect the best conditions possible. This way the algorithm will provide the best option, not the one which is the closest to the request of an end-user.

Esposte et al. [ 66 ] have proposed using a recommender system which would collect cybersecurity alerts gathered from various external sources and recommend them to any person with network administrator profile. The main idea behind this was that a network administrator may be flooded with security alerts, but not all of them are relevant. Based on the administrator’s preferences and ratings, the alerts could be filtered using a recommender system. An interesting assumption is to make sure that some items will always be recommended, even if the user has not provided any requirements yet. The system is a hybrid one; first, the general ranking scores of items are calculated using the Bayesian Average, then, the collaborative filtering and content-based filtering are applied. The greater the ranking scores are, the better ranked the item is going to be. Then, the ranking is computed again, after adding weights for the elements, e.g., adding the greatest weight to the critical votes, etc. In cases when there are no rated items, the most recent cybersecurity alerts are shown. Otherwise, collaborative filtering and content-based approaches are applied. In the CF part, the similarity of users to their neighbors is calculated using the Jaccard similarity coefficient. The greater the coefficient, the greater the similarity. Then, the list of top-N recommendations for a user is computed. The other list is created using the content-based method, where the user’s interest in items is used to weight them, and the items the user is interested in have their tags added to the user’s interest. The higher the weight of the item, the more interest the user has in it. After performing collaborative filtering and the content-based recommendations, a mixed hybrid approach is then used to come up with the list of the top-N recommended items.

As the authors say, their “work advances the state of the art in cyber security by proposing a new model for gathering relevant information on cyber security alerts based on recommender system methods [ 66 ]”. The model was evaluated in an offline experiment and the results show it could be applied in the cybersecurity alert recommendation process. It is worth noting that the authors redefine the elements of a recommender system to suit the needs of cybersecurity administrators: the “item” in this case means a security alert, and its content elements are “item attributes”. The “user” here is the network administrator. By a “transaction” in this context one understands the potential interaction between “users” and “items”.

Casey et al. [ 93 ] discuss a full implementation of a recommendation-verification system in the context of defense against malware. They argue that it is possible to employ machine learning methods in learning the trace features of malware families. They present the requirements such a system would have to meet, and emphasize the significance of its interpretability. In an experimental way they prove the feasibility of the solution they propose.

In their paper, Du et al. [ 94 ] present the problem of People-Readable Threat Intelligence (PRTI), the amount of information in which may be overwhelming for cyberdefenders. They notice that making recommendations in this particular field is challenging, as the data tend to expire and become outdated very quickly, traditional knowledge graphs are not really suitable for this purpose owing to large amounts of noise in data, and the language of PRTI is highly condensed and specific. Thus, they proposed a knowledge graph created specifically for the PRTI recommendation equipped with a denoising entity extraction module. Then, they propose a framework which uses a Knowledge-aware Long Short-term Memory neural network (KLSTM) for providing external knowledge for PRTI recommendation, using information from the knowledge graph. They prove experimentally that their method of combining a Latent Drichlet Model (LDA; a three-level hierarchical Bayesian model) topic model with a KG-aware LSTM proves effective and more accurate than in the case without external knowledge.

Sayan et al. [ 95 ] have presented the design of the architecture of an Intelligent Cyber Security Assistant (ICSA). Such a tool would aid human cyberdefenders in their tasks. The proposed architecture would detect attacks using machine learning and recommend the defense solutions. It gathers the network traffic data using existing monitoring tools. Then, the data are analyzed using the anomaly-based intrusion detection approach. The results are then used for making recommendations. The Intelligent Recommender Assistant (IRA) is a module that leverages machine learning methods for making recommendations. The authors decided to use the knowledge-based technique; thus, the knowledge scope is assessed, and knowledge base must first be built. Then, they apply feature engineering to transform the raw data into so that it better suits the predictive model. The learned model is then tested on unseen data, making recommendations concerning the mitigation of real-time attacks. The authors claim that through iterative machine learning, their system can make accurate predictions and expect it to keep learning, adapting and improving, which will make it possible to let it make automated or semi-automated cyberdefense actions.

Panda et al. have proposed a recommender system the aim of which would be to find which machine learning model is the most suitable for identifying attack models. The recommender system’s components were Naïve Bayes, Decision Trees, Ensemble learning, AdaBoost [ 96 ].

Gadepally et al. argue that recommender systems show promise in cybersecurity applications, as they might significantly lower the time of responding to the threat [ 97 ]. In the cybersecurity domain, the analysts must process massive amounts of information, which may lead to overlooking crucial data. According to authors, a recommendation system would help filter and prioritize information, as well as weigh multiple factors. They think that a recommender could be used to track hundreds of websites in an automated way, to learn from past user behavior. This would be used to prepare recommendations for the IT security team. If vulnerability’s severity and possible impact were assessed, they could be then used to make a recommendation concerning patching the vulnerability, e.g., if the patching might be postponed or should be immediate. Recommendation systems might also be used for tracking network anomalies, to draw the IT security teams’ attention to the possible issues which could be solved in a proactive way. Finally, the authors remark that (as of 2016), future work would be needed to adapt the recommender technologies to meet the specific requirements of the cybersecurity domain.

5. Discussion of the Results

This work has first presented a detailed overview of the concept of recommender systems, their types and techniques, advantages and disadvantages, security concerns, as well as possible fields of application, pointing it out that the use of recommender systems as an aid to cyberdefenders in hardly ever mentioned. Then, against this background, the state of the art of the applications of recommender systems in cybersecurity was indicated, gathered from the systematic review of the subject literature. The results of the conducted study helped answer the Research Questions.

5.1. The Answers to the Research Questions

The answer to Research Question 1: The comprehensive, broad study of the subject literature allowed the authors to conclude that recommender systems have indeed been applied for the sake of cybersecurity. They were also able to identify several specific applications, which have been presented in the previous Section of this paper. Most of the authors of the analyzed papers claim they were the first ones or one of the first ones to apply this technology to cybersecurity, but believe it shows promise and in the long run could effectively assist the human operators in the loop. They believe that recommender systems show promise as a tool for cyberdefenders, as they “have the potential in mission scenarios to shift computational support from being reactive to being predictive [ 97 ]”. It has also been remarked that more exploration should be made into using recommenders in the cyber domain [ 19 ].

The answer to Research Question 2: While analyzing the sources, the authors noticed that the papers did not present a unified taxonomy of recommender system types. By gathering and organizing the types presented in the analyzed papers, this work proposes a new, up-to-date list of recommender system types. Thus, this work has presented the most comprehensive list of the kinds of filtering among all the analyzed papers.

5.2. Threats to Validity

For the sake of this work, a substantial number of papers have been looked upon and considered for further analysis. Due to the fact that three of the five selected knowledge sources’ search engines returned hundreds or thousands of false positives, the authors analyzed the hits until the results ceased to seem relevant. Furthermore, the study let the authors identify about a dozen works showcasing actual implementations of recommender systems for cybersecurity. The investigated works come from different approaches and are not prone to comparison. It emphasizes the need for further research in this emerging field, a similar survey ran in a few years could probably help answer the research questions more thoroughly.

6. Conclusions

This paper has presented the results of a broad, systematic study of the potential use of recommender systems in cybersecurity. Several hundred literature items were marked as potentially relevant and then carefully analyzed. Several papers presenting the implementations of recommenders in cybersecurity were found and described.

All in all, the study showed that recommender systems could indeed be applied to support the human cyberdefender in their decisions, and contribute to a safer, more secure cyberspace.

This emerging field still needs plenty of research and in-depth consideration. It might also be beneficial to further explore enhancing the systems, e.g., by the application of graphs, or combining the method with other cybersecurity tools.

Furthermore, in the course of the study, a list of all the mentioned recommender system types was created, making it the most up-to-date and comprehensive division, as of this day.

Author Contributions

Investigation, R.K.; Supervision, R.S.C.; Writing—original draft, A.P.; Writing—review and editing, M.P. All authors have read and agreed to the published version of the manuscript.

This work is supported by the Ensuresec project, which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 883242.

Institutional Review Board Statement

Informed consent statement, data availability statement, conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Help | Advanced Search

Computer Science > Cryptography and Security

Title: manipulating recommender systems: a survey of poisoning attacks and countermeasures.

Abstract: Recommender systems have become an integral part of online services to help users locate specific information in a sea of data. However, existing studies show that some recommender systems are vulnerable to poisoning attacks, particularly those that involve learning schemes. A poisoning attack is where an adversary injects carefully crafted data into the process of training a model, with the goal of manipulating the system's final recommendations. Based on recent advancements in artificial intelligence, such attacks have gained importance recently. While numerous countermeasures to poisoning attacks have been developed, they have not yet been systematically linked to the properties of the attacks. Consequently, assessing the respective risks and potential success of mitigation strategies is difficult, if not impossible. This survey aims to fill this gap by primarily focusing on poisoning attacks and their countermeasures. This is in contrast to prior surveys that mainly focus on attacks and their detection methods. Through an exhaustive literature review, we provide a novel taxonomy for poisoning attacks, formalise its dimensions, and accordingly organise 30+ attacks described in the literature. Further, we review 40+ countermeasures to detect and/or prevent poisoning attacks, evaluating their effectiveness against specific types of attacks. This comprehensive survey should serve as a point of reference for protecting recommender systems against poisoning attacks. The article concludes with a discussion on open issues in the field and impactful directions for future research. A rich repository of resources associated with poisoning attacks is available at this https URL .

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Ready to accelerate your ML initiatives? Dive into Tecton's interactive demo

A Practical Guide to Building an Online Recommendation System

Recommender Systems

Have you ever wondered how TikTok can recommend videos to you that were uploaded minutes ago? Or how YouTube can pick up on your brand-new interest immediately after you watched one video about it? Or how Amazon can recommend products based on what you currently have in your shopping cart?

The answer is online recommendation systems (or recommender systems or recsys) . These systems can generate recommendations for users based on real-time contextual information, such as the latest item catalog, user behaviors, and session context. If you’re interested in building an online recommendation system or trying to take your existing system to the next level, then this blog post is for you.

I worked as a tech lead on the YouTube Homepage Recommendations team from 2018 to 2021, focusing on long-term user satisfaction metrics. While at YouTube, I learned from some of the best minds in the business, who had been building and refining the world’s largest recommendation engine for close to a decade. Since 2021, I’ve been with Tecton and have worked with many smaller companies building and refining their search and recommendation systems.

In this blog post, I will give an overview of online recommendation systems, the various approaches for building different subcomponents, and offer some guidance to help you reduce costs, manage complexity, and enable your team to ship ideas.

While online recommendation systems can vary substantially, they are all generally composed of the same high-level components. Those are:

Candidate generation

Feature retrieval, model inference, pointwise scoring and ranking, listwise ranking.

Candidate generation is responsible for quickly and cheaply narrowing the set of possible candidates down to a small enough set that can be ranked. For example, narrowing a billion possible YouTube videos down to ~hundreds based on a user’s subscriptions and recent interests. A good candidate generation system produces a relatively small number of diverse, high-quality candidates for the rest of the system.

There are many different options that can make effective candidate generators:

• E.g., query Postgres for a user’s recently purchased items.
• E.g., use Redis sorted sets to track popular items in a given locale, or
• run a daily batch job to generate a list of 500 candidates for every user and load those candidates into DynamoDB.
• Many different ML approaches that can learn “ embeddings ” for users and items. These approaches can be combined with vector search technologies like Faiss , Annoy , Milvus , and Elasticsearch to scale to online serving.
• See this blog post for an excellent discussion on the state of the art using approximate nearest neighbor search in conjunction with a two-tower deep neural network.
• Collaborative Filtering/Matrix Factorization also falls into this category. For lower scale or less mature systems, a basic collaborative filtering model may also be used directly in production.

Every approach will have its pros and cons with respect to complexity, freshness, candidate diversity, candidate quality, and cost. A good practice is to use a combination of approaches and then “union” the results in the online system. For example, collaborative filtering models can be used to generate high-quality candidates for learned users and items, but collaborative filtering models suffer from the cold-start problem . Your system could supplement a collaborative filtering model with recently added, popular items fetched from an operational database.

At one or more points, the recommendation system will need to look up or compute data/features for the user and the candidates being considered. This data will fall into three categories:

• E.g., product_category=clothing or average_review_rating_past_24h=4.21
• E.g., user_favorite_product_categories=[clothing, sporting_goods] or user_site_spend_past_month=408.10
• E.g., user_has_bought_this_product_before=False or product_is_in_user_favorite_categories=True
• Cross features like these can be very helpful with model performance without requiring the model to spend capacity to learn these relationships.

Your system does not necessarily need to use a purpose-built “feature store” for feature retrieval, but the data service needs to handle the following:

• Due to the user-to-item fanout, this service may receive 100-1000x the QPS of your recommendations service, and long-tail latency will heavily impact your overall system performance.
• Fortunately, in most recsys cases, query volume and cost can be significantly reduced at the expense of data freshness by caching item features. Item features (as opposed to user features) are usually well suited to caching because of their lower cardinality and looser freshness requirements. (It’s probably not an issue if your product_average_review_rating feature is a minute or two stale.) Feature caching may be done at multiple levels; e.g., a first-level, in-process cache and a second-level, out-of-process cache like Redis.
• Feature data fetched online must be consistent with the feature data that ranking models were trained with. For example, if the feature user_site_spend_past_month is pre-tax during offline training and post-tax during online inference, you may get inaccurate online model predictions.
• Your chosen data service should support serving the kinds of features and data that your models and heuristics require. For example, time-windowed aggregates (e.g., product_average_review_rating_past_24h or user_recent_purchase_ids ) are a popular and powerful class of features, and your data service should have a scalable approach to building and serving them.
• If your data service does not support these classes of features, then contributors will look for escape hatches, which may degrade system complexity, reliability, or performance.

Filtering is the process of removing candidates based on fetched data or model predictions. Filters primarily act as system guardrails for bad user experience. They fall into the following categories:

• E.g., an item_out_of_stock_filter that filters out-of-stock products.
• E.g., a recently_view_products_filter that filters out products that the user has viewed in the past day or in the current session.
• E.g., an unhelpful_review_filter that filters out item reviews if a personalized model predicts that the user would rate the review as unhelpful.

Even though filters act as guardrails and are typically very simple, they themselves may need some guardrails. Here are some recommended practices:

• Have checks in place to prevent a single filter (or a combination of filters) from filtering out all of the candidates. This is especially important for model-based filters, but even seemingly benign filters, like an item_out_of_stock_filter , can cause outages if the upstream data service has an outage.
• Have monitoring and tooling in place to track filters over time and debug changes. Just like model predictions, filter behavior can change based on data drift and will need to be re-tuned.

Finally, we get to the “normal” machine learning part of the system: model inference. Like with other ML systems, online model inference boils down to sending feature vectors to a model service to get predictions. Typically these models predict easily measurable downstream events, like Click Through Rate, Probability of a Purchase, Video Watch Time, etc. The exact ML method, model architecture, and feature engineering for these models are huge topics, and fully covering them is outside of the scope of this blog post. 

Instead, we will focus on a couple of practical topics that are covered less in data science literature.

• If you’ve decided to build an online recommendation system, then leveraging fresh features (i.e., features based on user or item events from the past several seconds or minutes) is probably a high priority for you.
• Some ML methods, like collaborative filtering, cannot incorporate fresh features like these into predictions. Other approaches, like hybrid content-collaborative filtering (e.g., LightFM ) or pure content-based filtering (e.g., built on XGBoost or a neural net) can take advantage of fresh features. Make sure your Data Science team is thinking in terms of online serving and fresh features when choosing ML methods.
• Model calibration is a technique used to fit the output distribution of an ML model to an empirical distribution. In other words, your model output can be interpreted as a real-world probability, and the outputs of different model versions are roughly comparable. 
• To understand what this means, consider two example uncalibrated Click Through Rate models. Due to different training parameters (e.g., negative label dropout), one model’s predictions vary between 0 and 0.2, and the second model’s predictions vary between 0 and 1. Both models have the same test performance (AUC) because their relative ordering of predictions is the same. However, their scores are not directly comparable. A score of 0.2 means a very high CTR for the first model and a relatively low CTR for the second model.
• If production and experimental models are not calibrated to the same distribution, then downstream systems (e.g., filters and ranking functions) that rely on the model score distributions will need to be re-tuned for every experiment, which quickly becomes unsustainable. Conversely, if all of your model versions are calibrated to the same distribution, then model experiments and launches can be decoupled from changes to the rest of the recommendation system.
• See this talk on the topic.

“Pointwise” ranking is the process of scoring and ranking items in isolation; i.e., without considering other items in the output. The item pointwise score may be as simple as a single model prediction (e.g., the predicted click through rate) or a combination of multiple predictions and heuristics. 

For example, at YouTube, the ranking score is a simple algebraic combination of many different predictions, like predicted “click through rate” and predicted “watch time”, and occasionally some heuristics, like a “small creator boost” to slightly bump scores for smaller channels.

Tuning the parameters X, Y, and Z will shift the system’s bias from one objective to another. If your model scores are calibrated, then this ranking function can be tuned separately from model launches. The parameters can even be tuned in a personalized way; e.g., new users may use different ranking function parameters than power users.

Have you ever noticed that your YouTube feed rarely has two similar videos next to each other? The reason is that item diversity is a major objective of YouTube’s “listwise” ranking. (The intuition being if a user has scrolled past two “soccer” videos, then it’s probably sub-optimal to recommend another “soccer” video in the third position.)

Listwise ranking is the process of ordering items in the context of other items in the list. ML-based and heuristic-based approaches can both be very effective for listwise optimization—YouTube has successfully used both. A simple heuristic approach that YouTube had success with is to greedily rank items based on their pointwise rank but apply a penalty to items that are too similar to the preceding N items.

If item diversity is an objective of your final output, keep in mind that listwise ranking can only be impactful if there are high-quality, diverse candidates in its input. This means that tuning the upstream system, and in particular candidate generation, to source diverse candidates is critical.

This blog post did not cover major recsys topics like feature engineering, experimentation, and metric selection, but I hope that it gave you some ideas about how to build or improve your system. If you’re interested in learning more, I recommend (pun intended) checking out the recordings for apply(recsys) , a free, virtual, half-day event focusing on everything recsys, with guest speakers from major recsys powerhouses such as Slack and ByteDance. You can also reach out to me ([email protected]) in the Tecton/Feast Slack if you want to chat more about recommendation system design.

Related Posts

High-scale feature serving at low cost with caching.

Learn the different use cases for caching features and see how the Tecton Serving Cache works in this blog post.

Product Recommender Systems

Q&A: Making the Move to Real-Time Machine Learning

In this post, we share some of the questions asked by attendees during a recent webinar about making the move from batch to real-time ML.

Operational & Real-Time Machine Learning

FanDuel Leverages Tecton’s Feature Platform for a Winning Machine Learning Strategy

FanDuel, the premier online gaming company in North America, shared at why and how they adopted Tecton to power their ML models and data-driven applications.

Customer Stories Events

Request a Demo

Unfortunately, Tecton does not currently support these clouds. We’ll make sure to let you know when this changes!

However, we are currently looking to interview members of the machine learning community to learn more about current trends.

If you’d like to participate, please book a 30-min slot with us here and we’ll send you a $50 amazon gift card in appreciation for your time after the interview.

Contact Sales

Interested in trying Tecton? Leave us your information below and we’ll be in touch.​

Request a free trial