research paper of apps

Advertisement

Mobile Data Science and Intelligent Apps: Concepts, AI-Based Modeling and Research Directions

• Published: 14 September 2020
• Volume 26 , pages 285–303, ( 2021 )

Cite this article

• Iqbal H. Sarker   ORCID: orcid.org/0000-0003-1740-5517 1 , 2 ,
• Mohammed Moshiul Hoque 2 ,
• Md. Kafil Uddin 1 &
• Tawfeeq Alsanoosy 3  

22k Accesses

70 Citations

13 Altmetric

Explore all metrics

Artificial intelligence (AI) techniques have grown rapidly in recent years in the context of computing with smart mobile phones that typically allows the devices to function in an intelligent manner. Popular AI techniques include machine learning and deep learning methods, natural language processing, as well as knowledge representation and expert systems, can be used to make the target mobile applications intelligent and more effective. In this paper, we present a comprehensive view on “ mobile data science and intelligent apps” in terms of concepts and AI-based modeling that can be used to design and develop intelligent mobile applications for the betterment of human life in their diverse day-to-day situation. This study also includes the concepts and insights of various AI-powered intelligent apps in several application domains, ranging from personalized recommendation to healthcare services, including COVID-19 pandemic management in recent days. Finally, we highlight several research issues and future directions relevant to our analysis in the area of mobile data science and intelligent apps. Overall, this paper aims to serve as a reference point and guidelines for the mobile application developers as well as the researchers in this domain, particularly from the technical point of view.

Similar content being viewed by others

Pathways to Artificial General Intelligence: A Brief Overview of Developments and Ethical Issues via Artificial Intelligence, Machine Learning, Deep Learning, and Data Science

Deep Learning for Contextual Mobile Data Analytics

Introduction to context-aware machine learning and mobile data analytics.

Avoid common mistakes on your manuscript.

1 Introduction

Due to the recent development of science and technology in the world, the smartphone industry has made exponential growth in the mobile phone application market [ 1 ]. These devices are well known as one of the most important Internet-of-Things (IoT) devices as well, according to their diverse capabilities including data storage and processing [ 2 ]. Today’s smartphone is also considered as “a next-generation, multi-functional cell phone that facilitates data processing as well as enhanced wireless connectivity”, i.e., a combination of “a powerful cell phone” and a “wireless-enabled PDA” [ 3 ]. In our earlier paper [ 4 ], we have shown that users’ interest on “Mobile Phones” is more and more than other platforms like “Desktop Computer” , “Laptop Computer” or “Tablet Computer” for the last five years from 2014 to 2019 according to Google Trends data [ 5 ], shown in Fig.  1 .

Users’ interest trends over time where x-axis represents the timestamp information and y-axis represents the popularity score in a range of 0 (min) to 100 (max)

In the real world, people use smartphones not only for voice communication between individuals but also for various activities with different mobile apps like e-mailing, instant messaging, online shopping, Internet browsing, entertainment, social media like Facebook, Linkedin, Twitter, or various IoT services like smart cities, health or transport services, etc. [ 2 , 6 ]. Smartphone applications differ from desktop applications due to their execution environment [ 7 ]. A desktop computer application is typically designed for a static execution environment, either in-office or home, or other static locations. However, this static precondition is generally not applicable to mobile services or systems. The reason is that the world around an application is changing frequently and computing is moving toward pervasive and ubiquitous environments [ 7 ]. Thus, mobile applications should adapt to the changing environment according to the contexts and behave accordingly, which is known as context-awareness [ 8 ].

Artificial intelligence (AI) techniques have grown rapidly in recent years in the context of computing with smart mobile phones that typically allows the devices to function in an intelligent manner. AI can be applied to various types of mobile data such as structured, semi-structured, and unstructured [ 9 ]. Popular AI techniques include machine learning (ML) and deep learning (DL) methods, natural language processing (NLP), as well as knowledge representation and expert systems (ES), can be used according to their data characteristics, in order to make the target mobile applications intelligent. AI-based models and their usage in practice can be seen in many intelligent mobile applications, such as personalized recommendation, virtual assistant, mobile business, healthcare services, and even the corona-virus COVID-19 pandemic management in recent days. A brief discussion of these apps and their relation with AI techniques within the area of mobile data science has been conducted in Section 6. This made a paradigm shift to context-aware intelligent computing , powered by the increasing availability of contextual smartphone data and the rapid progress of data analytics techniques. The intelligent smartphone applications and corresponding services are considered as “context-aware” because smartphones are able to know their users’ current contexts and situations, “adaptive” because of their dynamic changing capability depending on the users’ needs, and “intelligent” because of building the model based on data-driven artificial intelligence, which makes them able to assist the end-users intelligently according to their needs in their different day-to-day situations. Thus AI-based modeling for intelligent decision making, is the key to achieve our goal in this paper.

Based on the importance of AI in mobile apps, mentioned above, in this paper, we study on mobile data science and intelligent apps that covers how the artificial intelligence methods can be used to design and develop data-driven intelligent mobile applications for the betterment of human life in different application scenarios. Thus, the purpose of this paper is to provide a base reference for those academia and industry people who want to study and develop various AI-powered intelligent mobile apps considering these characteristics rather than traditional apps, in which we are interested.

The main contributions of this paper are listed as follows:

To provide a brief overview and concept of the mobile data science paradigm for the purpose of building data-driven intelligent apps. For this, we first briefly review the relevant methods and systems, to motivate our study in this area.

To present AI-based modeling for intelligent mobile apps where various machine learning and deep learning algorithms, the concept of natural language processing, as well as knowledge representation and rule-based expert systems, are used.

To discuss the usefulness of various AI-powered intelligent apps in several application domains, and the role of AI-based modeling in practice for the betterment of human life.

To highlight and summarize the potential research directions relevant to our study and analysis in the area of mobile data science and intelligent apps.

The rest of the paper is organized as follows. Section 2 motivates and defines the scope of our study. In Section 3, we provide a background of our study including traditional data science and context-aware mobile computing, and review the works related to data-driven mobile systems and services. We define and discuss briefly about mobile data science paradigm in Section 4. In Section 5, we present our AI-based modeling within the scope of our study. Various AI-powered intelligent apps are discussed and summarized in Section 6. In section 7, we highlight and summarize a number of research issues and potential future directions. In Section 8, we highlight some key points regarding our studies, and finally, Section 9 concludes this paper.

2 The motivation and scope of the study

In this section, our goal is to motivate the study of exploring mobile data analytics and artificial intelligence methods that work well together in data-driven intelligent modeling and mobile applications in the interconnected world, especially in the environment of today’s smartphones and Internet-of-Things (IoT), where these devices are well known as one of the most important IoT devices. Hence, we also present the scope of our study.

We are currently living in the era of Data Science (DS), Machine Learning (ML), Artificial Intelligence (AI), Internet-of-Things (IoT), and Cybersecurity, which are commonly known as the most popular latest technologies in the fourth industrial revolution (4IR) [ 10 , 11 ]. The computing devices like smartphones and corresponding applications are now used beyond the desktop, in diverse environments, and this trend toward ubiquitous and context-aware smart computing is accelerating. One key challenge that remains in this emerging research domain is the ability to effectively process mobile data and enhance the behavior of any application by informing it of the surrounding contextual information such as temporal context, spatial context, social context, environmental or device-related context, etc. Typically, by context, we refer to any information that characterizes a situation related to the interaction between humans, applications, and the surrounding environment [ 4 , 12 ].

For AI-based modeling, several machine learning and deep learning algorithms, the concept of natural language processing, as well as knowledge representation and rule-based expert systems, can be used according to their data characteristics, in order to make the target mobile applications intelligent. For instance, machine learning (ML) algorithms typically find the insights or natural patterns in mobile phone data to make better predictions and decisions in an intelligent systems [ 13 , 14 ]. Deep learning is a part of machine learning that allows us to solve complex problems even when using a diverse data set. Natural language processing (NLP) is also an important part of AI that derives intelligence from unstructured mobile content expressed in a natural language, such as English or Bengali [ 15 ]. Another important part of AI is knowledge representation and a rule-based expert system that is also considered in our analysis. Expert system (ES) typically emulates the decision-making ability of a human expert in an intelligent system that is designed to solve complex problems by reasoning through knowledge, represented mainly as IF-THEN rules rather than conventional procedural code.

Thus, the overall performance of the AI-based mobile applications depends on the nature of the contextual data, and artificial intelligence tasks that can play a significant role to build an effective model, in which we are interested in this paper. Overall, the reasons for AI-tasks in mobile applications and systems can be summarized as below -

to empower the evolution of the mobile industry by making smartphone apps as intelligent pieces of software that can predict future outcomes and make decisions according to users’ needs.

to learn from data including user-centric, and device-centric contexts, by analyzing the data patterns.

to deliver an enhanced personalized experience while adapting quickly to changing innovations and environments.

to better utilization of available resources with higher effectiveness and efficiency.

to understand the real-world problems and to provide intelligent and automated services accordingly as well as complex problems in this mobile domain.

to enable the smartphones more secured through predictive analytics by taking into account possible threats in real-time.

To achieve our goal, in this study, we mainly explore mobile data science and intelligent apps that aims at providing an overview of how AI-based modeling by taking into account various techniques’ that can be used to design and develop intelligent mobile apps for the betterment of human life in various application domains, briefly discussed in Section 5, and Section 6.

3 Background and related work

In this section, we give an overview of the related technologies of mobile data science that include the traditional data science, as well as the computing device and Internet, and context-aware mobile computing in the scope of our study.

3.1 Data science

We are living in the age of data [ 16 ]. Thus, relevant data-oriented technologies such as data science, machine learning, artificial intelligence, advanced analytics, etc. are related to data-driven intelligent decision making in the applications. Nowadays, many researchers use the term “data science” to describe the interdisciplinary field of data collection, pre-processing, inferring, or making decisions by analyzing the data. To understand and analyze the actual phenomena with data, various scientific methods, machine learning techniques, processes, and systems are used, which is commonly known as data science. According to Cao et al. [ 16 ] “data science is a new interdisciplinary field that synthesizes and builds on statistics, informatics, computing, communication, management, and sociology to study data and its environments, to transform data to insights and decisions by following a data-to-knowledge-to-wisdom thinking and methodology”. As a high-level statement, it is the study of data to provide data-driven solutions for the given problems, as known as “the science of data”.

3.2 Computing devices and internet

The advancement of mobile computing and the Internet have played a central role in the development of the current digital age. The use of the Internet with mobile devices makes it the most popular computing device, for the people in the real world.

Mobile devices have become one of the primary ways, in which people around the globe communicate with each other for various purposes. While mobile phones may come in various forms in the real world, in this paper, they refer to smartphones or mobile devices with the capability of computing and Internet access. These devices have incorporated a variety of significant and interesting features to facilitate better information access through smart computing and the proper utilization of the devices for the benefit of the users. In recent times, the smartphones are becoming more and more powerful in both computing and the data storage capacity. As such, in addition to being used as a communication device, these smart mobile phones are capable of doing a variety of things relevant to users’ daily life such as instant messaging, Internet or web browsing, e-mail, social network systems, online shopping, or various IoT services like smart cities, health or transport services [ 2 , 6 ]. The future smartphones will be more powerful than current devices, communicate more quickly, store more data, and integrate new interaction technologies.

3.3 Context-aware Mobile computing

The notion of context has been used in numerous areas, including mobile and pervasive computing, human-centered computing, and ambient intelligence [ 17 ]. In the area of mobile and pervasive computing, several early works on context-aware computing, or context-awareness referred context as the location of people and objects [ 18 ]. Moreover, locational context, or user activities [ 17 , 18 ], temporal information [ 4 , 19 ], environmental information [ 20 ], user’s identity [ 21 ], or social context [ 22 , 23 ] are taken into account as contexts for different purposes. The state of the surrounding information of the applications are also considered as contexts in [ 24 , 25 ]. In [ 26 ], Schilit et al. claim that the important aspects of context are: (i) where you are, (ii) whom you are with, and (iii) what resources are nearby. Dey et al. [ 12 ] define context, which is perhaps now the most widely accepted. According to Dey et al. [ 12 ] “Context is any information that can be used to characterize the situation of an entity. An entity is person, place, or object that is considered relevant to the interaction between a user and an application, including the user and the application themselves”. We can also define context äs a specific type of knowledge to adapt application behavior.”

Based on the contextual information defined above, context-awareness can be the spirit of pervasive computing [ 27 ]. In general, context-awareness has adapting capability in the applications with the movement of mobile phone users, and thecontext-aware computing refers to sense the surrounding physical environment, and able to adapt application behavior. Therefore, context-awareness simply represents the dynamic nature of the applications. The use of contextual information in mobile applications is thus able to reduce the amount of human effort and attention that is needed for an application to provide the services according to user’s needs or preferences, in a pervasive computing environment [ 28 ]. Different types of contexts might have a different impact on the applications that are discussed briefly in our earlier paper, Sarker et al. [ 4 , 29 ].

3.4 Mobile systems and services

Research that relies on mobile data collected from diverse sources is mostly application-specific, which differs from application-to-application. A number of research has been done on mobile systems and services considering diverse sources of data. For instance, phone call logs [ 30 , 31 ] that contain context data related to a user’s phone call activities. In addition to call-related metadata, other types of contextual information such as user location, thesocial relationship between the caller a callee identified by the individual’s unique phone contact number are also recorded by the smart mobile phones [ 31 ]. Mobile SMS Log contains all the message including the spam and non-spam text messages [ 32 ] or good content and bad content [ 33 ] with their related contextual information such as user identifier, date, time, and other SMS related metadata, which can be used in the task of automatic filtering SMS spam for different individuals in different contexts [ 31 , 32 ], or predicting good time or bad time to deliver such messages [ 33 ]. App usages log contains various contextual information such as date, time-of-the-day, battery level, profile type such as general, silent, meeting, outdoor, offline, charging state such as charging, complete, or not connected, location such as home, workplace, on the way, etc. and other apps relatedmetadata with various kinds of mobile apps [ 34 , 35 , 36 , 37 , 38 ]. The notification log contains the contextual information such as notification type, user’s various physical activity (still, walking, running, biking and in-vehicle), user location such as home, work, or other, date, time-of-the-day, user’s response with such notifications (dismiss or accept) and other notification related metadata [ 39 ]. Weblog contains the information about user mobile web navigation, web searching, e-mail, entertainment, chat, misc., news, TV, netting, travel, sport, banking, and related contextual information such as date, time-of-the-day, weekdays, weekends [ 40 , 41 , 42 ]. Game log contains the information about playing various types such games of individual mobile phone users, and related contextual information such as date, time-of-the-day, weekdays, weekends etc. [ 43 ].

The ubiquity of smart mobile phones and their computing capabilities for vairous real life purposes provide an opportunity of using these devices as a life-logging device, i.g., personal e-memories [ 44 ]. In a more technical sense, life-logs sense and store individual’s contextual information from their surrounding environment through a variety of sensors available in their smart mobile phones, which are the core components of life-logs such as user phone calls, SMS headers (no content), App use (e.g., Skype, Whatsapp, Youtube etc.), physical activities form Google play API, and related contextual information such as WiFi and Bluetooth devices in user’s proximity, geographical location, temporal information [ 44 ]. Several applications such as smart context-aware mobile communication, intelligent mobile notification management, context-aware mobile recommendation etc. are popular in the area of mobile analytics and applications. Smart context-aware mobile communication (e.g., intelligent phone call interruption management) is one of the most compelling and widely studied applications [ 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 ]. For mobile notification management, several research [ 39 , 56 , 57 , 58 , 59 ] has been done. Similarly, a number of research [ 34 , 60 , 61 , 62 , 65 ] has been done on recommendation system.

Various techniques are used in various applications, such as interruption management, activity recognition, recommendation system, mobile commerce, etc. in the area of mobile analytics. For instance, Seo et al. [ 66 ] design a context-aware configuration manager for smartphones PYP. An intelligent interruption management system is proposed in [ 48 ], use decision tree for making decisions. Bozanta et al. [ 67 ], Lee et al. [ 68 ] use classification technique to build a personalized hybrid recommender system. Turner et al. [ 59 , 69 ], Fogarty et al. [ 70 ] use classification technique in their interruptibility predictionand management system. In the area of transportation, Bedogni et al. [ 71 ] use classification techniques in their context-aware mobile applications. To adopt mobile learning, Tan et al. [ 72 ] investigates using a multi-layer perceptron model. In [ 43 ], Paireekreng et al. have proposed a personalization mobile game recommendation system. Moreover, regression techniques such as Linear regression [ 9 ], support vector regression [ 73 ], and ensemble learning techniques, such as Random Forest learning [ 74 ] are popular in the area of supervised learning.

Beside the above mentioned approaches, several researchers [ 34 , 35 , 39 ] use association rules that are used to build various context-aware mobile service according to users needs. A number of research [ 40 , 98 , 99 , 100 , 101 , 102 ] have been done based on clustering approach for different purposes in their study. Moreover, a significant amount of research [ 72 , 94 , 95 , 96 , 97 ] have been done on deep learning for various purposes in the area of mobile analytics. Moreover, context engineering including principal component analysis, or context correlation analysis [ 77 , 78 ] is another important issue to work in this area. In Table 1 , we have summarized this research based on the most popular approaches and data-driven tasks within the scope of our analysis.

Although various types of mobile phone data and techniques discussed above are used in the area of mobile analytics and systems for different purposes, a comprehensive AI-based modeling for building intelligent apps is being interested, according to the needs of the current in the community. Thus, in this paper, we focus on mobile data science and corresponding intelligent apps, where the most popular AI techniques including machine learning and deep learning methods, the concept of natural language processing, as well as knowledge representation and expert systems, can be used to build intelligent mobile apps in various application domains.

4 Mobile data science paradigm

In this section, we provide a brief overview of mobile data science and its related components within the scope of our study.

4.1 Understanding Mobile data

Mobile data science and data-driven intelligent apps are largely driven by the availability of data. Mobile datasets typically represent a collection of information records that consist of several attributes or contextual features and related facts. Thus,it’s important to understand the nature of mobile data containing various types of features and contexts. The reason is that raw data collected from relevant sources for a particular application can be used to analyze the various patterns or insight, to build a data-driven model to achieve our goal. Several datasets exist in the area of mobile analytics, such as phone call logs [ 30 ], apps logs [ 34 , 35 ], weblogs [ 40 ] etc. These context-rich historical mobile phone data are the collection of the past contextual information and users’ diverse activities [ 92 , 103 ]. Moreover, IoT data, smart cities data, business data, health data, mobile security data, or various sensors data associated with the mobile devices and target application can also be used as data sources. Intelligent apps are based on the extracted insight from such kinds of relevant datasets depending on apps characteristics. In the next, we summarize several characteristics of intelligent apps.

4.2 Intelligent apps characteristics

Intelligent apps offer personalized and adaptive user experiences, where artificial intelligence, the Internet-of-Things, and data analytics are the core components. Based on this, we have summarized the characteristics of intelligent apps to assist smartphone users in their daily life activities.

Action-Oriented: The foremost characteristic of intelligent apps is that these applications do not wait for users to make decisions in various situations. Rather, the apps can study user behavior and deliver personalized and actionable results using the power of predictive analytics.

Adaptive in Nature: The apps should be adaptive in nature. Every user is different in their use, the adaptability of the app plays a very crucial role. Meaning, they can easily upgrade their knowledge as per their surroundings to produce a highly-satisfying user experience.

Suggestive and Decision-Oriented: Generating suggestions and making decisions according to users’ needs and interests, could be an interesting characteristic of an intelligent app. Such suggestions may vary from user-to-user according to their interests and helps the users to decide what suits best for them.

Data-driven: Delivering a data-driven output is also one of the key features of intelligent apps. The intelligent apps gather data from a variety of sources, such as online, user interaction, sensors, etc. relevant to the target application and extracting data patterns, thus providing better user experience.

Context-awareness: Context awareness is the ability of an application to gather information about its surrounding environment at any given time and adapt behaviors accordingly. It makes the apps much smarter use by taking into account users contexts as well as the device’s contexts to proactively deliver highly relevant information and suggestions.

Cross-Platform Operation: The app also should have the ability to understand and process the desired output in a way that the users feel the same experience while working on cross platforms.

In this study, we take into account the above-discussed characteristics of mobile apps that could be able to intelligently assist the users in their diverse daily life activities. Based on these characteristics, in the next, we briefly discuss the concept of mobile data science and AI that can help to achieve the goal.

4.3 Mobile data science and AI

Data science is transforming the world’s industries. It is critically important for the future of intelligent mobile apps and services because of “apps intelligence is all about mobile data”. Traditionally, mobile application developers didn’t use data science techniques to make the apps intelligent considering the above characteristics. Although, a number of recent research [ 4 , 29 , 34 , 38 , 48 ] has been done based on machine learning techniques to model and build mobile applications, most of existing mobile applications are static or used custom-written rules like signatures, or manually defined heuristics for their different applications [ 47 , 66 ]. The main drawback of these custom-written rules-based approaches is that the knowledge or rules used by the applications are not automatically discovered; users need to define and maintain the rules manually. In general, users may not have the time, inclination, expertise, or interest to maintain rules manually. Although these rule-based approaches have their own merits in several cases, it needs too much manual work to keep up with the changing of userscontext landscape. On the contrary, data science can make a massive shift in technology and its operations, where AI techniques including machine learning and deep learning methods, natural language processing, as well as knowledge representation and expert systems, can be used to learn and making intelligent decisions. Thus, data science is considered as a practical application of machine learning, a major part of AI, with a complete focus on solving real-world problems. Overall, data science is a comprehensive process that involves data collection, pre-processing, data analysis, visualization, and decision making [ 16 ], whereas AI makes use of computer algorithms that can show human intelligence.

The concept of mobile data science incorporates the methods and techniques of machine learning and AI and data science as well as the context-aware computing to build intelligent mobile apps. The combination of these technologies has given birth to the term “mobile data science”, which refers to collect a large amount of mobile data from different sources and analyze it using machine learning techniques through the discovery of useful insights or the data-driven patterns, which is primarily defined in our earlier paper [ 104 ]. It is, however, worth remembering that mobile data science is not just about a collection of AI techniques. Mobile data science is a process that can help mobile application developers or analysts to scale and automate the target apps in a smart way and in a timely manner. Thus in a broader sense, we can say that “Mobile data science is research or working area existing at the intersection of context-aware mobile computing, data science, and artificial intelligence, which is mainly data-focused associated with target mobile apps, applies AI techniques for modeling, and eventually making intelligent decisions in applications. Thus it aims to seek for optimizing solutions to build automated and intelligent mobile applications to intelligently assist the users in their various daily activities.”. Several key modules, such as data collection, data processing, context and usage analysis, and building models, are involved in mobile data science, which are discussed briefly in our earlier paper [ 104 ]. In this paper, we mainly explore on AI-based modeling and its role in mobile apps in various application domains ranging from personalized services to healthcare services, which includes machine learning (ML) and deep learning (DL) methods, the concept of natural language processing (NLP), as well as knowledge representation, and rule-based expert systems (ES).

Overall, the outputs of mobile data science are typically mobile data products, which can be a data-driven AI-based model, potential mobile service and recommendation, or the corresponding intelligent mobile apps. In Section 6, we have discussed about AI-powered intelligent mobile apps in several application domains within the area of mobile data science.

4.4 Mobile security and privacy

Although we focus on intelligent apps from the perspective of artificial intelligence within the scope of our study discussed above, mobile security and privacy could be another part related to mobile data science in terms of data-driven security solutions. In the real world, most of the people including business people use smartphones not only to communicate but also to plan and organize their various kinds of daily works and also in their private life with family and friends. In most cases, both the business or personal information are stored on smartphones and people use such information when needed [ 105 , 106 ]. Thus, in addition to intelligent apps, mobile security and privacy is also important. Smartphones collect and analyze the sensitive information to which access must be controlled to protect the privacy of the user and the intellectual property of the organization or the company. Besides, there are several threats to mobile devices, including mobile malware, botnet, denial-of-service (DoS), eavesdropping, phishing, data breaches, etc. [ 106 , 107 , 108 ]. In terms of security analytics, in our earlier paper, Sarker et al. [ 10 ], we have discussed various types of security data and the effectiveness of the data-driven cybersecurity modeling based on artificial intelligence, particularly using machine learning techniques. Thus data-driven intelligent solutions through finding security insight could be effective to detect and mitigate such kind of mobile security threats.

5 AI-based modeling for Mobile services

As discussed earlier, mobile data science is data-focused, applies various artificial intelligence methods that eventually seek for intelligent decision making in mobile applications or services. In our analysis, we divide the artificial intelligence methods into several categories, such as basic machine learning and deep learning algorithms, natural language processing, knowledge representation and expert systems, within the scope of our study. These AI-based methods potentially can be used to make intelligent decisions in apps, which are discussed briefly in the following.

5.1 Machine learning modeling with Mobile data

Machine Learning (ML) including deep neural network learning is an important part of Artificial Intelligence (AI) which can empower mobile devices to learn, explore, and envisage outcomes automatically without user interference. For instance, machine learning algorithms can do the analysis of targeted user behavior patterns utilizing phone log data to make personalized suggestions as well as recommendations for mobile phone users. Typically, a machine learning model for building intelligent apps is a collection of target app-related data from relevant diverse sources, such as phone logs, sensors, or external sources, etc. and the chosen algorithms that work on that data in order to deduce the output.

To build a model utilizing collected data, supervised learning is performed when specific target classes are defined to reach from a certain set of inputs [ 13 ]. For instance, to classify or predict the future outcome, several popular algorithms such as Navies Bayes [ 109 ], Decision Trees [ 93 , 110 , 111 ], K-nearest neighbors [ 112 ], Support vector machines [ 73 ], Adaptive boosting [ 113 ], Logistic regression [ 114 ] etc. can be used. Such classification techniques are capable to build a prediction model ranging from predicting next usage to smartphone security, e.g., predicting mobile malware attack. Several feature engineering tasks, such as feature selection, extraction, etc., or context pre-modeling [ 78 ] can make the resultant predictive model more effective. On the other hand, in unsupervised learning, data is not labeled or classified, and it investigates similarity among unlabeled data [ 9 ]. Several clustering algorithms such as K-means [ 115 ], K-medoids [ 116 ], Single linkage [ 117 ], Complete linkage [ 118 ], BOTS [ 75 ] can be used for such modeling by taking into account certain similarity measures depending on the data characteristics. For instance, considering certain similarity in users’ preferences or behavioral activities, and to generate suggestions and recommendations accordingly, these algorithms can play a role to achieve the goal. Moreover, association rule learning techniques such as AIS [ 119 ], Apriori [ 120 ], FP-Tree [ 121 ], RARM [ 122 ], Eclat [ 123 ], ABC-RuleMiner [ 29 ] can be used for building rule-based machine learning model for the mobile phone users. In addition to these basic machine learning techniques, several deep neural learning methods such as recurrent neural network, long-short term memory, convolutional neural network, multilayer perceptron, etc. that are originated from an Artificial Neural Network (ANN) can be used in the learning process [ 9 , 13 ]. In these deep learning models, several hidden layers can be included to complete the overall process.

To understand and analyze the actual phenomena with mobile data, the above-discussed machine learning and deep learning techniques are useful to build AI-based modeling, depending on the target application and corresponding data characteristics. Thus the machine learning models and corresponding mobile apps that are close to the reality, are able to make data-driven intelligent decisions in apps and can behave according to users’ needs. Overall, the machine learning models can change the future of mobile applications and industry because of its learning capability from data. Therefore, machine learning methods including deep neural networks, on a global scale, is able to make mobile platforms more user-friendly, improve users’ experiences, and aid in building intelligent applications.

5.2 Natural language processing for Mobile content

Natural Language Processing (NLP) is an important branch of artificial intelligence that typically deals with the interaction between computers and humans using the natural language. One of the ultimate goals of NLP is to derive intelligence from unstructured data or content expressed in a natural language, such as English or Bengali. As each language has a unique set of grammar and syntax, and convention, NLP techniques can make it possible for computers to read text, hear speech, interpret it, measure sentiment or to mine opinions, and eventually determine which parts are important in an intelligent system [ 124 ]. For instance, to extract sentiments associated with positive, neutral, or negative polarities for specific subjects from a text document, an NLP-based methodology can be used. Thus, NLP can play a significant role to build intelligent apps when unstructured mobile content is available, and to be an important part within the scope of our study.

In recent days, a large amount of content read on mobile devices is text-based, such as emails, web pages, comments, blogs, or documents [ 15 ]. NLP techniques particularly, text mining extracts patterns and structured information from textual content that could make the apps smarter and intelligent, in which we are interested. For instance, browsing through large amounts of textual content on a small-screen mobile device may be tedious or time-consuming. In some cases, the important information might be easily overlooked due to the small screen of the devices. Thus, document summarization based on NLP might be the potential solution to provide a summary with high quality and minimal time.

Information extraction from mobile content could be another example of NLP based modeling. It typically identifies instances of a particular class of events, entities, or relationships in a natural language text and creates a structured representation of the discovered information [ 15 ]. For instance, this can be used to automatically find all the occurrences of a specific type of entity, such as ‘business’, and gather complementary information in the form of metadata around them. In addition to information extraction, NLP techniques can also be used when needed to develop the new mobile content. For instance, response generation while replying to an email, question answering, e.g., a company might need a mobile app that can answer questions about various products or services. Similarly, medical information extraction, personalized recommendation system through comments or text mining, context-aware chatbot, etc. are also included within the area. Thus, NLP techniques can play a significant role to build AI-based modeling depending on the target application and corresponding data type and characteristics.

5.3 Domain knowledge representation and Mobile expert system modeling

Due to the diversity of mobile users, contexts, increasing information, and variations in mobile computing platforms, mobile applications today are facing the challenges to provide the expected services. In artificial intelligence (AI), knowledge representation and expert system modeling is considered as another important part to minimize this issue, and to build knowledge-base intelligent systems.

5.3.1 Knowledge representation

In the real world, knowledge is considered as the information about a particular domain. It is typically a study of how the beliefs, intentions, and judgments of an intelligent agent can be expressed suitably for automated reasoning to solve problems. Thus, the main purpose of knowledge representation is modeling the intelligent behavior of an agent. It allows a machine to learn from that knowledge and behave intelligently like a human being. Instead of trying to understand from the bottom-up learning, its main goal is to understand the problems from the top-down, and to focus on what an associated agent needs to know in order to behave intelligently. Knowledge can be of several types:

Declarative Knowledge: known as descriptive knowledge that represents to know about something, which includes concepts, facts, and objects, and expressed in a declarative sentence.

Structural Knowledge: represents the basic knowledge to solve problems which describes the relationship between concepts and objects.

Procedural Knowledge: known as imperative knowledge that is responsible for knowing how to do something which includes rules, strategies, procedures, etc.

Meta Knowledge: represents knowledge about other types of knowledge.

Heuristic Knowledge: represents knowledge of some experts in a field or subject that could be based on previous experiences.

To represent knowledge in Artificial Intelligence (AI), “Ontology” in general has become popular as a paradigm by providing a methodology for easier development of interoperable and reusable knowledge bases (KB). Ontologies can be used to capture, represent knowledge and describe concepts and the relationship that holds between those concepts. In general, ontology is an explicit specification of conceptualization and a formal way to define the semantics of knowledge and data. According to [ 125 ], formally, an ontology is represented as “{ O  =  C ,  R ,  I ,  H ,  A }, where { C  =  C 1 ,  C 2 , …,  C n } represents a set of concepts, and { R  =  R 1 ,  R 2 , …,  R m } represents a set of relations defined over the concepts. I represents a set of instances of concepts, and H represents a Directed AcyclicGraph (DAG) defined by the subsumption relation between concepts, and A represents a set of axioms bringing additional constraints on the ontology”. Let’s consider an inference rules in ontologies for deductive reasoning. A rule may exist which states “If a mobile user accepts phone calls from family at work and a phone call is from his mother, then the call has been answered.” Then a program could deduce from a social relationship ontology that the user answers her mother’s incoming call at work. Thus, particular domain ontologies can help for building an effective semantic mobile application. Moreover, ontologies capturing complex dependencies between concepts for a particular problem domain provide a flexible and expressive tool for modeling high-level concepts and relations among the given attributes, which allows both the system and the user to operate the same taxonomy and play an important role to build an expert system [ 125 ].

5.3.2 Mobile expert system modeling

A mobile expert system is an example of a knowledge-based system, which is broadly divided into two subsystems, such as the inference engine and the knowledge base, shown in Fig.  2 . The knowledge base typically represents facts and rules, while the inference engine applies the rules to the known facts to deduce new facts. The knowledge-base module is the core of this expert system as it consists of knowledge of the target mobile application domain as well as operational knowledge of apps’ decision rules. The user interface accepts the original facts and invokes the inference engine to activate the decision rules in the knowledge base. The system uses expert knowledge represented mainly as IF-THEN rules, to offer recommendations or making decisions in relevant application areas. For instance, by using the expert system model, the process of selecting the semantic outcome for mobile users becomes more appropriate according to expert recommendations. A rule consists of two parts: the IF part, called the antecedent (premise or condition) and the THEN part called the consequent (conclusion or action).

A structure of a mobile expert system modeling

The basic syntax of a rule is:

IF < antecedent > THEN < consequent  > .

Such an IF-THEN rule-based expert system model can have the decision-making ability of a human expert in an intelligent system that is designed to solve complex problems as well through knowledge reasoning. To develop the knowledge base module, an ontology-based knowledge representation platform discussed earlier can play a major role to generate the conceptual rules. To provide a continuous supply of knowledge to a rule-based expert system, data mining, and machine learning techniques can be used. For instance, in our earlier approach “ABC-RuleMiner”, Sarker et al. [ 29 ], we have discovered a set of useful contextual rules for mobile phone users considering their behavioral patterns in the data. Domain experts having knowledge of business rules can then update and manage the rules according to the needs. Thus, the mobile expert systems can be used to make intelligent decisions in corresponding mobile applications.

6 AI-powered intelligent Mobile apps

An intelligent system typically tells what to do or what to conclude in different situations [ 126 ] and can act as an intelligent software agent. Thus, intelligent mobile apps are those applications that use AI-based modeling discussed above, in order to make intelligent decisions and to provide useful suggestions and recommendations. Based on this, the target mobile applications for various daily life services are outlined in the following subsections ranging from personalized to community services.

6.1 Personalized Mobile user experience

In the real world, people want their experience to be absolutely personalized these days. Thus, most of the mobile apps heavily rely on personalization to keep users engaged and interested. Users also now expect the applications to deliver unique experiences that may vary from user-to-user according to their own preferences. Thus understanding “user persona” is the key to creating personalized mobile applications that are based on users’ past experiences represented by users’ historical data. ML-based models can effectively discover useful insight from individuals’ phone data by taking into account users own behavioral activities, interactions, or preferences, and can be used to perform individual personalized services in various applications. For instance, an intelligent phone call interruption management system can be a real-life application based on the discovered rules, which handles the incoming phone calls automatically according to the behavior of an individual user [ 29 ]. Moreover, mobile notification management [ 58 , 59 ], apps usage prediction and management, etc. can be the real-life examples of personalized services for the end mobile phone users. Thus, the extracted insight from relevant contextual historical and real-time interaction data using ML-based models can be used to deliver rich and personalized experiences to the users in various day-to-day situations in their daily life activities. Similarly, a knowledge-based mobile expert system considering a set of context-aware IF-THEN rules, can also help to provide personalized services for individual users.

6.2 Mobile recommendation

Recommender systems are typically developed to overcome the problem of information overload by aiding users in the search for relevant information and helping them identify which items (e.g., media, product, or service) are worth viewing in detail. This task is also known as information filtering. According to [ 127 ], the most important feature of a recommender system is its ability to “guess” a user’s preferences and interests by analyzing the behavior of the user and/or the behavior of other users to generate personalized recommendations. In general, the traditional recommender systems mainly focus on recommending the most relevant items to users among a huge number of items [ 128 ]. However, mobile recommendation systems based on users’ contextual information such as temporal, spatial, or social etc. could be more interesting for the users [ 62 , 63 , 64 ]. The advanced mobile apps powered by predictive intelligent capabilities using ML-based models make recommend engines smart enough to analyze the user content preferences and cater to the appropriate content that the user is looking for. For instance, a mobile system generating shopping recommendations helps the user to find the most satisfying product by reducing search effort and information overload. Similarly, tourist guides [ 129 ], food or restaurant services [ 130 ], finding cheaper flights, accommodation, attractions, or leisure dissemination, etc. can be other real-life examples for the mobile phone users. Moreover, an NLP-based methodology can be a way to retrieve the best recommendation service based on public comments.

6.3 Mobile virtual assistance

An intelligent virtual assistant is also known as an intelligent personal assistant that is typically a software agent to perform tasks or services for an individual based on queries like commands or questions. The chatbot is sometimes used to refer to virtual assistants, which is a software application used to conduct an online chat conversation via text or text-to-speech. Several key advantages make the chatbots beneficial these days as they are able to provide 24*7 automated support, able to provide instant answers, good in handling customers or users, avoiding repetitive work, as well as save time and service cost. Intelligent mobile apps powered by AI are able to provide such services with higher accuracy. AI-based models including NLP and ML can be used to build such applications. Moreover, people are now typically spending more time on different messaging apps that are the platforms of communication and bots will be how their users access all sorts of services. Thus, chatbots can engage by answering basic questions in various services. For instance, online ordering, product suggestions, customer support, personal finance assistance, searching, and flight tracking, finding a restaurant, etc. A knowledge-based mobile expert system considering a set of IF-THEN rules, can also be applied to provide such service. Thus, different virtual assistant apps like voice assistants or chatbots offer interactive experiences to users, who are able to retrieve the necessary information effectively and efficiently according to their needs.

6.4 Internet of things (IoT) and smart cities

The Internet of Things (IoT) is typically a network of physical devices, and objects which utilize sensors, software, etc. for sending and receiving data. Smart cities use IoT devices as well to collect and analyze data, and become the most extensive application domain these days. In general, the smart city development is considered as a new way of thinking among cities, businesses, citizens, academia, industry people or others, who are the key stakeholders. As today’s smartphones are considered as one of the most important IoT devices [ 2 ], integrating mobile apps with IoT developments can dramatically improve the quality of human life. AI-based modeling in apps can provide relevant intelligent services in this domain, as well as can bring technology, government, and different layers of society together for the betterment of human life. For instance, machine learning-based modeling utilizing sensor data collected from parking places, or traffic signals, can be used for a better city planning for the governments. Similarly, a knowledge-based mobile expert system considering a set of IF-THEN rules, can help to make context-aware and timely decisions. Overall, AI-based modeling can assist the users in our most common daily life issues, such as questions, suggestions, general feedback, and reporting in various smart city services including smart governance, smart home, education, communication, transportation, retail, agriculture, health care, enterprise and many more.

6.5 Mobile business

Smart mobile apps have the potential to increase the operational excellence in the business-to-business as well as business-to-customer sectors. The new availability and advancement of AI and machine learning are causing a revolutionary shift in business and is considered as the new digital frontier for enterprises. Since, almost every organization deal with customer service, the businesses people think about intelligent interactions within mobile applications these days according to consumer demands. Businesses can leverage the data that are collected from various sources such as point-of-sale machines, online traffic, mobile devices, etc. to analyze and strategically improve the user experience. AI techniques can find trends from data and adjust the apps themselves to create more meaningful and context-rich opportunities to engage users. For instance, machine learning algorithms are capable to understand the customer behavior, interests, and provide them with more relevant product recommendations based on purchase history, fraud identification with credit cards, and visual search. By taking into account context-awareness, it can also empower businesses with prominent features, such as delivering precise location-based suggestions. Moreover, an NLP-based methodology of sentiment evaluation such as positive, neutral, or negetive sentiment (also known as opinion mining) on business data, e.g., review comments, can retrieve the best and perfect suggestions and product recommendations in terms of quality and quantity for the customers. AI positively impacts customer behavior by incorporating the chatbots as well in a mobile application, which may reduce the repetitive tasks and optimize manpower utilization. Similarly, knowledge-based mobile expert system considering a set of business IF-THEN rules, can make intelligent decisions. Overall, AI mobile applications in the business domain help in expanding businesses, introducing new products or services, identifying customer interests, and maintaining a prominent position in the global market.

6.6 Mobile healthcare and medicine

Intelligent mobile healthcare applications are bringing better opportunities for both the patients, medical practitioners, or related organizations through simplifying their physical interactions. These apps can provide opportunities to several health-related services such as medical diagnosis, medicine recommendation including e-prescription, suggesting primary precautions, remote health monitoring, or effectively patient management in the hospital. For assessing and strengthening health facilities, or building health management information systems (HMIS), various kinds of health data can be collecting from multiple sources on a wide variety of health topics to analyze [ 131 ]. With the help of AI methods including ML-based models, intelligent health services can be provided. Thus it may reduce the expense and time of the patients and clinics, as they offer customized medicines and drugs as well as give preventive measures through continuous information accumulation. Moreover, AI-powered mobile applications could also be applicable to find the best nearest doctor, to book a consultation, to keep reminders of medication, getting a basic knowledge of each medication, and more. Mobile healthcare app is also able to help doctors with remaining updates with real-time status of consultations, assigning duties to staff, ensuring the availability of equipment, maintaining a proper temperature for medicines, and more. In addition, the healthcare virtual assistant services like chatbots can be used to provide basic healthcare service as well, as these online programs can assist patients in many ways, such as scheduling appointments, answering common questions, aiding in the payment process, and even providing basic virtual diagnostics. Overall, AI-modeling based mobile healthcare services may create a new endeavor for all citizens in a country including the rural people of low-income countries.

6.7 The novel coronavirus COVID-19

Coronavirus disease (COVID-19) is an infectious disease caused by a newly discovered coronavirus [ 131 ]. COVID-19 apps typically are known as the mobile software applications that use digital contact tracing in response to the COVID-19 pandemic, i.e. the process of identifying persons (“contacts”) who may have been in contact with an infected person. According to the World Health Organization (WHO) [ 131 ], most people infected with the COVID-19 virus will experience mild to moderate respiratory illness and recover without requiring special treatment. Thus, in order to keep this infectious disease in control, “contact tracing” is an important factor. Smartphone apps are playing a big role in the response to the COVID-19 pandemic. These apps are being used to track infected people, social distancing, detecting COVID-19 symptoms, self-quarantine guidelines, the latest communication to the citizens, and ease the burden on healthcare staff. Thus, mobile apps are considered as an effective control strategy against the spread of COVID-19 or similar future pandemics, considering the patient and social sensing data. An intelligent framework and mobile application design will not only strengthen the fight against ongoing COVID-19 challenges based on the collected data by mobile phones, but also against similar disasters in a post-COVID world.

In addition to these application areas, AI-based models in mobile applications can also be applicable to several other domains, such as financial, manufacturing, smart robotics, security and privacy, and many more. Thus, the impact of AI-models in mobile app development and user experience is significant in these days and can be considered as next-generation mobile learning.

7 Research issues and future directions

With the rapid development of smartphones, Internet-of-Things (IoT), and AI technologies, the most fundamental challenge is to explore the relevant data collected from diverse sources and to extract useful insights for future actions. Thus, in this section, we highlight and analyze the main challenges and research issues in the scope of our study. In the following, the issues that we identified and corresponding future directions are discussed briefly.

According to our study in this paper, source datasets are the primary component to work in the area of mobile data science. Thus, collecting real-world data such as categorical, numerical, or textual relevant to a particular application is the first step for building an intelligent smartphone apps, which may vary from service to service. For instance, to manage mobile interruptions, the relevant contextual information and an individual’s behavioral data is needed to be analyzed [ 4 ]. Similarly, for smart healthcare services, patient data and corresponding contextual information might be useful. Thus, to facilitate the extraction of reliable insight from the data using AI techniques and to use the knowledge in context-aware applications, integrating and effective management of mobile data is important. The reason is that AI methods particularly machine learning techniques highly impact on data [ 9 ]. Therefore, establishing a large number of recent datasets from diverse sources and to integrate and manage such information for effective data analysis is needed, which could be one of the major challenges to work in the area of mobile data science and data-driven intelligent applications.

The next challenge is an effective modeling of mobile users and their activities from the relevant data. The main goal of mobile user modeling is the customization and adaptation of systems to the user’s specific needs. The system needs to output the ‘right’ outcome at the ‘right’ time or contexts in the ‘right’ way [ 4 ]. Thus, several aspects such as context-dependency, individual user behavior, and their preferences in different contexts are needed to take into account for an effective user modeling and to build corresponding intelligent apps. The reason is that usage patterns of mobile phones vary greatly between individuals behaving differently in different contexts. Thus considering various contexts, such as temporal, spatial, social, etc. and their effective modeling based on these contexts are important to build an intelligent app [ 93 ]. For this purpose, data preparation, discretization of contexts, and discovery of useful insights are the key issues [ 4 ]. Moreover, the concept of RecencyMiner [ 76 ] can be more effective because of considering the recent pattern-based insights. Therefore, effectively modeling mobile users considering these aspects, could be another research issue in the area of mobile data science and intelligent applications.

The context-sensitive features in mobile data and their patterns are of high interest to be discovered and analyzed to make context-aware intelligent decisions for a particular application in a pervasive computing environment. The traditional analytical techniques including data science and machine learning may not be applicable to make real-time decisions for analyzing smartphone data, because of a large number of data processing that may reduce the performance of mobile phones. For instance, the association rule mining technique [ 120 ] may discover a large number of redundant rules that become useless and make the decision-making process complex and ineffective [ 29 ]. Such traditional techniques may not be applicable for analyzing smartphone data. Thus, a deeper understanding is necessary on the strengths and weaknesses of state-of-the-art big data processing and analytics systems to realize large-scale context-awareness and to build a smart context-aware model. Therefore effectively building a data-driven context-aware model for intelligent decision-making on smartphones, could be another research issue in the area of mobile data science and intelligent applications.

Real-life mobile phone datasets may contain many features or high-dimensions of contexts, which may cause several issues such as increases model complexity, may arise over-fitting problem, and consequently decreases the outcome of the resultant AI-based model [ 77 ]. The reason is that the performance of AI methods particularly machine learning algorithms heavily depends on the choice of features or data representation. Having irrelevant features or contextual information in the data makes the model learn based on irrelevant features that consequently decrease the accuracy of the models [ 132 ]. Thus the challenge is to effectively select the relevant and important features or extracting new features that are known as feature optimization. In the area of AI, particularly data science and machine learning, feature optimization problem is considered as an important pre-processing step that helps to build an effective and simplified model and consequently improves the performance of the learning algorithms by removing the redundant and irrelevant features [ 111 ]. Therefore, feature optimization could be a significant research issue in the area of mobile data science and intelligent applications.

The next challenge is the extraction of the relevant and accurate information from the unstructured or semi-structured data on mobile phones. A large amount of content such as emails, web pages, or documents is read on these devices frequently that is text-based [ 15 ]. Thus the problem of information overload arises due to the small screen of the devices rather than the desktop computer. Therefore effectively mining the contents or texts considering these aspects, could be another research issue in the area of mobile data science and intelligent applications. Natural language processing (NLP) techniques can help to make such text-based apps smarter, by automatically analyzing the meaning of content and taking appropriate actions on behalf of their users. Due to the devices’ limited input and processing capabilities rather than desktop computers, it is then needed to develop novel approaches that can bring NLP power to smartphones. Several NLP tasks such as automatic summarization, information extraction, or new content development, etc. could be useful to minimize the issue.

Mobile expert system uses expert knowledge represented mainly as IF-THEN rules, to offer recommendations or making decisions in relevant application areas. However, the development of large-scale rule-based systems may face numerous challenges. For instance, the reasoning process can be very complex, and designing of such systems becomes hard to manage [ 133 ]. There is still a lack of lightweight rule-based inference engines that will allow for reasoning on mobile devices [ 133 ]. Thus a set of concise and effective rules will be beneficial in terms of outcome and simplicity for such a rule-based expert system for mobile devices. Moreover, ontologies [ 125 ] capturing complex dependencies between concepts for a particular problem domain provides a flexible and expressive tool for modeling high-level concepts and relations among the given attributes, which allows both the system and the user to operate the same taxonomy and play an important role to build an expert system. This is where the ontological modeling and reasoning is useful. Thus, an effective design of ontology, or knowledge representation model for the respective problem domain could be another research issue.

The mobility of computing devices, e.g., smartphones, applications, and users leads to highly dynamic computing environments. Unlike desktop applications, which rely on a carefully configured, and largely static set of resources, pervasive computing applications are subjected to changes in available resources such as network connectivity, user contexts, etc. Moreover, they are frequently required to cooperate spontaneously and opportunistically with previous unknown software services to accomplish tasks on behalf of users. Thus, pervasive computing software must be highly adaptive and flexible. As an example, an application may need to modify it’s style of output following a transition from an office environment to a moving vehicle, to be less intrusive [ 4 ]. Thus to effectively adapt to the changing environment according to users’ needs is important, which is important in the area of mobile data science and intelligent applications. Context-awareness represents the ability of mobile devices to sense their physical environment and adapt their behavior accordingly, incorporating this property in the applications could be a potential solution to overcome this issue.

8 Discussion

Although several research efforts have been directed towards intelligent mobile apps, discussed throughout the paper, this paper presents a comprehensive view of mobile data science and intelligent apps in terms of concepts and AI-based modeling. For this, we have conducted a literature review to understand the contexts, mobile data, context-aware computing, data science, intelligent apps characteristics, and different types of mobile systems and services, as well as the used techniques, related to mobile applications. Based on our discussion on existing work, several research issues related to mobile datasets, user modeling, intelligent decision making, feature optimization, mobile text mining based on NLP, mobile expert system, and context-aware adaptation, etc. are identified that require further research attention in the domain of mobile data science and intelligent apps.

The scope of mobile data science is broad. Several data-driven tasks, such as personalized user experience, mobile recommendations, virtual assistant, mobile business, and even mobile healthcare system including the COVID-19 smartphone app, etc. can be considered as the scope of mobile data science. Traditionally mobile app development mostly focused on knowledge that is not automatically discovered [ 47 , 66 ]. Taking the advantage of large amounts of data with rich information, AI is expected to help with studying much more complicated yet much closer to real-life applications, which then leads to better decision making in relevant applications. Considering the volume of collected data and the features, one can decide whether the standalone or cloud-based application is more suitable to provide the target service. Thus, the output of AI-based modeling can be used in many application areas such as mobile analytics, context-aware computing, pervasive computing, health analytics, smart cities, as well as the Internet of things (IoT). Moreover, intelligent data-driven solutions could also be effective in AI-based mobile security and privacy, where AI works with huge volumes of security event data to extract the useful insights using machine learning techniques [ 10 ].

Although the intelligent apps discussed in this paper can play a significant role in the betterment of human life in different directions, several dependencies may pose additional challenges, such as the availability of network and the data transfer speeds or the battery life of mobile devices. Moreover, privacy and security issues may become another challenge while considering the data collection and processing over the cloud or within the device. Taking the advantages of these issues considering the application type and target goal, we believe this analysis and guidelines will be helpful for both the researchers and application developers to work in the area of mobile data science and intelligent apps.

9 Conclusion

In this paper, we have studied on mobile data science and reviewed the motivation of using AI in mobile apps to make it intelligent. We aimed to provide an overview of how artificial intelligence can be used to design and develop data-driven intelligent mobile applications for the betterment of human life. For this, we have presented an AI-based modeling that includes machine learning and deep learning methods, the concept of natural language processing, as well as knowledge representation and expert systems. Such AI-based modeling can be used to build intelligent mobile applications ranging from personalized recommendations to healthcare services including COVID-19 pandemic management, that are discussed briefly in this paper. A successful intelligent mobile system must possess the relevant AI-based modeling depending on the data characteristics. The sophisticated algorithms then need to be trained through collected data and knowledge related to the target application before the system can assist the users with suggestions and decision making. We have concluded with a discussion about various research issues and future directions relevant to our analysis in the area of mobile data science and intelligent apps, that can help the researchers to do future research in the identified directions.

Peng M, Zeng G, Sun Z, Huang J, Wang H, Tian G (2018) Personalized app recommendation based on app permissions. World Wide Web 21(1):89–104

Google Scholar  

El Khaddar MA, Boulmalf M (2017) Smartphone: the ultimate iot and ioe device. Smartphones from an applied research perspective , page 137

Zheng P, Ni LM (2006) Spotlight: the rise of the smart phone. IEEE Distributed Systems Online 7(3):3–3

Sarker IH (2019) Context-aware rule learning from smartphone data: survey, challenges and future directions. Journal of Big Data 6(1):1–25

MathSciNet   Google Scholar  

Google trends. In https://trends.google.com/trends/ , 2019

Pejovic V, Musolesi M (2014) Interruptme: designing intelligent prompting mechanisms for pervasive applications. In Proceedings of the International Joint Conference on Pervasive and Ubiquitous Computing, Seattle, WA, USA, 13–17 September, pp.∼897–908. ACM, New York, USA

Finin T, Joshi A, Kagal L, Ratsimore O, Korolev V, Chen H (2001) Information agents for mobile and embedded devices. Cooperative Information Agents V, pages 264–286

Damiao Ribeiro de Almeida, Cláudio de Souza Baptista, Elvis Rodrigues da Silva, Cláudio EC Campelo, Hugo Feitosa de Figueirêdo, and Yuri Almeida Lacerda (2006) A context-aware system based on service-oriented architecture. In Advanced Information Networking and Applications, 2006 . AINA 2006 . 20th International Conference on , volume 1, pages 6–pp. IEEE

Han J, Pei J, Kamber M (2011) Data mining: concepts and techniques. Elsevier, Amsterdam

MATH   Google Scholar  

Sarker IH, Kayes ASM, Badsha S, Alqahtani H, Watters P, Ng A (2020) Cybersecurity data science: an overview from machine learning perspective. Journal of Big Data 7(1):1–29

Ślusarczyk B (2018) Industry 4.0: Are we ready? Polish Journal of Management Studies , 17

Dey AK (2001) Understanding and using context. Pers Ubiquit Comput 5(1):4–7

Sarker IH, Kayes ASM, Watters P (2019) Effectiveness analysis of machine learning classification models for predicting personalized context-aware smartphone usage. Journal of Big Data 6(1):1–28

Witten IH, Frank E, Trigg LE, Hall MA, Holmes G, Cunningham SJ (1999) Weka: Practical machine learning tools and techniques with java implementations

Sateli B, Cook G, Witte R (2013) Smarter mobile apps through integrated natural language processing services. In International Conference on Mobile Web and Information Systems , pages 187–202. Springer

Cao L (2017) Data science: a comprehensive overview. ACM Computing Surveys (CSUR) 50(3):43

Dourish P (2004) What we talk about when we talk about context. Pers Ubiquit Comput 8(1):19–30

Schilit BN, Theimer MM (1994) Disseminating active map information to mobile hosts. IEEE Netw 8(5):22–32

Brown PJ, Bovey JD, Chen X (1997) Context-aware applications: from the laboratory to the marketplace. IEEE Pers Commun 4(5):58–64

Brown PJ (1995) The stick-e document: a framework for creating context-aware applications. Electronic Publishing-Chichester 8:259–272

Ryan N, Pascoe J, Morse D (1999) Enhanced reality fieldwork: the context aware archaeological assistant. Bar International Series 750:269–274

Franklin D, Flaschbart J (1998) All gadget and no representation makes jack a dull environment. In Proceedings of the AAAI 1998 Spring Symposium on Intelligent Environments , pages 155–160

Hull R, Neaves P, Bedford-Roberts J (1997) Towards situated computing. In Wearable Computers, 1997 . Digest of Papers., First International Symposium on , pages 146–153. IEEE

Ward A, Jones A, Hopper A (1997) A new location technique for the active office. IEEE Pers Commun 4(5):42–47

Rodden T, Cheverst K, Davies K, Dix A (1998) Exploiting context in hci design for mobile systems. In Workshop on human computer interaction with mobile devices , pages 21–22. Glasgow

Schilit B, Adams N, Want R (1994) Context-aware computing applications. In Mobile Computing Systems and Applications, 1994 . WMCSA 1994 . First Workshop on , pages 85–90. IEEE

Shi Y (2006) Context awareness, the spirit of pervasive computing. In Pervasive Computing and Applications, 2006 1st International Symposium on , pages 6–6. IEEE

Christos Anagnostopoulos, Athanasios Tsounis, and Stathes Hadjiefthymiades (2005) Context management in pervasive computing environments. In Pervasive Services, 2005 . ICPS’05 . Proceedings. International Conference on , pages 421–424. IEEE

Sarker IH, Kayes ASM (2020) Abc-ruleminer: User behavioral rule-based machine learning method for context-aware intelligent services. Journal of Network and Computer Applications , page 102762

Phithakkitnukoon S, Dantu R, Claxton R, Eagle N (2011) Behavior-based adaptive call predictor. ACM Transactions on Autonomous and Adaptive Systems 6(3):21:1–21:28

Eagle N, Pentland AS (2006) Reality mining: sensing complex social systems. Personal and ubiquitous computing 10(4):255–268

Almeida TA, Hidalgo JMG, Yamakami A (2011) Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 11th ACM symposium on Document engineering , pages 259–262. ACM

Fischer JE, Yee N, Bellotti V, Good N, Benford S, Greenhalgh C (2010) Effects of content and time of delivery on receptivity to mobile interruptions. In Proceedings of the 12th international conference on Human computer interaction with mobile devices and services , pages 103–112. ACM

Zhu H, Chen E, Xiong H, Yu K, Cao H, Tian J (2014) Mining mobile user preferences for personalized context-aware recommendation. ACM Transactions on Intelligent Systems and Technology (TIST) 5(4):58

Srinivasan V, Moghaddam S, Mukherji A (2014) Mobileminer: Mining your frequent patterns on your phone. In Proceedings of the International Joint Conference on Pervasive and Ubiquitous Computing, Seattle, WA, USA, 13–17 September, pp.∼389–400. ACM, New York, USA

Kim J, Mielikäinen T (2014) Conditional log-linear models for mobile application usage prediction. In Machine Learning and Knowledge Discovery in Databases , pages 672–687. Springer

Liao Z-X, Pan Y-C, Peng W-C, Lei P-R (2013) On mining mobile apps usage behavior for predicting apps usage in smartphones. In Proceedings of the 22nd International Conference on Information & Knowledge Management , pages 609–618. ACM

Zhu H, Chen E, Xiong H, Cao H, Tian J (2014) Mobile app classification with enriched contextual information. IEEE Trans Mob Comput 13(7):1550–1563

Mehrotra A, Hendley R, Musolesi M (2016) Prefminer: mining user’s preferences for intelligent mobile notification management. In Proceedings of the International Joint Conference on Pervasive and Ubiquitous Computing, Heidelberg, Germany, 12–16 September, pp. ∼ 1223–1234. ACM, New York, USA.

Halvey M, Keane MT, Smyth B (2005) Time based segmentation of log data for user navigation prediction in personalization. In Proceedings of the International Conference on Web Intelligence, Compiegne, France, 19–22 September, pp. ∼ 636–640. IEEE Computer Society, Washington, DC, USA.

Halvey M, Keane MT, Smyth B (2006) Time based patterns in mobile-internet surfing. In Proceedings of the SIGCHI Conference on Human Factors in computing systems, Montreal, Quebec, Canada, 22–27 April, pp.∼31–34. ACM, New York, USA

Bordino I, Donato D (2012) Extracting interesting association rules from toolbar data. In International Conference on Information and Knowledge Management . ACM

Paireekreng W, Rapeepisarn K, Wong KW (2009) Time-based personalised mobile game downloading. In Transactions on Edutainment II, pp. ∼ 59–69

Rawassizadeh R, Tomitsch M, Wac K, Tjoa AM (2013) Ubiqlog: a generic mobile phone-based life-log framework. Personal and ubiquitous computing 17(4):621–637

Danninger M, Kluge T, Stiefelhagen R (2006) Myconnector: analysis of context cues to predict human availability for communication. In Proceedings of the 8th International Conference on Multimodal Interfaces , pages 12–19. ACM

Khalil A, Connelly K (2005) Improving cell phone awareness by using calendar information. In Human-Computer Interaction , pages 588–600. Springer

Dekel A, Nacht D, Kirkpatrick S (2009) Minimizing mobile phone disruption via smart profile management. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services , page 43. ACM

Zulkernain S, Madiraju P, Ahamed SI, Stamm K (2010) A mobile intelligent interruption management system. J. UCS 16(15):2060–2080

Pielot M (2014) Large-scale evaluation of call-availability prediction. In Proceedings of the International Joint Conference on Pervasive and Ubiquitous Computing , pages 933–937. ACM

Knittel J, Shirazi AS, Henze N, Schmidt A (2013) Utilizing contextual information for mobile communication. In Extended Abstracts on Human Factors in Computing Systems , pages 1371–1376. ACM

Smith J, Dulay N (2014) Ringlearn: Long-term mitigation of disruptive smartphone interruptions. In International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops) , pages 27–35. IEEE

Stern H, Pammer V, Lindstaedt SN (2011) A preliminary study on interruptibility detection based on location and calendar information. Proceedings of Context-Systems Design, Evaluation and Optimisation(CoSDEO)

Vilwock W, Madiraju P, Ahamed SI (2013) A system implementation of interruption management for mobile devices. In Proceedings of the 16th International Conference on Computational Science and Engineering , pages 181–187. IEEE

Bohmer M, Lander C, Gehring S, Brumby DP, Kruger A (2014) Interrupted by a phone call: exploring designs for lowering the impact of call notifications for smartphone users. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , pages 3045–3054. ACM

Grandhi SA, Jones Q (2015) Knock knock whos there? putting the user in control of managing interruptions. International Journal of Human-Computer Studies 79:35–50

Shirazi AS, Henze N, Dingler T, Pielot M, Weber D, Schmidt A (2014) Large-scale assessment of mobile notifications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , pages 3055–3064. ACM

Iqbal ST, Horvitz E (2010) Notifications and awareness: a field study of alert usage and preferences. In Proceedings of the 2010 ACM conference on Computer supported cooperative work , pages 27–30. ACM

Kanjo E, Kuss DJ, Ang CS (2017) Notimind: Utilizing responses to smart phone notifications as affective sensors. IEEE Access 5:22023–22035

Turner LD, Allen SM, Whitaker RM (2015) Push or delay? decomposing smartphone notification response behaviour. In Human Behavior Understanding , pages 69–83. Springer

Park M-H, Hong J-H, Cho S-B (2007) Location-based recommendation system using bayesian user’s preference model in mobile devices. In International Conference on Ubiquitous Intelligence and Computing , pages 1130–1139. Springer

Zheng VW, Cao B, Zheng Y, Xie X, Yang Q (2010) Collaborative filtering meets mobile recommendation: A user-centered approach. In AAAI , volume 10, pages 236–241

Kim K-j, Ahn H, Jeong S (2010) Context-aware recommender systems using data mining techniques. In Proceedings of world academy of science, engineering and technology , volume 64, pages 357–362

Liu Q, Ge Y, Li Z, Chen E, Xiong H (2011) Personalized travel package recommendation. In Data Mining (ICDM), 2011 IEEE 11th International Conference on , pages 407–416. IEEE

Shin D, Lee J-w, Yeon J (2009) Context-aware recommendation by aggregating user context. In IEEE Conference on Commerce and Enterprise Computing, Vienna, Austria, Austria, 20–23 July, pp.∼423–430. IEEE Computer Society, Washington, DC, USA

Liu B, Kong D, Cen L, Gong NZ, Jin H, Xiong H (2015) Personalized mobile app recommendation: Reconciling app functionality and user privacy preference. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining , pages 315–324. ACM

Seo S-s, Kwon A, Kang J-M, Strassner J (2011) Pyp: design and implementation of a context-aware configuration manager for smartphones. In International Workshop on Smart Mobile Applications

Bozanta A, Kutlu B (2018) Developing a contextually personalized hybrid recommender system. Mob Inf Syst 2018:1–13

Lee W-P (2007) Deploying personalized mobile services in an agent-based environment. Expert Syst Appl 32(4):1194–1207

Turner LD, Allen SM, Whitaker RM (2015) Interruptibility prediction for ubiquitous systems: conventions and new directions from a growing field. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing , pages 801–812. ACM

Fogarty J, Hudson SE, Atkeson CG, Avrahami D, Forlizzi J, Kiesler S, Lee JC, Yang J (2005) Predicting human interruptibility with sensors. ACM Transactions on Computer-Human Interaction (TOCHI) 12(1):119–146

Bedogni L, Di Felice M, Bononi L (2016) Context-aware android applications through transportation mode detection techniques. Wirel Commun Mob Comput 16(16):2523–2541

Tan GW-H, Ooi K-B, Leong L-Y, Lin B (2014) Predicting the drivers of behavioral intention to use mobile learning: A hybrid sem-neural networks approach. Computers in Human Behavior 36:198–213

Sathiya Keerthi S, Shevade SK, Bhattacharyya C, Murthy KRK (2001) Improvements to platt’s smo algorithm for svm classifier design. Neural computation 13(3):637–649

Breiman L (2001) Random forests. Mach Learn 45(1):5–32

Sarker IH, Colman A, Kabir MA, Han J (2018) Individualized time-series segmentation for mining mobile phone user behavior. The Computer Journal, Oxford University, UK 61(3):349–368

Sarker IH, Colman A, Han J (2019) Recencyminer: mining recency-based personalized behavior from contextual smartphone data. Journal of Big Data 6(1):1–21

Sarker IH, Abushark YB, Khan AI (2020) Contextpca: Predicting context-aware smartphone apps usage based on machine learning techniques. Symmetry 12(4):499

Sarker IH, Alqahtani H, Alsolami F, Khan AI, Abushark YB, Siddiqui MK (2020) Context pre-modeling: an empirical analysis for classification based user-centric context-aware predictive modeling. Journal of Big Data 7(1):1–23

Pielot M, De Oliveira R, Kwak H, Oliver N (2014) Didn’t you see my message?: predicting attentiveness to mobile instant messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , pages 3319–3328. ACM

Bayat A, Pomplun M, Tran DA (2014) A study on human activity recognition using accelerometer data from smartphones. Procedia Computer Science 34:450–457

Ayu MA, Ismail SA, Matin AFA, Mantoro T (2012) A comparison study of classifier algorithms for mobile-phone’s accelerometer based activity recognition. Procedia Engineering 41:224–229

Fetter M, Seifert J, Gross T (2011) Predicting selective availability for instant messaging. In IFIP Conference on Human-Computer Interaction , pages 503–520. Springer

Fisher R, Simmons R (2011) Smartphone interruptibility using density-weighted uncertainty sampling with reinforcement learning. In 2011 10th International Conference on Machine Learning and Applications and Workshops , volume 1, pages 436–441. IEEE

Swati K, Patankar AJ (2014) Effective personalized mobile search using knn. In 2014 International Conference on Data Science & Engineering (ICDSE) , pages 157–160. IEEE

Middleton SE, Shadbolt NR, De Roure DC (2004) Ontological user profiling in recommender systems. ACM Transactions on Information Systems (TOIS) 22(1):54–88

Anagnostopoulos T, Anagnostopoulos C, Hadjiefthymiades S, Kyriakakos M, Kalousis A (2009) Predicting the location of mobile users: a machine learning approach. In Proceedings of the 2009 international conference on Pervasive services , pages 65–72. ACM

Riboni D, Bettini C (2011) Cosar: hybrid reasoning for context-aware activity recognition. Pers Ubiquit Comput 15(3):271–289

Zhong E, Tan B, Mo K, Yang Q (2013) User demographics prediction based on mobile data. Pervasive and mobile computing 9(6):823–837

Wang Y, Feng D, Li D, Chen X, Zhao Y, Niu X (2016) A mobile recommendation system based on logistic regression and gradient boosting decision trees. In 2016 International Joint Conference on Neural Networks (IJCNN) , pages 1896–1902. IEEE

Ernsting C, Dombrowski SU, Oedekoven M, Julie LO, Kanzler M, Kuhlmey A, Gellert P et al (2017) Using smartphones and health apps to change and manage health behaviors: a population-based survey. Journal of medical Internet research 19(4):e101

Sarker IH (2019) A machine learning based robust prediction model for real-life mobile phone data. Internet of Things 5:180–193

Hong J, Suh E-H, Kim J, Kim SY (2009) Context-aware system for proactive personalized service based on context history. Expert Syst Appl 36(4):7448–7457

Sarker IH, Colman A, Han J, Khan AI, Abushark YB, Salah K (2019) Behavdt: A behavioral decision tree learning to build user-centric context-aware predictive model. Mobile Networks and Applications , pages 1–11

Alawnah S, Sagahyroon A (2017) Modeling of smartphones’ power using neural networks. EURASIP Journal on Embedded Systems 2017(1):22

Leong L-Y, Hew T-S, Tan GW-H, Ooi K-B (2013) Predicting the determinants of the nfc-enabled mobile credit card acceptance: a neural networks approach. Expert Syst Appl 40(14):5604–5620

Chong AY-L (2013) Predicting m-commerce adoption determinants: a neural network approach. Expert Syst Appl 40(2):523–530

Rajashekar D, Nur Zincir-Heywood A, Heywood MI (2016) Smart phone user behaviour characterization based on autoencoders and self organizing maps. In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW) , pages 319–326. IEEE

Kandasamy K, Kumar CS (2015) Modified pso based optimal time interval identification for predicting mobile user behaviour in location based services. Indian Journal of Science and Technology 8(S7):185–193

Hartono RN, Pears R, Kasabov N, Worner SP (2014) Extracting temporal knowledge from time series: A case study in ecological data. In Proceedings of the International Joint Conference on Neural Networks, Beijing, China, 6–11 July, pp. ∼ 4237–4243. IEEE Computer Society, Washington, DC, USA

Keogh E, Chu S, Hart D, Pazzani M (2004) Segmenting time series: a survey and novel approach. Data mining in time series databases 57:1–22

Shokoohi-Yekta M, Chen Y, Campana B, Hu B, Zakaria J, Keogh E (2015) Discovery of meaningful rules in time series. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, 10–13 August, pp. ∼ 1085–1094. ACM, New York, USA

Zhang G, Liu X, Yang Y (2015) Time-series pattern based effective noise generation for privacy protection on cloud. IEEE Trans Comput 64(5):1456–1469

Cao H, Bao T, Yang Q, Chen E, Tian J (2010) An effective approach for mining mobile user habits. In Proceedings of the International Conference on Information and knowledge management, Toronto, ON, Canada, 26–30 October, pp. ∼ 1677–1680. ACM, New York, USA

Iqbal H Sarker (2018) Mobile data science: Towards understanding data-driven intelligent mobile applications. EAI Endorsed Transactions on Scalable Information Systems , 5(19)

La Polla M, Martinelli F, Sgandurra D (2012) A survey on security for mobile devices. IEEE communications surveys & tutorials 15(1):446–471

Otrok H, Mizouni R, Bentahar J et al. (2014) Mobile phishing attack for android platform pages 18–23

Dunham K (2008) Mobile malware attacks and defense

Lin J, Yu W, Zhang N, Yang X, Zhang H, Zhao W (2017) A survey on internet of things: architecture, enabling technologies, security and privacy, and applications. IEEE Internet Things J 4(5):1125–1142

John GH, Langley P (1995) Estimating continuous distributions in bayesian classifiers. In Proceedings of the Eleventh conference on Uncertainty in artificial intelligence , pages 338–345. Morgan Kaufmann Publishers Inc.

Quinlan JR (1993) C4.5: Programs for machine learning. Machine Learning

Sarker IH, Abushark YB, Alsolami F, Khan AI (2020) Intrudtree: A machine learning based cyber security intrusion detection model. Symmetry 12(5):754

Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Machine learning 6(1):37–66

Freund Y, Schapire RE et al. (1996) Experiments with a new boosting algorithm. In Icml , volume 96, pages 148–156. Citeseer

Le Cessie S, Van Houwelingen JC (1992) Ridge estimators in logistic regression. Journal of the Royal Statistical Society: Series C (Applied Statistics) 41(1):191–201

MacQueen J (1967) Some methods for classification and analysis of multivariate observations. In Fifth Berkeley symposium on mathematical statistics and probability , volume 1

Rokach L (2010) A survey of clustering algorithms. In Data Mining and Knowledge Discovery Handbook , pages 269–298. Springer

Sneath PHA (1957) The application of computers to taxonomy. Journal of General Microbiology , 17(1)

Sorensen T (1948) Method of establishing groups of equal amplitude in plant sociology based on similarity of species. Biol. Skr. , 5

Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. In ACM SIGMOD Record , volume 22, pages 207–216. ACM

Agrawal R, Srikant R (1994) Fast algorithms for mining association rules. In Proceedings of the International Joint Conference on Very Large Data Bases, Santiago Chile, pp. ∼ 487–499. , volume 1215

Han J, Pei J, Yin Y (2000) Mining frequent patterns without candidate generation. In ACM Sigmod Record , volume 29, pages 1–12. ACM

Das A, Ng W-K, Woon Y-K (2001) Rapid association rule mining. In Proceedings of the tenth international conference on Information and knowledge management , pages 474–481. ACM

Zaki MJ (2000) Scalable algorithms for association mining. IEEE transactions on knowledge and data engineering 12(3):372–390

Sun S, Luo C, Chen J (2017) A review of natural language processing techniques for opinion mining systems. Information fusion 36:10–25

Maedche A, Staab S (2001) Ontology learning for the semantic web. IEEE Intell Syst 16(2):72–79

Grosan C, Abraham A (2011) Rule-based expert systems. Int Underw Syst Des, pages 149–185

Lu J, Wu D, Mao M, Wang W, Zhang G (2015) Recommender system application developments: a survey. Decision Support Systems 74:12–32

Bobadilla J, Ortega F, Hernando A, Gutiérrez A (2013) Recommender systems survey. Knowl-Based Syst 46:109–132

Marine-Roig E, Martin-Fuentes E, Daries-Ramon N (2017) User-generated social media events in tourism. Sustainability 9(12):2250

Belanche D, Flavián M, Pérez-Rueda A (2020) Mobile apps use and wom in the food delivery sector: The role of planned behavior, perceived security and customer lifestyle compatibility. Sustainability 12(10):4275

World health organization: Who. http://www.who.int/

Yi B-J, Lee D-G, Rim H-C (2015) The effects of feature optimization on high-dimensional essay data. Math Probl Eng 2015:1–12

Bobek S, Nalepa GJ, Ślażyński M (2019) Heartdroid†rule engine for mobile and context-aware expert systems. Expert Systems 36(1):e12328

Download references

Author information

Authors and affiliations.

Swinburne University of Technology, Melbourne, VIC, 3122, Australia

Iqbal H. Sarker & Md. Kafil Uddin

Chittagong University of Engineering and Technology, Chittagong, 4349, Bangladesh

Iqbal H. Sarker & Mohammed Moshiul Hoque

RMIT University, Melbourne, VIC, 3000, Australia

Tawfeeq Alsanoosy

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Iqbal H. Sarker .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Sarker, I.H., Hoque, M.M., Uddin, M.K. et al. Mobile Data Science and Intelligent Apps: Concepts, AI-Based Modeling and Research Directions. Mobile Netw Appl 26 , 285–303 (2021). https://doi.org/10.1007/s11036-020-01650-z

Download citation

Published : 14 September 2020

Issue Date : February 2021

DOI : https://doi.org/10.1007/s11036-020-01650-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mobile data science
• Artificial intelligence
• Machine learning
• Natural language processing
• Expert system
• Data-driven decision making
• Context-awareness
• Intelligent mobile apps
• Find a journal
• Publish with us
• Track your research

• Event Website Publish a modern and mobile friendly event website.
• Registration & Payments Collect registrations & online payments for your event.
• Abstract Management Collect and manage all your abstract submissions.
• Peer Reviews Easily distribute and manage your peer reviews.
• Conference Program Effortlessly build & publish your event program.
• Virtual Poster Sessions Host engaging virtual poster sessions.
• Customer Success Stories
• Wall of Love ❤️

Top 11 Apps for Researchers in 2023

Published on 01 May 2022

The evolution of new technologies has caused a digital transformation in almost every industry and field of interest, including academia. Technology has changed the way that academics conduct research, document findings, and collaborate with peers. 

Academics can now rely on new avenues of collaboration that didn’t even exist when they launched their careers. Networks like SSRN and Mendeley provide opportunities for researchers to share their work for increased collaboration, and abstract management tools streamline the peer review process required by legitimate academic conferences and journals. 

As this digital transformation accelerates, researchers can now access a vast array of apps aimed at simplifying their workflows and facilitating information sharing. While these apps have the potential to improve the way scientists conduct and share their research, the selection can be overwhelming. 

Based on our experience and extensive research, here are the 11 best apps available for researchers in 2023.

1. Fourwaves 

Fourwaves is a conference management software for researchers. Their free web application allows you to create a complete event website, manage abstract submissions, peer reviews, host virtual poster sessions , manage registrations and more. 

It’s the easiest way to organize scientific events as the tool was crafted with researchers in mind every step of the way. 

Fourwaves can be used not only for in-person events but also for hybrid and virtual conferences . They offer a complete virtual venue to access live streams, chat or call other participants and attend virtual poster sessions.

You can go as far as mass email your attendees, automatically generate your event schedule or even print out your name tags; everything you need for your event is in one place.

Most interesting features:

• Ready-to-go event website ; all you have to do is enter your event’s content and you’re ready to publish. 
• Abstract management & Peer review tool ; you can easily collect submissions, review them according to your criterias, email authors and publish your material and the full conference schedule online.
• Registration and payment management ; attendees can easily register to your event and pay online on your Fourwaves event website. 

2. R Discovery: Academic Research

R Discovery is a free app that empowers researchers to save time wading through a sea of academic research papers by finding the articles that are most relevant to your work and delivering them to you each day. It curates over 96 million research articles which includes over 24 million open access articles. 

The app is mobile-only, available for download on the Google Play App Store and the Apple App store for mobile use on your Android device, iPhone or iPad. The app scans papers from all major disciplines in the arts and sciences. 

• As soon as you sign up and submit your areas of interest, R Discovery will serve you the top three related articles in a news feed each day.
• R Discovery uses AI to learn your reading interests over time and populate your news feed with content increasingly tailored to your specific interests.
• The app provides export functions for easy integration with reference managers to organize your citations.

3. LabArchives  

LabArchives is a web-based application that acts as a digital lab notebook, helping researchers keep their work and notes organized to improve productivity in their labs. Users can access LabArchives to make notes, store images and data, and use the search feature for simple access to all of their material. 

There are also Android and iOS versions of this app available in the Apple App Store and Google Play App Store that allow users to access their digital notebooks from their Android devices, iPhones and iPads and have instant access to all of their data, from anywhere. While there are Premium and Enterprise versions of the platform for more advanced use and collaboration, individuals and small teams can access a free version that still includes unlimited notebooks and 1GB of storage. 

Most interesting features: 

• Makes it easy to store and share data between your team members, with user-friendly search functions. You can even share DNA sequence files in over 30 formats! 
• Access information from your desktop or your phone, thanks to the free iOS app for your iPhone or iPad. There is also an Android app available in the Google Play store, but based on reviews it appears that functionality is limited. 
• Data security that lets you determine file access and sharing limitations, so you know exactly who is viewing your files and when.

Typeset is a web-based application that was created to help researchers write, collaborate, format and submit research papers for publication. Typeset allows you to upload your work to their platform, and use their AI to reformat your research and submissions to meet the publication requirements of various journal and conference organizers. 

Typeset works seamlessly with reference management software like Mendeley, Zotero, Paperpile and more. It allows users to choose from over 45,000 verified journal formats and export your work to Word, LaTex and PDF formats. 

Typeset does not offer mobile apps for Apple or Android devices. There are a variety of subscription levels available with pricing ranging from free to $20 per month. 

• Editing features that increase the chances of being published.
• Integrations that enable you to submit research for publication directly from the app.
• Plagiarism and grammar checker for increased quality and peace of mind.

5. BenchSci

The BenchSci platform was built to use advanced biomedical AI to help source the materials that scientific researchers need to move forward with their work. 

Once the app user enters their protein target into the BenchSci platform, the app will sift through thousands of reliable information sources like websites and scientific publications, delivering options that will help determine the antibody or reagent needed. BenchSci is a web-based application that is not available for Android or iOS. It is used by more than 48,000 individual scientists and over 4,000 institutions. BenchSci boasts that their tools can accelerate projects through their AI-powered reagent and antibody selection process, cutting the selection time from 12 weeks to 30 seconds. By empowering researchers to find the antibodies and reagents they need easier and faster, BenchSci reduces the number of materials they need to purchase and experiment with, therefore reducing costs. 

• AI-Assisted Reagent Selection, which uses AI and automation to reduce the errors and inefficiencies in the reagent and model system selection for scientists. 
• AI-Assisted Antibody Selection, which follows the same principle as the reagent selection but focuses on antibodies. This feature is free for you to use if you are a student or researcher at an academic, government, or nonprofit institution. 
• Things change quickly, so the platform is constantly updated to add new antibody and reagent products to ensure that users can access everything available.

6. eLabJournal

There are many Electronic Lab Notebooks (ELNs) available on the market, but the eLabJournal takes the concept of ELNs to the next step. eLabJournal was designed to increase productivity and efficiency in your research lab and simplify the process of organizing and locating data, collaborating with peers, and exporting files into a variety of formats. 

This is a web-based application with mobile versions available on the Google Play and Apple App Stores. Academics can purchase a subscription to the eLabJournal for $15.55 per month, while Industry users are charged $41.95 per month. 

• This ELN uses a simple, intuitive interface that was specifically designed to meet the needs of those in the life science research and development field. 
• Facilitates the ability to link data with functionality to upload images (via the Android and iPhone apps) and a wide range of file types. 
• Seamlessly integrates with eLab’s other products through their SDK and APIs, providing extensive customization opportunities to meet the specific needs of your lab.

7. Connected Papers

Connected Papers is a web-based application that provides a uniquely visual representation of the published research available in a certain field. This helps researchers and scientists browse the information available related to their field of study and ensure that nothing is being missed as they prepare their work for submission. 

The app works when a scientist enters their research topic into the search bar. Within seconds, Connected Papers reviews tens of thousands of papers related to that topic, and creates a visual map showcasing all of the work available for the scientist to review and consider in their research. Connected Papers is currently not available on the Apple App Store or Google Play App Store. It is completely free to use. 

• The visual maps create an easy-to-follow pathway that showcases how closely related particular sources are to the work you’re conducting.
• The app creates clusters that groups papers based on their level of similarities, and pushes less relevant papers away.
• Connected works scans the citations used by various sources and classified papers to be closely related based on how many citations overlap. 

8. Papership

The Papership app allows you to store, annotate, manage and share research papers from anywhere. Available on your Mac, iPhone, and iPad, Papership syncs with popular web-based platforms Zotero and Mendeley to allow app users to access their curated research libraries stored in their Zotero and Mendeley accounts conveniently and remotely. 

• You can choose a free version of the app which can integrate with annotation apps like Evernote, or purchase the annotation function of Papership for $9.99 per month.
• Documents annotated through Papership can be shared via email, SMS, iMessage, Facebook and Twitter. 
• Papership provides quantitative measurements of the significance of a publication to alert the reader as to the legitimacy of the research. It measures both peer-reviewed and non-peer reviewed sources. 

9. GanttPRO

Ganttpro is a web-based project management application that helps research teams plan and organize projects through the use of collaborative Gantt charts. By providing the ability to create interactive Gantt charts online, GanttPRO makes it possible to plan and control many projects at the same time. It empowers researchers to organize and schedule tasks, set deadlines, identify dependencies and manage resources, all while making this information readily available to all collaborators. GanttPRO is available in a mobile version that can be downloaded for your Android and Apple mobile devices. The company offers a free trial and once that is complete different app packages are available that range from $7.99 to $19.99 per month. 

• Drag and drop capabilities to make it simple to organize and reorganize as inputs, outputs and priorities change
• Allows for the creation of multiple workspaces to separate personal tasks from overall team projects
• Collaborative functions make it easy to track the progress of each team member and step in to help whenever needed. 

Trello is an app that can be used by academics, researchers, marketers, computer scientists and basically any other student, professor or business person interested in seamlessly collaborating and managing projects on-the-go. Trello is organized in boards, lists and cards that are customizable and expandable as the project and team grows. Trello easily integrates with other popular apps like Dropbox, Slack, Chrome, Teams and more. It is available for Android and Apple mobile devices on the App Store and Google Play App Store. 

• Timelines that allow all team members to stay on track and be held accountable to deadlines
• Table views that connect work across a variety of related Trello boards
• A handy Dashboard that highlights usage and engagement stats for all of your boards.

11. Researcher

The Researcher app was built to make it easier for researchers to find academic articles relevant to their work. By aggregating over 19,000 sources that include peer-reviewed academic journals, blogs, podcasts and recordings from live events, Researcher helps scientists stay up-to-date on emerging trends and information related to any given field of study or interest. The creators of Researcher claim that their app is “like social media, but better.” The Researcher app is free to use and is available for download on the Apple App Store, the Google Play App Store and the AppInChina App Store. 

• Filter options that allow you to sift through tens of thousands of sources in seconds
• Notification options to ensure that any time a new source is published that relates to your stated interests, you’ll find out about it right away.
• Bookmarks that make it easy for you to come back to an interesting piece when the time is right, without having to search.

Conclusion 

The apps listed above can help you be more efficient, collaborate better with your colleagues, and get more organized. We hope one or more of them considerably help you with your research. Let us know if we missed any! 

5 Best Event Registration Platforms for Your Next Conference

By having one software to organize registrations and submissions, a pediatric health center runs aro...

5 Essential Conference Apps for Your Event

In today’s digital age, the success of any conference hinges not just on the content and speakers bu...

Your personal research assistant

Zotero is a free, easy-to-use tool to help you collect, organize, annotate, cite, and share research.

Available for Mac, Windows, Linux, and iOS

Just need to create a quick bibliography? Try ZoteroBib .

Meet Zotero.

Collect with a click..

Zotero automatically senses research as you browse the web. Need an article from JSTOR or a preprint from arXiv.org? A news story from the New York Times or a book from a library? Zotero has you covered, everywhere.

Organize your way.

Zotero helps you organize your research any way you want. You can sort items into collections and tag them with keywords. Or create saved searches that automatically fill with relevant materials as you work.

Cite in style.

Zotero instantly creates references and bibliographies for any text editor, and directly inside Word, LibreOffice, and Google Docs. With support for over 10,000 citation styles, you can format your work to match any style guide or publication.

Stay in sync.

Zotero can optionally synchronize your data across devices, keeping your files, notes, and bibliographic records seamlessly up to date. If you decide to sync, you can also always access your research from any web browser.

Collaborate freely.

Zotero lets you co-write a paper with a colleague, distribute course materials to students, or build a collaborative bibliography. You can share a Zotero library with as many people you like, at no cost.

Zotero is open source and developed by an independent, nonprofit organization that has no financial interest in your private information. With Zotero, you always stay in control of your own data.

Still not sure which program to use for your research? See why we think you should choose Zotero .

Ready to try Zotero?

R Discovery: Academic Research 4+

Access 100m+ research papers, cactus communications private limited, designed for ipad.

• 4.8 • 645 Ratings

Screenshots

Description.

R Discovery is a free app for students and researchers to find and read research papers. This literature search and reading app for researchers curates an academic reading library based on your interests so you stay updated on latest academic research with access to scholarly articles, scientific journals, open access articles, and peer reviewed articles. With R Discovery, you can do a literature search like on Google Scholar, refseek, Research Gate, or Academia.edu, or let our AI generate separate feeds of relevant scholarly articles for you. We search, you read. It’s that simple! R Discovery gives you access to: • 250M+ Research articles (journal articles, clinical trials, conference papers & more) • 40M+ Open access articles (world’s largest OA journal articles library) • 3M+ Preprints from arXiv, bioRxiv, medRxiv & other preprint servers • 9.5M+ Research topics • 14M+ Authors • 32K+ Academic journals • 100K+ Universities & Institutions • Content from Microsoft Academic, PubMed, PubMed Central, CrossRef, Unpaywall, OpenAlex, etc. See how R Discovery’s personalized research reading feed and unique features save time and improve your literature reading! Largest repository of open access articles Access the largest library of open access journal articles and preprints on mobile, with 40M+ open access articles from top publishers and global research databases. Unlock full-text papers with institutional access Use your university credentials to log in and access paywalled journal articles for your thesis research with our GetFTR & Libkey integrations. Most reliable, cleanest research database Read science articles from the most trustworthy global research paper database, cleaned to remove duplication, eliminate ambiguities in journal, publisher, author names, and exclude predatory content. Curated research feeds Benefit from our AI-curated research feeds dedicated to the Top 100 papers, open access articles, preprints, paywalled papers, journal feeds, etc. Coming up: New feeds on patents, conferences & seminars. Reading lists from the research community Access and share research recommendations by a community of peers in your field; these lists allow for quick, easy, relevant research discovery and better literature reading. Collaborative reading lists Save, view, and share your reading lists with co-researchers on your study. Easy knowledge sharing via our premium collaborative reading list feature helps accelerate innovation; so invite your peers to join now. Audio streaming Amplify your reading with audio listening for library lists, research paper titles & abstracts. This Prime feature lets you create audio playlists and delve into research articles on the go. Research paper translation Read research articles in your own language with our academic translation Prime feature. Choose a paper to read and click on the translate option to read in your chosen language. Auto sync library with Zotero, Mendeley Our auto sync Prime feature integrates your research paper topics and research library with Mendeley, Zotero, updating it every time you save or remove papers. Coming up: Endnote integrations! Easy accessibility, summaries & notifications Read research that matters with alerts on Just Published research papers and assess relevance with research summaries. Bookmark articles on the research app and read on the web. R Discovery partners with research publications, including Elsevier, Wiley, IOP, Springer Nature, Sage, Taylor & Francis, Hindawi, NEJM, Emerald Publishing, Duke University Press, Intech Open, AIAA, Karger, Underline.io, SAGE, JStage for the best content. Enjoy free research discovery or upgrade to R Discovery Prime to unlock unlimited use of our premium features. Join 2.4M+ academics and redefine the way you read on R Discovery, the highest rated app in this space (rated 4.6+ on App Store). Get it now!

Version 3.2.3

In this release, you can search research papers without mandatory onboarding. Now find best research papers for you without any hassle. You can now pause your Prime membership and again restart it when you need it in your project. You can also now read the research papers under a minute with quick summaries available in story like format for quick readings. Improvements and bug fixes.

Ratings and Reviews

645 Ratings

Keeps improving

The app keeps getting better and provides a good UI. I like the fact that I get to pick concepts I am interested in and can focus on multiple areas at the same time.

Developer Response ,

Hi, thanks for sticking with the app and sharing about the improvements we keep making to it! We liked to hear that it fulfills your need to read about multiple areas of interest :) Do keep sharing feedback and suggestions for improvement by writing to us at [email protected].

I want to love this app…

The app is great and has a lot of features that are great for students and researchers. I love that I can select topics and journals that are relevant to me and that it provides me plenty of OpenAccess options. However, the app is super clunky and slow. It seems bloated on the back end and it makes the app almost unusable. I think I may just use it to get a reading list going and then read the articles on another app like Zotero or Reader.

Hi! Thanks for taking the time and sharing your feedback with us. We sincerely apologize for any inconvenience you faced with R Discovery. We request you to please write to us at [email protected] with your registered email ID, and we will try to fix this as soon as possible for you. Regards.

Great info and very helpful

Hi there, thanks so much for the review and for rating our app 5 stars. We're happy to know that you're finding R Discovery helpful. Please do let us know if you have any other feedback/suggestions/feature requests to share with us by writing to us at [email protected]. We'd love to hear from you!

App Privacy

The developer, Cactus Communications Private Limited , indicated that the app’s privacy practices may include handling of data as described below. For more information, see the developer’s privacy policy .

Data Used to Track You

The following data may be used to track you across apps and websites owned by other companies:

• Contact Info
• Search History
• Identifiers

Data Linked to You

The following data may be collected and linked to your identity:

Data Not Linked to You

The following data may be collected but it is not linked to your identity:

• Diagnostics

Privacy practices may vary, for example, based on the features you use or your age. Learn More

Information

• App Support
• Privacy Policy

More By This Developer

Cactus Quest Learning

You Might Also Like

Researcher: Discover & Discuss

Academia.edu

Web of Science MyRA

Persian Translator Offline

EITA Mobile

• MCLA Library Home
• Research Guides
• Apps for Research & Writing
• Recommended Apps

Apps for Research & Writing: Recommended Apps

• EBSCO eBooks
• Overdrive eBooks
• Internet Archive
• Research Articles
• Note-Taking
• Group Work/Collaboration
• Mind-Mapping
• Help This link opens in a new window

Using This Guide

This guide will cover apps that I find useful for research, ebook reading,  and organizing references and notes.

iPhone/iPad Apps

• Evernote Create text, photo and audio notes ● Auto-synchronize your notes to your Mac, PC, and Web ● Magically makes text within snapshots searchable ● All notes include geo-location information for mapping and search
• Dropbox Dropbox is a free service that lets you bring your photos, docs, and videos anywhere and share them easily.
• QR Reader Barcode and QR code scanning app.

Android Apps

• Xodo PDF Reader & Editor Xodo is an all-in-one PDF reader and PDF editor. With Xodo, you can read, annotate, sign, and share PDFs and fill in PDF forms, open .docx/.pptx as PDFs, plus sync with Google Drive, Dropbox and OneDrive.
• QR Droid Code Scanner Barcode and QR code scanning app
• Next: eBooks >>
• Last Updated: Jan 18, 2023 12:47 PM
• URL: https://library.mcla.edu/apps

Home → Productivity → Boost Your Productivity: The Ultimate Guide to Apps for Researchers

Boost Your Productivity: The Ultimate Guide to Apps for Researchers

Jordan Kruszynski

• January 4, 2024

Have you ever thought about how apps can help researchers?

We use apps every day, sometimes for fun and sometimes for business, but when it comes to research, they can be an impressively efficient tool that allows us to do our work more easily and productively. In academia, there are always deadlines to meet, papers to write, and experiments to run – in other words, plenty of opportunities for apps to help you simplify matters. That’s why we decided to put together a handy guide to apps for researchers, so you can sit back, relax, and ruminate on which one of these technical tools is going to help you streamline your work process.

Ready to start managing your workload, get organised, and boost your productivity?

Make a cup of tea, pull up a chair, and let’s get started.

The Benefits of Using Apps for Research

Before we dive into the top apps for researchers, let’s first discuss some of the benefits of using apps for research.

At the top of the heap has to be organisation . With so many different sources of information available to us, it can be difficult to keep track of everything. Websites, journal articles, webinars, and all those evasive little internet links that can pop up at any moment – the list is a long one. Note-taking and reference management apps can help here by allowing you to keep all of your information in one place.

Another big benefit is the ability to collaborate with others. Research is often a team effort, and apps that allow for collaboration with colleagues can help streamline the process.

Additionally, using apps can save you time . Instead of manually searching for sources or organising your notes, you can use apps to automate these tasks, freeing up time for more important work.

Top Productivity Apps for Researchers

Now that we’ve discussed the benefits of using apps for research, let’s take a closer look at our picks for organisation, collaboration and time-saving.

Note-taking Apps for Researchers

One of the most important aspects of research is taking notes. Note-taking apps can help you keep all of your notes in one place (and save paper), making it far easier to find and reference them later. Some of them even include several nifty features that set them apart from the competition:

• Evernote – Evernote is a popular note-taking app that allows you to create notes, to-do lists, and reminders. You can capture notes (including hand-drawn ones!) across an array of media. It also allows you to organise your notes into themed notebooks and tag them for easy searching.
• Google Keep – Google Keep is a note-taking app developed by Google. It allows you to create notes, to-do lists, and reminders. It also allows you to colour code your notes for easy organisation – a great one for visual learners.
• Audemic – Audemic lets you drop any academic paper into the app, breaking it down into sections and giving you a complete audio version as well as text. You can switch between sections on the fly, and if you hear something you need to make a note on, simply click the corresponding part of the text. Easy sharing options and a slick interface complete the package.

Reference Management Apps for Researchers

Referencing can be a beast, but it’s something that we researchers need to keep on top of almost every day. Reference management apps can really help you keep track of your sources and ensure that you’re citing them correctly. Some of the top reference management apps include:

• Zotero – Zotero is a reference management app that allows you to collect, organise, and cite your sources across almost all media types, in most cases with a single click. It also has a plugin for Microsoft Word that allows you to easily insert citations into your papers.
• Mendeley – Mendeley, just like Zotero, allows you to collect, organise, and cite your sources. If that makes them sound identical, then think again, as there’s a fairly vicious rivalry between users of both over which has the better features. Mendeley features a nice social networking component that allows you to connect with other researchers in your field.
• EndNote – Don’t forget EndNote! More than just a third wheel in the ongoing scuffle between Zotero and Mendeley, EndNote has a comprehensive set of features that have made it the go-to referencing app for some. It comes equipped ready to cover an enormous range of citation styles – a useful bonus.

Writing and Editing Apps for Researchers

Once you’ve collected your sources and taken your notes, it’s time to start writing. Writing and editing apps can help you streamline the process and sharpen your style, ensuring that your papers are well-written and error-free. Some of the top writing and editing apps for researchers include:

• Grammarly – Probably the most popular editor, Grammarly is a writing and editing app that checks your grammar and spelling. It also provides suggestions for improving your writing. This can be a big help for researchers whose second language is English, as careful observation of the app’s suggestions should improve your own skills over time.
• Hemingway Editor – Hemingway Editor is a writing and editing app that helps you simplify your writing. It highlights complex sentences and suggests simpler alternatives. Helpful for those who have a hard time getting out of ‘academic mode’ when you need to write something more general.
• Scrivener – Scrivener is a writing and editing app that allows you to organise your writing into sections and chapters. In a nice touch, it also has a distraction-free mode that can help you focus on your writing.

Collaboration Apps for Researchers

As I mentioned earlier, research is often a team effort. Collaboration apps can help you work with others more efficiently. Some of the top collaboration apps for researchers include:

• Slack – The relaxed, colourful and intuitive Slack is a messaging app that allows you to communicate with your team in real-time. It also allows you to share files and collaborate on projects.
• Trello – Trello is a project management app that allows you to organise your projects into boards, lists, and cards. It also allows you to assign tasks to team members and track their progress. Naturally, the Japanese-inspired kanban element of the app works well for those who favour efficiency.
• Asana – Asana is a project management app that allows you to divide your projects into tasks and subtasks. It offers you the chance to view the progress of your projects through multiple lenses and filters, so you can choose the perspective that suits you most.

Productivity Hacks for Using Research Apps

To close our guide to the top apps for researchers, let’s talk about some productivity hacks for using them.

First , make sure you’re using the right apps for your research needs. Not all apps are created equal, and it’s important to choose apps that align with your research goals.

Second , take advantage of app integrations . Many apps have integrations with other tools and software, such as Microsoft Word or Google Drive. Audemic , for example, has full integration with Zotero to make things double quick. These integrations can help streamline your workflow and save you even more time.

Finally , don’t forget to take breaks . It’s important to give your brain (and eyes) a rest and recharge. Use apps like Pomofocus to remind you to take breaks and stay focused.

Final Thoughts

So there you have it – there are a host of apps on the market that can help you to sharpen up your day-to-day research tasks, many of which are free to use (or at least to trial). They all have the capacity to make your work quicker and easier, each one bringing a different set of features to the table.

What are you waiting for? Be inspired by our guide and start investigating all these impressive app options – it won’t take long before you find the ones that work for you. Choose the right apps for your research needs and you’ll be powering towards your goals in no time.

Keep striving, researchers! ✨

Table of Contents

Related articles.

How to Publish a Research Paper: A Step-by-Step Guide

You’re in academia. You’re going steady. Your research is going well and you begin to wonder: ‘How exactly do I get a

Behind the Scenes: What Does a Research Assistant Do?

Have you ever wondered what goes on behind the scenes in a research lab? Does it involve acting out the whims of

How to Write a Research Paper Introduction: Hook, Line, and Sinker

Want to know how to write a research paper introduction that dazzles? Struggling to hook your reader in with your opening sentences?

Blog Podcast

Privacy policy Terms of service

Subscribe to our newsletter!

Discover more from Audemic: Access any academic research via audio

Subscribe now to keep reading and get access to the full archive.

Type your email…

Continue reading

About this app

Data safety.

Ratings and reviews

• Flag inappropriate

What's new

App support, similar apps.

• Introduction
• Conclusions
• Article Information

Screenshots of the smartphone cognitive tasks developed by Datacubed Health and included in the ALLFTD Mobile App. Details about the task design and instructions are included in the eMethods in Supplement 1. A, Flanker (Ducks in a Pond) is a task of cognitive control requiring participants to select the direction of the center duck. B, Go/no-go (Go Sushi Go!) requires participants to quickly tap on pieces of sushi (go) but not to tap when they see a fish skeleton (no-go). C, Card sort (Card Shuffle) is a task of cognitive flexibility requiring participants to learn rules that change during the task. D, The adaptative, associative memory task (Humi’s Bistro) requires participants to learn the food orders of several restaurant tables. E, Stroop (Color Clash) is a cognitive inhibition paradigm requiring participants to inhibit their tendency to read words and instead respond based on the color of the word. F, The 2-back task (Animal Parade) requires participants to determine whether animals on a parade float match the animals they saw 2 stimuli previously. G, Participants are asked to complete 3 testing sessions over 2 weeks. Shown in dark blue, they have 3 days to complete each testing session with a washout day between sessions on which no tests are available. Session 2 always begins on day 5 and session 3 on day 9. Screenshots are provided with permission from Datacubed Health.

Forest plots present internal consistency and test-retest reliability results in the discovery and validation cohorts, as well as an estimate in a combined sample of discovery and validation participants. ICC indicates interclass correlation coefficient.

A and B, Correlation matrices display associations of in-clinic criterion standard measures and ALLFTD mobile App (mApp) test scores in discovery and validation cohorts. Below the horizontal dashed lines, the associations among app tests and between app tests and demographic characteristics convergent clinical measures, divergent cognitive tests, and neuroimaging regions of interest can be viewed. Most app tests show strong correlations with each other and with age, convergent clinical measures, and brain volume. The measures show weaker correlations with divergent measures of visuospatial (Benson Figure Copy) and language (Multilingual Naming Test [MINT]) abilities. The strength of convergent correlations between app measures and outcomes is similar to the correlations between criterion standard neuropsychological scores and these outcomes, which can be viewed by looking across the rows above the horizontal black line. C and D, In the discovery and validation cohorts, receiver operating characteristics curves were calculated to determine how well a composite of app tests, the Uniform Data Set, version 3.0, Executive Functioning Composite (UDS3-EF), and the Montreal Cognitive Assessment (MoCA) discriminate individuals without symptoms (Clinical Dementia Rating Scale plus National Alzheimer’s Coordinating Center FTLD module sum of boxes [CDR plus NACC-FTLD-SB] score = 0) from individuals with the mildest symptoms of FTLD (CDR plus NACC-FTLD-SB score = 0.5). AUC indicates area under the curve; CVLT, California Verbal Learning Test.

eMethods. Instruments and Statistical Analysis

eResults. Participants

eTable 1. Participant Characteristics and Test Scores in Original and Validation Cohorts

eTable 2. Comparison of Diagnostic Accuracy for ALLFTD Mobile App Composite Score Across Cohorts

eTable 3. Number of Distractions Reported During the Remote Smartphone Testing Sessions

eTable 4. Qualitative Description of the Distractions Reported During Remote Testing Sessions

eFigure 1. Scatterplots of Test-Retest Reliability in a Mixed Sample of Adults Without Functional Impairment and Participants With FTLD

eFigure 2. Comparison of Test-Retest Reliability Estimates by Endorsement of Distractions

eFigure 3. Comparison of Test-Retest Reliability Estimates by Operating System

eFigure 4. Correlation Matrix in the Combined Cohort

eFigure 5. Neural Correlates of Smartphone Cognitive Test Performance

eReferences

Nonauthor Collaborators

Data Sharing Statement

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Staffaroni AM , Clark AL , Taylor JC, et al. Reliability and Validity of Smartphone Cognitive Testing for Frontotemporal Lobar Degeneration. JAMA Netw Open. 2024;7(4):e244266. doi:10.1001/jamanetworkopen.2024.4266

Manage citations:

© 2024

• Permissions

Reliability and Validity of Smartphone Cognitive Testing for Frontotemporal Lobar Degeneration

• 1 Department of Neurology, Memory and Aging Center, Weill Institute for Neurosciences, University of California, San Francisco
• 2 Department of Neurology, Columbia University, New York, New York
• 3 Department of Neurology, Mayo Clinic, Rochester, Minnesota
• 4 Department of Quantitative Health Sciences, Division of Biomedical Statistics and Informatics, Mayo Clinic, Rochester, Minnesota
• 5 Department of Neurology, Case Western Reserve University, Cleveland, Ohio
• 6 Department of Neurosciences, University of California, San Diego, La Jolla
• 7 Department of Radiology, University of North Carolina, Chapel Hill
• 8 Department of Neurology, Indiana University, Indianapolis
• 9 Department of Neurology, Vanderbilt University, Nashville, Tennessee
• 10 Department of Neurology, University of Washington, Seattle
• 11 Department of Psychiatry and Psychology, Mayo Clinic, Rochester, Minnesota
• 12 Department of Neurology, Institute for Precision Health, University of California, Los Angeles
• 13 Department of Neurology, Knight Alzheimer Disease Research Center, Washington University, Saint Louis, Missouri
• 14 Department of Psychiatry, Knight Alzheimer Disease Research Center, Washington University, Saint Louis, Missouri
• 15 Department of Neuroscience, Mayo Clinic, Jacksonville, Florida
• 16 Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia
• 17 Division of Neurology, University of British Columbia, Musqueam, Squamish & Tsleil-Waututh Traditional Territory, Vancouver, Canada
• 18 Department of Neurosciences, University of California, San Diego, La Jolla
• 19 Department of Neurology, Nantz National Alzheimer Center, Houston Methodist and Weill Cornell Medicine, Houston Methodist, Houston, Texas
• 20 Department of Neurology, UCLA (University of California, Los Angeles)
• 21 Department of Neurology, University of Colorado, Aurora
• 22 Department of Neurology, David Geffen School of Medicine, UCLA
• 23 Department of Neurology, University of Alabama, Birmingham
• 24 Tanz Centre for Research in Neurodegenerative Diseases, Division of Neurology, University of Toronto, Toronto, Ontario, Canada
• 25 Department of Neurology, Massachusetts General Hospital and Harvard Medical School, Boston
• 26 Department of Epidemiology and Biostatistics, University of California, San Francisco
• 27 Department of Psychological & Brain Sciences, Washington University, Saint Louis, Missouri

Question   Can remote cognitive testing via smartphones yield reliable and valid data for frontotemporal lobar degeneration (FTLD)?

Findings   In this cohort study of 360 patients, remotely deployed smartphone cognitive tests showed moderate to excellent reliability comparedwith criterion standard measures (in-person disease severity assessments and neuropsychological tests) and brain volumes. Smartphone tests accurately detected dementia and were more sensitive to the earliest stages of familial FTLD than standard neuropsychological tests.

Meaning   These findings suggest that remotely deployed smartphone-based assessments may be reliable and valid tools for evaluating FTLD and may enhance early detection, supporting the inclusion of digital assessments in clinical trials for neurodegeneration.

Importance   Frontotemporal lobar degeneration (FTLD) is relatively rare, behavioral and motor symptoms increase travel burden, and standard neuropsychological tests are not sensitive to early-stage disease. Remote smartphone-based cognitive assessments could mitigate these barriers to trial recruitment and success, but no such tools are validated for FTLD.

Objective   To evaluate the reliability and validity of smartphone-based cognitive measures for remote FTLD evaluations.

Design, Setting, and Participants   In this cohort study conducted from January 10, 2019, to July 31, 2023, controls and participants with FTLD performed smartphone application (app)–based executive functioning tasks and an associative memory task 3 times over 2 weeks. Observational research participants were enrolled through 18 centers of a North American FTLD research consortium (ALLFTD) and were asked to complete the tests remotely using their own smartphones. Of 1163 eligible individuals (enrolled in parent studies), 360 were enrolled in the present study; 364 refused and 439 were excluded. Participants were divided into discovery (n = 258) and validation (n = 102) cohorts. Among 329 participants with data available on disease stage, 195 were asymptomatic or had preclinical FTLD (59.3%), 66 had prodromal FTLD (20.1%), and 68 had symptomatic FTLD (20.7%) with a range of clinical syndromes.

Exposure   Participants completed standard in-clinic measures and remotely administered ALLFTD mobile app (app) smartphone tests.

Main Outcomes and Measures   Internal consistency, test-retest reliability, association of smartphone tests with criterion standard clinical measures, and diagnostic accuracy.

Results   In the 360 participants (mean [SD] age, 54.0 [15.4] years; 209 [58.1%] women), smartphone tests showed moderate-to-excellent reliability (intraclass correlation coefficients, 0.77-0.95). Validity was supported by association of smartphones tests with disease severity ( r range, 0.38-0.59), criterion-standard neuropsychological tests ( r range, 0.40-0.66), and brain volume (standardized β range, 0.34-0.50). Smartphone tests accurately differentiated individuals with dementia from controls (area under the curve [AUC], 0.93 [95% CI, 0.90-0.96]) and were more sensitive to early symptoms (AUC, 0.82 [95% CI, 0.76-0.88]) than the Montreal Cognitive Assessment (AUC, 0.68 [95% CI, 0.59-0.78]) ( z of comparison, −2.49 [95% CI, −0.19 to −0.02]; P  = .01). Reliability and validity findings were highly similar in the discovery and validation cohorts. Preclinical participants who carried pathogenic variants performed significantly worse than noncarrier family controls on 3 app tasks (eg, 2-back β = −0.49 [95% CI, −0.72 to −0.25]; P  < .001) but not a composite of traditional neuropsychological measures (β = −0.14 [95% CI, −0.42 to 0.14]; P  = .32).

Conclusions and Relevance   The findings of this cohort study suggest that smartphones could offer a feasible, reliable, valid, and scalable solution for remote evaluations of FTLD and may improve early detection. Smartphone assessments should be considered as a complementary approach to traditional in-person trial designs. Future research should validate these results in diverse populations and evaluate the utility of these tests for longitudinal monitoring.

Frontotemporal lobar degeneration (FTLD) is a neurodegenerative pathology causing early-onset dementia syndromes with impaired behavior, cognition, language, and/or motor functioning. 1 Although over 30 FTLD trials are planned or in progress, there are several barriers to conducting FTLD trials. Clinical trials for neurodegenerative disease are expensive, 2 and frequent in-person trial visits are burdensome for patients, caregivers, and clinicians, 3 a concern magnified in FTLD by behavioral and motor impairments. Given the rarity and geographical dispersion of eligible participants, FTLD trials require global recruitment, 4 particularly for those that are far from expert FTLD clinical trial centers. Furthermore, criterion standard neuropsychological tests are not adequately sensitive until symptoms are already noticeable to families, limiting their usefulness as outcomes in early-stage FTLD treatment trials. 4

Reliable, valid, and scalable remote data collection methods may help surmount these barriers to FTLD clinical trials. Smartphones are garnering interest across neurological conditions as a method for administering remote cognitive and motor evaluations. Preliminary evidence supports the feasibility, reliability, and/or validity of unsupervised smartphone cognitive and motor testing in older adults at risk for Alzheimer disease, 5 - 8 Parkinson disease, 9 and Huntington disease. 10 The clinical heterogeneity of FTLD necessitates a uniquely comprehensive smartphone battery. In the ALLFTD Consortium (Advancing Research and Treatment in Frontotemporal Lobar Degeneration [ARTFLD] and Longitudinal Evaluation of Familial Frontotemporal Dementia Subjects [LEFFTDS]), the ALLFTD mobile Application (ALLFTD-mApp) was designed to remotely monitor cognitive, behavioral, language, and motor functioning in FTLD research. Taylor et al 11 recently reported that unsupervised ALLFTD-mApp data collection through a multicenter North American FTLD research network was feasible and acceptable to participants. Herein, we extend that work by investigating the reliability and validity of unsupervised remote smartphone tests of executive functioning and memory in a cohort with FTLD that has undergone extensive phenotyping.

Participants were enrolled from ongoing FTLD studies requiring in-person assessment, including participants from 18 centers from the ALLFTD study study 12 and University of California, San Francisco (UCSF) FTLD studies. To study the app in older individuals, a small group of older adults without functional impairment was recruited from the UCSF Brain Aging Network for Cognitive Health. All study procedures were approved by the UCSF or Johns Hopkins Central Institutional Review Board. All participants or legally authorized representatives provided written informed consent. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology ( STROBE ) reporting guideline.

Inclusion criteria were age 18 years or older, having access to a smartphone, and reporting English as the primary language. Race and ethnicity were self reported by participants using options consistent with the National Alzheimer’s Coordinating Center (NACC) Uniform Data Set (UDS) and were collected to contextualize the generalizability of these results. Participants were asked to complete tests on their own smartphones. Informants were encouraged for all participants and required for those with symptomatic FTLD (Clinical Dementia Rating Scale plus NACC FTLD module [CDR plus NACC-FTLD] global score ≥1). Recruitment targeted individuals with CDR plus NACC-FTLD global scores less than 2, but sites had discretion to enroll more severely impaired participants. Exclusion criteria were consistent with the parent ALLFTD study. 12

Participants were enrolled in the ALLFTD-mApp study within 90 days of annual ALLFTD study visits (including neuropsychological and neuroimaging data collection). Site research coordinators (including J.C.T., A.B.W., S.D., and M.M.) assisted participants with app download, setup, and orientation and observed participants completing the first questionnaire. All cognitive tasks were self-administered without supervision (except pilot participants, discussed below) in a predefined order with minor adjustments throughout the study. Study partners of participants with symptomatic FTLD were asked to remain nearby during participation to help navigate the ALLFTD-mApp but were asked not to assist with testing.

The baseline participation window was divided into three 25- to 35-minute assessment sessions occurring over 11 days. All cognitive tests were repeated in every session to enhance task reliability 6 , 13 and enable assessment of test-retest reliability, except for card sort, which was administered once every 6 months due to expected practice effects. Adherence was defined as the percentage of all available tasks that were completed. Participants were asked to complete the triplicate of sessions every 6 months for the duration of the app study. Only the baseline triplicate was analyzed in this study.

Replicability was tested by dividing the sample into a discovery cohort (n = 258) comprising all participants enrolled until the initial data freeze (October 1, 2022) and a validation cohort (n = 102) comprising participants enrolled after October 1, 2022, and 18 pilot participants 11 who completed the first session in person with an examiner present during cognitive pretesting. Sensitivity analyses excluded this small pilot cohort.

ALLFTD investigators partnered with Datacubed Health 14 to develop the ALLFTD-mApp on Datacubed Health’s Linkt platform. The app includes cognitive, motor, and speech tasks. This study focuses on 6 cognitive tests developed by Datacubed Health 11 comprising an adaptive associative memory task (Humi’s Bistro) and gamified versions of classic executive functioning paradigms: flanker (Ducks in a Pond), Stroop (Color Clash), 2-back (Animal Parade), go/no-go (Go Sushi Go!), and card sort (Card Shuffle) ( Figure 1 and eMethods in Supplement 1 ). Most participants with symptomatic FTLD (49 [72.1%]) were not administered Stroop or 2-back, as pilot studies identified these as too difficult. 11 The app test results were summarized as a composite score (eMethods in Supplement 1 ). Participants completed surveys to assess technological familiarity (daily or less than daily use of a smartphone) and distractions (present or absent).

Criterion standard clinical data were collected during parent project visits. Syndromic diagnoses were made according to published criteria 15 - 19 based on multidisciplinary conferences that considered neurological history, neurological examination results, and collateral interview. 20

The CDR plus NACC-FTLD module is an 8-domain rating scale based on informant and participant report. 21 A global score was calculated to categorize disease severity as asymptomatic or preclinical if a pathogenic variant carrier (0), prodromal (0.5), or symptomatic (1.0-3.0). 22 A sum of the 8 domain box scores (CDR plus NACC-FTLD sum of boxes) was also calculated. 22

Participants completed the UDS Neuropsychological Battery, version 3.0 23 (eMethods in Supplement 1 ), which includes traditional neuropsychological measures and the Montreal Cognitive Assessment (MoCA), a global cognitive screen. Executive functioning and processing speed measures were summarized into a composite score (UDS3-EF). 24 Participants also completed a 9-item list-learning memory test (California Verbal Learning Test, 2nd edition, Short Form). 25 Most (339 [94.2%]) neuropsychological evaluations were conducted in person. In a subsample (n = 270), motor speed and dexterity were assessed using the Movement Disorder Society Uniform Parkinson Disease Rating Scale 26 Finger Tapping subscale (0 indicates no deficits [n = 240]).

We acquired T1-weighted brain magnetic resonance imaging for 199 participants. Details of image acquisition, harmonization, preprocessing, and processing are provided in eMethods in Supplement 1 and prior publications. 27 Briefly, SPM12 (Statistical Parametric Mapping) was used for segmentation 28 and Large Deformation Diffeomorphic Metric Mapping for generating group templates. 29 Gray matter volumes were calculated in template space by integrating voxels and dividing by total intracranial volume in 2 regions of interest (ROIs) 30 : a frontoparietal and subcortical ROI and a hippocampal ROI. Voxel-based morphometry was used to test unbiased voxel-wise associations of volume with smartphone tests (eMethods in Supplement 1 ). 31 , 32

Participants in the ALLFTD study underwent genetic testing 33 at the University of California, Los Angeles. DNA samples were screened using targeted sequencing of a custom panel of genes previously implicated in neurodegenerative diseases, including GRN ( 138945 ) and MAPT ( 157140 ). Hexanucleotide repeat expansions in C9orf72 ( 614260 ) were detected using both fluorescent and repeat-primed polymerase chain reaction analysis. 34

Statistical analyses were conducted using Stata, version 17.0 (StataCorp LLC), and R, version 4.4.2 (R Project for Statistical Computing). All tests were 2 sided, with a statistical significance threshold of P < .05.

Psychometric properties of the smartphone tests were explored using descriptive statistics. Comparisons between CDR plus NACC-FTLD groups (ie, asymptomatic or preclinical, prodromal, and symptomatic) for continuous variables, including demographic characteristics and cognitive task scores (first exposure to each measure), were analyzed by fitting linear regressions. We used χ 2 difference tests for frequency data (eg, sex and race and ethnicity).

Internal consistency, which measures reliability within a task, was estimated for participants’ first exposure to each test using Cronbach α (details in eMethods in Supplement 1 ). Test-retest reliability was estimated using intraclass correlation coefficients for participants who completed a task at least twice; all exposures were included. Reliability estimates are described as poor (<0.500), moderate (0.500-0.749), good (0.750-0.890), and excellent (≥0.900) 35 ; these are reporting rules of thumb, and clinical interpretation should consider raw estimates. We calculated 95% CIs via bootstrapping with 1000 samples.

Validity analyses used participants’ first exposure to each test. Linear regressions were fitted in participants without symptoms with age, sex, and educational level as independent variables to understand the unique contribution of each demographic factor to cognitive test scores. Correlations and linear regression between the app-based tasks and disease severity (CDR plus NACC-FTLD sum of boxes score), neuropsychological test scores, and gray matter ROIs were used to investigate construct validity in the full sample. Demographic characteristics were not entered as covariates because the primary goal was to assess associations between app-based measures and criterion standards, rather than understand the incremental predictive value of app measures. To address potential motor confounds, associations with disease severity were evaluated in a subsample without finger dexterity deficits on motor examination (using the Movement Disorder Society Uniform Parkinson Disease Rating Scale Finger Tapping subscale). To complement ROI-based neuroimaging analysis based on a priori hypotheses, we conducted voxel-based morphometry (eMethods in Supplement 1 ) to uncover other potential neural correlates of test performance. 31 , 32 Finally, we evaluated the association of the number of distractions and operating system with reliability and validity, controlling for age and disease severity, which are predictive factors associated with test performance in correlation analyses.

To evaluate the app’s ability to select participants with prodromal or symptomatic FTLD for trial enrollment, we tested discrimination of participants without symptoms from those with prodromal and symptomatic FTLD. To understand the app’s utility for screening early cognitive impairment, we fit receiver operating characteristics curves testing the predictive value of the app composite, UDS3-EF, and MoCA for differentiating participants without symptoms and those with preclinical FTLD from those with prodromal FTLD; areas under the curves (AUC) for the app and MoCA were compared using the DeLong test in participants with results for both predictive factors.

We compared app performance in preclinical participants who carried pathogenic variants with that in noncarrier controls using linear regression adjusted for age (a predictive factor in earlier models). For this analysis, we excluded those younger than 45 years to remove participants likely to be years from symptom onset based on natural history studies. 4 We analyzed memory performance in participants who carried MAPT pathogenic variants, as early executive deficits may be less prominent. 34 , 36

Of 1163 eligible participants, 360 were enrolled, 439 were excluded, and 364 refused to participate (additional details are provided in the eResults in Supplement 1 ). Participant characteristics are reported in Table 1 for the full sample. The discovery and validation cohorts did not significantly differ in terms of demographic characteristics, disease severity, or cognition (eTable 1 in Supplement 1 ). In the full sample, there were 209 women (58.1%) and 151 men (41.9%), and the mean (SD) age was 54.0 (15.4) years (range, 18-89 years). The mean (SD) educational level was 16.5 (2.3) years (range, 12-20 years). Among the 358 participants with racial and ethnic data available, 340 (95.0%) identified as White. For the 18 participants self-identifying as being of other race or ethnicity, the specific group was not provided to protect participant anonymity. Among the 329 participants with available CDR plus NACC-FTLD scores ( Table 1 ), 195 (59.3%) were asymptomatic or preclinical (Global Score, 0), 66 (20.1%) were prodromal (Global score, 0.5), and 68 (20.7%) were symptomatic (global score, 1.0 or 2.0). Of those with available genetic testing results (n = 222), 100 (45.0%) carried a pathogenic familial FTLD pathogenic variant, including 63 of 120 participants without symptoms and with available results. On average, participants completed 78% of available smartphone measures over a mean (SD) of 2.6 (0.6) sessions.

Descriptive statistics for each task are presented in Table 2 . Ceiling effects were not observed for any tests. A small percentage of participants were at the floor for flanker (19 [5.3%]), go/no-go (13 [4.0%]), and card sort (9 [3.3%]) scores. Floor effects were only observed in participants with prodromal or symptomatic FTLD.

Except for go/no-go, internal consistency estimates ranged from good to excellent (Cronbach α range, 0.84 [95% CI, 0.81-0.87] to 0.99 [95% CI, 0.99-0.99]), and test-retest reliabilities were moderate to excellent (interclass correlation coefficient [ICC] range, 0.77 [95% CI, 0.69-0.83] to 0.95 [95% CI, 0.93-0.96]), with slightly higher estimates in participants with prodromal or symptomatic FTLD ( Table 2 , Figure 2 , and eFigure 1 in Supplement 1 ). Go/no-go reliability was particularly poor in participants without symptoms (ICC, 0.10 [95% CI, −0.37 to 0.48]) and was removed from subsequent validation analyses except the correlation matrix ( Figure 3 A and B). The 95% CIs for reliability estimates overlapped in the discovery and validation cohorts ( Figure 2 ). Reliability estimates showed overlapping 95% CIs regardless of distractions (eFigure 2 in Supplement 1 ) or operating systems (eFigure 3 in Supplement 1 ), with a pattern of slightly lower reliability estimates when distractions were endorsed for all comparisons except Stroop (Cronbach α).

In 57 participants without symptoms who did not carry pathogenic variants, older age was associated with worse performance on all measures (β range,  − 0.40 [95 CI, −0.68 to −0.13] to −0.78 [95 CI, −0.89 to −0.52]; P ≤ .03), except card sort (β = −0.22 [95% CI, −0.54 to 0.09]; P  = .16) and go-no/go (β = −0.15 [95% CI, −0.44 to 0.14]; P  = .31), though associations were in the expected direction. Associations with sex and educational level were not statistically significant.

Cognitive tests administered using the app showed evidence of convergent and divergent validity (eFigure 4 in Supplement 1 ), with very similar findings in discovery ( Figure 3 A) and validation cohorts ( Figure 3 B). App–based measures of executive functioning were generally correlated with criterion standard in-person measures of these domains and less with measures of other cognitive domains ( r range, 0.40-0.66). For example, the flanker task was associated with the UDS3-EF composite (β = 0.58 [95% CI, 0.48-0.68]; P  < .001) and measures of visuoconstruction (β for Benson Figure Copy, 0.43 [95% CI, 0.32-0.54]; P  = .01) and naming (β for Multilingual Naming Test, 0.25 [95% CI, 0.14-0.37]; P  < .001). The app memory test was associated with criterion standard memory and executive functioning tests.

Worse performance on all app measures was associated with greater disease severity on CDR plus NACC-FTLD ( r range, 0.38-0.59) ( Table 1 , Figure 3 , and eFigure 4 in Supplement 1 ). The same pattern of results was observed after excluding those with finger dexterity issues. Except for go/no-go, performance of participants with prodromal FTLD was statistically significantly worse than that of participants without symptoms on all measures ( P  < .001).

The AUC for the app composite to distinguish participants without symptoms from those with dementia was 0.93 (95% CI, 0.90-0.96). The app also accurately differentiated participants without symptoms from those with prodromal or symptomatic FTLD (AUC, 0.87 [95% CI, 0.84-0.92]). Compared with the MoCA (AUC, 0.68 [95% CI, 0.59-0.78), app composite performance (AUC, 0.82 [95% CI, 0.76-0.88]) more accurately differentiated participants without symptoms and with prodromal FTLD ( z of comparison, −2.49 [95% CI, −0.19 to −0.02]; P  = .01), with similar accuracy to the UDS3-EF (AUC, 0.81 [95% CI, 0.73-0.88]); highly similar results (eTable 2 in Supplement 1 ) were observed in the discovery ( Figure 3 C) and validation ( Figure 3 D) cohorts.

In 56 participants without symptoms who were older than 45 years, those carrying GRN , C9orf72 , or another rare pathogenic variants performed significantly worse on 3 of 4 executive tests compared with noncarrier controls, including flanker (β = −0.26 [95% CI, −0.46 to −0.05]; P  = .02), card sort (β = −0.28 [95% CI, −0.54 to −0.30]; P  = .03), and 2-back (β = −0.49 [95% CI, −0.72 to −0.25]; P  < .001). The estimated scores of participants who carried pathogenic variants were on average lower than those of carriers on a composite of criterion standard in-person tests, but the difference was not statistically significant (UDS3-EF β = −0.14 [95% CI, −0.42 to 0.14]; P  = .32). Participants who carried preclinical MAPT pathogenic variants scored higher than noncarriers on the app Memory test, though the difference was not statistically significant (β = 0.21 [95% CI, −0.50 to 0.58]; P  = .19).

In prespecified ROI analyses, worse app executive functioning scores were associated with lower frontoparietal and/or subcortical volume ( Figures 3 A and B) (β range, 0.34 [95% CI, 0.22-0.46] to 0.50 [95 CI, 0.40-0.60]; P < .001 for all) and worse memory scores with smaller hippocampal volume (β = 0.45 [95% CI, 0.34-0.56]; P  < .001). Voxel-based morphometry (eFigure 5 in Supplement 1 ) suggested worse app performance was associated with widespread atrophy, particularly in frontotemporal cortices.

Only for card sort were distractions (eTables 3 and 4 in Supplement 1 ) associated with task performance; those experiencing distractions unexpectedly performed better (β = 0.16 [95% CI, 0.05-0.28]; P  = .005). The iPhone operating system was associated with better performance on 2 speeded tasks: flanker (β = 0.16 [95% CI, 0.07-0.24]; P  < .001) and go/no-go (β = 0.16 [95% CI, 0.06-0.26]; P  = .002). In a sensitivity analysis, associations of all app tests with disease severity, UDS3-EF, and regional brain volumes remained after covarying for distractions and operating system, as did the models differentiating participants who carried preclinical pathogenic variants and noncarrier controls.

There is an urgent need to identify reliable and valid digital tools for remote neurobehavioral measurement in neurodegenerative diseases, including FTLD. Prior studies provided preliminary evidence that smartphones collect reliable and valid cognitive data in a variety of age-related and neurodegenerative illnesses. This is the first study, to our knowledge, to provide analogous support for the reliability and validity of remote cognitive testing via smartphones in FTLD and preliminary evidence that this approach improves early detection relative to traditional in-person measures.

Reliability, a prerequisite for a valid clinical trial end point, indicates measurements are consistent. In 2 cohorts, we found smartphone cognitive tests were reliable within a single administration (ie, internally consistent) and across repeated assessments (ie, test-retest reliability) with no apparent differences by operating system. For all measures except go/no-go, reliability estimates were moderate to excellent and on par with other remote digital assessments 5 , 6 , 10 , 37 , 38 and in-clinic criterion standards. 39 - 41 Go/no-go showed similar within- and between-person variability in participants without symptoms (ie, poor reliability), and participant feedback suggested instructions were confusing and the stimuli disappeared too quickly. Those endorsing distractions tended to have lower reliability, though 95% CIs largely overlapped; future research detailing the effect of the home environment on test performance is warranted.

Construct validity was supported by strong associations of smartphone tests with demographics, disease severity, neuroimaging, and criterion standard neuropsychological measures that replicated in a validation sample. These associations were similar to those observed among the criterion standard measures and similar to associations reported in other validation studies of smartphone cognitive tests. 5 , 6 , 10 Associations with disease severity were not explained by motor impairments. The iPhone operating system was associated with better performance on 2 time-based measures, consistent with prior findings. 6

A composite of brief smartphone tests was accurate in distinguishing dementia from cognitively unimpaired participants, screening out participants without symptoms, and detecting prodromal FTLD with greater sensitivity than the MoCA. Moreover, carriers of preclinical C9orf72 and GRN pathogenic variants performed significantly worse than noncarrier controls on 3 tests, whereas they did not significantly differ on criterion standard measures. These findings are consistent with previous studies showing digital executive functioning paradigms may be more sensitive to early FTLD than traditional measures. 42 , 43

This study has some limitations. Validation analyses focused on participants’ initial task exposure. Future studies will explore whether repeated measurements and more sophisticated approaches to composite building (current composite assumes equal weighting of tests) improve reliability and sensitivity, and a normative sample is being collected to better adjust for demographic effects on testing. 24 Longitudinal analyses will explore whether the floor effects in participants with symptomatic FTLD will affect the utility for monitoring. The generalizability of the findings is limited by the study cohort, which comprised participants who were college educated on average, mostly White, and primarily English speakers who owned smartphones and participated in the referring in-person research study. Equity in access to research is a priority in FTLD research 44 , 45 ; translations of the ALLFTD-mApp are in progress, cultural adaptations are being considered, and devices have been purchased for provisioning to improve the diversity of our sample.

The findings of this cohort study, coupled with prior reports indicating that smartphone testing is feasible and acceptable to patients with FTLD, 11 suggest that smartphones may complement traditional in-person research paradigms. More broadly, the scalability, ease of use, reliability, and validity of the ALLFTD-mApp suggest the feasibility and utility of remote digital assessments in dementia clinical trials. Future research should validate these results in diverse populations and evaluate the utility of these tests for longitudinal monitoring.

Accepted for Publication: February 2, 2024.

Published: April 1, 2024. doi:10.1001/jamanetworkopen.2024.4266

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2024 Staffaroni AM et al. JAMA Network Open .

Corresponding Author: Adam M. Staffaroni, PhD, Weill Institute for Neurosciences, Department of Neurology, Memory and Aging Center, University of California, San Francisco, 675 Nelson Rising Ln, Ste 190, San Francisco, CA 94158 ( [email protected] ).

Author Contributions: Dr Staffaroni had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Staffaroni, A. Clark, Taylor, Heuer, Wise, Forsberg, Miller, Hassenstab, Rosen, Boxer.

Acquisition, analysis, or interpretation of data: Staffaroni, A. Clark, Taylor, Heuer, Sanderson-Cimino, Wise, Dhanam, Cobigo, Wolf, Manoochehri, Mester, Rankin, Appleby, Bayram, Bozoki, D. Clark, Darby, Domoto-Reilly, Fields, Galasko, Geschwind, Ghoshal, Graff-Radford, Hsiung, Huey, Jones, Lapid, Litvan, Masdeu, Massimo, Mendez, Miyagawa, Pascual, Pressman, Ramanan, Ramos, Rascovsky, Roberson, Tartaglia, Wong, Kornak, Kremers, Kramer, Boeve, Boxer.

Drafting of the manuscript: Staffaroni, A. Clark, Taylor, Heuer, Wolf, Lapid.

Critical review of the manuscript for important intellectual content: Staffaroni, Taylor, Heuer, Sanderson-Cimino, Wise, Dhanam, Cobigo, Manoochehri, Forsberg, Mester, Rankin, Appleby, Bayram, Bozoki, D. Clark, Darby, Domoto-Reilly, Fields, Galasko, Geschwind, Ghoshal, Graff-Radford, Hsiung, Huey, Jones, Lapid, Litvan, Masdeu, Massimo, Mendez, Miyagawa, Pascual, Pressman, Ramanan, Ramos, Rascovsky, Roberson, Tartaglia, Wong, Miller, Kornak, Kremers, Hassenstab, Kramer, Boeve, Rosen, Boxer.

Statistical analysis: Staffaroni, A. Clark, Taylor, Heuer, Sanderson-Cimino, Cobigo, Kornak, Kremers.

Obtained funding: Staffaroni, Rosen, Boxer.

Administrative, technical, or material support: A. Clark, Taylor, Heuer, Wise, Dhanam, Wolf, Manoochehri, Forsberg, Darby, Domoto-Reilly, Ghoshal, Hsiung, Huey, Jones, Litvan, Massimo, Mendez, Miyagawa, Pascual, Pressman, Ramanan, Kramer, Boeve, Boxer.

Supervision: Geschwind, Miyagawa, Roberson, Kramer, Boxer.

Conflict of Interest Disclosures: Dr Staffaroni reported being a coinventor of 4 ALLFTD mobile application tasks (not analyzed in the present study) and receiving licensing fees from Datacubed Health; receiving research support from the National Institute on Aging (NIA) of the National Institutes of Health (NIH), Bluefield Project to Cure FTD, the Alzheimer’s Association, the Larry L. Hillblom Foundation, and the Rainwater Charitable Foundation; and consulting for Alector Inc, Eli Lilly and Company/Prevail Therapeutics, Passage Bio Inc, and Takeda Pharmaceutical Company. Dr Forsberg reported receiving research support from the NIH. Dr Rankin reported receiving research support from the NIH and the National Science Foundation and serving on the medical advisory board for Eli Lilly and Company. Dr Appleby reported receiving research support from the Centers for Disease Control and Prevention (CDC), the NIH, Ionis Pharmaceuticals Inc, Alector Inc, and the CJD Foundation and consulting for Acadia Pharmaceuticals Inc, Ionis Pharmaceuticals Inc, and Sangamo Therapeutics Inc. Dr Bayram reported receiving research support from the NIH. Dr Domoto-Reilly reported receiving research support from NIH and serving as an investigator for a clinical trial sponsored by Lawson Health Research Institute. Dr Bozoki reported receiving research funding from the NIH, Alector Inc, Cognition Therapeutics Inc, EIP Pharma, and Transposon Therapeutics Inc; consulting for Eisai and Creative Bio-Peptides Inc; and serving on the data safety monitoring board for AviadoBio. Dr Fields reported receiving research support from the NIH. Dr Galasko reported receiving research funding from the NIH; clinical trial funding from Alector Inc and Esai; consulting for Esai, General Electric Health Care, and Fujirebio; and serving on the data safety monitoring board of Cyclo Therapeutics Inc. Dr Geschwind reported consulting for Biogen Inc and receiving research support from Roche and Takeda Pharmaceutical Company for work in dementia. Dr Ghoshal reported participating in clinical trials of antidementia drugs sponsored by Bristol Myers Squibb, Eli Lilly and Company/Avid Radiopharmaceuticals, Janssen Immunotherapy, Novartis AG, Pfizer Inc, Wyeth Pharmaceuticals, SNIFF (The Study of Nasal Insulin to Fight Forgetfulness) study, and A4 (The Anti-Amyloid Treatment in Asymptomatic Alzheimer’s Disease) trial; receiving research support from Tau Consortium and the Association for Frontotemporal Dementia; and receiving funding from the NIH. Dr Graff-Radford reported receiving royalties from UpToDate; reported participating in multicenter therapy studies by sponsored by Biogen Inc, TauRx Therapeutics Ltd, AbbVie Inc, Novartis AG, and Eli Lilly and Company; and receiving research support from the NIH. Dr Grossman reported receiving grant support from the NIH, Avid Radiopharmaceuticals, and Piramal Pharma Ltd; participating in clinical trials sponsored by Biogen Inc, TauRx Therapeutics Ltd, and Alector Inc; consulting for Bracco and UCB; and serving on the editorial board of Neurology . Dr Hsiung reported receiving grant support from the Canadian Institutes of Health Research, the NIH, and the Alzheimer Society of British Columbia; participating in clinical trials sponsored by Anavax Life Sciences Corp, Biogen Inc, Cassava Sciences, Eli Lilly and Company, and Roche; and consulting for Biogen Inc, Novo Nordisk A/S, and Roche. Dr Huey reported receiving research support from the NIH. Dr Jones reported receiving research support from the NIH. Dr Litvan reported receiving research support from the NIH, the Michael J Fox Foundation, the Parkinson Foundation, the Lewy Body Association, CurePSP, Roche, AbbVie Inc, H Lundbeck A/S, Novartis AG, Transposon Therapeutics Inc, and UCB; serving as a member of the scientific advisory board for the Rossy PSP Program at the University of Toronto and for Amydis; and serving as chief editor of Frontiers in Neurology . Dr Masdeu reported consulting for and receiving research funding from Eli Lilly and Company; receiving personal fees from GE Healthcare; receiving grant funding and personal fees from Eli Lilly and Company; and receiving grant funding from Acadia Pharmaceutical Inc, Avanir Pharmaceuticals Inc, Biogen Inc, Eisai, Janssen Global Services LLC, the NIH, and Novartis AG outside the submitted work. Dr Mendez reported receiving research support from the NIH. Dr Miyagawa reported receiving research support from the Zander Family Foundation. Dr Pascual reported receiving research support from the NIH. Dr Pressman reported receiving research support from the NIH. Dr Ramos reported receiving research support from the NIH. Dr Roberson reported receiving research support from the NIA of the NIH, the Bluefield Project, and the Alzheimer’s Drug Discovery Foundation; serving on a data monitoring committee for Eli Lilly and Company; receiving licensing fees from Genentech Inc; and consulting for Applied Genetic Technologies Corp. Dr Tartaglia reported serving as an investigator for clinical trials sponsored by Biogen Inc, Avanex Corp, Green Valley, Roche/Genentech Inc, Bristol Myers Squibb, Eli Lilly and Company/Avid Radiopharmaceuticals, and Janssen Global Services LLC and receiving research support from the Canadian Institutes of Health Research (CIHR). Dr Wong reported receiving research support from the NIH. Dr Kornak reported providing expert witness testimony for Teva Pharmaceuticals Industries Ltd, Apotex Inc, and Puma Biotechnology and receiving research support from the NIH. Dr Kremers reported receiving research funding from NIH. Dr Kramer reported receiving research support from the NIH and royalties from Pearson Inc. Dr Boeve reported serving as an investigator for clinical trials sponsored by Alector Inc, Biogen Inc, and Transposon Therapeutics Inc; receiving royalties from Cambridge Medicine; serving on the Scientific Advisory Board of the Tau Consortium; and receiving research support from NIH, the Mayo Clinic Dorothy and Harry T. Mangurian Jr. Lewy Body Dementia Program, and the Little Family Foundation. Dr Rosen reported receiving research support from Biogen Inc, consulting for Wave Neuroscience and Ionis Pharmaceuticals, and receiving research support from the NIH. Dr Boxer reported being a coinventor of 4 of the ALLFTD mobile application tasks (not the focus of the present study) and previously receiving licensing fees; receiving research support from the NIH, the Tau Research Consortium, the Association for Frontotemporal Degeneration, Bluefield Project to Cure Frontotemporal Dementia, Corticobasal Degeneration Solutions, the Alzheimer’s Drug Discovery Foundation, and the Alzheimer’s Association; consulting for Aeovian Pharmaceuticals Inc, Applied Genetic Technologies Corp, Alector Inc, Arkuda Therapeutics, Arvinas Inc, AviadoBio, Boehringer Ingelheim, Denali Therapeutics Inc, GSK, Life Edit Therapeutics Inc, Humana Inc, Oligomerix, Oscotec Inc, Roche, Transposon Therapeutics Inc, TrueBinding Inc, and Wave Life Sciences; and receiving research support from Biogen Inc, Eisai, and Regeneron Pharmaceuticals Inc. No other disclosures were reported.

Funding/Support: This work was supported by grants AG063911, AG077557, AG62677, AG045390, NS092089, AG032306, AG016976, AG058233, AG038791, AG02350, AG019724, AG062422, NS050915, AG032289-11, AG077557, K23AG061253, and K24AG045333 from the NIH; the Association for Frontotemporal Degeneration; the Bluefield Project to Cure FTD; the Rainwater Charitable Foundation; and grant 2014-A-004-NET from the Larry L. Hillblom Foundation. Samples from the National Centralized Repository for Alzheimer’s Disease and Related Dementias, which receives government support under cooperative agreement grant U24 AG21886 from the NIA, were used in this study.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Group Information: A complete list of the members of the ALLFTD Consortium appears in Supplement 2 .

Data Sharing Statement: See Supplement 3 .

Additional Contributions: We thank the participants and study partners for dedicating their time and effort, and for providing invaluable feedback as we learn how to incorporate digital technologies into FTLD research.

Additional Information: Dr Grossman passed away on April 4, 2023. We want to acknowledge his many contributions to this study, including data acquisition, and design and conduct of the study. He was an ALLFTD site principal investigator and contributed during the development of the ALLFTD mobile app.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

Help | Advanced Search

Computer Science > Computation and Language

Title: realm: reference resolution as language modeling.

Abstract: Reference resolution is an important problem, one that is essential to understand and successfully handle context of different kinds. This context includes both previous turns and context that pertains to non-conversational entities, such as entities on the user's screen or those running in the background. While LLMs have been shown to be extremely powerful for a variety of tasks, their use in reference resolution, particularly for non-conversational entities, remains underutilized. This paper demonstrates how LLMs can be used to create an extremely effective system to resolve references of various types, by showing how reference resolution can be converted into a language modeling problem, despite involving forms of entities like those on screen that are not traditionally conducive to being reduced to a text-only modality. We demonstrate large improvements over an existing system with similar functionality across different types of references, with our smallest model obtaining absolute gains of over 5% for on-screen references. We also benchmark against GPT-3.5 and GPT-4, with our smallest model achieving performance comparable to that of GPT-4, and our larger models substantially outperforming it.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 25 March 2024
• Correction 27 March 2024

Weird new electron behaviour in stacked graphene thrills physicists

• Dan Garisto

You can also search for this author in PubMed   Google Scholar

You have full access to this article via your institution.

Electrons in stacked sheets of staggered graphene collectively act as though they have fractional charges at ultralow temperatures. Credit: Ramon Andrade 3DCiencia/Science Photo Library

Minneapolis, Minnesota

Last May, a team led by physicists at the University of Washington in Seattle observed something peculiar. When the scientists ran an electrical current across two atom-thin sheets of molybdenum ditelluride (MoTe 2 ), the electrons acted in concert, like particles with fractional charges. Resistance measurements showed that, rather than the usual charge of –1, the electrons behaved similar to particles with charges of –2/3 or –3/5, for instance. What was truly odd was that the electrons did this entirely because of the innate properties of the material, without any external magnetic field coaxing them. The researchers published the results a few months later, in August 1 .

Strange topological materials are popping up everywhere physicists look

The same month, this phenomenon, known as the fractional quantum anomalous Hall effect (FQAHE), was also observed in a completely different material. Researchers led by Long Ju, a condensed-matter physicist at the Massachusetts Institute of Technology (MIT) in Cambridge, saw the effect when they sandwiched five layers of graphene between sheets of boron nitride. They published their results in February this year 2 — and physicists are still buzzing about it.

At the American Physical Society (APS) March Meeting, held in Minneapolis, Minnesota, from 3 to 8 March, Ju presented the team’s findings, which haven’t yet been replicated by other researchers. Attendees, including Raquel Queiroz, a theoretical physicist at Columbia University in New York City, said that they thought the results were convincing, but were scratching their heads over the discovery. “There is a lot we don’t understand,” Queiroz says. Figuring out the exact mechanism of the FQAHE in the layered graphene will be “a lot of work ahead of theorists”, she adds.

Although the FQAHE might have practical applications down the line — fractionally charged particles are a key requirement for a certain type of quantum computer — the findings are capturing physicists’ imagination because they are fundamentally new discoveries about how electrons behave.

“I don’t know anyone who’s not excited about this,” says Pablo Jarillo-Herrero, a condensed-matter physicist at MIT who was not involved with the studies. “I think the question is whether you’re so excited that you switch all your research and start working on it, or if you’re just very excited.”

Strange maths

Strange behaviour by electrons isn’t new.

In some materials, usually at temperatures near absolute zero, electrical resistance becomes quantized. Specifically, it’s the material’s transverse resistance that does this. (An electrical current encounters opposition to its flow in both the same direction as the current — called longitudinal resistance — and in the perpendicular direction — what’s called transverse resistance.)

Quantized ‘steps’ in the transverse resistance occur at integer multiples of electron charge: 1, 2, 3 and so on. These plateaus are the result of a strange phenomenon: the electrons maintain the same transverse resistance even as charge density increases. That’s a little like vehicles on a road moving at the same speed, even with more traffic. This is known as the quantum Hall effect.

In a different set of materials, with less disorder, the transverse resistance can even display plateaus at fractions of electron charge: 2/5, 3/7 and 4/9, for example. The plateaus take these values because the electrons collectively act like particles with fractional charges — hence the fractional quantum Hall effect (FQHE).

Key to both phenomena is a strong external magnetic field, which prevents electrons from crashing into each other and enables them to interact.

(Left to right) Long Ju, Zhengguang Lu, Yuxuan Yao and Tonghang Han are all part of the team at MIT that demonstrated the fractional quantum anomalous Hall effect in layered graphene. Credit: Jixiang Yang

The FQHE, discovered in 1982, revealed the richness of electron behaviour. No longer could physicists think of electrons as single particles; in delicate quantum arrangements, the electrons could lose their individuality and act together to create fractionally charged particles. “I think people don’t appreciate how different [the fractional] is from the integer quantum Hall effect,” says Ashvin Vishwanath, a theoretical physicist at Harvard University in Cambridge. “It’s a new world.”

Over the next few decades, theoretical physicists came up with models to explain the FQHE and predict its effects. During their exploration, a tantalizing possibility appeared: perhaps a material could exhibit resistance plateaus without any external magnetic field. The effect, now dubbed the quantum anomalous Hall effect — ‘anomalous’, for the lack of a magnetic field — was finally observed in thin ferromagnetic films by a team at Tsinghua University in Beijing, in 2012 3 .

Carbon copy

Roughly a decade later, the University of Washington team reported the FQAHE for the first time 1 , in a specially designed 2D material: two sheets of MoTe 2 stacked on top of one another and offset by a twist.

This arrangement of MoTe 2 is known as a moiré material. Originally used to refer to a patterned textile, the term has been appropriated by physicists to describe the patterns in 2D materials created from atom-thin lattices when they are stacked and then twisted, or staggered atop one another. The slight offset between atoms in different layers of the material shifts the hills and valleys of its electric potential. And it effectively acts like a powerful magnetic field, taking the place of the one needed in the quantum Hall effect and the FQHE.

Xiaodong Xu, a condensed-matter physicist at the University of Washington, talked about the MoTe 2 discovery at the APS meeting. Theory hinted that the FQAHE would appear in the material at about a 1.4º twist angle. “We spent a year on it, and we didn’t see anything,” Xu told Nature .

Source: Adapted from Ref. 2.

Then, the researchers tried a larger angle — a twist of about 4º. Immediately, they began seeing signs of the effect. Eventually, they measured the electrical resistance and spotted the signature plateaus of the FQAHE. Soon afterwards, a team led by researchers at Shanghai Jiao Tong University in China replicated the results 4 .

Meanwhile, at MIT, Ju was perfecting his technique, sandwiching graphene between layers of boron nitride. Similar to graphene, the sheets of boron nitride that Ju’s team used were a mesh of atoms linked together in a hexagonal pattern. The material’s lattice has a slightly different size from graphene’s; the mismatch creates a moiré pattern (see ‘Anomalous behaviour’).

Last month, Ju published a report 2 about seeing the characteristic plateaus. “It is a really amazing result,” Xu says. “I'm very happy to see there’s a second system.” Since then, Ju says, he’s also seen the effect when using four and six layers of graphene.

Both moiré systems have their pros and cons. MoTe 2 exhibited the effect at a few kelvin, as opposed to 0.1 kelvin for the layered graphene sandwich. (Low temperatures are required to minimize disorder in the systems.) But graphene is a cleaner and higher-quality material that is easier to measure. Experimentalists are now trying to replicate the results in graphene and find other materials that behave similarly.

Moiré than bargained for

Theorists are relatively comfortable with the MoTe 2 results, for which the FQAHE was partly predicted. But Ju’s layered graphene moiré was a shock to the community, and researchers are still struggling to explain how the effect happens. “There’s no universal consensus on what the correct theory is,” Vishwanath says. “But they all agree that it’s not the standard mechanism.” Vishwanath and his colleagues posted a preprint proposing a theory that the moiré pattern might not be that important to the FQAHE 5 .

Welcome anyons! Physicists find best evidence yet for long-sought 2D structures

One reason to doubt the importance of the moiré is the location of the electrons in the material: most of the activity is in the topmost layer of graphene, far away from the moiré pattern between the graphene and boron nitride at the bottom of the sandwich that is supposed to most strongly influence the electrons. But B. Andrei Bernevig, a theoretical physicist at Princeton University in New Jersey, and a co-author of another preprint proposing a mechanism for the FQAHE in the layered graphene 6 , urges caution about theory-based calculations, because they rely on currently unverified assumptions. He says that the moiré pattern probably matters, but less than it does in MoTe 2 .

For theorists, the uncertainty is exciting. “There are people who would say that everything has been seen in the quantum Hall effect,” Vishwanath says. But these experiments, especially the one using the layered graphene moiré, show that there are still more mysteries to uncover.

Nature 628 , 16-17 (2024)

doi: https://doi.org/10.1038/d41586-024-00832-z

Updates & Corrections

Correction 27 March 2024 : An earlier version of this story spelled researcher Tonghang Han’s name incorrectly in the photo caption.

Park, H. et al. Nature 622 , 74–79 (2023).

Article   PubMed   Google Scholar  

Lu, Z. et al. Nature 626 , 759–764 (2024).

Chang, C.-Z. et al. Science 340 , 167–170 (2013).

Xu, F. et al. Phys. Rev. X 13 , 031037 (2023).

Article   Google Scholar  

Dong, J. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2311.05568 (2023).

Kwan, Y. H. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2312.11617 (2023).

Download references

Reprints and permissions

Related Articles

• Materials science

Exclusive: official investigation reveals how superconductivity physicist faked blockbuster results

News 06 APR 24

Phase-change memory via a phase-changeable self-confined nano-filament

Article 03 APR 24

Shell buckling for programmable metafluids

A quirky fluid that has robotic capabilities

News & Views 03 APR 24

Heat flows enrich prebiotic building blocks and enhance their reactivity

Postdoctoral Fellow (Aging, Metabolic stress, Lipid sensing, Brain Injury)

Seeking a Postdoctoral Fellow to apply advanced knowledge & skills to generate insights into aging, metabolic stress, lipid sensing, & brain Injury.

Dallas, Texas (US)

UT Southwestern Medical Center - Douglas Laboratory

High-Level Talents at the First Affiliated Hospital of Nanchang University

For clinical medicine and basic medicine; basic research of emerging inter-disciplines and medical big data.

Nanchang, Jiangxi, China

The First Affiliated Hospital of Nanchang University

POSTDOCTORAL Fellow -- DEPARTMENT OF Surgery – BIDMC, Harvard Medical School

The Division of Urologic Surgery in the Department of Surgery at Beth Israel Deaconess Medical Center and Harvard Medical School invites applicatio...

Boston, Massachusetts (US)

Director of Research

Applications are invited for the post of Director of Research at Cancer Institute (WIA), Chennai, India.

Chennai, Tamil Nadu (IN)

Cancer Institute (W.I.A)

Postdoctoral Fellow in Human Immunology (wet lab)

Join Atomic Lab in Boston as a postdoc in human immunology for universal flu vaccine project. Expertise in cytometry, cell sorting, scRNAseq.

Boston University Atomic Lab

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

New Modeling App To Assist Farmers With Forecasting

Being able to see into the future would be a handy trick for anybody, but that ability could be indispensable in helping farmers navigate the ups and downs of Mother Nature and markets.

Scientists with  Texas A&M AgriLife Research  in the  Texas A&M College of Agriculture and Life Sciences  are developing a tool that could give agricultural producers a glimpse into the future for planning purposes.

Dr. Raghavan Srinivasan, professor in the Texas A&M Department of Ecology and Conservation Biology  and  Biological and Agricultural Engineering , received more than $750,000 in grant funding from the National Institute of Food and Agriculture and is leading a team to develop an integrated decision support system (IDSS) — a modeling tool that can forecast potential cropping conditions and economic results for producers.

The tool will utilize existing technology, data collection tools and data — including weather, market prices, farm production costs and revenues, water conservation practices and water movement through watersheds — to project scenario-based outcomes for producer operations based on possible fluctuations within those factors.

“It would be an incredibly powerful tool for farmers and agricultural operations to have in their toolbox,” Srinivasan said. “This tool won’t tell us the future, but it will give us the range of potential outcomes based on factors like continuing drought or changes in input costs or commodity market conditions. The goal is to help farmers consider economic, environmental and production challenges together in one place.”

Modeling App Development Underway

This type of tool could be especially valuable as producers in the U.S. and around the world are facing challenges related to increasingly variable weather, including droughts.

The team, which includes researchers Dr. Jean-Claude Bizimana, Dr. Samuel Zapata and Dr. Anthony Baffoe-Bonnie, all with the Texas A&M Department of Agricultural Economics , and a water resource engineering expert from Oregon State University, plans to develop an IDSS, “ECO-HAWQS@Farm,” that tightly couples economic and watershed models within an accessible user interface.

The new application is expected to give producers insights on potential outcomes and help them plan in ways that help mitigate risks, Bizimana said. The project will test the performance of ECO-HAWQS by working with small- and medium-sized farming operations in Texas’ Lower Rio Grande Valley and Oregon’s Umatilla River Basin.

Bizimana said team members are engaging with a “focus group” of growers in the Rio Grande Valley to assess factors like acreage, cropping options and rotations, and the use of conservation practices like cover crops or no-till cropping. Researchers’ discussions with growers will help them set parameters for the new application in ways that provide useful decision support in “real-world” conditions.

The research team is close to having enough feedback from farmers and is ready to input data. The next step will be to engage with a larger pool of farmers in the region with small- to medium-sized operations to expand the model’s datasets, Bizimana said.

Bizimana said the goal is to produce a decision support tool that growers across the country and globe can access and use easily.

“We’ll be recruiting farmers for feedback,” he said. “We’re in the early stages of the project, but we all understand the value a tool like this represents for growers, especially smaller operations, to capitalize on opportunities and avoid major setbacks.”

This article by Adam Russell originally appeared on AgriLife Today .

Related Stories

An Aggie Approach To Expanding The Family Farm

Volleman’s Family Farm gives back to help shape the future of the dairy industry.

Texas A&M AgriLife To Lead USDA-Backed Organic Agriculture Program In Texas

As part of the Transition To Organic Partnership Program, AgriLife personnel will recruit, train and mentor farmers across the state.

Board Of Regents Approves Center For Greenhouse Gas Management In Agriculture And Forestry

The new center will focus scientific expertise to minimize emissions through climate-smart systems and practices.

Recent Stories

Student Affairs Staff Members Earn National Honors

Two Aggie professionals were recognized at the 2024 NASPA (Student Affairs Administrators in Higher Education) Annual Conference

Pets And The Solar Eclipse

A Texas A&M vet says the sudden increase in darkness during what should be the middle of the day may be puzzling and disorienting for pets.

Total Solar Eclipse: Astronomers Break Down The Science Behind The Rare Phenomenon

Make the most of Monday’s rare astronomical event by learning about the science behind the eclipse, safety tips and other key insights from Texas A&M experts.

Subscribe to the Texas A&M Today newsletter for the latest news and stories every week.

Aggregate Implications of Changing Industrial Trends in Japan

April 8, 2024 Toyoichiro Shirota *1 Satoshi Tsuchida *2

• Full Text [PDF 2,653KB]

This study examines the extent to which the long-term declining trend in Japan's GDP growth rate is attributable to factors common to all the industries or those specific to individual industries. By applying Japan's 1958-2019 data to a multi-industry network model, we obtained the following results. First, common factors explain approximately 60% of the variation in Japan's long-term GDP growth rate. This result contrasts to that in the US: common factors explain only about 30% of the secular trend in US GDP growth. Second, however, the impact of industry-specific factors is non-negligible. In particular, machinery-industry-specific factors explain much of the low growth in the past 20 years. Finally, the spillover effects from individual industries to the aggregate GDP depend on the role of each industry in the production network, and in Japan, the influence of investment-related industries such as the machinery industries and construction is substantial.

The authors thank Kosuke Aoki, Ichiro Fukunaga, Yoshihiko Hogen, Ryo Jinnai, Takashi Nagahata, Jouchi Nakajima, Yoichi Ueno for comments and discussions. Shirota is grateful for financial support from JSPS KAKENHI Grant-in-Aid for Scientific Research(C) No. 21K01396. Any remaining errors are the authors' own. The views expressed in this paper are those of the authors and do not necessarily reflect those of the authors' affiliations, including the Bank of Japan.

• *1 College of Economics, Aoyama Gakuin University E-mail : [email protected]
• *2 Research and Statistics Department, Bank of Japan E-mail : [email protected]

Papers in the Bank of Japan Working Paper Series are circulated to stimulate discussion and comment. Views expressed are those of the authors and do not necessarily reflect those of the Bank. If you have any comments or questions on a paper in the Working Paper Series, please contact the authors. When making a copy or reproduction of the content for commercial purposes, please contact the Public Relations Department ([email protected]) at the Bank in advance to request permission. When making a copy or reproduction, the Bank of Japan Working Paper Series should explicitly be credited as the source.

An official website of the United States Government

• Kreyòl ayisyen
• Search Toggle search Search Include Historical Content - Any - No Include Historical Content - Any - No Search
• Menu Toggle menu
• INFORMATION FOR…
• Individuals
• Business & Self Employed
• Charities and Nonprofits
• International Taxpayers
• Federal State and Local Governments
• Indian Tribal Governments
• Tax Exempt Bonds
• FILING FOR INDIVIDUALS
• How to File
• When to File
• Where to File
• Update Your Information
• Get Your Tax Record
• Apply for an Employer ID Number (EIN)
• Check Your Amended Return Status
• Get an Identity Protection PIN (IP PIN)
• File Your Taxes for Free
• Bank Account (Direct Pay)
• Payment Plan (Installment Agreement)
• Electronic Federal Tax Payment System (EFTPS)
• Your Online Account
• Tax Withholding Estimator
• Estimated Taxes
• Where's My Refund
• What to Expect
• Direct Deposit
• Reduced Refunds
• Amend Return

Credits & Deductions

• INFORMATION FOR...
• Businesses & Self-Employed
• Earned Income Credit (EITC)
• Child Tax Credit
• Clean Energy and Vehicle Credits
• Standard Deduction
• Retirement Plans

Forms & Instructions

• POPULAR FORMS & INSTRUCTIONS
• Form 1040 Instructions
• Form 4506-T
• POPULAR FOR TAX PROS
• Form 1040-X
• Circular 230

Get an extension to file your tax return

If you need more time to file your taxes, you can  request an extension through October 15. You must file your request by the April tax filing due date to get the extension. Make sure you pay any tax you owe by the April filing date. The extension is only for filing your return.

Automatic extensions

You may qualify for an automatic tax filing extension if you’re:

• In a federally declared disaster area
• In the military stationed abroad or serving in a combat zone
• A citizen or resident alien living outside the U.S.

Get an extension

There are 3 ways to get until October 15 to file your tax return.

Pay online and check the box

Pay what you owe using an online payment option and check the box that you are paying as part of filing for an extension. You don’t have to file a separate extension form and you’ll receive a confirmation number of your extension for your records.

Use Free File (no income limit for extensions)

Use IRS Free File to electronically request an automatic tax-filing extension. 

Request an extension by mail

1. File Form 4868, Application for Automatic Extension of Time To File U.S. Individual Income Tax Return . You can file by mail, online with an IRS e-filing partner or through a tax professional. 2. Estimate how much tax you owe for the year on the extension form: Subtract the taxes you already paid for the filing year.

Find more on extensions for: Businesses or corporations Form 7004, Application for Automatic Extension of Time to File Certain Business Income Tax, Information, and Other Returns Form 1138, Extension of Time for Payment of Taxes by a Corporation Expecting a Net Operating Loss Carryback U.S. citizens or resident aliens living outside the U.S. Form 2350, Application for Extension of Time to File U.S. Income Tax Return (For U.S. Citizens and Resident Aliens Abroad Who Expect To Qualify for Special Tax Treatment) Exempt organizations Form 8868, Application for Extension of Time To File an Exempt Organization Return   Retirement plans Form 5558, Application for Extension of Time to File Certain Employee Plan Returns Estates Form 4768, Application for Extension of Time to File a Return and/or Pay U.S. Estate (and Generation-Skipping Transfer) Taxes Form 8892, Application for Automatic Extension of Time to File Form 709 and/or Payment of Gift/Generation-Skipping Transfer Tax Information return filers Form 8809, Application for Extension of Time to File Information Returns

•  Facebook
•  Twitter
•  Linkedin

• Asia Pacific
• Latin America
• Middle East & Africa
• North America
• Australia & New Zealand

Mainland China

• Hong Kong SAR, China
• Philippines
• Taiwan, China
• Channel Islands
• Netherlands
• Switzerland
• United Kingdom
• Saudi Arabia
• South Africa
• United Arab Emirates
• United States

From startups to legacy brands, you're making your mark. We're here to help.

• Innovation Economy Fueling the success of early-stage startups, venture-backed and high-growth companies.
• Midsize Businesses Keep your company growing with custom banking solutions for middle market businesses and specialized industries.
• Large Corporations Innovative banking solutions tailored to corporations and specialized industries.
• Commercial Real Estate Capitalize on opportunities and prepare for challenges throughout the real estate cycle.
• Community Impact Banking When our communities succeed, we all succeed. Local businesses, organizations and community institutions need capital, expertise and connections to thrive.
• International Banking Power your business' global growth and operations at every stage.
• Client Stories

Prepare for future growth with customized loan services, succession planning and capital for business equipment.

• Asset Based Lending Enhance your liquidity and gain the flexibility to capitalize on growth opportunities.
• Equipment Financing Maximize working capital with flexible equipment and technology financing.
• Trade & Working Capital Experience our market-leading supply chain finance solutions that help buyers and suppliers meet their working capital, risk mitigation and cash flow objectives.
• Syndicated Financing Leverage customized loan syndication services from a dedicated resource.
• Employee Stock Ownership Plans Plan for your business’s future—and your employees’ futures too—with objective advice and financing.

Institutional Investing

Serving the world's largest corporate clients and institutional investors, we support the entire investment cycle with market-leading research, analytics, execution and investor services.

• Institutional Investors We put our long-tenured investment teams on the line to earn the trust of institutional investors.
• Markets Direct access to market leading liquidity harnessed through world-class research, tools, data and analytics.
• Prime Services Helping hedge funds, asset managers and institutional investors meet the demands of a rapidly evolving market.
• Global Research Leveraging cutting-edge technology and innovative tools to bring clients industry-leading analysis and investment advice.
• Securities Services Helping institutional investors, traditional and alternative asset and fund managers, broker dealers and equity issuers meet the demands of changing markets.
• Financial Professionals
• Liquidity Investors

Providing investment banking solutions, including mergers and acquisitions, capital raising and risk management, for a broad range of corporations, institutions and governments.

• Center for Carbon Transition J.P. Morgan’s center of excellence that provides clients the data and firmwide expertise needed to navigate the challenges of transitioning to a low-carbon future.
• Corporate Finance Advisory Corporate Finance Advisory (“CFA”) is a global, multi-disciplinary solutions team specializing in structured M&A and capital markets. Learn more.
• Development Finance Institution Financing opportunities with anticipated development impact in emerging economies.
• Sustainable Solutions Offering ESG-related advisory and coordinating the firm's EMEA coverage of clients in emerging green economy sectors.
• Mergers and Acquisitions Bespoke M&A solutions on a global scale.
• Capital Markets Holistic coverage across capital markets.
• Capital Connect
• In Context Newsletter from J.P. Morgan
• Director Advisory Services

Accept Payments

Explore Blockchain

Client Service

Process Payments

Manage Funds

Safeguard Information

Banking-as-a-service

Send Payments

• Partner Network

A uniquely elevated private banking experience shaped around you.

• Banking We have extensive personal and business banking resources that are fine-tuned to your specific needs.
• Investing We deliver tailored investing guidance and access to unique investment opportunities from world-class specialists.
• Lending We take a strategic approach to lending, working with you to craft the fight financing solutions matched to your goals.
• Planning No matter where you are in your life, or how complex your needs might be, we’re ready to provide a tailored approach to helping your reach your goals.

Whether you want to invest on your own or work with an advisor to design a personalized investment strategy, we have opportunities for every investor.

• Invest on your own Unlimited $0 commission-free online stock, ETF and options trades with access to powerful tools to research, trade and manage your investments.
• Work with our advisors When you work with our advisors, you'll get a personalized financial strategy and investment portfolio built around your unique goals-backed by our industry-leading expertise.
• Expertise for Substantial Wealth Our Wealth Advisors & Wealth Partners leverage their experience and robust firm resources to deliver highly-personalized, comprehensive solutions across Banking, Lending, Investing, and Wealth Planning.
• Why Wealth Management?
• Retirement Calculators
• Market Commentary

Who We Serve

Explore a variety of insights.

Global Research

• Newsletters

Insights by Topic

Explore a variety of insights organized by different topics.

Insights by Type

Explore a variety of insights organized by different types of content and media.  

• All Insights

We aim to be the most respected financial services firm in the world, serving corporations and individuals in more than 100 countries.

Key takeaways

• Cocoa prices are rising due to a global supply shortage, chronic underinvestment in cocoa farms and investor speculation.
• Chocolate brands are grappling with the impact of higher cocoa costs, and this is resulting in price hikes and shrinkflation.
• J.P. Morgan Research projects cocoa prices will come down slightly over the medium term, tracking around the $6,000 mark.

Cocoa prices have skyrocketed in recent months, reaching historical highs of nearly $10,000 per metric ton in March 2024. What’s driving this unprecedented increase? How will this impact chocolate brands? And where is the market headed?  

Why are cocoa prices rising? 

The rise in cocoa prices is largely due to a global cocoa shortage. Climate change -induced drought has ravaged crops in West Africa, which contributes around 80% of the world’s cocoa output. According to the International Cocoa Organization, global cocoa supply will decline by almost 11% over the 2023/2024 season.

There are also deep-rooted structural issues at play, including chronic underinvestment in cocoa farms. Today, the crop is still largely cultivated by smallholder farmers, many of whom struggle to make a living income and lack the means to reinvest in their land — which translates to lower yields over time. “Cocoa is a market where the grower produces a very high-value good but receives a very low share of the actual value chain. As a result, replanting rates are very low and cocoa trees are ageing,” said Tracey Allen, an Agricultural Commodities Strategist at J.P. Morgan. 

This is all further compounded by investor speculation, which is driving prices up. “What was a structurally led, supply-side issue has been exacerbated by dry weather and a strong Harmattan [a season in West Africa]. It has now manifested in an investor-driven parabolic move in prices, especially over the last six weeks,” Allen added. “For instance, non-commercial investors now hold over 60% of total open interest across cocoa futures and options in the New York market, which is an historical high. Consumers are now scrambling to hedge forward exposure in thin liquidity.”

“We have seen some shifting away from chocolate to other products, whether that’s cookies or salty snacks. We think chocolate’s losing a bit of share, so consumers are certainly reacting to higher prices.”

Ken Goldman

Lead Equity Research Analyst for U.S. Food Producers and Food Retailers, J.P. Morgan

What’s the impact on chocolate makers? 

Chocolate brands are grappling with the impact of higher cocoa costs, and many are passing on the burden to consumers in the form of price hikes.

“In the U.S., Hershey has been very clear that list pricing is still one of the most important arrows in their quiver to offset inflation . Over the next year or two, they will probably pass on more cocoa inflation, and consumers will see higher prices for their chocolate as a result,” said Ken Goldman, Lead Equity Research Analyst for U.S. Food Producers and Food Retailers at J.P. Morgan. “I’m sure Hershey’s competitors like Mars and Lindt are going to do the same — it’s just a matter of time.”

This could, however, dampen consumer demand. “Companies will have to be careful about how they manage the elasticity of demand for chocolate, especially as consumers might not be able to take the price hikes needed to offset the huge increase in cocoa costs. We see an impact on volume whenever prices are raised, even in other categories that are deemed inelastic,” said Celine Pannuti, Head of European Staples & Beverages at J.P. Morgan.

Indeed, chocolate sales have already taken a hit. “In the U.S., snacks in general have been slumping a little bit over the last two months, likely because they are a little bit more discretionary in nature. Within the category itself, we have seen some shifting away from chocolate to other products, whether that’s cookies or salty snacks,” Goldman added. “We think chocolate’s losing a bit of share, so consumers are certainly reacting to higher prices.”

In a bid to regain market share, some manufacturers are innovating with recipes that call for less cocoa — for instance, chocolate bars containing a higher proportion of fruits and nuts — while others are reducing the size of their products. “We’ve seen cases of shrinkflation, so for the same price, the chocolate bar is much smaller,” Pannuti noted. “But while these are some avenues that companies are taking, consumers will ultimately have to face up to higher prices.”

“Companies will have to be careful about how they manage the elasticity of demand for chocolate, especially as consumers might not be able to take the price hikes needed to offset the huge increase in cocoa costs.”

Celine Pannuti

Head of European Staples & Beverages, J.P. Morgan

Will cocoa prices continue to surge? 

While J.P. Morgan Research projects cocoa prices will remain elevated, they are expected to come down slightly over the medium term, tracking around the $6,000 mark.

“The market has been well overshot, so I wouldn’t be surprised if prices move off the peak before stabilizing at an elevated level, until there is a genuine supply-side response,” Allen noted.

The latter could materialize thanks to global climate patterns , which could increase yields in some cocoa-producing regions. The ongoing El Niño phenomenon (the warming of sea surface temperatures) is gradually weakening, which could herald a transition to La Niña (the cooling of sea surface temperatures) in the summer. “Under this scenario, weather threats would diminish and we should receive improved rainfall across West Africa and Asia,” Allen said. “This would be supportive of main crop output from over 80% of the world’s cocoa-producing countries including Côte d’Ivoire and Ghana, paving the way for cocoa prices to ease off the historic highs. Increasing cacao plantings will be critical to boost longer-term supply.”

How are rising cocoa prices impacting chocolate brands?

Cocoa prices have soared to an all-time high due to a global supply shortage. What’s the impact on chocolate makers, and what are they doing to adapt? Here, Ken Goldman, Lead Equity Research Analyst for U.S. Food Producers and Food Retailers, and Celine Pannuti, Head of European Staples and Beverages, share their thoughts with Shirley Wang, a credit analyst covering the investment-grade and high-yield consumer and retail sector.

How are Rising Cocoa Prices Impacting Chocolate Brands?

Shirley Wang: Cocoa prices have reached multi-decade highs due to a global supply shortage. How is this effecting chocolate makers? And is consumer demand suffering as a result? Welcome to Research Recap on J.P. Morgan's Making Sense Podcast channel. I'm Shirley Wang, a credit analyst covering the investment-grade and high-yield consumer and retail sector at J.P. Morgan. Today, I'm joined by my colleagues, Ken Goldman, lead equity research analyst for US food producers and food retailers, and Céline Pannuti, head of European staples and beverages, and we're here to discuss all things chocolate. Ken and Céline, thanks so much for joining us today.

Ken Goldman: Thank you for having us.

Céline Pannuti: Thank you so much.

Shirley Wang: To kick things off, Ken, can you start us off by providing some context? How is the ongoing cocoa shortage affecting the prices of chocolate on supermarket shelves?

Ken Goldman: Up and about to go up more. Chocolate prices are high, obviously, like every other food item, but whereas most food inflation, we're probably done with and prices will remain high, because cocoa is rising so much, chocolate prices will have to rise more. Now, this may take some time. Hershey, for example, has said that right now they are not taking a lot of incremental list pricing, meaning it won't raise the actual price that they charge pre-discount to their customers. But they've also been very clear recently that list pricing is still one of the most important arrows in their quiver to offset inflation. And so, I think over the next year, two years, they will pass on as much as they possibly can of cocoa inflation. And as a result we as consumers we'll see higher prices for our chocolate. And I'm sure Hershey's competitors like Mars and Lindt and everyone else who's big in the United States is going to do the same. It's just a matter of time.

Shirley Wang: And Céline, what are you noticing from some of the companies you cover?

Céline Pannuti: From my standpoint, I concur with what Ken said. Nestlé, which I cover, is active in chocolate. They own Kit Kat, for example. We've seen pricing for them really accelerating by the last two years, but what you're referring to is really a huge spike in cocoa prices that we are seeing now in the last four, five months. And that will take time to work its way through. But effectively, there will be more pricing to be taken. At the same time, I think companies will have to be careful on how they manage the elasticity because I presume consumers won't be able to take as much pricing as what they probably need to take to offset this huge increase in cocoa prices.

Shirley Wang: Great, thanks for that. So, both of you mentioned pricing. But, apart from that how else are companies within your coverage being affected by the higher cocoa prices. Are they using less chocolate in their recipes for example?

Ken Goldman: So, some of the companies that I cover, I think you will see them lean a little bit in their marketing away from some chocolate items, right? And it's tough for Mondelēz because most of what they sell outside the US is chocolate. But Hershey has items like Twizzlers and Jolly Ranchers, and they can certainly try and shift some of their consumer consumption toward those confectionery items that are not based on chocolate. And as we think about the ingredients themselves and the products themselves, yeah, it's possible, right? It's possible to use a little bit less cocoa. I don't think these companies really wanna do that, but at times in the past, maybe the distant past, some of these companies that we cover have shifted that mix a little bit. There's also a way to lean people more toward the chocolate bars that have nuts in them and fruit in them, or add more fruit and nuts. So if you're a fan of chocolate bars that have all those sorts of croutons in them, that's the kind of side benefit you get from this. But in general, I think it's safe to say that there's only so much the companies can do in terms of the product itself to offset cocoa inflation.

Céline Pannuti: Yeah, I think the innovation will be a way to try to re-engineer some of those recipes. What we've seen as well is shrinkflation. So for the same price of a chocolate bar, the size of that bar will be much smaller. So that sometimes has infuriated consumers. Those clearly could be some of the avenues that our companies are going to take, but agreed that ultimately you will have to face up to a much higher price nonetheless.

Shirley Wang: So looking at the consumer, are people buying less chocolate products as a result, or is demand still resilient?

Ken Goldman: So snacks in general in the United States have been slumping a little bit for the last two months. Both sweet snacks and salty have been off a little bit more than usual as a percentage of total food at home sales in the US. It's a little surprising. And there's some controversy or some questions about why that is. Some people might think that it's because these are a little bit more discretionary in nature, right? It's not a meal to have a chocolate bar or a bag of potato chips. And if you as a consumer go into a grocery store and you have $50 to spend, and the cost of everything else that you must have in your basket has gone up, you may not have an extra dollar for a candy bar. So that's a broader answer. In terms of chocolate specifically, we have seen a little bit of shifting away within snacks in general, from chocolate to other products, whether it's cookies or salty snacks a little bit. So within snacking in general, we think that chocolate's losing a little bit of share as well. So consumers are certainly reacting to higher prices. It's one of the reasons why I think some of the companies that we cover are a little hesitant to take as much pricing as they normally would at this time. In a typical environment, either Hershey or Mars may have already taken more price. Now, Hershey has other reasons not to take price right now. They're implementing a new ERP system. They don't wanna complicate things as they do that, but it's a little surprising that some of their competitors haven't taken pricing either. And I think that's a large part because of some of the elasticity we've seen in the category.

Shirley: Celine, do you have anything you’d like to add on that?

Céline Pannuti: What I would say concerning, the volume performance for Nestlé specifically, we've seen that post-COVID where the chocolate or their confectionery business had been impacted by less impulse obviously during COVID, we've really seen a re-acceleration with the reopening. So we have had two good years of better volume growth and at the same time companies have raised prices. So probably the category on the global basis has been quite resilient. But like Ken said, the question mark going forward is how much more consumers can take in terms of price increases. It is true that it's a treat. But what we've seen in other categories, like coffee for instance, is that even in categories that are deemed inelastic, when prices are raised so much, we started to see this impact in volume. So I think we have to be a bit careful about what seems to have been quite resilient volume over the past couple of years if our corporates need to raise prices further. And I would expect that yes, maybe consumers could be a bit more choosy because price points ultimately would lead to them looking at alternatives, some other categories could benefit in that.

Shirley Wang: Great. So, Ken, and Céline, you've mentioned somewhat uncertain volume and strong pricing for chocolate by manufacturers passed through cost inflation, as well as mixed consumer sentiment. As we wrap up today’s podcast, how do you see the overall impact for food manufacturers?

Ken Goldman: Well, I think, you know, higher inflation in general for food producers presents a challenge, right? It's not always easy to pass on all of your higher costs to your customers. They don't wanna always take it because then they have to pass it on to their consumers. And especially when you have recession-like symptoms for lack of a better word, among lower-end consumers, it creates even more of a challenge. So, certainly we've seen Hershey guide to flat earnings for 2024. A large part of that is because of cocoa inflation and potentially because of their limited ability to pass it on immediately. As we look to 2025, a lot of my clients, institutional investors, have become concerned that higher cocoa will affect Hershey's EPS in 2025 as well. And the, the, the reason it's so far out is Hershey and other food companies, including Mondelēz, typically don't buy most of their cocoa needs on spot markets. They will buy them on future markets. Hershey has said they will go anywhere from 3 to 24 months out in terms of their purchases. So it's difficult for us to be precise in forecasting exactly when a higher inflation will hit their income statement, but at some point down the road, and a lot of investors think it's 2025, they will have another headwind incrementally above what they have this year, and they'll have to figure out how to deal with that, but it's not going to be something they can fully offset.

Céline Pannuti: Yeah. And, and as far as I'm concerned, Nestlé only has 10% of their sales tied to confectionery, so it's not exactly relevant on the overall EPS growth, but effectively they have been constrained in terms of their EPS delivery in the past for higher cost inflation. So I would imagine that the confectionery profit pool is probably going to be constrained even if it's only a certain percentage of the total for Nestlé.

Shirley Wang: Ken and Céline, thanks so much for sharing your thoughts today.

Ken Goldman: Thank you.

Céline Pannuti: Thank you.

Shirley Wang: And thank you to all our listeners for tuning in. We hope you join us again next time.

[END OF EPISODE]

Related insights

Leveraging cutting-edge technology and innovative tools to bring clients industry-leading analysis and investment advice.

16:12 - Economy

Research Recap | El Niño and the impact on agriculture

July 26, 2023

In this episode of Research Recap, discover how El Niño will affect crop yields and other commodities.

Climate change and the impact on agriculture

August 02, 2022

How is climate change and extreme weather affecting crop production and prices? J.P. Morgan Research examines the effects on agricultural commodities.

This communication is provided for information purposes only. Please read J.P. Morgan research reports related to its contents for more information, including important disclosures. JPMorgan Chase & Co. or its affiliates and/or subsidiaries (collectively, J.P. Morgan) normally make a market and trade as principal in securities, other financial products and other asset classes that may be discussed in this communication. This communication has been prepared based upon information, including market prices, data and other information, from sources believed to be reliable, but J.P. Morgan does not warrant its completeness or accuracy except with respect to any disclosures relative to J.P. Morgan and/or its affiliates and an analyst's involvement with any company (or security, other financial product or other asset class) that may be the subject of this communication. Any opinions and estimates constitute our judgment as of the date of this material and are subject to change without notice. Past performance is not indicative of future results. This communication is not intended as an offer or solicitation for the purchase or sale of any financial instrument. J.P. Morgan Research does not provide individually tailored investment advice. Any opinions and recommendations herein do not take into account individual client circumstances, objectives, or needs and are not intended as recommendations of particular securities, financial instruments or strategies to particular clients. You must make your own independent decisions regarding any securities, financial instruments or strategies mentioned or related to the information herein. Periodic updates may be provided on companies, issuers or industries based on specific developments or announcements, market conditions or any other publicly available information. However, J.P. Morgan may be restricted from updating information contained in this communication for regulatory or other reasons. Clients should contact analysts and execute transactions through a J.P. Morgan subsidiary or affiliate in their home jurisdiction unless governing law permits otherwise. This communication may not be redistributed or retransmitted, in whole or in part, or in any form or manner, without the express written consent of J.P. Morgan. Any unauthorized use or disclosure is prohibited. Receipt and review of this information constitutes your agreement not to redistribute or retransmit the contents and information contained in this communication without first obtaining express permission from an authorized officer of J.P. Morgan. 

You're now leaving J.P. Morgan

J.P. Morgan’s website and/or mobile terms, privacy and security policies don’t apply to the site or app you're about to visit. Please review its terms, privacy and security policies to see how they apply to you. J.P. Morgan isn’t responsible for (and doesn’t provide) any products, services or content at this third-party site or app, except for products and services that explicitly carry the J.P. Morgan name.