mobile application research paper

• Open access
• Published: 07 May 2013

Usability of mobile applications: literature review and rationale for a new usability model

• Rachel Harrison 1 ,
• Derek Flood 1 &
• David Duce 1  

Journal of Interaction Science volume  1 , Article number:  1 ( 2013 ) Cite this article

188k Accesses

346 Citations

3 Altmetric

Metrics details

The usefulness of mobile devices has increased greatly in recent years allowing users to perform more tasks in a mobile context. This increase in usefulness has come at the expense of the usability of these devices in some contexts. We conducted a small review of mobile usability models and found that usability is usually measured in terms of three attributes; effectiveness, efficiency and satisfaction. Other attributes, such as cognitive load, tend to be overlooked in the usability models that are most prominent despite their likely impact on the success or failure of an application. To remedy this we introduces the PACMAD (People At the Centre of Mobile Application Development) usability model which was designed to address the limitations of existing usability models when applied to mobile devices. PACMAD brings together significant attributes from different usability models in order to create a more comprehensive model. None of the attributes that it includes are new, but the existing prominent usability models ignore one or more of them. This could lead to an incomplete usability evaluation. We performed a literature search to compile a collection of studies that evaluate mobile applications and then evaluated the studies using our model.

Introduction

Advances in mobile technology have enabled a wide range of applications to be developed that can be used by people on the move. Developers sometimes overlook the fact that users will want to interact with such devices while on the move. Small screen sizes, limited connectivity, high power consumption rates and limited input modalities are just some of the issues that arise when designing for small, portable devices. One of the biggest issues is the context in which they are used. As these devices are designed to enable users to use them while mobile, the impact that the use of these devices has on the mobility of the user is a critical factor to the success or failure of the application.

Current research has demonstrated that cognitive overload can be an important aspect of usability [ 1 , 2 ]. It seems likely that mobile devices may be particularly sensitive to the effects of cognitive overload, due to their likely deployment in multiple task settings and limitations of size. This aspect of usability is often overlooked in existing usability models, which are outlined in the next section, as these models are designed for applications which are seldom used in a mobile context. Our PACMAD usability model for mobile applications, which we then introduce, incorporates cognitive load as this attribute directly impacts and may be impacted by the usability of an application.

A literature review, outlined in the following section, was conducted as validation of the PACMAD model. This literature review examined which attributes of usability, as defined in the PACMAD usability model, were used during the evaluation of mobile applications presented in a range of papers published between 2008 and 2010. Previous work by Kjeldskov & Graham [ 3 ] has looked at the research methods used in mobile HCI, but did not examine the particular attributes of usability incorporated in the PACMAD model. We also present the results of the literature review.

The impact of this work on future usability studies and what lessons other researchers should consider when performing usability evaluations on mobile applications are also discussed.

Background and literature review

Existing models of usability.

Nielsen [ 4 ] identified five attributes of usability:

  Efficiency : Resources expended in relation to the accuracy and completeness with which users achieve goals;

  Satisfaction : Freedom from discomfort, and positive attitudes towards the use of the product.

  Learnability : The system should be easy to learn so that the user can rapidly start getting work done with the system;

  Memorability : The system should be easy to remember so that the casual user is able to return to the system after some period of not having used it without having to learn everything all over again;

  Errors : The system should have a low error rate, so that users make few errors during the use of the system and that if they do make errors they can easily recover from them. Further, catastrophic errors must not occur.

In addition to this Nielsen defines Utility as the ability of a system to meet the needs of the user. He does not consider this to be part of usability but a separate attribute of a system. If a product fails to provide utility then it does not offer the features and functions required; the usability of the product becomes superfluous as it will not allow the user to achieve their goals. Likewise, the International Organization for Standardization (ISO) defined usability as the “Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [ 5 ]. This definition identifies 3 factors that should be considered when evaluating usability.

  User : Person who interacts with the product;

  Goal : Intended outcome;

  Context of use : Users, tasks, equipment (hardware, software and materials), and the physical and social environments in which a product is used.

Each of the above factors may have an impact on the overall design of the product and in particular will affect how the user will interact with the system. In order to measure how usable a system is, the ISO standard outlines three measurable attributes:

  Effectiveness : Accuracy and completeness with which users achieve specified goals;

Unlike Nielsen’s model of usability, the ISO standard does not consider Learnability, Memorability and Errors to be attributes of a product’s usability although it could be argued that they are included implicitly within the definitions of Effectiveness, Efficiency and Satisfaction. For example, error rates can be argued to have a direct effect on efficiency.

Limitations for mobile applications

The models presented above were largely derived from traditional desktop applications. For example, Nielsen’s work was largely based on the design of telecoms systems, rather than computer software. The advent of mobile devices has presented new usability challenges that are difficult to model using traditional models of usability. Zhang and Adipat [ 6 ] highlighted a number of issues that have been introduced by the advent of mobile devices:

  Mobile Context : When using mobile applications the user is not tied to a single location. They may also be interacting with nearby people, objects and environmental elements which may distract their attention.

  Connectivity : Connectivity is often slow and unreliable on mobile devices. This will impact the performance of mobile applications that utilize these features.

  Small Screen Size : In order to provide portability mobile devices contain very limited screen size and so the amount of information that can be displayed is limited.

  Different Display Resolution : The resolution of mobile devices is reduced from that of desktop computers resulting in lower quality images.

  Limited Processing Capability and Power : In order to provide portability, mobile devices often contain less processing capability and power. This will limit the type of applications that are suitable for mobile devices.

  Data Entry Methods : The input methods available for mobile devices are different from those for desktop computers and require a certain level of proficiency. This problem increases the likelihood of erroneous input and decreases the rate of data entry.

From our review it is apparent that many existing models for usability do not consider mobility and its consequences, such as additional cognitive load. This complicates the job of the usability practitioner, who must consequently define their task model to explicitly include mobility. One might argue that the lack of reference to a particular context could be a strength of a usability model provided that the usability practitioner has the initiative and knows how to modify the model for a particular context.

The PACMAD usability model aims to address some of the shortcomings of existing usability models when applied to mobile applications. This model builds on existing theories of usability but is tailored specifically for applications that can be used on mobile devices. The PACMAD usability model is depicted in Figure  1 side by side with Nielsen’s and the ISO’s definition of usability. The PACMAD usability model incorporates the attributes of both the ISO standard and Nielsen’s model and also introduces the attribute of cognitive load which is of particular importance to mobile applications. The following section introduces the PACMAD usability model and describes in detail each of the attributes of usability mentioned below as well as the three usability factors that are part of this model: user, task and context.

Comparison of usability models.

The PACMAD usability model for mobile applications identifies three factors (User, Task and Context of use) that should be considered when designing mobile applications that are usable. Each of these factors will impact the final design of the interface for the mobile application. In addition to this the model also identifies seven attributes that can be used to define metrics to measure the usability of an application. The following section outlines each of these factors and attributes in more detail.

Factors of usability

The PACMAD usability model identifies three factors which can affect the overall usability of a mobile application: User , Task and Context of use . Existing usability models such as those proposed by the ISO [ 5 ] and Nielsen [ 4 ] also recognise these factors as being critical to the successful usability of an application. For mobile applications Context of use plays a critical role as an application may be used in multiple, very different contexts.

User It is important to consider the end user of an application during the development process. As mobile applications are usually designed to be small, the traditional input methods, such as a keyboard and mouse, are no longer practical. It is therefore necessary for application designers to look at alternative input methods. Some users may find it difficult to use some of these methods due to physical limitations. For example it has been shown [ 7 ] that some Tetraplegic users who have limited mobility in their upper extremities tend to have high error rates when using touch screens and this may cause unacceptable difficulties with certain (usually small) size targets.

Another factor that should be considered is the user’s previous experience. If a user is an expert at the chosen task then they are likely to favour shortcut keys to accomplish this task. On the other hand novice users may prefer an interface that is intuitive and easy to navigate and which allows them to discover what they need. This trade-off must be considered during the design of the application.

Task The word task refers here to the goal the user is trying to accomplish with the mobile application. During the development of applications, additional features can be added to an application in order to allow the user to accomplish more with the software. This extra functionality comes at the expense of usability as these additional features increase the complexity of the software and therefore the user’s original goal can become difficult to accomplish.

For example, consider a digital camera. If a user wants to take a photograph, they must first select between different modes (e.g. video, stills, action, playback, etc.) and then begin to line up the shot. This problem is further compounded if the user needs to take a photograph at night and needs to search through a number of menu items to locate and turn on a flashlight.

Context of use The word context refers here to the environment in which the user will use the application. We want to be able to view context separately from both the user and the task. Context not only refers to a physical location but also includes other features such as the user’s interaction with other people or objects (e.g. a motor vehicle) and other tasks the user may be trying to accomplish. Research has shown that using mobile applications while walking can slow down the walker’s average walking speed [ 8 ]. As mobile applications can be used while performing other tasks it is important to consider the impact of using the mobile application in the appropriate context.

Attributes of usability

The PACMAD usability model identifies 7 attributes which reflect the usability of an application: Effectiveness , Efficiency , Satisfaction , Learnability , Memorability , Errors and Cognitive load . Each of these attributes has an impact on the overall usability of the application and as such can be used to help assess the usability of the application.

Effectiveness Effectiveness is the ability of a user to complete a task in a specified context. Typically effectiveness is measured by evaluating whether or not participants can complete a set of specified tasks.

Efficiency Efficiency is the ability of the user to complete their task with speed and accuracy. This attribute reflects the productivity of a user while using the application. Efficiency can be measured in a number of ways, such as the time to complete a given task, or the number of keystrokes required to complete a given task.

Satisfaction Satisfaction is the perceived level of comfort and pleasantness afforded to the user through the use of the software. This is reflected in the attitudes of the user towards the software. This is usually measured subjectively and varies between individual users. Questionnaires and other qualitative techniques are typically used to measure a user’s attitudes towards a software application.

Learnability A recent survey of mobile application users [ 9 ] found that users will spend on average 5 minutes or less learning to use a mobile application. There are a large number of applications available on mobile platforms and so if users are unable to use an application they may simply select a different one. For this reason the PACMAD model includes the attribute Learnability as suggested by Nielsen.

Learnability is the ease with which a user can gain proficiency with an application. It typically reflects how long it takes a person to be able to use the application effectively. In order to measure Learnability, researchers may look at the performance of participants during a series of tasks, and measure how long it takes these participants to reach a pre-specified level of proficiency.

Memorability The survey also found that mobile applications are used on an infrequent basis and that participants used almost 50% of the applications only once a month [ 9 ]. Thus there may be a large period of inactivity between uses and so participants may not easily recall how to use the application. Consequently the PACMAD usability model includes the attribute of Memorability as also suggested by Nielsen.

Memorability is the ability of a user to retain how to use an application effectively. Software might not be used on a regular basis and sometimes may only be used sporadically. It is therefore necessary for users to remember how to use the software without the need to relearn it after a period of inactivity. Memorability can be measured by asking participants to perform a series of tasks after having become proficient with the use of the software and then asking them to perform similar tasks after a period of inactivity. A comparison can then be made between the two sets of results to determine how memorable the application was.

Errors The PACMAD usability model extends the description of Errors, first proposed by Nielsen, to include an evaluation of the errors that are made by participants while using mobile apps. This allows developers to identify the most troublesome areas for users and to improve these areas in subsequent iterations of development. This attribute is used to reflect how well the user can complete the desired tasks without errors. Nielsen [ 4 ] states that users should make few errors during the use of a system and that if they do make errors they should be able to easily recover from them. The error rate of users may be used to infer the simplicity of a system. The PACMAD usability model considers the nature of errors as well as the frequency with which they occur. By understanding the nature of these errors it is possible to prevent these errors from occurring in future versions of the application.

Cognitive load The main contribution of the PACMAD model is its inclusion of Cognitive Load as an attribute of usability. Unlike traditional desktop applications, users of mobile applications may be performing additional tasks, such as walking, while using the mobile device. For this reason it is important to consider the impact that using the mobile device will have on the performance of the user of these additional tasks. For example a user may wish to send a text message while walking. In this case the user’s walking speed will be reduced as they are concentrating on sending the message which is distracting them from walking.

Cognitive load refers to the amount of cognitive processing required by the user to use the application. In traditional usability studies a common assumption is that the user is performing only a single task and can therefore concentrate completely on that task. In a mobile context users will often be performing a second action in addition to using the mobile application [ 8 , 10 ]. For example a user may be using a stereo while simultaneously driving a car. In this scenario it is important that the cognitive load required by the mobile application, in this case the stereo, does not adversely impact the primary task.

While the user is using the application in a mobile context it will impact both the user’s ability to move and to operate the mobile application. Therefore it is important to consider both dimensions when studying the usability of mobile applications. One way this can be measured is through the NASA Task Load Index (TLX) [ 11 ]. This is a subjective workload assessment tool for measuring the cognitive workload placed on a user by the use of a system. In this paper we adopt a relatively simple view of cognitive load. For a more accurate assessment it may be preferable to adopt a more powerful multi-factorial approach [ 1 , 12 ] but this is beyond the scope of this paper.

Literature review

In order to evaluate the appropriateness and timeliness of the PACMAD usability model for mobile applications, a literature review was conducted to review current approaches and to determine the need for a comprehensive model that includes cognitive load. We focused on papers published between 2008 and 2010 which included an evaluation of the usability of a mobile application.

Performing the literature review

The first step in the literature review was to collect all of the publications from the identified sources. These sources were identified by searching the ACM digital library, IEEE digital library and Google Scholar. The search strings used during these searches were “ Mobile Application Evaluations ”, “ Usability of mobile applications ” and “ Mobile application usability evaluations ”. The following conferences and journals were identified as being the most relevant sources: the Mobile HCI conference (MobileHCI), the International Journal of Mobile Human Computer Interaction (IJMHCI), the ACM Transactions on Computer-Human Interaction (TOCHI), the International Journal of Human Computer Studies (IJHCS), the Personal and Ubiquitous Computing journal (PUC), and the International Journal of Human-Computer Interaction (IJHCI). We also considered the ACM Conference on Human Factors in Computing Systems (CHI) and the IEEE Transactions on Mobile Computing (IEEE TOMC). These sources were later discarded as very few papers (less than 5% of the total) were relevant.

The literature review was limited to the publications between the years 2008 and 2010 due to the emergence of smart phones during this time. Table  1 shows the number of publications that were examined from each source.

The sources presented above included a number of different types of publications (Full papers, short papers, doctoral consortium, editorials, etc.). We focused the study only on full or short research papers from peer reviewed sources. This approach was also adopted by Budgen et al. [ 13 ]. Table  2 shows the number of remaining publications by source.

The abstract of each of the remaining papers was examined to determine if the paper:

Conducted an evaluation of a mobile application/device;

Contained some software component with which the users interact;

Conducted an evaluation which was focused on the interaction with the application or device;

Publications which did not meet the above criteria were removed.

The following exclusion criteria were u sed to exclude papers:

Focused only on application development methodologies and techniques;

Contained only physical interaction without a software component;

Examined only social aspects of using mobile applications;

Did not consider mobile applications.

Each abstract was reviewed by the first two authors to determine if it should be included within the literature review. When a disagreement arose between the reviewers it was discussed until mutual agreement was reached. A small number of relevant publications were unavailable to the authors. Table  3 shows the number of papers included within the literature review by source.

Each of the remaining papers was examined by one reviewer (either the first or second author of this paper). The reviewer examined each paper in detail and identified for each one:

 The attribute of usability that could be measured through the collected metrics;

 The focus of the research presented.

 The type of study conducted;

To ensure the quality of the data extraction performed the first and second author independently reviewed a 10% sample and compared these results. When a disagreement arose it was discussed until an agreement was reached.

Twenty papers that were identified as being relevant did not contain any formal evaluations of the proposed technologies. The results presented below exclude these 20 papers. In addition to this some papers presented multiple studies. In these cases each study was considered independently and so the results based on the number of studies within the evaluated papers rather than the number of papers.

Limitations

This literature review is limited for a number of reasons. Firstly a small number of papers were unavailable to the researchers (8 out of 139 papers considered relevant). This unavailability of less than 6% of the papers probably does not have a large impact on the results presented. By omitting certain sources from the study a bias may have been introduced. We felt that the range of sources considered was a fair representation of the field of usability of mobile applications although some outlying studies may have been omitted due to limited resources. Our reviews of these sources led us to believe that the omitted papers were of borderline significance. Ethical approval for this research was given by Oxford Brookes University Research Ethics Committee.

Research questions

To evaluate the PACMAD usability model three Research Questions (RQ1 to RQ3) were established to determine how important each of the factors and attributes of usability are in the context of mobile applications.

RQ1: What attributes are used when considering the usability of mobile applications?

This research question was established to discover what attributes are typically used to analyse mobile applications and which metrics are associated with them. The answers to this question provide evidence and data for the PACMAD usability model.

RQ2: To what extent are the factors of usability considered in existing research?

In order to determine how research in mobile applications is evolving, RQ2 was established to examine the current research trends into mobile applications, with a particular focus on the factors that affect usability.

In addition to this we wanted to establish which research methods are most commonly used when evaluating mobile applications. For this reason, a third research question was established.

RQ3: What research methodologies are used to evaluate the usability of mobile applications?

There are many ways in which mobile applications can be evaluated including controlled studies, field studies, ethnography, experiments, case-studies, surveys, etc. This research question aims to identify the most common research methodologies used to evaluate mobile apps. The answers to this question will throw light on the maturity of the mobile app engineering field.

The above research questions were answered by examining the literature on mobile applications. The range of literature on the topic of mobile applications is so broad it was important to limit the literature review to the most relevant and recent publications and to limit the publication interval to papers published between 2008 and 2010.

Table  4 shows the percentage of studies that include metrics, such as time to complete a given task, which either directly or indirectly assesses the attributes of usability included within the PACMAD usability model. In some cases the studies evaluated multiple attributes of usability and therefore the results above present both the percentage and the number of studies in which each attribute was considered. These studies often do not explicitly cite usability or any usability related criteria, and so the metrics used for the papers’ analyses were used to discover the usability attributes considered. This lack of precision is probably due to a lack of agreement as to what constitutes usability and the fact that the attributes are not orthogonal. The three most common attributes, Effectiveness, Efficiency and Satisfaction, correspond to the attributes identified by the ISO’s standard for usability.

One of the reasons these attributes are so widely considered is their direct relationship to the technical capabilities of the system. Both Effectiveness and Efficiency are related to the design and implementation of the system and so are usually tested thoroughly. These attributes are also relatively easy to measure. In most cases the Effectiveness of the system is evaluated by monitoring whether a user can accomplish a pre-specified task. Efficiency can be measured by finding the time taken by the participant to complete this task. Questionnaires and structured interviews can be used to determine the Satisfaction of users towards the system. Approximately 22% of the papers reviewed evaluated all three of these attributes.

The focus on these attributes of usability implies that Learnability, Memorability, Errors, and Cognitive load, are considered to be of less importance than Effectiveness, Efficiency and Satisfaction. Learnability, Memorability, Errors, and Cognitive load are not easy to evaluate and this may be why their assessment is often overlooked. As technology matures designers have begun to consider usability earlier in the design process. This is reflected to a certain extent by technological changes away from command line towards GUI based interfaces.

The aspects of usability that were considered least often in the papers reviewed are Learnability and Memorability. There are numerous reasons for this. The nature of these attributes demands that they are evaluated over periods of time. To effectively measure Learnability, users’ progress needs to be checked at regular intervals or tracked over many completions of a task. In the papers reviewed, Learnability was usually measured indirectly by the changes in effectiveness or efficiency over many completions of a specified task.

Memorability was only measured subjectively in the papers reviewed. One way to objectively measure Memorability is to examine participants’ use of the system after a period of inactivity with the system. The practical problem of recruiting participants who are willing to return multiple times to participate in an evaluation is probably one of the reasons why this attribute is not often measured objectively.

What differentiates mobile applications from more traditional applications is the ability of the user to use the application while moving. In this context, the users’ attention is divided between the act of moving and using the application. About 26% of the studies considered cognitive load. Some of these studies used the change in performance of the user performing the primary task (which was usually walking or driving) as an indication of the cognitive load. Other studies used the NASA TLX [ 11 ] to subjectively measure cognitive load.

Table  5 shows the current research trends within mobile application research. It can be seen that the majority of work is focused on a task approximately 47% of the papers reviewed focus on allowing users to complete a specific task. The range of tasks considered is too broad to provide a detailed description and so we present here only some of the most dominant trends seen within the literature review.

The integration of cameras into mobile devices has enabled the emergence of a new class of application for mobile devices known as augmented reality. For example Bruns and Bimber [ 14 ] have developed an augmented reality application which allows users to take a photograph of an exhibit at an art gallery which allows the system to find additional information about the work of art. Similar systems have also been developed for Points of Interest (POIs) for tourists [ 15 ].

While using maps is a traditional way of navigating to a destination, mobile devices incorporating GPS (Global Positioning Satellite) technology have enabled researchers to investigate new ways of helping users to navigate. A number of systems [ 16 , 17 ] have proposed the use of tactile feedback to help guide users. Through the use of different vibration techniques the system informs users whether they should turn left, right or keep going straight. Another alternative to this is the use of sound. By altering the spatial balance and volume of a user’s music, Jones et al. [ 18 ] have developed a system for helping guide users to their destination.

One of the biggest limitations to mobile devices is the limited input modalities. Developers of apps do not have a large amount of space for physical buttons and therefore researchers are investigating other methods of interaction. This type of research accounts for approximately 29% of the studies reviewed.

The small screen size found on mobile applications has meant that only a small fraction of a document can be seen in detail. When mobile devices are used navigating between locations, this restriction can cause difficulty for users. In an effort to address this issue Burigat et al. [ 19 ] have developed a Zoomable User Interface with Overview (ZUIO). This interface allows a user to zoom into small sections of a document, such as a map, while displaying a small scale overview of the entire document so that the user can see where on the overall document they are. This type of system can also be used with large documents, such as web pages and images.

Audio interfaces [ 20 ] are a type of interface that is being investigated to assist drivers to use in-car systems. Traditional interfaces present information to users by visual means, but for drivers this distraction has safety critical implications. To address this issue audio inputs are common for in-vehicle systems. The low quality of voice recognition technology can limit its effectiveness within this context. Weinberg et al. [ 21 ] have shown that multiple push-to-talk buttons can improve the performance of users of such systems. Other types of interaction paradigms in these papers include touch screens [ 22 ], pressure based input [ 23 ], spatial awareness [ 24 ] and gestures [ 25 ]. As well as using these new input modalities a number of researchers are also looking at alternative output modes such as sound [ 26 ] and tactile feedback [ 27 ].

In addition to considering the specific tasks and input modalities, a small number of researchers are investigating ways to assist specific types of users, such as those suffering from physical or psychological disabilities, to complete common tasks. This type of research accounts for approximately 9% of the evaluated papers. Approximately 8% of the papers evaluated have focused on the context in which mobile applications are being used. The remaining 6% of studies are concerned with new development and evaluation methodologies for mobile applications. These include rapid prototyping tools for in-car systems, the effectiveness of expert evaluations and the use of heuristics for evaluating mobile haptic interfaces.

RQ3 was posed to investigate how usability evaluations are currently conducted. The literature review revealed that 7 of the papers evaluated did not contain any usability evaluations. Some of the remaining papers included multiple studies to evaluate different aspects of a technology or were conducted at different times during the development process. Table  6 shows the percentage of studies that were conducted using each research methodology.

By far the most dominant research methodology used in the examined studies was controlled experiments, accounting for approximately 59% of the studies. In a controlled experiment, all variables are held constant except the independent variable, which is manipulated by the experimenter. The dependant variable is the metric which is measured by the experimenter. In this way a cause and effect relationship may be investigated between the dependant and independent variables. Causality can be inferred from the covariation of the independent and dependent variables, temporal precedence of the cause as the manipulation of the independent variable and the elimination of confounding factors though control and internal validity tests.

Although the most common approach is the use of controlled experiments, other research methodologies were also used. A number of studies evaluated the use of new technologies through field studies. Field studies are conducted in a real world context, enabling evaluators to determine how users would use a technology outside of a controlled setting. These studies often revealed issues that would not be seen in a controlled setting.

For example a system designed by Kristoffersen and Bratteberg [ 28 ] to help travellers get to and from an airport by train without the use of paper tickets was deployed. This system used a credit card as a form of ticket for a journey to or from the airport. During the field study a number of usability issues were experienced by travellers. One user wanted to use a card to buy a ticket for himself and a companion; the system did not include this functionality as the developers of the system had assumed each user would have their own credit card and therefore designed the system to issue each ticket on a different credit card.

The evaluation also revealed issues relating to how the developers had implemented the different journey types, i.e. to and from the airport. When travelling to the airport users are required to swipe their credit card at the beginning and end of each journey, whereas when returning from the airport the user only needs to swipe their card when leaving the airport. One user found this out after he had swiped his card to terminate a journey from the airport, but was instead charged for a second ticket to the airport.

Although controlled experiments and field studies account for almost 90% of the studies, other strategies are also used. Surveys were used to better understand how the public reacted to mobile systems. Some of these studies were specific to a new technology or paradigm, [ 29 ] while others considered uses such as working while on the move [ 30 ]. In two cases (1% of the studies) archival research was used to investigate a particular phenomena relating to mobile technologies. A study conducted by Fehnert and Kosagowsky [ 31 ] used archival research to investigate the relationship between expert evaluations of user experience quality of mobile phones and subsequent usage figures. Lacroix et al. [ 32 ] used archival research to investigate the relationship between goal difficulty and performance within the context of an on-going activity intervention program.

In some cases it was found that no formal evaluation was conducted but instead the new technology presented in the paper was evaluated informally with colleagues of the developers. These evaluations typically contained a small number of participants and provide anecdotal evidence of a system’s usability.

The results obtained during the literature review reinforced the importance of cognitive load as an attribute of usability. It was found that almost 23% of the studies measured the cognitive load of the application under evaluation. These results show that current researchers in the area of mobile applications are beginning to recognise the importance of cognitive load in this domain and as such there is sufficient evidence for including it within the PACMAD model of usability.

The results also show that Memorability is not considered an important aspect of usability by many researchers. Only 2% of the studies evaluated Memorability. If an application is easy to learn then users may be willing to relearn how to use the application and therefore Memorability may indeed not be significant. On the other hand, some applications have a high learning curve and as such require a significant amount of time to learn. For these applications Memorability is an important attribute.

The trade-off between Learnability and Memorability is a consideration for application developers. Factors such as the task to be accomplished and the characteristics of the user should be considered when making this decision. The PACMAD model recommends that both factors should be considered although it also recognises that it may be adequate to evaluate only one of these factors depending on the application under evaluation. The literature review has also shown that the remaining attributes of usability are considered extensively by current research. Effectiveness, Efficiency and Satisfaction were included in over 50% of the studies. It was also found the Errors were evaluated in over 30% of these studies.

When considering the factors that can affect usability, it was found that the task is the most dominant factor being researched. Over 45% of the papers examined focused primarily on allowing a user to accomplish a task. When the interaction with an application is itself considered as a task this figure rises to approximately 75%. Context of use and the User were considered in less than 10% of the papers. Context of use can vary enormously and so should be considered an important factor of usability [ 5 , 33 ]. Our results indicate that context is not extensively researched and this suggests a gap in the literature.

It was revealing that some components of the PACMAD model occur only infrequently in the literature. As mentioned above Learnability and Memorability are rarely investigated, perhaps suggesting that researchers expected users to be able to learn to use apps without much difficulty., This finding could also be due to the difficulty of finding suitable subjects willing to undergo experiments on these attributes or the lack of standard research methods for these attributes. Effectiveness, Efficiency, Satisfaction and Errors were investigated more frequently, possibly because these attributes are widely recognised as important, and also possibly because research methods for investigating these attributes are well understood and documented. Almost a quarter of the studies investigated discussed Cognitive Load. It is surprising that this figure is not higher although this could again be due to the lack of a well-defined research methodology for investigating this attribute.

Conclusions

The range and availability of mobile applications is expanding rapidly. With the increased processing power available on portable devices, developers are increasing the range of services that they provide. The small size of mobile devices has limited the ways in which users can interact with them. Issues such as the small screen size, poor connectivity and limited input modalities have an effect on the usability of mobile applications.

The prominent models of usability do not adequately capture the complexities of interacting with applications on a mobile platform. For this reason, this paper presents our PACMAD usability model which augments existing usability models within the context of mobile applications.

To prove the concept of this model a literature review has been conducted. This review has highlighted the extent to which the attributes of the PACMAD model are considered within the mobile application domain. It was found that each attribute was considered in at least 20% of studies, with the exception of Memorability. It is believed one reason for this may be the difficulty associated with evaluating Memorability.

The literature review has also revealed a number of novel interaction methods that are being researched at present, such as spatial awareness and pressure based input. These techniques are in their infancy but with time and more research they may eventually be adopted.

Appendix A: Papers used in the literature review

Apitz, G., F. Guimbretière, and S. Zhai, Foundations for designing and evaluating user interfaces based on the crossing paradigm. ACM Trans. Comput.-Hum. Interact., 2008. 17(2): p. 1–42.

Arning, K. and M. Ziefle, Ask and You Will Receive: Training Novice Adults to use a PDA in an Active Learning Environment. International Journal of Mobile Human Computer Interaction (IJMHCI), 2010. 2(1): p. 21–47.

Arvanitis, T.N., et al., Human factors and qualitative pedagogical evaluation of a mobile augmented reality system for science education used by learners with physical disabilities. Personal Ubiquitous Comput., 2009. 13(3): p. 243–250.

Axtell, C., D. Hislop, and S. Whittaker, Mobile technologies in mobile spaces: Findings from the context of train travel. Int. J. Hum.-Comput. Stud., 2008. 66(12): p. 902–915.

Baber, C., et al., Mobile technology for crime scene examination. Int. J. Hum.-Comput. Stud., 2009. 67(5): p. 464–474.

Bardram, J.E., Activity-based computing for medical work in hospitals. ACM Trans. Comput.-Hum. Interact., 2009. 16(2): p. 1–36.

Bergman, J., J. Kauko, and J. Keränen, Hands on music: physical approach to interaction with digital music, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Bergman, J. and J. Vainio, Interacting with the flow, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Bertini, E., et al., Appropriating Heuristic Evaluation for Mobile Computing International Journal of Mobile Human Computer Interaction (IJMHCI), 2009. 1(1): p. 20–41.

Böhmer, M. and G. Bauer, Exploiting the icon arrangement on mobile devices as information source for context-awareness, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Bostr, F., et al., Capricorn - an intelligent user interface for mobile widgets, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Brewster, S.A. and M. Hughes, Pressure-based text entry for mobile devices, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Bruns, E. and O. Bimber, Adaptive training of video sets for image recognition on mobile phones. Personal Ubiquitous Comput., 2009. 13(2): p. 165–178.

Brush, A.J.B., et al., User experiences with activity-based navigation on mobile devices, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Burigat, S., L. Chittaro, and S. Gabrielli, Navigation techniques for small-screen devices: An evaluation on maps and web pages. Int. J. Hum.-Comput. Stud., 2008. 66(2): p. 78–97.

Büring, T., J. Gerken, and H. Reiterer, Zoom interaction design for pen-operated portable devices. Int. J. Hum.-Comput. Stud., 2008. 66(8): p. 605–627.

Buttussi, F., et al., Using mobile devices to support communication between emergency medical responders and deaf people, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Chen, N.Y., F. Guimbretière, and C.E. Löckenhoff, Relative role of merging and two-handed operation on command selection speed. Int. J. Hum.-Comput. Stud., 2008. 66(10): p. 729–740.

Chen, T., Y. Yesilada, and S. Harper, What input errors do you experience? Typing and pointing errors of mobile Web users. Int. J. Hum.-Comput. Stud., 2010. 68(3): p. 138–157.

Cherubini, M., et al., Text versus speech: a comparison of tagging input modalities for camera phones, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Chittaro, L. and A. Marassi, Supporting blind users in selecting from very long lists of items on mobile phones, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Chittaro, L. and D. Nadalutti, Presenting evacuation instructions on mobile devices by means of location-aware 3D virtual environments, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Clawson, J., et al., Mobiphos: a collocated-synchronous mobile photo sharing application, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Cockburn, A. and C. Gutwin, A model of novice and expert navigation performance in constrained-input interfaces. ACM Trans. Comput.-Hum. Interact., 2010. 17(3): p. 1–38.

Cox, A.L., et al., Tlk or txt? Using voice input for SMS composition. Personal Ubiquitous Comput., 2008. 12(8): p. 567–588.

Crossan, A., et al., Instrumented Usability Analysis for Mobile Devices International Journal of Mobile Human Computer Interaction (IJMHCI), 2009. 1(1): p. 1–19.

Cui, Y., et al., Linked internet UI: a mobile user interface optimized for social networking, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Cummings, M.L., et al., Supporting intelligent and trustworthy maritime path planning decisions. Int. J. Hum.-Comput. Stud., 2010. 68(10): p. 616–626.

Dahl, Y. and D. Svan, A comparison of location and token-based interaction techniques for point-of-care access to medical information. Personal Ubiquitous Comput., 2008. 12(6): p. 459–478.

Dai, L., A. Sears, and R. Goldman, Shifting the focus from accuracy to recallability: A study of informal note-taking on mobile information technologies. ACM Trans. Comput.-Hum. Interact., 2009. 16(1): p. 1–46.

Decle, F. and M. Hachet, A study of direct versus planned 3D camera manipulation on touch-based mobile phones, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Duh, H.B.-L., V.H.H. Chen, and C.B. Tan, Playing different games on different phones: an empirical study on mobile gaming, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Dunlop, M.D. and M.M. Masters, Investigating five key predictive text entry with combined distance and keystroke modelling. Personal Ubiquitous Comput., 2008. 12(8): p. 589–598.

Ecker, R., et al., pieTouch: a direct touch gesture interface for interacting with in-vehicle information systems, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Eslambolchilar, P. and R. Murray-Smith, Control centric approach in designing scrolling and zooming user interfaces. Int. J. Hum.-Comput. Stud., 2008. 66(12): p. 838–856.

Fehnert, B. and A. Kosagowsky, Measuring user experience: complementing qualitative and quantitative assessment, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Fickas, S., M. Sohlberg, and P.-F. Hung, Route-following assistance for travelers with cognitive impairments: A comparison of four prompt modes. Int. J. Hum.-Comput. Stud., 2008. 66(12): p. 876–888.

Froehlich, P., et al., Exploring the design space of Smart Horizons, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Gellersen, H., et al., Supporting device discovery and spontaneous interaction with spatial references. Personal Ubiquitous Comput., 2009. 13(4): p. 255–264.

Ghiani, G., B. Leporini, and F. Patern, Vibrotactile feedback as an orientation aid for blind users of mobile guides, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Gostner, R., E. Rukzio, and H. Gellersen, Usage of spatial information for selection of co-located devices, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Goussevskaia, O., M. Kuhn, and R. Wattenhofer, Exploring music collections on mobile devices, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Greaves, A. and E. Rukzio, Evaluation of picture browsing using a projector phone, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Hachet, M., et al., Navidget for 3D interaction: Camera positioning and further uses. Int. J. Hum.-Comput. Stud., 2009. 67(3): p. 225–236.

Hall, M., E. Hoggan, and S. Brewster, T-Bars: towards tactile user interfaces for mobile touchscreens, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Hang, A., E. Rukzio, and A. Greaves, Projector phone: a study of using mobile phones with integrated projector for interaction with maps, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Hardy, R., et al., Mobile interaction with static and dynamic NFC-based displays, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Heikkinen, J., T. Olsson, and K. Väänänen-Vainio-Mattila, Expectations for user experience in haptic communication with mobile devices, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Henze, N. and S. Boll, Evaluation of an off-screen visualization for magic lens and dynamic peephole interfaces, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Herbst, I., et al., TimeWarp: interactive time travel with a mobile mixed reality game, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Hinze, A.M., C. Chang, and D.M. Nichols, Contextual queries express mobile information needs, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Hutter, H.-P., T. Müggler, and U. Jung, Augmented mobile tagging, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Jones, M., et al., ONTRACK: Dynamically adapting music playback to support navigation. Personal Ubiquitous Comput., 2008. 12(7): p. 513–525.

Joshi, A., et al., Rangoli: a visual phonebook for low-literate users, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Jumisko-Pyykk, S. and M.M. Hannuksela, Does context matter in quality evaluation of mobile television?, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Kaasinen, E., User Acceptance of Mobile Services. International Journal of Mobile Human Computer Interaction (IJMHCI), 2009. 1(1): p. 79–97 pp.

Kaasinen, E., et al., User Experience of Mobile Internet: Analysis and Recommendations. International Journal of Mobile Human Computer Interaction (IJMHCI), 2009. 1(4): p. 4–23.

Kane, S.K., J.O. Wobbrock, and I.E. Smith, Getting off the treadmill: evaluating walking user interfaces for mobile devices in public spaces, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Kang, N.E. and W.C. Yoon, Age- and experience-related user behavior differences in the use of complicated electronic devices. Int. J. Hum.-Comput. Stud., 2008. 66(6): p. 425–437.

Kanjo, E., et al., MobGeoSen: facilitating personal geosensor data collection and visualization using mobile phones. Personal Ubiquitous Comput., 2008. 12(8): p. 599–607.

Kawsar, F., E. Rukzio, and G. Kortuem, An explorative comparison of magic lens and personal projection for interacting with smart objects, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Keijzers, J., E.d. Ouden, and Y. Lu, Usability benchmark study of commercially available smart phones: cell phone type platform, PDA type platform and PC type platform, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Kenteris, M., D. Gavalas, and D. Economou, An innovative mobile electronic tourist guide application. Personal Ubiquitous Comput., 2009. 13(2): p. 103–118.

Komninos, A. and M.D. Dunlop, A calendar based Internet content pre-caching agent for small computing devices. Personal Ubiquitous Comput., 2008. 12(7): p. 495–512.

Kratz, S., I. Brodien, and M. Rohs, Semi-automatic zooming for mobile map navigation, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Kray, C., et al., Bridging the gap between the Kodak and the Flickr generations: A novel interaction technique for collocated photo sharing. Int. J. Hum.-Comput. Stud., 2009. 67(12): p. 1060–1072.

Kristoffersen, S. and I. Bratteberg, Design ideas for IT in public spaces. Personal Ubiquitous Comput., 2010. 14(3): p. 271–286.

Lacroix, J., P. Saini, and R. Holmes, The relationship between goal difficulty and performance in the context of a physical activity intervention program, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Lavie, T. and J. Meyer, Benefits and costs of adaptive user interfaces. Int. J. Hum.-Comput. Stud., 2010. 68(8): p. 508–524.

Lee, J., J. Forlizzi, and S.E. Hudson, Iterative design of MOVE: A situationally appropriate vehicle navigation system. Int. J. Hum.-Comput. Stud., 2008. 66(3): p. 198–215.

Liao, C., et al., Papiercraft: A gesture-based command system for interactive paper. ACM Trans. Comput.-Hum. Interact., 2008. 14(4): p. 1–27.

Lin, P.-C. and L.-W. Chien, The effects of gender differences on operational performance and satisfaction with car navigation systems. Int. J. Hum.-Comput. Stud., 2010. 68(10): p. 777–787.

Lindley, S.E., et al., Fixed in time and “time in motion”: mobility of vision through a SenseCam lens, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Liu, K. and R.A. Reimer, Social playlist: enabling touch points and enriching ongoing relationships through collaborative mobile music listening, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Liu, N., Y. Liu, and X. Wang, Data logging plus e-diary: towards an online evaluation approach of mobile service field trial, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Liu, Y. and K.-J. Räihä, RotaTxt: Chinese pinyin input with a rotator, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Lucero, A., J. Keränen, and K. Hannu, Collaborative use of mobile phones for brainstorming, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Luff, P., et al., Swiping paper: the second hand, mundane artifacts, gesture and collaboration. Personal Ubiquitous Comput., 2010. 14(3): p. 287–299.

Mallat, N., et al., An empirical investigation of mobile ticketing service adoption in public transportation. Personal Ubiquitous Comput., 2008. 12(1): p. 57–65.

McAdam, C., C. Pinkerton, and S.A. Brewster, Novel interfaces for digital cameras and camera phones, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

McDonald, D.W., et al., Proactive displays: Supporting awareness in fluid social environments. ACM Trans. Comput.-Hum. Interact., 2008. 14(4): p. 1–31.

McKnight, L. and B. Cassidy, Children’s Interaction with Mobile Touch-Screen Devices: Experiences and Guidelines for Design. International Journal of Mobile Human Computer Interaction (IJMHCI), 2010. 2(2): p. 1–18.

Melto, A., et al., Evaluation of predictive text and speech inputs in a multimodal mobile route guidance application, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Miyaki, T. and J. Rekimoto, GraspZoom: zooming and scrolling control model for single-handed mobile interaction, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Moustakas, K., et al., 3D content-based search using sketches. Personal Ubiquitous Comput., 2009. 13(1): p. 59–67.

Oakley, I. and J. Park, Motion marking menus: An eyes-free approach to motion input for handheld devices. Int. J. Hum.-Comput. Stud., 2009. 67(6): p. 515–532.

Oulasvirta, A., Designing mobile awareness cues, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Oulasvirta, A., S. Estlander, and A. Nurminen, Embodied interaction with a 3D versus 2D mobile map. Personal Ubiquitous Comput., 2009. 13(4): p. 303–320.

Ozok, A.A., et al., A Comparative Study Between Tablet and Laptop PCs: User Satisfaction and Preferences. International Journal of Human-Computer Interaction, 2008. 24(3): p. 329–352.

Park, Y.S., et al., Touch key design for target selection on a mobile phone, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Peevers, G., G. Douglas, and M.A. Jack, A usability comparison of three alternative message formats for an SMS banking service. Int. J. Hum.-Comput. Stud., 2008. 66(2): p. 113–123.

Preuveneers, D. and Y. Berbers, Mobile phones assisting with health self-care: a diabetes case study, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Puikkonen, A., et al., Practices in creating videos with mobile phones, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Reischach, F.v., et al., An evaluation of product review modalities for mobile phones, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Reitmaier, T., N.J. Bidwell, and G. Marsden, Field testing mobile digital storytelling software in rural Kenya, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Robinson, S., P. Eslambolchilar, and M. Jones, Exploring casual point-and-tilt interactions for mobile geo-blogging. Personal and Ubiquitous Computing, 2010. 14(4): p. 363–379.

Rogers, Y., et al., Enhancing learning: a study of how mobile devices can facilitate sensemaking. Personal Ubiquitous Comput., 2010. 14(2): p. 111–124.

Rohs, M., et al., Impact of item density on the utility of visual context in magic lens interactions. Personal Ubiquitous Comput., 2009. 13(8): p. 633–646.

Sá, M.d. and L. Carriço, Lessons from early stages design of mobile applications, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Sadeh, N., et al., Understanding and capturing people’s privacy policies in a mobile social networking application. Personal Ubiquitous Comput., 2009. 13(6): p. 401–412.

Salvucci, D.D., Rapid prototyping and evaluation of in-vehicle interfaces. ACM Trans. Comput.-Hum. Interact., 2009. 16(2): p. 1–33.

Salzmann, C., D. Gillet, and P. Mullhaupt, End-to-end adaptation scheme for ubiquitous remote experimentation. Personal Ubiquitous Comput., 2009. 13(3): p. 181–196.

Schildbach, B. and E. Rukzio, Investigating selection and reading performance on a mobile phone while walking, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Schmid, F., et al., Situated local and global orientation in mobile you-are-here maps, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Schröder, S. and M. Ziefle, Making a completely icon-based menu in mobile devices to become true: a user-centered design approach for its development, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Scott, J., et al., RearType: text entry using keys on the back of a device, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Seongil, L., Mobile Internet Services from Consumers’ Perspectives. International Journal of Human-Computer Interaction, 2009. 25(5): p. 390–413.

Sharlin, E., et al., A tangible user interface for assessing cognitive mapping ability. Int. J. Hum.-Comput. Stud., 2009. 67(3): p. 269–278.

Sintoris, C., et al., MuseumScrabble: Design of a Mobile Game for Children’s Interaction with a Digitally Augmented Cultural Space. International Journal of Mobile Human Computer Interaction (IJMHCI), 2010. 2(2): p. 53–71.

Smets, N.J.J.M., et al., Effects of mobile map orientation and tactile feedback on navigation speed and situation awareness, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Sodnik, J., et al., A user study of auditory versus visual interfaces for use while driving. Int. J. Hum.-Comput. Stud., 2008. 66(5): p. 318–332.

Sørensen, C. and A. Al-Taitoon, Organisational usability of mobile computing-Volatility and control in mobile foreign exchange trading. Int. J. Hum.-Comput. Stud., 2008. 66(12): p. 916–929.

Stapel, J.C., Y.A.W.d. Kort, and W.A. IJsselsteijn, Sharing places: testing psychological effects of location cueing frequency and explicit vs. inferred closeness, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Streefkerk, J.W., M.P.v. Esch-Bussemakers, and M.A. Neerincx, Field evaluation of a mobile location-based notification system for police officers, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Takayama, L. and C. Nass, Driver safety and information from afar: An experimental driving simulator study of wireless vs. in-car information services. Int. J. Hum.-Comput. Stud., 2008. 66(3): p. 173–184.

Takeuchi, Y. and M. Sugimoto, A user-adaptive city guide system with an unobtrusive navigation interface. Personal Ubiquitous Comput., 2009. 13(2): p. 119–132.

Tan, F.B. and J.P.C. Chou, The Relationship Between Mobile Service Quality, Perceived Technology Compatibility, and Users’ Perceived Playfulness in the Context of Mobile Information and Entertainment Services. International Journal of Human-Computer Interaction, 2008. 24(7): p. 649–671.

Taylor, C.A., N. Samuels, and J.A. Ramey, Always On: A Framework for Understanding Personal Mobile Web Motivations, Behaviors, and Contexts of Use. International Journal of Mobile Human Computer Interaction (IJMHCI), 2009. 1(4): p. 24–41.

Turunen, M., et al., User expectations and user experience with different modalities in a mobile phone controlled home entertainment system, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Vartiainen, E., Improving the User Experience of a Mobile Photo Gallery by Supporting Social Interaction International Journal of Mobile Human Computer Interaction (IJMHCI), 2009. 1(4): p. 42–57.

Vuolle, M., et al., Developing a questionnaire for measuring mobile business service experience, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Weinberg, G., et al., Contextual push-to-talk: shortening voice dialogs to improve driving performance, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Wilson, G., C. Stewart, and S.A. Brewster, Pressure-based menu selection for mobile devices, in Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. 2010, ACM: Lisbon, Portugal.

Wobbrock, J.O., B.A. Myers, and H.H. Aung, The performance of hand postures in front- and back-of-device interaction for mobile computing. Int. J. Hum.-Comput. Stud., 2008. 66(12): p. 857–875.

Xiangshi, R. and Z. Xiaolei, The Optimal Size of Handwriting Character Input Boxes on PDAs. International Journal of Human-Computer Interaction, 2009. 25(8): p. 762–784.

Xu, S., et al., Development of a Dual-Modal Presentation of Texts for Small Screens. International Journal of Human-Computer Interaction, 2008. 24(8): p. 776–793.

Yong, G.J. and J.B. Suk, Development of the Conceptual Prototype for Haptic Interface on the Telematics System. International Journal of Human-Computer Interaction, 2010. 26(1): p. 22–52.

Yoo, J.-W., et al., Cocktail: Exploiting Bartenders’ Gestures for Mobile Interaction. International Journal of Mobile Human Computer Interaction (IJMHCI), 2010. 2(3): p. 44–57.

Yoon, Y., et al., Context-aware photo selection for promoting photo consumption on a mobile phone, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

You, Y., et al., Deploying and evaluating a mixed reality mobile treasure hunt: Snap2Play, in Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. 2008, ACM: Amsterdam, The Netherlands.

Yu, K., F. Tian, and K. Wang, Coupa: operation with pen linking on mobile devices, in Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 2009, ACM: Bonn, Germany.

Authors’ note

This research is supported by Oxford Brookes University through the central research fund and in part by Lero - the Irish Software Engineering Research Centre ( http://www.lero.ie ) grant 10/CE/I1855.

Adams R: Decision and stress: cognition and e-accessibility in the information workplace. Springer Universal Access in the Information Society 2007, 5 (4):363–379. 10.1007/s10209-006-0061-9

Article   Google Scholar  

Adams R: Applying advanced concepts of cognitive overload and augmentation in practice; the future of overload. In Foundations of augmented cognition . 2nd edition. Edited by: Schmorrow D, Stanney KM, Reeves LM. Arlington, VA: Springer Berlin Heidelberg; 2006:223–229.

Google Scholar  

Kjeldskov J, Graham C: A review of mobile HCI research methods . Udine, Italy: 5th International Symposium, Mobile HCI 2003; 2003. September 8–11, 2003, Proceedings

Book   Google Scholar  

Nielsen J: Usability engineering. Morgan Kaufmann Pub 1994.

ISO 9241: Ergonomics Requirements for Office Work with Visual Display Terminals (VDTs) International Standards Organisation, Geneva 1997.

Zhang D, Adipat B: Challenges, methodologies, and issues in the usability testing of mobile applications. International Journal of Human-Computer Interaction 2005, 18 (3):293–308. 10.1207/s15327590ijhc1803_3

Guerreiro TJV, Nicolau H, Jorge J, Gonçalves D Proceedings of the 12th international conference on Human computer interaction with mobile devices and services. In Assessing mobile touch interfaces for tetraplegics . Lisbon, Portugal: ACM; 2010. 2010

Chapter   Google Scholar  

Schildbach B, Rukzio E Proceedings of the 12th international conference on human computer interaction with mobile devices and services. In Investigating selection and reading performance on a mobile phone while walking . Lisbon, Portugal: ACM; 2010. 2010

Flood D, Harrison R, Duce D, Iacob C: Evaluating Mobile Applications: A Spreadsheet Case Study. International Journal of Mobile Human Computer Interaction (IJMHCI) 2013, 4 (4):37–65. 10.4018/jmhci.2012100103

Salvucci DD: Predicting the effects of in-car interface use on driver performance: an integrated model approach. International Journal of Human-Computer Studies 2001, 55 (1):85–107. 10.1006/ijhc.2001.0472

Hart SG, Staveland LE: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human mental workload 1988, 1 (3):139–183.

Flood D, Germanakos P, Harrison R, Mc Caffery F: Estimating cognitive overload in mobile applications for decision support within the medical domain . Wroclaw, Poland: 14th International conference on Enterprise Information Systems (ICEIS 2012); 2012.

Budgen D, Burn AJ, Brereton OP, Kitchenham BA, Pretorius R: (2010) Empirical evidence about the UML: a systematic literature review . Software: Practice and Experience; 2010.

Bruns E, Bimber O: Adaptive training of video sets for image recognition on mobile phones. Personal Ubiquitous Comput 2009, 13 (2):165–178. 10.1007/s00779-008-0194-3

Schinke T, Henze N, Boll S Proceedings of the 12th international conference on human computer interaction with mobile devices and services, September 07–10, 2010. In Visualization of off-screen objects in mobile augmented reality . Portugal: Lisbon; 2010.

Smets NJJM, Brake GM, Neerincx MA, Lindenberg J Proceedings of the 10th international conference on human computer interaction with mobile devices and services. In Effects of mobile map orientation and tactile feedback on navigation speed and situation awareness . Amsterdam, The Netherlands: ACM; 2008.

Ghiani G, Leporini B, Patern F Proceedings of the 10th international conference on human computer interaction with mobile devices and services. In Vibrotactile feedback as an orientation aid for blind users of mobile guides . Amsterdam, The Netherlands: ACM; 2008.

Jones M, Jones S, Bradley G, Warren N, Bainbridge D, Holmes G: ONTRACK: Dynamically adapting music playback to support navigation. Personal Ubiquitous Computing 2008, 12 (7):513–525. 10.1007/s00779-007-0155-2

Burigat, S, Chittaro, L, Parlato, E, ACM In proceedings of the 10th international conference on Human computer interaction with mobile devices and services (pp. 147–156). Map, diagram, and web page navigation on mobile devices: the effectiveness of zoomable user interfaces with overviews 2008. September

Sodnik J, Dicke C, Tomaic S, Billinghurst M: A user study of auditory versus visual interfaces for use while driving. Int. J. Hum.-Comput. Stud 2008, 66 (5):318–332. 10.1016/j.ijhcs.2007.11.001

Weinberg G, Harsham B, Forlines C, Medenica Z Proceedings of the 12th international conference on human computer interaction with mobile devices and services. In Contextual push-to-talk: shortening voice dialogs to improve driving performance . Lisbon, Portugal: ACM; 2010. 2010

Park YS, Han SH, Park J, Cho Y Proceedings of the 10th international conference on human computer interaction with mobile devices and services. In Touch key design for target selection on a mobile phone . Amsterdam, The Netherlands: ACM; 2008.

Brewster SA, Hughes M Proceedings of the 11th international conference on human-computer interaction with mobile devices and services. In Pressure-based text entry for mobile devices . Bonn, Germany: ACM; 2009.

Oakley I, Park J: Motion marking menus: an eyes-free approach to motion input for handheld devices. Int J Hum.-Comput. Stud 2009, 67 (6):515–532. 10.1016/j.ijhcs.2009.02.002

Hall M, Hoggan E, Brewster S Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. In T-Bars: towards tactile user interfaces for mobile touchscreens . Amsterdam, The Netherlands: ACM; 2008. 2008

McAdam C, Pinkerton C, Brewster SA Proceedings of the 12th international conference on human computer interaction with mobile devices and services. In Novel interfaces for digital cameras and camera phones . Lisbon, Portugal: ACM; 2010. 2010

Heikkinen J, Olsson T, Väänänen-Vainio-Mattila K Proceedings of the 11th international conference on human-computer interaction with mobile devices and services. In Expectations for user experience in haptic communication with mobile devices . Bonn, Germany: ACM; 2009.

Kristoffersen S, Bratteberg I: Design ideas for IT in public spaces. Personal Ubiquitous Comput 2010, 14 (3):271–286. 10.1007/s00779-009-0255-2

Mallat N, Rossi M, Tuunainen VK, Oörni A: An empirical investigation of mobile ticketing service adoption in public transportation. Personal Ubiquitous Comput 2008, 12 (1):57–65.

Axtell C, Hislop D, Whittaker S: Mobile technologies in mobile spaces: findings from the context of train travel. International Journal of Human Computer Studies 2008, 66 (12):902–915. 10.1016/j.ijhcs.2008.07.001

Fehnert B, Kosagowsky A Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. In Measuring user experience: complementing qualitative and quantitative assessment . Amsterdam, The Netherlands: ACM; 2008.

Lacroix J, Saini P, Holmes R Proceedings of the 10th international conference on Human computer interaction with mobile devices and services. In The relationship between goal difficulty and performance in the context of a physical activity intervention program, . Amsterdam, The Netherlands: ACM; 2008.

Maguire M: Context of use within usability activities. International Journal of Human-Computer Studies 2001, 55 (4):453–483. 2001 10.1006/ijhc.2001.0486

Download references

Author information

Authors and affiliations.

Oxford Brookes University, Oxford, UK

Rachel Harrison, Derek Flood & David Duce

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rachel Harrison .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

DF performed the literature review, helped to propose the PACMAD model and drafted the manuscript. RH assisted the literature review, proposed the PACMAD model and drafted the limitations section. DAD helped to refine the conceptual framework and direct the research. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, rights and permissions.

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Harrison, R., Flood, D. & Duce, D. Usability of mobile applications: literature review and rationale for a new usability model. J Interact Sci 1 , 1 (2013). https://doi.org/10.1186/2194-0827-1-1

Download citation

Received : 10 March 2013

Accepted : 10 March 2013

Published : 07 May 2013

DOI : https://doi.org/10.1186/2194-0827-1-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mobile Phone
• Mobile Device
• Cognitive Load
• Augmented Reality
• Usability Model

Mobile application development process: A practical experience

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Open access
• Published: 09 August 2021

The use of mobile applications in higher education classes: a comparative pilot study of the students’ perceptions and real usage

• David Manuel Duarte Oliveira   ORCID: orcid.org/0000-0002-8763-6997 1 ,
• Luís Pedro 1 &
• Carlos Santos 1  

Smart Learning Environments volume  8 , Article number:  14 ( 2021 ) Cite this article

22k Accesses

6 Citations

3 Altmetric

Metrics details

This paper was developed within the scope of a PhD thesis that intends to characterize the use of mobile applications by the students of the University of Aveiro during class time. The main purpose of this paper is to present the results of an initial pilot study that aimed to fine-tune data collection methods in order to gather data that reflected the practices of the use of mobile applications by students in a higher education institution during classes. In this paper we present the context of the pilot, its technological settings, the analysed cases and the discussion and conclusions carried out to gather mobile applications usage data logs from students of an undergraduate degree on Communication Technologies.

Our study gathered data from 77 participants, taking theoretical classes in the Department of Communication and Arts at the University of Aveiro. The research was based on the Grounded Theory method approach aiming to analyse the logs from the access points of the University. With the collected data, a profile of the use of mobile devices during classes was drawn.

The preliminary findings suggest that the use of apps during the theoretical classes of the Department of Communication and Art is quite high and that the most used apps are Social networks like Facebook and Instagram. During this pilot the accesses during theoretical classes corresponded to approximately 11,177 accesses per student. We also concluded that the students agree that accessing applications can distract them during these classes and that they have a misperception about their use of online applications during classes, as the usage time is, in fact, more intensive than what participants reported.

Introduction

The use of mobile devices by higher education students has grown in the last years (GMI, 2019 ). Technological advancements are also pushing society with consequent rapidly changing environments. Higher Education Institutions (HEIs) are not exempted from these technological changes and advancements, and it is compulsory that they follow this technological evolution so that the teaching-learning process is improved and enriched.

HEI’s are trying to integrate digital devices such as mobile phones and tablets, and informal learning situations, assuming that the use of these technologies may result in a different learning approach and increase students’ motivation and proficiency (Aagaard, 2015 ).

In a study by Magda, & Aslanian ( 2018 ), students report that they access course documents and communicate with the faculty via their mobile devices, such as smartphones. Over 40% say they perform searches for reports and access institutions E-Learning systems via mobile devices (Magda, & Aslanian, 2018 ). The EDUCAUSE Horizon Report - 2019 Higher Education Edition (Alexander et al., 2019 ) also mentions M-Learning as the main development in the use of technology in higher education. However, teachers believe students use their gadgets less than they actually do, and mobile devices also challenge teaching practices. Students use devices for off-task (Jesse, 2015 ) or parallel activities and there may be inaccurate references to their actual use of mobile devices.

Mobile device users have very different usage habits of their devices and their applications, and it is important to study and characterize these behaviours in different contexts, as explained below. The reports that usually support these studies are made with questions directed to the users themselves asking them questions about the apps they have on the devices and the reasons for using them. However, Gerpott & Thomas ( 2014 ) argue that other types of studies are needed to properly support this type of research.

Studies are usually conducted in organizations, based on the opinion of the participants, and cannot be replicated and generalized, for example, regarding the use of the internet or mobile applications by the general public, because these devices, unlike desktop devices, can be used anywhere and at any time (Gerpott & Thomas, 2014 ).

Furthermore, in mobile contexts, it becomes difficult for people to remember what they have used, because mobile applications can be used for various tasks, in various contexts, whether professional or personal, and the variety of applications, the use made, the periods of use are usually so wide and differentiated, that it can become difficult for users to refer which services or applications they have used, under which circumstances and how often. (Boase & Ling, 2013 ).

Thus, it is relevant, for several areas and especially for this research area, to have studies that cross-reference reported usage with actual usage. One of the ways to achieve this is with the use of logs of the use of mobile devices and applications, as mentioned by De Reuver & Bouwman ( 2015 ):

Using this approach this pilot study aims to create and validate a methodology:

i) to show the profile of these users,

ii) to reveal the kind of applications they use in the classroom and when they are in the institutions,

iii) and also, to compare the users’ perceptions with the real use of mobile applications.

Knowing the real usage and the usage students mention may provide valuable insights to teachers and HEIs and use this data for decision making about institutional applications to support students and teachers in their teaching and learning activities. Such information can also bring insights on the integration of M-Learning strategies, promoting interaction, communication, access to courses and the completion of assignments using students’ devices.

The central focus of this study is, therefore, to show preliminary results of the use of applications by students in class time during theoretical classes, through logs collected during class time.

The paper is divided into five parts. In the first part, relevant theoretical considerations are addressed, having in mind the current state of the art in terms of the literature and empirical work in this field. The second part outlines the study methodology. In the third part, the technological setting is highlighted. The cases and the results of the data analysis are described in the fourth part. Lastly, the results are interpreted, connected and crossed with the preliminary considerations.

Literature review

The massive use of mobile devices has created new forms of social interaction, significantly reducing the spatial difficulties that could exist, and today people can be reached and connected anytime and anywhere (Monteiro et al., 2017 ). This also applies to the school environment, where students bring small devices (smartphones, tablets and e-book readers) with them, which, thanks to easy access to an Internet connection, keep them permanently connected, even during classes.

In HEIs there is also a growing tendency among members of the academic community to use mobile devices in their daily activities (Oliveira et al., 2017 ), and students expect these devices to be an integral part of their academic tasks, too (Dobbin et al., 2011 ). A great number of users take advantage of mobile devices to search information and, since they do not always have computers available, these devices allow them an easy access to academic and institutional information (Vicente, 2013 ).

One of the challenges educational institutions face today has to do with the ubiquitous character of these devices and with finding ways in which they can be useful for learning, thus approaching a new educational paradigm: Mobile Learning (M-Learning) (Ryu & Parsons, 2008 ).

M-learning allows learning to take place in multiple places, in several ways and when the learner wants to learn. As learning does not necessarily have to occur within school buildings and schedules, M-Learning reduces the limitations of learning confined to the classroom (Sharples, M., Corlett, D. & Westmancott,  2002 ), leading UNESCO to consider that M-Learning, in fact, increases the reach of education and may promote equality in education (UNESCO, 2013 ). The EDUCAUSE Horizon Report - 2019 Higher Education Edition (Alexander et al., 2019 ) also mentions M-Learning as the main development in the use of technology in higher education and, therefore, it becomes increasingly relevant to rethink learning spaces in a more open perspective, both physically and methodologically, namely through mobile learning that places the student at the centre of the learning process.

Quite often studies that intend to determine the use of mobile applications focus on general questions, but the most common ones are related to the frequency and duration of the use of these devices, for example, questions such as “how many SMS or calls are made?” or “how often do you use the device?”

In fact, instruments like questionnaires are widely used in this type of studies. However, since mobile devices are completely integrated in our daily life and we use them quite extensively, it is difficult to retain and define with plausible accuracy the actual use that we make of them.

It is therefore relevant to effectively understand how these students use these devices, more specifically the applications installed on them. To this end, most studies have been based on designs that are focused on the users’ perceptions and based are on these reports.

Thus, it was important to understand if what users report using corresponds to what they actually use, and if this use does not occur for distraction or entertainment, for example.

Considering the above, some studies have focused on the validity of the use of these instruments. One of these first studies, carried out by Parslow et al. ( 2003 ), aimed at determining the number of calls made and received in the days, weeks or months preceding the date of the questionnaire, and their duration. The answers were compared with the logs of the operators and it was concluded that self-report questionnaires do not always represent the actual pattern of use.

Finally, in self-report instruments, which refer to questions of daily activity on mobile devices, this activity may not represent a general pattern of activity, since from individual to individual the patterns of daily use may vary considerably and thus reflect a very irregular use.

In a study by Boase & Ling ( 2013 ), the authors mentioned that about 40% of studies on mobile device use, based on articles published in journals (41 articles between 2003 and 2010), are based on questionnaires.

The questions asked aim to estimate how long or what type of use they have made of their devices on a daily basis, and sometimes aim to know about time periods of several days. In most of these studies, 40% of papers use at least one measure of frequency of use and 27% a measure of duration of use that users make. Another factor that is mentioned is that users do not always report their usage completely accurately. On the other hand, the same study mentions that users may over or under report the use they make for reasons of sociability (Boase & Ling, 2013 ).

Given the moderate correlation between self-report instruments and data from records or logs (Boase & Ling, 2013 ), the author considers that researchers can significantly improve the results if they use, together with other instruments, data from logs to make their studies more accurate and rigorous. Another suggestion would be the use of mobile applications that record these usage behaviours (Raento et al., 2009 ).

Indeed, this kind of instrument is widely used in this type of studies. However, given that mobile devices are fully integrated into our daily lives and we use them quite extensively, it becomes difficult to retain and define with plausible accuracy the use we make of them. In addition to the factors mentioned in the previous paragraph, it is important that these types of studies are validated with other methods, such as the use of logs, as presented in this study. The logs in this study refer to the capture records of the mobile device traffic made by the students.

This article therefore aims to present preliminary results with an approach that uses cross-checking of log data with questionnaire results.

Methodology

This article intends to present and discuss preliminary results of a study that aims to characterize the use of mobile applications at the University of Aveiro through collected logs, crossing its results with questionnaires answered by students during the classes, and also with an observation grid with data from the analysed class and questions to teachers related to what teachers recommend regarding the use of mobile phones during class time.

The research question that motivated this article is: which digital applications/services are most frequently used on mobile devices by the students of the University of Aveiro during their classes?

The study was composed of 40 students, that answered the questionnaires.

The research was based on the Grounded Theory method aiming to analyse the logs from the access points of the University. With the collected data, a usage profile of mobile devices during classes was drawn.

Figure  1 presents a diagrammatic representation of the created methodological process.

General diagram of the study

Therefore 3 instruments were used for the data collection: a questionnaire, an observation grid and logs collected through mobile traffic in the wi-fi network of the university.

The questionnaire allowed for a quantitative assessment of the profile of the participants and collected data on the use that participants claimed to make of their mobile devices. The observation grid served as a guide for the implementation of the study, allowing to record data on the classes where the collections took place and to verify whether certain items were present, such as permission to use mobile devices or planning to use them by teachers. The observation grid would also serve to make the link between use and content in class, but in this pilot, it was not possible to make this link between the class content and the usage of mobile applications, because the author could not observe the applications used by students.

The database containing the usage records enabled the analysis of the logs, resulting in the quantification and verification of the type of activity that each (anonymous) participant made of their device.

The 3 instruments used aimed to i) determine which application(s) students were really using during the classes, through the analysis of the data logs collected from the Wi-Fi network of the University; ii) identify the participants’ representations of their activities by means of several questions regarding mobile usage during class time; iii) observe students’ behaviour and focus via an observation grid that was used by the researcher/observer when he was attending the classes.

The group who participated in this pilot study was selected in accordance with the professors and classes available, so it is considered a convenience sample. The group was constituted by students of undergraduate classes from the Communication and Arts Department of the University of Aveiro.

Table  1 summarizes the schedule of the pilots carried out, the curricular units where they took place, their duration and the instruments used. For ease of management, all the pilots took place in the same department of the University.

The Table  2 summarizes the collected data from questionnaires and logs.

This pilot aimed to build an approach to data analysis, close to the Grounded Theory methodology, in which a provisional theory is built based on the observed and analysed data (Alves et al., 2017 ; Long et al., 1993 ). The data collected in this pilot will serve to define a more complete methodology to be used in a larger study.

This chapter is divided into three parts: context, technological setting and cases analysed. In the context part, the classes which are part of the study will be described, relating the answers from the questionnaires with the teachers’ recommendations about the use of mobile devices. In the technological scenario section, it is intended to describe the technological background underlying the collection process of the logs and in the last part, analysed cases, the objective was to validate if the data to be collected matched the outlined objectives.

In the questionnaire, the questions were divided into two main groups: aspects related to the participant’s profile and aspects directly related to the use of the applications. Aspects related to participants were intended to characterize them. Regarding the use of applications, we aimed to find out the students’ perception of the applications they use in their daily routine, inside and outside of the classroom, and how they do it. Data were collected using a Google Forms form and processed using Microsoft Excel.

In this subchapter, through the data collected from the students’ answers to the questionnaires, and by crossing this information with the data collected from the teachers in the observation grid, we try to describe the context of the pilot.

All of the teachers stated that they allowed their students to use mobile phones during class time, but that they did not plan that use. They also stated that in most part of the classes several students use their mobile phones and apps to search for class related materials. The teachers also showed curiosity about knowing, with more detail, the mobile phone use their students actually have.

In the three classes analysed (Aesthetics, Scriptwriting and Music in History and Culture), when asked about the possibility of using mobile applications as a pedagogical complementary resource 43%, 47% and 55% of students fully agreed that these should be used. In these three classes, 31%, 44%, and 67% of students showed a more moderate opinion: they agreed (but not in such an assertive way) that these should be used.

Another conclusion is that most of the students used a smartphone (88,9%, 75%, 52%) during class time, but many of them also used a computer (66,7%, 100%, 84%). The percentage use of tablets is much lower (11,1%, 0%, 15%).

In the analysed scenario, the majority of the students used the android operating system and 94% also agreed that mobile applications could help to manage the academic tasks, except in the case of the “Aesthetics Curricular Unit”.

When it comes to the time of use, per week, in classes, 53%, 58%, and 22% of the students answered they used these devices between 4 to 5 days a week and 15%, 40% and 70% said they used them between 1 to 3 days a week.

Students were also asked about how frequently they accessed mobile applications during class time and, in all, 77% of the respondents reported accessing apps at least between 1 to 5 times per class. About 20% referred they accessed apps from 6 to 10 times per class.

As for the purposes of accessing apps during classes, most students mentioned categories related i) to support the class / to research (70%, 100%, 77,8%), ii) to access institutional platforms (47.4%, 66.7%, 89, 9%), iii) to access to information (47.4%, 50%, 66.7%) and iv) to work (36.8%, 50%, 44.4%).

Interestingly, the categories communication (52.6%, 41.7%, 22.3%), collaboration (10.5%, 16.7%, 0%), access to institutional services (5.3%, 0% 0%) and “I do not use them” (10.5%, 0%, 0%) presented very low percentages, namely the last one.

When questioned about the use of mobile devices that did not include academic reasons, many students referred to the categories “to be linked/connected” or “to be updated” (42.1%, 66.7%, 33.3%), “to communicate” (57.7% 75.7%, 66.7%), “to share and access content” (31.6%, 58.3%, 33.3%), but few mentioned “for entertainment” (26.3%, 16.7%, 22.2%), “as a habit or routine” (10.5%, 41.7%, 11.1%) and “I do not use them” (10.5%, 0%, 11.1%).

When asked about which mobile applications are most used in an academic context, the most relevant category was “to research / to study” (73.7%, 58.3%, 89.9%), “to check the calendar” (31.6%, 25%, 66.7% %) and “to surf the web” (47.4%, 50%, 55.6%). Again, categories such as “to work” (36.8%, 33.3%, 33.3%), “to take notes” (26,2%, 33.3%, 55.6%) and “to create content” (31.6%, 25%, 11.1%) presented relatively low percentages. It should also be noted that the respondents presented answers that created categories which were not expected such as “to watch films” (10.5%, 8.3%, 0%), “to listen to music” (31.6%, 33.3%, 33.3%), “to take photos” (10.5%, 0%, 0%) and “to play games” (5.3%, 0%, 0%) All the students said that they used applications during classes in at least one of the categories. In fact, in the three courses no one stated “not to use them” (0% in all).

When asked about the teachers’ permission to use the mobile devices in the classroom, most of the students said that teachers allowed free use (52.6%, 100%, 77.8%). Only a few stated that teachers allowed using them specifically when planned (41, 1%, 0%, 22.2%). The respondents of one course stated that teachers did not allow the use of devices (Aesthetics - 5.3%). Finally, when asked about the usefulness of integrating mobile applications in class, there was an overwhelming majority of respondents (100%, 78,9%, 100%) saying they believed that such integration could be enriching and useful.

Below is presented a table describing the most used mobile apps during class activities. It should be noted that only the two answers with the greatest relevance for each category were considered.

Table  3 systematizes what the results have been showing until now: there is an important part of students that use mobile phones during their classes and, even when teachers advise them not to use them, they ignore the recommendations and use them anyway. The main purposes stated were: to be in contact with others through social networking but also to access different kinds of information in browsers. Moreover, the classes where the use of devices is not recommended by the teachers seems to be the one where some applications are most used.

Technological setting

In this section we intend to describe the technological background underlying the process of collecting the logs. The first goal was to register and capture logs from the wi-fi network of the university, which consists of a wireless network that users can access using their universal user credentials.

In order to do that a meeting was scheduled with the university’s technology services, as our main concern was the anonymization of the data collected in order (i) to confer more neutrality to the data treatment, and (ii) to comply with European data protection legislation. Another issue for discussion was the need of powerful machines so that they could process the large amount of data collected.

In this meeting the necessary steps were agreed in order to guarantee the users’ privacy, the authorization of the university’s central services to do the study and the registration method of the logs. The overall procedure demanded several experiences of data collection to fine-tune the final pilot, which works as the basis capture setting for all the main study.

The Wi-Fi traffic capture software (Wireshark) was selected to work both with Android and IOS devices and it was possible to understand the functionalities of the software.

The pilot also helped to understand and solve additional problems that appeared during the previous tests, related to the anonymization of the users’ data. It was necessary to ensure that the users’ personal data were not identifiable, which was a commitment: in fact, only HTTPS Footnote 1 traffic was captured, being all the other information encrypted.

After the first tests, an initial data collection pilot took place in a classroom context. A specific capture system was created to allow the capture of mobile application logs used only by a certain group of students, from a designated Curricular Unit. A specific scenario was set up to ensure that only those students communicating through the IP Footnote 2 defined for the scenario and during that class time were considered and treated under the scope of this study:

If the traffic of the concerned student is communicating through one of the APs (Access Points) covering the room, then the device will be assigned a “Room network” IP;

If the student’s traffic is not communicating through one of the APs covering the room, then the device will be assigned a “Non Room network” IP;

If the student traffic does not belong to the group to be analysed and the device in question is communicating through one of the APs covering the room, then the device will be assigned an IP from a “normal eduroam network”;

In the final steps we resolved the IP’s in Wireshark (software used for the capture) and the unsolved IP’s where filtered in a PHP Footnote 3 script, through the gethostbyaddr method where the unsolved ones are incrementally added.

Finally, using an IP list, we performed a comparison to resolve any unresolved names;

This step allowed to fine tune the process and to make the final test.

Analysed cases

After performing these tests, a scenario for this final pilot was set up to validate if the data to be collected matched the outlined objectives. In this final pilot, logs were collected in a classroom so that the scenario was as close to the desired collection as possible. In this pilot, it was possible to verify that the collected data fulfilled the requirements. At this point, in addition to the HTTPS traffic packets, the packets referring to DNS Footnote 4 traffic were also included. This option made the HTTPS traffic more easily understandable. Furthermore, the researcher could conclude that all authenticated devices belonged to separate accounts.

The results show that the pre-tests/pilots and the final pilot turned out very well and in a very reliable way since they allowed to verify the main problems that could occur and helped to certify that the traffic anonymity condition was respected. In fact, only the HTTPS was considered, and all other communication was encrypted with no risk of corruption of private data. Moreover, this option had an important justification: the fact that HTTPS traffic could be more easily understandable and the fact that it allowed certifying that all the authenticated devices of the wireless network belonged to separate accounts.

To process and create output visualization of the data, the choice was an integrated solution, both for the processing stage and for creating visualisations. Given the variety of tools available, several were tried out and Tableau Software® (Tableau Prep® and Tableau Desktop®) was chosen. Tableau Software is an interactive data processing and visualisation tool that belongs to the Salesforce company and, although it is paid software, it allows for an academic licence that was used in this project.

This solution, besides allowing working with a large amount of data, also allows for a very interactive data treatment and visualisation. This software also allows the importation of data from various sources, which in the case of this study was also an advantage.

This solution allowed us to work with large amounts of data but it also allowed for a very interactive data treatment and visualization. In the case of Tableau Prep, the file with the logs was imported in a CSV format Footnote 5 and treated iteratively in a dynamic way, being refined to the desired data in a second stage. As an example, we can mention the separation of the field “time duration” in hours, minutes and seconds fields; all the IPs were converted to a generic name “student”; all the destinations visited by the students were grouped in main categories, as for instance “Facebook”, as each application had numerous distinct destinations.

About 30 changes in data treatment and in data flow “cleaning” were performed, which were, later, exported to Tableau Desktop. Each file imported to Tableau Prep, in addition to the changes applied to the previous file, was refined with more changes, in an iterative process.

After treating the data on Tableau prep the generated data flow was imported to Tableau Desktop so that dynamic data visualizations were created. At this stage, dimensions, measurements, and filters were created according to the desired data visualization. The software has the big advantage of creating dynamic visualizations of the logs’ data which allows for a different and richer perspective on the data obtained, in order to deepen further studies about the same topic.

Discussion and conclusions

This paper aimed to describe the process of a pilot to carry out a larger study where we wanted to cross-reference actual usage data (logs) of mobile applications in the classroom with data from student questionnaires. In this article we also present the main results of this pilot, both from the point of view of the process of the pilot and from the point of view of the data of use of mobile applications by students in the classroom.

From the preliminary data analysis of this pilot, we can infer that the most used apps are Facebook, Google and Instagram, as we can see in Fig.  2 and Fig.  3 , although some variations between the attendees of the courses were registered when it comes to other apps. For example, in the case of the Design course, there are alternative apps being used such as YouTube or Vimeo.

General use of applications in Scriptwriting class

General use of applications in Aesthetics classe

Another noticeable preliminary result is that students use Facebook more at the beginning of classes and Instagram is used more at the end, as we can see in Fig.  4 and Fig.  5 .

Use of Facebook per hour in Scriptwriting class

Use of Instagram per hour in Scriptwriting class

In addition, the developed model was used in the main study with a bigger convenience sampling approach, which may provide a more accurate representation of the population of mobile-phone-users in the study field.

The visualizations created in a dynamic way during this study showed that the use of logs as a complementary data provider to other instruments, such as questionnaires, can be an added value for this research field.

On the other hand, this pilot contradicts (sometimes slightly, others considerably) the results of the questionnaires answered by the students and whose logs were collected and analysed. Logs show that:

there is a common use of mobile applications during the classes;

the purpose of the access is different: participants report that they use mobile applications mostly for academic reasons, but it can be noted that there is a general use of other mobile applications such as social networks and Youtube;

the usage time is much longer than what participants reported;

the frequency is also different: students stated that they use mobile applications in classes only 1–3 days a week, but we found that, in the analysed classes, there is an almost constant use of them, and finally

students report that they do not use social networks much in class, but the use is, in fact, massive.

The students’ perception of the “use of mobile devices and applications during lessons”, and as already mentioned, during a teaching activity - 70% of the students refer using the applications between 1 to 5 times, 22% between 6 to 10 times and 4% more than 10 times. It should also be noted, as previously mentioned, that only 4% mention not using them. With regards to the use during the week, 56% of the students refer using them between 4 to 5 days per week and 39% between 1 to 3 days per week. There is also a relatively low percentage of students mentioning that they use the devices during class more than ten times (4%).

However, analysis of the logs shows that this use appears to be much more intensive. We performed a calculation based on the average number of accesses, from which we removed 40% of potential automatic accesses and divided by the average number of accesses each application had in the initial test. The results present 6.6 accesses to the device per class/student in the class with the fewest accesses, and for the highest case, 313 accesses to the device per class/student.

This result is reinforced by results from other studies, such as the Mobile Survey Report, which states that students make regular use of laptops and smartphones during lessons (Seilhamer et al., 2018 ).

These conclusions lead us to some very serious insights on this subject. Apparently, even older students have a misperception of their use of online applications during classes. There is a serious discrepancy and incongruency between the behaviours that they claim to adopt and those they actually engage in during the classes. There are authors, who argue for the need for other types of studies that support this type of approach (Gerpott & Thomas, 2014 ), because the perception reported by users may not correspond to the actual use. It means that this gap deserves a deeper reflection. Why does it happen? Are students not motivated in higher education? Is the world offered online more interesting than the one in the physical campus? We will try to answer these questions in the main study.

Availability of data and materials

Some of the visualizations created are publicly available at https://public.tableau.com/profile/davidoliveiraua

HTTPS It is a protocol used for secure communication over a computer network, and is widely used on the Internet

IP is the s a numerical label assigned to each device connected to a computer network that uses the Internet Protocol for communication

PHP is a general-purpose scripting language especially suited to web development

DNS is naming system for computers, services, or other resources connected to the Internet

Unformatted file where values are separated by commas

Abbreviations

Higher Education Institutions

Access Points

Hypertext Transfer Protocol Secure

Internet Protocol

Hypertext Preprocessor

Domain Name System

Comma-separated values

Aagaard, J. (2015). Drawn to distraction: A qualitative study of off-task use of educational technology, Computers & Education, Elsevier

Alexander, B., Ashford-Rowe, K., Barajas-Murphy, N., Dobbin, G., Knott, J., McCormack, M., Pomerantz, J., Seilhamer, R., & Weber, N. (2019). EDUCAUSE Horizon Report: 2019 Higher Education Edition. https://library.educause.edu/-/media/files/library/2019/4/2019horizonreport.pdf?la=en&hash=C8E8D444AF372E705FA1BF9D4FF0DD4CC6F0FDD1

Alves, A. G., Martins, C. A., Pinho, E. S., & Tobias, G. C. (2017). A Teoria Fundamentada em dados como ferramenta de análise em pesquisa qualitativa. 6o Congresso Ibero-Americano Em Investigação Qualitativa - CIAIQ2017, 1, 499–507

Boase, J., & Ling, R. (2013). Measuring Mobile phone use: Self-report versus log data. Journal of Computer-Mediated Communication , 18 (4), 508–519 https://doi.org/10.1111/jcc4.12021 .

Article   Google Scholar  

De Reuver, M., & Bouwman, H. (2015). Dealing with self-report bias in mobile internet acceptance and usage studies. Information Management , 52 (3), 287–294 https://doi.org/10.1016/j.im.2014.12.002 .

Dobbin, G., Dahlstrom, E., Arroway, P., & Sheehan, M. C. (2011). Mobile IT in higher education. Educause , 1–33.

Gerpott, T. J. and Thomas, S. (2014) Empirical research on mobile internet usage: A meta- analysis of the literature, telecommunications policy, 38 (3): 291-310. 2014. [online]. Available: http://www.sciencedirect.com/science/article/pii/S030859611300175Accessed on: Dec., 20, 2016.

GMI. (2019). Mobile Learning Market Size | Global Growth Statistics 2020–2026. https://www.gminsights.com/industry-analysis/mobile-learning-market

Jesse, G. R. (2015). Smartphone and app usage among college students: Using smartphones effectively for social and educational needs. Proceedings of the EDSIG Conference , 2015 , 1–13 http://proc.iscap.info/2015/pdf/3424.pdf

Long, D. R., Strauss, A., & Corbin, J. (1993). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. The Modern Language Journal , 77(2), 235 https://doi.org/10.2307/328955

Magda, A. J., & Aslanian, C. B. (2018). Online college students 2018. Comprehensive Data on Demands and Preferences. https://www.learninghouse.com/knowledge-center/research-reports/ocs2018/

Monteiro, A., Bento, M., Lencastre, J., Pereira, M., Ramos, A., Osório, A. J., & Silva, B. (2017). Challenges of mobile learning – A comparative study on use of mobile devices in six European schools: Italy, Greece, Poland, Portugal, Romania and Turkey. Revista de Estudios e Investigación En Psicología y Educación , 13 , 352. https://doi.org/10.17979/reipe.2017.0.13.3229 –357.

Oliveira, D., Tavares, R., & Laranjeiro, D. (2017). Estudo de avaliação de aplicações móveis de instituições de ensino superior português. https://www.researchgate.net/publication/307981176

Parslow, R. C., Hepworth, S. J., & McKinney, P. A. (2003). Recall of past use of mobile phone handsets. Radiat. Prot. Dosim., 106(3), pp. 233–240. http://search.ebscohost.com/login.aspx?direct=true&db=cmedm&AN=14690324&site=ehost-live , DOI: https://doi.org/10.1093/oxfordjournals.rpd.a006354

Raento, M., Oulasvirta, A., & Eagle, N. (2009). Smartphones: An emerging tool for social scientists. Sociological Methods & Research , 37 (3), 426–454 https://doi.org/10.1177/0049124108330005 .

Ryu, H., & Parsons, D. (2008). Innovative Mobile learning: Techniques and technologies. Information Science Reference - Imprint of: IGI Publishing.

Seilhamer, R., Chen, B., DeNoyelles, A., Raible, J., Bauer, S., & Salter, A. (2018). 2018 Mobile Survey Report. https://digitallearning.ucf.edu/msi/research/mobile/survey2018/

Sharples, M., Corlett, D. & Westmancott, O. (2002). The design and implementation of a mobile learning platform based on android. Proceedings - 2013 International Conference on Information Science and Cloud Computing Companion, ISCC-C 2013, 345–350. https://doi.org/10.1109/ISCC-C.2013.136

UNESCO. (2013). Policy guidelines for mobile learning. In policy guidelines for mobile learning . https://doi.org/ISBN 978-92-3-001143-7. France

Vicente, F. (2013). WelcomeUA: Desenvolvimento de interface e avaliação da usabilidade. 136. https://ria.ua.pt/handle/10773/12403

Download references

Acknowledgements

Not applicable.

The authors declare that they have no funding in this project.

Author information

Authors and affiliations.

Communication and Arts Department, Universidade de Aveiro, Campus Universitário de Santiago, 3810-193, Aveiro, Portugal

David Manuel Duarte Oliveira, Luís Pedro & Carlos Santos

You can also search for this author in PubMed   Google Scholar

Contributions

DO wrote the introduction and discussion, and saw to the article structure, wrote the method section and conducted the data analysis. LP and CS conducted the literature review. All authors contributed to the discussion and conclusion sections, and the overall flow of the article. All authors read and approved the final manuscript.

Corresponding author

Correspondence to David Manuel Duarte Oliveira .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Oliveira, D.M.D., Pedro, L. & Santos, C. The use of mobile applications in higher education classes: a comparative pilot study of the students’ perceptions and real usage. Smart Learn. Environ. 8 , 14 (2021). https://doi.org/10.1186/s40561-021-00159-6

Download citation

Received : 20 January 2021

Accepted : 16 July 2021

Published : 09 August 2021

DOI : https://doi.org/10.1186/s40561-021-00159-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mobile application
• Mobile usage
• Higher education

Advertisement

A Study of Mobile App Use for Teaching and Research in Higher Education

• Original research
• Open access
• Published: 05 June 2022
• Volume 28 , pages 1271–1299, ( 2023 )

Cite this article

You have full access to this open access article

• Annika Hinze   ORCID: orcid.org/0000-0002-7383-1134 1 ,
• Nicholas Vanderschantz 2 ,
• Claire Timpany 2 ,
• Sally Jo Cunningham 1 ,
• Sarah-Jane Saravani 3 &
• Clive Wilkinson 3  

19k Accesses

7 Citations

Explore all metrics

The exponential growth in the use of digital technologies and the availability of mobile software applications (apps) has been well documented over the past decade. Literature on the integration of mobile technology into higher education reveals an increasing focus on how mobile devices are used within the classroom environment, both physical and online, rather than on how mobile applications may be used for either teaching or the research process. Our study surveyed staff and higher degree research students at a New Zealand university using an online questionnaire to gain insight into the use of mobile apps for tertiary teaching and research, seeking information, particularly on which apps were used for which tasks and what obstacles hindered their use. The online survey used 29 questions and ran in 2016/2017. 269 participants completed the survey, nearly 20% of the potential sample. We found that mobile apps were used by academics and students for both teaching and research, primarily in the form of document and data storage and exchange, and communication. Very little app use was recorded for in-class activities (teaching) or in-field activities (research). Apps use resulted from personal motivation rather than institutional planning. Both students and academics reported that institutional support and flexibility would likely provide motivation and lead to increased app use for both research and teaching.

Similar content being viewed by others

Adoption of online mathematics learning in Ugandan government universities during the COVID-19 pandemic: pre-service teachers’ behavioural intention and challenges

Geofrey Kansiime & Marjorie Sarah Kabuye Batiibwe

Assistive technology for the inclusion of students with disabilities: a systematic review

José María Fernández-Batanero, Marta Montenegro-Rueda, … Inmaculada García-Martínez

Barriers to Online Learning in the Time of COVID-19: A National Survey of Medical Students in the Philippines

Ronnie E. Baticulon, Jinno Jenkin Sy, … John Carlo B. Reyes

Avoid common mistakes on your manuscript.

1 Introduction

Mobile learning has been claimed as the future of learning (Bowen & Pistilli, 2012 ) yet surprisingly little specific empirical investigation of mobile application use in tertiary settings is available in the literature. While digital devices are prevalent in the higher education environment, the use and uptake of mobile apps for tertiary teaching and research by academic staff has only begun to be studied (Lai & Smith, 2018 ; Shraim & Crompton, 2015 ).

1.1 Technology Availability to Students

The 2019 ECAR survey of Undergraduate Students and Information Technology found that students see technology as a means for better engagement with study material, instructors and peers in the classroom (Galanek, & Gierdowski, 2019 ). The 2020 survey found that 75% of students who connect to campus WiFi are using two or more devices (Gierdowski et al., 2020 ). The 2018 survey reveals 95% of students have access to smartphones and 91% to laptops (Galanek et al., 2018 ). The downloading of mobile software applications (apps) in recent years shows a similar pattern of increase, rising from 84 billion downloads from the Apple App Store/Google Play in 2016 to 105 billion in 2018 (Sensor Tower). The third most popular Apple App Store category in May 2019, was education at 8.52% (Statista, 2019 ). Studies of higher education students in Southeast Asian universities reveal even higher percentages, for example, 100% of Hong Kong undergraduates in a 2018 study possessed mobile phones, of whom 85% also used apps for their academic studies (Shuk Han Wai et al., 2016 ). Thus previously held concerns that not all students will have access to a smartphone is not supported by the wealth of recent research investigating technology availability (Anderson, 2015 ).

For some time there has been the suggestion that technological advancement of mobile devices and the increased availability of mobile apps may prove central to academic teaching and research (Hahn, 2014 ; Canuel & Chrichton, 2015 ; MacNeill, 2015 ). Specific empirical investigation that discusses mobile app use as opposed to mobile device or more generally information technology use in tertiary teaching or research is extremely limited. Of the few specific discussions of mobile app use in academia, we identify library studies that have been conducted on the selection, use or development of mobile apps (Wong, 2012 ; Hennig, 2014 ; van Arnhem, 2015 ). These studies have often had a focus on the delivery of information or data about library services. Practitioner research in library and education have also included work describing apps and app features for research or teaching—an example being apps for ethnographic field research (van Arnhem, 2015 ). Work has investigated undergraduate student perceptions of mobile apps and mobile devices. An early study of tertiary student use of mobile note-taking software by undergraduate students (Schepman et al., 2012 ) saw widespread positive perception and adoption of these mobile tools by students. Studies exploring the impact the integration of mobile computing devices is having on higher education teaching and learning reveal an increasing engagement with content, collaboration with classmates and information creation and sharing outside the formal learning spaces (Bell et al., 2019 ; Compton & Burke, 2018 ; Gikas & Grant, 2013 ). Systematic literature reviews (Burch & Mohammed, 2019 ; Singh & Hardaker, 2014 ) and reports or investigations of academics’ perspectives of technology use in tertiary classrooms (Galanek & Gierdowski, 2019 ) provide insights into the broad picture but have provided little advice regarding app use for research or teaching.

1.2 Technology Use in Academia

Research on the integration of mobile technology in higher education is focussed on how mobile devices are used within the classroom environment, rather than on their application to the research process (Morris et al., 2016 ; Pedro et al., 2018 ; Schepman et al., 2012 ; Shuk Han Wai et al., 2016 ). MacNeill ( 2015 ) outlines techniques and strategies for the use of apps to support learning, teaching and research. The perspectives are self-reflective and provide insights into tools that have been trialled by the author with recommendations for educators to dedicate time to explore the wealth of available applications for teaching and research inside and outside the classroom. In the higher education classroom, mobile devices in higher education can provide new opportunities for information gathering and use, content access, communication, collaboration and reflection (Beddall-Hill et al., 2011 ; Bowen & Pistilli, 2012 ).

Lai and Smith ( 2018 ) identify a paucity of research on technology use in higher education. We identified two previous surveys of mobile technology use in tertiary teaching and learning. They focussed either on how socio-demographic factors influenced the perception of teaching staff (Lai & Smith, 2018 ), or the perceptions of the pedagogical affordances for mobile devices in teaching (Shraim & Compton, 2015 ). The survey of 308 tertiary teaching staff by Lai and Smith ( 2018 ) found that while many of the respondents were positive about the benefits that mobile technology could provide for their teaching, many felt they lacked the confidence to apply the technology effectively. “When implementing a mobile application in curriculum, instructors need to clearly state the goals of using the application to make sure the students understand the purpose of using the application for coursework, how it is connected to the curriculum, and how it will improve their learning” (Chen et al., 2013 , p.339). Other surveys of academics on their use of apps and mobile devices have focussed on teaching. The survey of faculty members use of mobile devices for teaching by Shraim and Crompton ( 2015 ) found that there are positive perceptions of the opportunities that mobile devices provide for teaching, but were focussed on the opportunities that the device itself provided (mobile connectivity, linking of formula and informal teaching, increasing enjoyment and connecting to real-world problems), rather than apps. The most important finding related to app use was the concerns that academics held about finding time to select appropriate apps and develop their teaching plans to incorporate them (Shraim & Crompton, 2015 ). Their scope was wider than app use, but only asked academics about their app use in their teaching, not their research.

Mobile devices provide opportunities to undertake research and fieldwork while enabling the collection, manipulation and sharing of data in real-time (Beddall-Hill et al., 2011 ). To date, the investigation of digital tools for research has focused on opportunities and challenges such as technical issues (e.g. battery life, data security or data inaccuracies) and considerations such as the preparation of future researchers to leverage the capacity of digital tools for research (Carter et al., 2015 ; Davidson et al., 2016 ; Garcia et al., 2016 ; Raento et al., 2009 ). The benefits of using mobile devices for research are described by Chen ( 2011 ), as including; immediacy of response, better enablement of longitudinal research, capturing of location information for context and the inclusion of an additional touchpoint to provide a more well-rounded research picture. Carlos ( 2012 ) suggests that mobile devices and mobile applications provide three main benefits for use in research: ready availability and familiarity, easy use, and always-on internet connections. A counter perspective is provided by McGeeney ( 2015 ) who observed a number of constraints for using mobile apps, compared to Web browsers. They found lower response rates, increased costs, and usability issues such as limited navigation and data entry options in mobile survey tools. Similarly, it is suggested that with mobile apps the time and effort required to learn how to use an app effectively can result in lower response rates than web-based data collection (Pew Research Center, 2015).

1.3 Institutional Expectations and Support

Many Institutions and academic libraries encourage mobile device use in educational contexts (Canuel et al., 2016 ; Hanbridge et al., 2018 ; Morris et al., 2016 ). Academics are encouraged to provide learning experiences that include “mobile-friendly content, multi-device syncing, and anywhere/anytime access” (EDUCAUSE, 2019 , p. 8). However, the 2019 Horizon Report has identified a need for sustained support and professional development to take advantage of the new teaching opportunities afforded by digital devices (EDUCAUSE, 2019 ). While academics are largely confident with mobile technologies, they need greater awareness of how these technologies can be incorporated effectively to take full advantage of the affordances mobile devices can offer in teaching ((Shraim & Compton, 2015 ). Several studies found that a lack of faculty training was a source for faculty dissatisfaction with classroom technology (Galanek & Gierdowski, 2019 ) and (mobile) IT integration into teaching (Burch & Mohammed, 2019 ; Shraim & Crompton, 2015 ). A number of academic libraries promote the use of mobile software to academics through digital or technological literacy training (Canuel & Chrichton, 2015 ; Hennig, 2014 ). However, research in the area of mobile application in academic libraries almost exclusively focused on the delivery of library services to mobile devices (Aher et al., 2017 ; Breeding, 2019 ; Singh Negi, 2014 ) or the integration of responsive design in web-based service (Kim, 2013 ; Tidal, 2017 ). A sample scan of university library websites indicates that it has become increasingly common for research university libraries to include guidance and instruction on the use of mobile apps for research. Such guidance usually takes the form of a brief preamble followed by a list of the various apps with a brief description of the features, functions and purpose of the app with links to the vendor website. Contextually, little indication is provided as to how or why such a list was curated or, more importantly, how the library supports the integration into learning of such mobile apps through training or instruction. A notable exception is the service offered by Stony Brook University Library, which assists in selecting and using mobile apps for research (Saragossi et al., 2018 , p. 202).

1.4 Research Questions and Focus

We identified a number of shortcomings in the existing literature on app use in tertiary contexts. Research on technology use in teaching and learning rarely focuses on (the experience of) app use, but rather on device capabilities and opportunities of technology use. Previous surveys predominantly analysed the undergraduate students’ perceptions of app use in teaching. Use of apps for academic research is little discussed beyond app use for specific projects, and general technology benefits or issues. While many tertiary institutions actively encourage academics to use mobile apps (and other technology) for teaching, the impact of such expectations on the academic experience is not well studied.

The research reported here attempts to understand more widely how apps are being used in tertiary teaching and research, including what are the perceived benefits and barriers. To provide insights into how mobile apps may be used by students and staff in teaching and research a university-wide survey on mobile app use in a tertiary setting was conducted. The survey design was guided by the following research questions:

RQ1: Are academics using mobile apps for tertiary teaching and research at University of Waikato? RQ2: Which apps are used by academics for which teaching and research tasks? RQ3: What is the experience of app use by academics: what obstacles/opportunities do they identify?

The survey was made available to staff and higher-degree students (collectively referred to as academics) across the University to capture their perspective on mobile app use for teaching and research.

This article presents our study data, and analyses these with respect to the three research questions posed above. The remainder of this article is structured as follows, in Sect.  2 we introduce our method, an online survey of staff and higher degree students at a New Zealand university. Section  3 provides results and analysis of the responses to this survey. We discuss the findings in light of our research questions in Sect.  4 and conclude this article in Sect.  5 . Initial analysis results were presented elsewhere (Hinze et al. 2017a , b ), and primarily focussed on the responses from higher degree research students. The results reported in this paper cover all responses to the survey including those of higher degree students and staff.

We performed an online survey of staff and higher degree students of the University of Waikato in New Zealand. The survey was designed to get a university-wide view of how mobile apps were being used for teaching, research, and learning purposes. The survey was performed over two consecutive years in order to capture the widest sample of participants.

2.1 Context of Study Environment

This New Zealand University is typical of western universities offering qualifications across multiple academic divisions including, but not limited to; the arts, computing, education, management, and the sciences. The majority of staff and students work on campus yet mobile and electronic learning is supported at all learning levels. The university provides Google apps for email, file storage, and word processing. A number of digital resources and technologies are supported depending on the needs of researchers and teachers in academic disciplines. A well-resourced library supports students and staff with print and electronic holdings. There are no required or mandated mobile apps at this university.

2.2 Data Collection

A location-restricted online, self-administered survey tool was developed in the Qualtrics Survey Software. The survey was made available to participants at the University of Waikato in New Zealand from 3rd to 19th August 2016 and again from 31st August to 6th October 2017. The potential sample size was approximately 820 enrolled masters or PhD thesis students and 580 staff (including academics, researchers, and research administrators). All responses were anonymous.

2.3 Participant Recruitment

Higher-degrees students and staff from across the university were invited to participate. We engaged the University’s research office to forward invitations to all departmental administrators, with whom we personally followed up with to distribute the survey invitation to all the University’s academic staff and researchers via email. We further followed up these email invitations with in-person invitations by one of the research team at Faculty and School meetings. The higher-degree students were engaged by the School of Graduate Research through email and social media. Our study had a potential pool of 1400 staff and higher-degrees students.

The survey was done in two stages (same target group, self-selected participants, initial and repeat attempt to engage participants), we present in this article the aggregated result of both stages. 288 survey entries were received, out of which 19 contained no further data and were excluded from the analysis. The survey was thus completed by 269 participants, or nearly 20% of the potential sample of university staff and higher-degree students.

2.4 Survey Tool

Our online tool was a 24-item survey that incorporated a combination of Likert scale tools, radio button responses, and free text questions. This tool was conceptualised in three sections which (1) requested demographic data, (2) surveyed previous experience and use of mobile apps, and (3) reviewed device and operating system use. The survey invited reflection by the participants on their use of mobile apps and whether they believed that their use or lack of use had influenced research or teaching practice. The survey also required participants to give information regarding their reasons for non-use in cases where participants indicated that they had not used, and were not intending to use, mobile apps. To review the survey questions please refer to Appendix 1.

2.5 Definitions Used in the Survey

In the survey we included the following definitions for clarity for the participants:

Mobile app—is a software application developed primarily, although not exclusively, for use on small computing devices, such as smartphones or tablets. Examples include WhatsApp, Evernote, and Flipboard. Other examples might include mobile app versions of programs such as Dropbox or EndNote.

Academic purposes—includes all teaching and/or research activities engaged in while a member of the University community.

2.6 Data Analysis

The results were analysed using default and cross-tabulation report functions provided by the Qualtrics software before manual manipulation, tabulation, and analysis using Excel. We have undertaken basic descriptive statistical analysis (means testing and T test for cohort comparison) and provide tables, graphs, mean values and probability values (where appropriate) along with our reporting in the Results section.

3 Results and Analysis

We present our results structured by the three research questions. After demographic information in Sects. 3.1 , 3.2 , 3.3 address the first question ( are academics using mobile apps for tertiary teaching and research) , while Sects. 3.4 , 3.5 address the second question ( which apps are used for tasks ), and finally Sects. 3.6 , 3.7 , 3.8 address the third question ( academic experience of app use: obstacles and opportunities ).

3.1 Demographic Attributes

The university staff and postgraduate students at the time of the two instances of the survey was reasonably stable at about 1400 (580 academic staff and 820 higher-degree research students), which forms the potential participant pool. 269 of these 1400 responded to our invitation, with most of our study participants being academic staff (N = 163), followed by doctoral students (N = 83), see Fig.  1 .

Participant roles (multiple selections possible)

Out of the 269 participants, 141 were female (52%) and 125 were male (46%); 2 did not specify gender (1%), and 1 selected other.

63% of the participants were younger than 40 years old, see Fig.  2 . The participants represent a range of schools and faculties, as shown in Fig.  3 . The other university areas mentioned by participants were administration and technical support. Five participants selected two options.

Participant age

Participants by school/faculty (multiple selections possible)

3.2 Use of Mobile Apps

With 172, the majority of the 269 participants (64%) had used mobile apps for academic purposes such as teaching or research, see Fig.  4 for details. We note that the percentages among Academic staff and doctoral students were comparable at 67% and 69%, respectively, while only 25% of Master’s students had used apps for research. Four participants provided no data (Fig. 5 ).

Prior use of apps for academic purposes (multiple roles possible)

Academic use of mobile apps by participant age range

Of the 172 participants who had used mobile apps for academic purposes, the age cohort that showed the strongest engagement were the 21–30 year-olds (71%). This was followed by the group of 31–40 year-olds (64%). If broken down by gender, 62% of the 141 female participants and 67% of the 125 male participants had used apps for academic purposes ( p  = 0.3975, i.e., there was no significant gender difference in app use), see Fig.  6 .

Academic mobile app usage by participant gender

Out of the participants who had used apps for academic purposes, most (19%) were in the Faculty of Computing and Mathematical Sciences, followed closely by both the Faculty of Education (18%) and science and engineering (18%); details are shown in Fig.  7 .

Academic mobile app usage by participant school/faculty

We surveyed the 172 participants who had used mobile apps for academic purposes to inquire which types of devices they used mobile apps with (multiple selections were possible). 303 responses were collected. The majority (79%) used smartphones, followed by iPad and Android tablet devices (together 70%), details see Fig.  8 . The named other devices were laptops and PCs, and one sporting device.

Type of mobile device used (multiple selections possible)

90 of the 172 app users gave details about operating systems with 114 selections; for details see Fig.  9 . Under ‘Other’ participants listed ChromeOS and Microsoft system (surface tablet). As expected based on mobile phone ownership data, Android and iOS emerged as the preferred operating systems.

Operating system used on mobile device (multiple selections possible)

Finally, we also asked if participants had been involved in the development of any mobile apps that might be used for academic purposes, and to explain their purpose. We received 60 answers: 50 no, 5 n/a, and the 5 positive answers: driving support (1), for teaching (2), indigenous language learning (1), and a personal digital library (1).

3.3 Purpose of Mobile App Usage

In order to investigate the mobile app use-cases in the tertiary environment, we asked participants about the situations that they had used these. Participants could select either or both teaching/supervision, and/or research. Ninety-five (56%) of the 171 respondents to this question had used a mobile app for teaching/supervision purposes; 146 (85%) had used one for research purposes. Of these, 70 (41%) selected that they had used apps for both (see Fig.  10 a, top).

Mobile app usage: ( a) by purpose (top), ( b) by gender (bottom)

More female participants are using apps than male participants (see Fig.  10 b, bottom). For teaching, there was not significantly more male respondents using apps than female respondents ( p  = 0.96). For research, more female participants were found to be usings apps than male participants, though this was still not significant ( p  = 0.69). We further note that female respondents tended to use apps for research or for teaching only (63.2% of 87 female compared to 54.7% of 84 male). Conversely more male respondents used apps across both categories (marked in gray). However, the difference between male and female use of apps for both purposes was not significant ( p  = 0.59). The majority of the participants who had used apps for teaching or supervision were academic staff (86 of 95). A small number of participants who had used mobile apps for teaching identified as doctoral students (15 of 95), none as Master’s students, 7 as Other (multiple selections possible). 88 Academics, 55 Doctoral students, 2 Master’s students and 18 Others reported using mobile apps for research purposes.

We observe that higher percentages of academic staff used a mobile app for teaching and supervision purposes compared to research purposes (see Fig.  11 ). Conversely, doctoral students were more likely to use apps for research purposes than for teaching/supervision purposes. Quite predictably, Master’s students and other participants were more likely to use mobile apps for research.

User roles for mobile app users

Only 6 participants reported being asked by their lecturer or supervisor to use mobile apps for academic purposes (50 reported having not been asked, 214 provided no answer). They named the following app purposes: document sharing, storage, referencing, communication; bookshelf app for recommended lecture text; conference presentation app; google drive and dropbox for backups of theses, and app examples to explore for research on interactive tour guides.

3.4 Apps for Teaching/Supervision

Ninety-five participants reported using mobile apps for academic purposes for teaching or supervision related activities. Unsurprisingly, the majority of these participants reported themselves as teaching staff. At this university, it is not atypical for staff to work across roles in a university, and for some higher degrees students to be contributing to teaching initiatives at various levels and therefore some doctoral students and participants in the ‘Other’ category had also used apps for teaching purposes. These 95 participants were asked to select from a shortlist of possible academic-related apps (see Fig.  12 ) the mobile apps that they used for teaching or supervision purposes. Also shown in Fig.  15 , the participants were asked if these apps were used by themselves or by students under their supervision. There was a substantial number of Other options named, including Google Drive (8), Google Docs (5), Facebook (4), Google Sheet (3), Kahoot (3), and Kindle (3) and a further 11 programmes named twice, and 65 programmes named once showing that a diverse range of apps were used (not shown in Fig.  12 ).

Apps used for teaching/supervision (multiple selections possible)

Mobile apps for teaching purposes were reported as being used by 95 participants, the specific purposes for using apps for teaching are elaborated on in Fig.  13 . The aspects teaching staff most engaged in were sharing or storing documents, as well as communication with colleagues. Other tasks mentioned were communication with students, in-class surveys, or keeping up with recent blogs.

Use of mobile apps in teaching practice (multiple selections possible)

There were 95 participants that had used a mobile app for teaching/supervision, of which 71 had requested their students to do the same. These participants were asked to state the purpose for making this request; results are summarised in Fig.  14 . The responses in the ‘Other’ category included quizzes, vocabulary practise, feedback, class activities, creative practice. Figure  14 shows that the primary reason for asking students to use mobile apps was for the purposes of communicating with others, sharing documents, followed by accessing course information.

Mobile apps recommended to students (multiple selections possible)

3.5 Apps for Research

Of the 172 participants who had used mobile apps for academic purposes, 146 did so for research purposes (85%), one participant provided no answer. This group of 146 participants were asked what academic-related mobile apps they had used for research purposes from a list of possibilities provided. The results, summarised in Fig.  15 , show the file-hosting app Dropbox was extremely popular and used by 62% of researchers (N = 91). There was a substantial number of participants (65) who provided ‘other’ options, with many participants naming up to 6 or 7 apps, including Google apps (N = 22, among which were Drive: 12, Docs: 3, Keep: 3, Slides: 2, Gmail: 3), Mendeley (N = 5), Skype (4), voice recording (3), Twitter (3). Participants also mention apps that had been written by themselves or their students.

Mobile apps used for research (multiple selections possible)

Participants were also asked what research purposes they used mobile apps for (see Fig.  16 ). Storage and sharing of documents, as well as searching and note-taking were the main reasons for researchers using mobile apps. Only 22 ‘Other’ answers were collected, mostly naming different uses such as reading (6), recording of various data, such as interviews (2) and notes on whiteboards (1), and app development (2).

Purpose of mobile app use for research (multiple selections possible)

3.6 Impact of Apps on Academic Experience

All participants who had indicated that they used mobile apps for academic purposes were asked to respond to questions on their use of mobile apps for their teaching/supervision or research, their knowledge of apps, and their use of mobile apps. The response required from participants was on a 5-point Likert scale from strongly agree (1) to strongly disagree (5), see Fig.  17 . The factors that participants reported to most strongly agree with was “my research or teaching benefited from the use of mobile apps” (mean = 1.72) and they “had no problems finding a suitable app for my research or teaching” (mean = 2.40). The attitude statement that participants most strongly disagreed with was “I experienced difficulties in using mobile apps” (mean = 3.60). Other responses regarding the attitude towards app use were; “the outcome of my research or teaching was impacted by the use of mobile apps” (mean = 2.49), “my research or teaching practice was conducted differently as a result of using mobile apps” (mean = 2.53), and “I know where to go to get help with mobile apps” (mean = 2.62).

Attitude to mobile app use: data out of 100% = 172 participants

Only 20% of participants experienced difficulties when using mobile apps in an academic setting. 45 to 60% of participants knew where to seek help and where to find suitable apps (vs 15–25% who did not; 7% no answer). A similar observation holds for the perceived impact of using apps for research and teaching both in terms of change of practice and outcomes.

However, nearly 90% of mobile app users responded that they felt they had benefited from, or felt neutral about, the inclusion of mobile apps in their academic activity (2% slightly disagreed, 8% no answer).

3.7 Experience of Users

The survey provided an opportunity for participants to provide any further comments they wished on mobile app usage in an academic setting. 60 participants provided comments, 18 from participants who had not used apps for academic purposes, and 42 from participants who had experience with such apps. Participants will be referred to by identifiers P1 to P269. Many of the concerns voiced were brought up by non-users and users alike. We, therefore, do not discuss their comments separately but indicate which category a participant falls into next to the identifier (P U for users and P N for non-users).

In comments provided by non-users, distrust in app/technical reliability were expressed, such as by participant P N 124: “Technology moves so fast that planned obsolescence is commonplace. New apps have a track record of failure in their first years: this does not look good to students if suddenly the app for their course falls over”. Similarly, P N 146 comments “I wish people would switch their bloody mobile phones off, and get a life really.”, and P N 231 “I do not have a mobile”.

Participants also discussed mobile app usefulness from a pedagogical viewpoint , stating that “[…] we have gone into more and more web-based teaching, and moodle etc. However, I have seen that … students who will end up as designers in some companies do not gain much from these approaches. In my judgement and experience … use of white board and limited amount of notes uploaded will work well, with [a] lot of laboratory type hands-on elements. I strongly believe that if we [lose] the 'human touch" in [the] classroom setting, it will gradually and negatively affect the quality of the graduates we produce” (P N 128).

Several users commented that they are planning to do more or feel still at the beginning of their journey and wish for more support : P U 233: “I've been reluctant because of time, planning and other flexibility related restrictions it places”, P U 254: “Most of the learning on this is on my own. more exposure is needed through seminar etc.” P U 87: “Would be great to get some training on this)”. Some expressed reservations about institutional support, for example, P U 108: “Help with mobile apps seems to be largely found in internet searches of forum posts and vendor provided documentation”.

Participants expressed that guidance on choosing apps was needed as “It would also be great if there was some sort of online resource on the uni website that lists and briefly explains some of the apps that might be useful when conducting research” (P U 39), and the concern that “There is simply not the capacity in ITS to support mobile app usage” (P N 124). Similarly, non-users wished for more support: P N 108: “Help with mobile apps seems to be largely found in internet searches of forum posts and vendor provided documentation”,

Some participants considered app use inconvenient , claiming “In many instances and situations a well thought out website enhanced for use on mobile will be more useful and less cumbersome than an app. I despise having to download and constantly update several apps, plus they come with intrusive permissions” (P71). Or participants felt that apps were “only useful where use of a real computer is impossible”. The context within which apps could be integrated into the learning environment caused some uncertainty, with several comments highlighting this reservation, “It is sometimes challenging to find the most appropriate app to meet a specific teaching purpose” and “The challenge will be to develop apps or modify existing apps to suit the purpose of the user and the context of the user”.

Finally, some participants expressed a dislike or unfamiliarity with/for phones and technology in general: “I wish people would switch their bloody mobile phones off, and get a life really [..]” (P N 146) and a distrust in apps as they expressed concern that “they need to be reliable enough that researchers can be confident that they will not suffer data losses if they use just apps” (P U 105). Similarly, worries about the hardware were expressed: “Our devices need updating. Phones are personally owned and my ipad is too old for some of the apps I want to use.” (P U 155).

Some comments seemed to be expressions of undisclosed fears that were channelled into the reasons given. For example, P U 104 raised the issue that “One can only move as fast as students are able. One can only do so much introducing of new technology—you can get to a point where you have built a learning task for example on a particular resource and then find that half the class cannot even access it”.

A theme that was detected in the responses received reinforced the mobile nature of both tertiary education and academic publishing today. This can be specifically seen in the discussion of mobile and on-the-go teaching, learning, and research. Participants listed the importance of being able to collect data, take notes, as well as communicate with peers, participants, and users in a variety of situations. One participant noted, “I've largely found it useful for mobility rather than anything else.” Another participant, whose complaint we noted earlier regarding screen size making viewing information less pleasurable for them compared to a computer, did note “at least information is available and accessible when on the move”. A further PhD student stated that “mobile apps are great. If you are in tedious work meetings you can work on easy bits of your thesis and people just think you are diligently taking notes”.

Similar numbers of participants believed their work-life was or was not impacted by mobile apps as participants who believed their teaching or research practices were different today because of their mobile app use. Investigation of the impact of technologies including mobile devices and applications on traditional pedagogies and research practices and processes warrants further empirical investigation.

Significant discussion related to use of apps for teaching rather than research. With some being enthusiastic: “We are moving into the new generation Apps is the tool to connect with the students. / Let’s not hesitate. We need to be engaging successfully to create a sense of new age.” (P U 81), while others are quite reserved about technology use, including “ web based teaching, and moodle” (P N 128). P U 78 described challenges: “It is sometimes challenging to find the most appropriate app to meet a specific teaching purpose”. P N 203 teaches online papers and comments “it would be great to have a way for students to access discussion groups and to have virtual communication through a mobile app”.

Several participants explicitly wished for apps that allowed access to library resources such as eBook readers (P U 33, P U 7), library search (P U 42, P U 120), and a personal library (P U 202). Some participants were very enthusiastic about the potential of apps in the academic environment, such as “We are moving into the new generation of Apps is the tool to connect with the students. Let’s not hesitate. We need to be engaging successfully to create a sense of new age” (P U 81) and “Apps greatly increases my ability to store quotes and research links” (P U 67). Conversely, some participants used the open feedback option to comment on the shortcomings of their personal phones (P N 169: “I find the real estate of my mobile device is too small [..] my tablet is too slow”), on perceived shortcomings of innovation management (P N 182: “endless workshops”) or even expressed fears about the motivation of the survey (P N 166: “The outcome of studies like this can be deeply political”); conveying a sense of fear about potentially being forced to use apps for research and teaching.

3.8 Experience of Non-Users

In Sect.  4.2 we report that 35% of the participants had not used mobile apps for academic purposes (N = 93). More than half of these non-users (N = 50/93) indicated that they did not intend using mobile apps for academic purposes in the future. Forty-four percent (41/93) of these non-users reported that they do plan to use apps in the future. We asked non-responders what their reasons for non-use of mobile apps in academic contexts were. Forty-seven people responded to this question. Nearly half of the 47 participants reported they lacked knowledge about how they might use mobile apps for their purposes. Further to this approximately one-third of the participants confessed their disinterest in apps, while approximately a third considered them to be irrelevant for their teaching or research needs. Eight participants reported a lack of apps for their purposes and 7 participants discussed the perceived lack of support from the university. Other opinions suggested that computers and large screen devices serve their needs better than mobile devices for academic purposes. One academic responder twice noted planned obsolescence as a factor hindering their use of mobile apps in the academic context. Some participants also named specific fields for which they believed mobile apps or small screen devices would not be suitable.

We noted earlier that 44% of non-users had indicated they might use mobile apps in the future. We asked these non-users to select what factors might influence their future use (multiple selections were possible). 40 participants responded to this question (1 provided no data) resulting in 145 selections (see Fig.  18 ). Non-users were most interested in mobile apps that supported them to share or communicate with others (selected 23 times), see Fig.  19 . The option ‘Other’ included participant sign-up, reading, engaging with students in and out of lectures (3), and the possibility of so far unforeseen usages (3).

Reasons for intended non-use of mobile apps (multiple selections possible)

Non-users intended future use of mobile apps

The 41 participants who reported not using mobile apps were asked how helpful the six factors shown in Fig.  20 might be in facilitating the uptake of mobile app usage for academic purposes. This question was posed as a 5-point Likert scale from very helpful (1) to very unhelpful (5), for which 38 of 41 people responded. Responses show that “more appropriate apps” (mean = 1.54) and “easier to use apps” (mean = 1.55) were the factors most likely to facilitate uptake with app non-users. This was followed by; “more practical support” (mean = 1.58), “more institutional support (mean = 1.66), “more information about apps” (mean = 1.66) and better access to appropriate devices (mean = 1.81).

Factors facilitating uptake of mobile apps

Of the six factors posed, the two that were defined as very helpful and helpful were factors relating to “easier to use apps” and “more appropriate apps”.

4 Discussion

We here discuss our findings in light of our research questions, their implications and opportunities. Our research was motivated by three questions, which we will answer here based on our study results. We will compare and contrast our findings with the related work, giving specific relation to two related surveys of academic use of mobile technology use (Lai & Smith, 2018 ; Shraim & Crompton, 2015 ).

4.1 Answering RQ1: Are Academics Using Mobile Apps?

Our study had 269 participants from a potential pool of 1400 staff and HRD students (19.2% response rate, including 28% academics and 10%). The two related surveys had similar response rates of 24% among teaching staff (Lai & Smith, 2018 ) and 29% (Shraim & Crompton, 2015 ), with similar distributions across gender (i.e. a slight to significant majority of male respondents).

172 of our 269 participants (64%) had used mobile apps for academic purposes such as teaching or research. The percentages among academic staff and doctoral students were comparable at 67% and 69%, respectively, but much lower for Master’s students. 95 of 172 (56%) had used a mobile app for teaching/supervision purposes; 146 (85%) had used one for research purposes; and 70 (41%) had used apps for both. By contrast, Lai and Smith ( 2018 ) found that the majority (75–90% for comparable categories) of their respondents had not used any mobile technology for teaching. Shraim and Crompton ( 2015 ) did not report previous app use for academic purposes.

We found that 62% of the 141 female participants and 67% of the 125 male participants had used apps for academic purposes. By contrast, Lai and Smith ( 2018 ) found that more female teachers used mobile technologies for teaching than male teachers. They hypothesised that the reason may have been that the female teachers were younger than the male teachers in their response cohort. They also found that junior teachers are more willing to learn to use new technologies than senior teachers. We similarly found the strongest engagement with mobile technology among the 21–30 year-olds (71%), followed by the 31–40 year-olds (64%). Shraim and Crompton ( 2015 ) noted that three-quarters of their respondents were aged between 25 and 45, going so far as to suggest that older faculty chose not to respond, perhaps being less inclined to use mobile technology as part of their teaching. Some of our participants were of a generation where the technology may be seen as a hindrance or unfamiliar tool. For example, participant P N 191 stated “I think strategic training is really necessary for people like myself who is not a digital native—what are the benefits? How to develop greater usage in daily work and life?” However, very few participants who had used apps did report technical difficulties (see our discussion in Sect.  4.3 ). We conclude that the study participants who did use apps for teaching and research were proficient, while the extent to which non-users experienced difficulties is hard to gauge.

Our findings support the related literature that academics are using mobile technology and mobile apps for teaching and research. These findings imply there is a need to more deeply understand the reasons for app use/non-use by academics across tertiary institutions. From there, an exploration can be started of how appropriate support can be provided.

4.2 Answering RQ2: Which Apps are Used by Academics for Teaching and Research?

Apps for Supervision/Teaching Participants reported app use for tasks that involve sharing or storing documents, as well as for communication with colleagues and students, and some use for in-class surveys, or keeping up with recent blogs. Similarly, it was reported that teachers required students to use apps primarily for communication and information storage or delivery purposes. This is in line with many studies that suggest that mobile devices in higher education may provide new opportunities for information gathering and use, content access, communication, collaboration and reflection (Beddall-Hill et al., 2011 ; Bowen & Pistilli, 2012 ). The tool that academics most reported as being used by themselves and by students was Dropbox, a file sharing and storing app that facilitates collaboration and information dissemination. Neither of the two related surveys (Lai & Smith, 2018 ; Shraim & Crompton, 2015 ) focussed on the use of mobile apps or specific software, but rather on technology use.

However, many participants reported a lack of time, resources, and control as reasons why they have not successfully implemented mobile apps into their teaching for use by or with students. Participant P U 233 noted “I've been looking at Kahoot at the like for teaching. I've been reluctant because of time, planning and other flexibility related restrictions it places”, while P U 154 reported “at the moment, I am just using the iPad to save paper. It hasn't really impacted how I teach. I am aware that there is far more I could do with it, but I do not have a lot of control over what/how I teach.” Lack of time, resources and knowledge are well-known issues for academic use of technology that were observed in other studies as well (Ajjan & Hartshorne, 2008 ; Lai & Smith, 2018 ; Shraim & Crompton, 2015 )).

Another interesting aspect was the perception that teachers need to restrict students’ screen time (P N 187): “with my overseas students (English language learners) … I try to promote personal f2f interaction in my lessons and try to get the young students away from their screens!” While this was not a prevalent theme, it deserves consideration in future research.

Apps for Research Sixty-four percent (172 of 269) of participants had used apps for research or teaching. A number of apps were listed for participants to select from. The participant was able to select multiple apps that they had used for their research. The research team had hypothesised a number of bibliographic, file sharing, and document creation tools for participants to select from. While file hosting and sharing was reported as being used significantly by participants, it was interesting to note that social media (Twitter), communication (Skype), as well as file creation and storage solutions (Drive, Google apps, voice recording) were also listed by numerous participants. If we consider the nature of research and international connectedness that is expected in universities today, it is unsurprising that a number of these apps that allow for asynchronous collaboration and long-distance telecommunication are listed as central to the modern research framework. This is summed up by one participant (P U 206) who commented “the survey seems to focus on information management. Apps also allow easy access to communication and collaboration channels.”

Higher numbers of academics and students reported using apps in the early phases of the research process for tasks such as note-taking (64 participants), search (66), research planning (59), communication (43), data collection (60), and document sharing (73), compared to later phases of the research process such as data analysis (21), presentation (30), and publishing (16).

App use for research was not considered in the related surveys (Lai & Smith, 2018 ; Shraim & Crompton, 2015 ). To the best of our knowledge, no comparable data has been collected so far. The implications of these findings is that work to support and develop appropriate mobile applications that service academics during all phases of the teaching and research process are required.

4.3 Answering RQ3: What is the Experience of Academic App Use?

We here discuss first the experience of respondents who had used apps, and then those of respondents who did not use apps but had identified obstacles.

Impact on Academic Experience The majority of our participants did not encounter any issues with finding and identifying relevant apps. Our participants also did not encounter major technical difficulties when using apps. For example, only three explicit comments called for technical support and only 20% of participants mentioned technical difficulties. Most observed that using apps influenced the way they did their teaching and research. The vast majority of mobile app users (90%) in our study felt that they had benefited from, or were neutral about, the inclusion of mobile apps in their academic activity. Both Lai and Smith ( 2018 ) and Shraim and Compton ( 2015 ) also explored teachers’ attitudes towards mobile technology use in the classroom but did not ask if teachers experienced the technology as having been helpful. Like most other publications on technology use for teaching (Hahn, 2014 ; Canuel & Chrichton, 2015 ; MacNeill, 2015 ), they asked instead about the teachers’ beliefs in the opportunity of enhanced learning, which may not align with the actual experience of using mobile apps. As a potential drawback, they named students becoming less critical, or increasing their workload (Lai & Smith, 2018 ). Given the low percentage of mobile technology use (< 25%), this feedback is largely not based on the academics’ experience. While they reported that their departments supported the use of mobile technology, it remains unclear if this describes a positive attitude or practical help (Lai & Smith, 2018 ).

Lack of support A common theme was a lack of support by the institution for mobile device and mobile app use for teaching, learning, and research purposes. Participants noted a need to be supported in identifying apps of relevance and suitability to their teaching and research. One academic participant discussed a “notable lack of support for adequate apps, a case in point being that the Uni does not provide apps suitable for reading online books” (P N 124). The results of both studies suggest that non-users may be more willing to use mobile apps if institutional support and guidance were provided. This desire for institutional support came from both academics and students, with higher degree student P U 33 reporting “it would be very beneficial to have an online list, or equivalent, of useful apps for students, varying from note-taking, referencing, data collection right through to ones specific to different fields of study. Many of the apps I now use would have been extremely useful had I known about them when I began this degree.” This reporting by academics of a need for institutional and wider support in selecting and using apps to support their pedagogy, classroom practice, and research is in line with Horizon Report Preview (EDUCAUSE, 2019 ) that calls for sustained support and professional development in order to take advantage of the new teaching practice opportunities afforded by the inclusion of digital devices within the education environment. A similar sentiment has been mirrored in other studies (Ajjan & Hartshorne, 2008 ; Chen et al., 2015 ; Lai & Smith, 2018 ; Shraim & Crompton, 2015 ).

Non-use 35% of our participants reported not having used mobile apps for academic purposes. Furthermore, approximately half these reported they had no intention to use apps in the future. Disinterest in mobile apps for teaching, or a view of mobile apps as being irrelevant to the participant, were common reasons for these responses. Some also noted a preference for desktop solutions for these tasks. This is summed up by P N 174 who noted “I don't like/prefer to use apps for academic purposes. I feel more comfortable on desktop/laptop when having to access content relating to academic needs”, while another participant stated “computers have more options than mobiles” and the perennial concern “screen size makes viewing information not as pleasurable as computer”.

Of the non-users, slightly under half suggested they might use or were willing to use mobile apps in the future for academic purposes. Non-users of apps were primarily interested in the potential ability to communicate or share with others. It is interesting to note that the communication affordances are of high interest because in both surveys the view that there is no use for apps besides for communication was a common criticism for mobile apps. Perceived potential benefits of mobile apps by non-users were features such as participant sign-up, reading and engaging with students in and out of lectures. Another feature that participants noted as a potential positive for mobile apps was the perceived convenience of managing, capturing, collecting, and storing information.

Many participants saw a need for future development, advancement, and indeed further research such as that we offer here. Almost all non-users identified “easier to use [apps]” and “more appropriate [apps]” as important or helpful. One participant summed this up “the challenge will be to develop apps or modify existing apps to suit the purpose of the user and the context of the user”, while another stated “I think [mobile apps] have some good potential for engaging students in classrooms and out of classrooms. I also don't think they are the be all, end all of engagement (i.e., necessary but not sufficient for good engagement).” There appears, in addition, to have been a perception that because software or apps are open source that they do not require coordinated technical support or training from the University. Through conducting this survey we found that there is a need to provide support and information to users for both subscription software as well as open-source alternatives.

The survey has highlighted that current users have typical usage patterns and generally feel confident with the use of mobile apps for a range of purposes. There was also a group of non-users and low-users that did not feel confident. We feel the implications of our findings are the need to support academics to locate and use mobile apps during teaching and research and the desire from academics for this support as well as for new mobile apps to meet their needs.

4.4 Limitations

This study was based at a single university in New Zealand; however, its results and recommendations for engagement and need for ongoing support are potentially widely applicable for a western tertiary education environment due to similarities in academic environments. One may expect differences in the specifics of apps used, such as the prevalence of Google tools in this sample, vs the use of OneDrive for similar tasks in universities with Microsoft contacts.

A participation rate is in keeping with typical response rates for similar online studies (Fosnacht et al., 2017 ; Nulty, 2008 ; Van Mol, 2017 ). As the participants were self-selected, it is unclear to what extent our sample accurately reflects the university situation. As our participants were self-selected, they did not necessarily constitute a representative sample of the whole university but rather reflected the feedback of people who felt strongly enough to engage in the process. While both users and non-users of mobile apps were explicitly targeted, the resulting sample consisted of predominantly mobile app users (66%). We hypothesise that non-users may have been less inclined to respond to a survey about app usage.

The study ran at the same time in two consecutive years. One notable difference was the proportion of academic vs student participants within the studies. However, there did not appear to be overall a significant variation between the results obtained in the first year vs the results from the second year, which were therefore presented here together. We noted that some participants commented also on the use of web applications. In order to keep the study results comparable, we did not change any questions. However, in future studies we would wish to include both mobile apps and web apps (i.e., software as a service), thus addressing the use of any software services away from the office or lab environment.

5 Conclusion

While students and academics use digital devices in the higher education environment, the uptake of mobile apps for tertiary teaching and research has only begun to be studied. Research on technology use in teaching and learning rarely focuses on the experience of app use by academics. The impact and experience of the institution’s expectations regarding apps by academics is not well studied. Our research attempts to understand how apps are being used in tertiary teaching and research, including what are the perceived benefits and barriers. Our study used an online survey, aiming to answer three research questions. Our study here is unique in that it has investigated students and academics’ attitudes to mobile apps in both the tertiary classroom and the research environment.

The contributions of our research presented here are the following: We conducted the first study into the experience of mobile app use for teaching and research by academics. Findings from our research are as follows: (1) Mobile apps were used by academics and students for both teaching and research, primarily in the form of document & data storage and exchange, and communication. Furthermore, the stated primary motivators for future mobile app use for both teaching and research were again the ability to communicate, collaborate and share with others. (2) Very little app use was recorded for in-class activities (teaching) or in-field activities (research). (3) Our study results and related work show that at present academics and students use mobile apps due to intrinsic personal motivations rather than institutional support or provision. There remain, consequently, opportunities for better support of mobile app use. (4) Both students and academics reported that institutional support and flexibility would likely provide motivation and lead to increased app use for both research and teaching.

Many of the apps named in our study were mobile versions of web apps (such as Dropbox, Evernote, Google Drive). Some participants may even have interpreted mobile app use to mean both mobile apps and web apps (e.g., for bibliographic software Zotero and Endnote). This interplay of mobile apps and web apps (or mobile access to web apps) has not been explored for the academic context so far and should be studied in a follow-up survey. Extending this survey with consideration of software as a service (SaaS) used on mobile devices may shine a light on some of these wider-reaching applications which also facilitate teaching and research.

Our study is the first of its kind, exploring the practical experience of academics using mobile apps for teaching and research. Data such as ours can inform academic management to better support students and staff with mobile app selection and use in the academic context.

Availability of Data and Material

Data not available for privacy reasons.

Code Availability

Not applicable.

Ajjan, H., & Hartshorne, R. (2008). Investigating faculty decisions to adopt Web 2.0 technologies: Theory and emperical tests. Internet and Higher Education, 11 , 71–80.

Article   Google Scholar  

Anderson, M. (2015). Technology device ownership: 2015. Report by Pew Research Center. http://www.pewinternet.org/2015/10/29/technology-device-ownership-2015/

Aher, R. S., Mayekar, A., & Malik, N. (2017). Mobile technology in library service. Sansmaran Research Journal, 7 (2), 63–69.

Google Scholar  

Beddall-Hill, N. L., Jabbar, A., & Al Shehri, S. (2011). Social mobile devices as tools for qualitative research in education: iPhones and iPads in ethnography, interviewing, and design-based research. Journal of the Research Center of Educational Technology, 7 (1), 67–90.

Bell, A., Headrick Taylor, K., Riesland, E., & Hays, M. (2019). Learning to see the familiar: Technological assemblages in a higher education (non)classroom setting. British Journal of Educational Technology, 50 (4), 1573–1588. https://doi.org/10.1111/bjet.12800

Bowen, K., & Pistilli, M. D. (2012). Student preferences for mobile app usage. Edu-cause Research Bulletin.

Breeding, M. (2019). Investing in enhanced digital experience. Computers in Libraries, 39 (8), 9–11.

Burch, Z. A., & Mohammed, S. (2019). Exploring Faculty Perceptions about Classroom Technology Integration and Acceptance: A literature Review. The Richard W. Riley College of Education and Leadership Publications. 196. https://scholarworks.waldenu.edu/cel_pubs/196

Canuel, R., & Chrichton, C. (2015). Leveraging apps for research and learning: A survey of Canadian academic libraries. Library Hi Tech, 33 (1), 2–14. https://doi.org/10.1108/LHT-12-2014-0115

Canuel, R., MacKenzie, E., Senior, A., & Torabi, N. (2016). Apps for academic success: Developing digital literacy and awareness to increase usage. Education for Information, 33 (3), 1–18. https://doi.org/10.3233/EFI-160085

Carlos, A. (2012). Research on the go: Mobile tools for conducting research. The Reference Librarian, 53 (4), 433–440.

Carter, A., Liddle, J., Hall, W., & Chenery, H. (2015). Mobile phones in research and treatment: Ethical guidelines and future directions. JMIR Mhealth and UHealth, 3 (4), 1.

Chen, G. (2011). Mobile research: Benefits, applications and outlooks. Internationalization, Design and Global Development, 1 , 11–16.

Chen, B., Sivo, S., Seilhamer, R., Sugar, A., & Mao, J. (2013). User acceptance of mobile technology: A campus-wide implementation of blackboard’s MobileTM learn application. Journal of Educational Computing Research, 49 (3), 327–343. https://doi.org/10.2190/EC.49.3.c

Chen, B., Seilhamer, R., Bennet, L., & Bauer, S. (2015). Students’ mobile learning practices in higher education: A multi-year study. Educause Review. http://er.educause.edu/articles/2015/6/students-mobile-learning-practices-in-higher-education-a-multiyear-study

Compton, H., & Burke, D. (2018). The use of mobile learning in higher education: A systematic review. Computers & Education, 123 , 53–64. https://doi.org/10.1016/j.compedu.2018.04.007

Davidson, J., Paulus, T., & Jackson, K. (2016). Speculating on the future of digital tools for qualitative research. Qualitative Inquiry, 22 (7), 606–610.

EDUCAUSE (2019). Horizon Report Preview: 2019 Higher Education Edition . https://library.educause.edu/-/media/files/library/2019/2/2019horizonreportpreview.pdf

Fosnacht, K., Sarraf, S., Howe, E., & Peck, L. K. (2017). How important are high response rates for college surveys? The Review of Higher Education, 40 (2), 245–265. https://doi.org/10.1353/rhe.2017.0003

Galanek, J. D., Gierdowski, D. C., & Brooks, D. C. (2018). ECAR study of undergraduate students and information technology, 2018 . Research report. Louisville, CO: ECAR. https://library.educause.edu/-/media/files/library/2018/10/studentitstudy2018.pdf?la=en&hash=C590C1F6C62B77792711BFAC1F642254A5618590

Galanek, J. D., & Gierdowski, D. C. (2019). ECAR Study of Faculty and Information Technology, 2019. Research report. Louisville, CO: ECAR. https://www.educause.edu/-/media/files/library/2019/12/facultystudy2019.pdf?la=en&hash=F0F8AF83BD870239A7E5CD285FBDBE157AA51135

Gierdowski, D. C., Brooks, D. C, & Galanek, J. D. (2020) EDUCAUSE 2020 Student Technology Report: Supporting the Whole Student. Research report. CO: ECAR. https://www.educause.edu/ecar/research-publications/student-technology-report-supporting-the-whole-student/2020/technology-use-and-environmental-preferences

Garcia, B., Welford, J., & Smith, B. (2016). Using a smartphone app in qualitative research: The good, the bad and the ugly. Qualitative Research, 16 (5), 508–525.

Gikas, J., & Grant, M. M. (2013). Mobile computing devices in higher education: Student perspectives on learning with cellphones, smartphones & social media. The Internet and Higher Education, 19 , 18–26. https://doi.org/10.1016/j.iheduc.2013.06.002

Hahn, J. (2014). Undergraduate research support with optical character recognition apps. Reference Services Review , 42 (2), 336-350.

Hanbridge, A. S., Tin, T., & Sanderson, N. (2018). Information literacy skills on the go: Mobile learning innovation. Journal of Information Literacy, 12 (1), 118–136.

Hennig, N. (2014). Apps for librarians: Using the best mobile technology to educate, create and engage. Libraries Unlimited. https://ebookcentral-proquest-com.ezproxy.waikato.ac.nz/lib/waikato/detail.action?docID=1813321

Hinze, A., Vanderschantz, N., Timpany, C., Cunningham, S. J., Saravani, S. J., & Wilkinson, C. (2017a). Use of mobile apps for teaching and research . (Working Paper Series 01/2017a). Department of Computer Science, The University of Waikato.

Hinze, A., Vanderschantz, N., Timpany, C., Saravani, S. J., Cunningham, S. J., & Wilkinson, C. (2017b). Use of mobile apps for teaching and research–implications for digital literacy. In International Conference on Asian Digital Libraries (pp. 173–184). Springer.

Kim, B. (2013). Responsive web design, discoverability, and mobile challenge. The library mobile experience: Practices and user expectations. Library Technology Reports, 49 (6), 29–39. https://doi.org/10.5860/ltr.49n6

Lai, K.-W., & Smith, L. (2018). Socio-demographic factors relating to perception and use of mobile technologies in tertiary teaching. British Journal of Educational Technology, 49 , 492–504. https://doi.org/10.1111/bjet.12544

MacNeill, F. (2015). Approaching apps for learning, teaching and research. In: A. Middleton (Ed.), Smart learning: Teaching and learning with smartphones and tablets in post compulsory education (pp. 238–264).

McGeeney, K. (2015). What we learned about surveying with mobile apps. Report by Pew Research Center. http://www.pewresearch.org/fact-tank/2015/04/02/what-we-learned-about-surveying-with-mobile-apps/

Morris, N. P., Lambe, J., Ciccone, J., & Swinnerton, B. (2016). Mobile technology: Students perceived benefits of apps for learning neuroanatomy. Journal of Computer Assisted Learning, 32 , 430–442. https://doi.org/10.1111/jcal.12144

Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment & Evaluation in Higher Education, 33 (3), 301–314. https://doi.org/10.1080/02602930701293231

Pedro, L., Barbosa, C., & Santos,C. (2018). A critical review of mobile learning integration in formal educational contexts. International Journal of Educational Technology in Higher Education, 15 (10), 1. https://doi.org/10.1186/s41239-018-0091-4

Perrin, A. (2021). Mobile Technology and Home Broadband 2021. Report by Pew Research Center. https://www.pewresearch.org/internet/2021/06/03/mobile-technology-and-home-broadband-2021/

Raento, M., Oulasvirta, A., & Eagle, N. (2009). Smartphones: An emerging tool for so-cial scientists. Sociological Methods & Research, 37 (3), 426–454.

Saragossi, J., Kasten, K., & Costello, L. (2018). Mobile applications in academic libraries. Library Faculty Publications, 21 . https://commons.library.stonybrook.edu/library_articles/21

Schepman, A., Rodway, P., Beattie, C., & Lambert, J. (2012). An observational study of undergraduate students’ adoption of (mobile) note-taking software. Computers in Human Behavior, 28 , 308–317. https://doi.org/10.1016/j.chb.2011.09.014

Sensor Tower. (2019). Global app revenue grew 23% in 2018 to more than $71 billion on iOS and Google Play . https://sensortower.com/blog/app-revenue-and-downloads-2018

Shraim, K., & Crompton, H. (2015). Perceptions of using smart mobile devices in higher education teaching: A case study from palestine. Contemporary Educational Technology, 6 (4), 301–318.

Shuk Han Wai, I., Sze Yan Ng, S., Chiu, D. K. W., Ho, K. K. W., & Lo, P. (2016). Exploring undergraduate students’ usage pattern of mobile apps for education. Journal of Librarianship and Information Science. https://doi.org/10.1177/0961000616662699

Singh Negi, D. (2014). Using mobile technologies in libraries and information centers. Library Hi Tech News, 31 (5), 14–16.

Singh, G., & Hardaker, G. (2014). Barriers and enablers to adoption and diffusion of eLearning: A systematic review of the literature–a need for an integrative approach. Education+ Training, 56 (2/3), 105–121.

Statista. (2019). Most popular Apple App Store categories in May 2019, by share of available apps . https://www.statista.com/statistics/270291/popular-categories-in-the-app-store/

Tidal, J. (2017). One site to rule them all, redux: The second round of usability testing of a responsively designed website. Journal of Web Librarianship, 11 (1), 16–34. https://doi.org/10.1080/19322909.2016.1243458

van Arnhem, J-P. (2015). Apps and gear for ethnographic field research. The Charleston Advisor, 17 (2), 58-64.

Van Mol, C. (2017). Improving web survey efficiency: The impact of an extra reminder and reminder content on web survey response. International Journal of Social Research Methodology, 20 (4), 317–327. https://doi.org/10.1080/13645579.2016.1185255

Wong, S. H. R. (2012). Which platform do our users prefer: Website of mobile app? Reference Services Review, 40 (1), 103-115.

Download references

Open Access funding enabled and organized by CAUL and its Member Institutions.

Author information

Authors and affiliations.

Department of Software Engineering, University of Waikato, Hamilton, New Zealand

Annika Hinze & Sally Jo Cunningham

Department of Design, University of Waikato, Hamilton, New Zealand

Nicholas Vanderschantz & Claire Timpany

University Library, University of Waikato, Hamilton, New Zealand

Sarah-Jane Saravani & Clive Wilkinson

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Annika Hinze .

Ethics declarations

Conflict of interest, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Hinze, A., Vanderschantz, N., Timpany, C. et al. A Study of Mobile App Use for Teaching and Research in Higher Education. Tech Know Learn 28 , 1271–1299 (2023). https://doi.org/10.1007/s10758-022-09599-6

Download citation

Accepted : 10 March 2022

Published : 05 June 2022

Issue Date : September 2023

DOI : https://doi.org/10.1007/s10758-022-09599-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mobile apps
• Academic survey
• Tertiary teaching and research
• Information behavior
• Twenty-first century abilities
• Find a journal
• Publish with us
• Track your research

mobile applications Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Protecting Privacy on Mobile Apps: A Principal–Agent Perspective

Although individuals increasingly use mobile applications (apps) in their daily lives, uncertainty exists regarding how the apps will use the information they request, and it is necessary to protect users from privacy-invasive apps. Recent literature has begun to pay much attention to the privacy issue in the context of mobile apps. However, little attention has been given to designing the permission request interface to reduce individuals’ perceived uncertainty and to support their informed decisions. Drawing on the principal–agent perspective, our study aims to understand the effects of permission justification, certification, and permission relevance on users’ perceived uncertainty, which in turn influences their permission authorization. Two studies were conducted with vignettes. Our results show that certification and permission relevance indeed reduce users’ perceived uncertainty. Moreover, permission relevance moderates the relationship between permission justification and perceived uncertainty. Implications for theory and practice are discussed.

Classifying Mobile Applications Using Word Embeddings

Modern application stores enable developers to classify their apps by choosing from a set of generic categories, or genres, such as health, games, and music. These categories are typically static—new categories do not necessarily emerge over time to reflect innovations in the mobile software landscape. With thousands of apps classified under each category, locating apps that match a specific consumer interest can be a challenging task. To overcome this challenge, in this article, we propose an automated approach for classifying mobile apps into more focused categories of functionally related application domains. Our aim is to enhance apps visibility and discoverability. Specifically, we employ word embeddings to generate numeric semantic representations of app descriptions. These representations are then classified to generate more cohesive categories of apps. Our empirical investigation is conducted using a dataset of 600 apps, sampled from the Education, Health&Fitness, and Medical categories of the Apple App Store. The results show that our classification algorithms achieve their best performance when app descriptions are vectorized using GloVe, a count-based model of word embeddings. Our findings are further validated using a dataset of Sharing Economy apps and the results are evaluated by 12 human subjects. The results show that GloVe combined with Support Vector Machines can produce app classifications that are aligned to a large extent with human-generated classifications.

Privacy Rating of Mobile Applications Based on Crowdsourcing and Machine Learning

With the advent of the 5G network era, the convenience of mobile smartphones has become increasingly prominent, the use of mobile applications has become wider and wider, and the number of mobile applications. However, the privacy of mobile applications and the security of users' privacy information are worrying. This article aims to study the ratings of data and machine learning on the privacy security of mobile applications, and uses the experiments in this article to conduct data collection, data analysis, and summary research. This paper experimentally establishes a machine learning model to realize the prediction of privacy scores of Android applications. The establishment of this model is based on the intent of using sensitive permissions in the application and related metadata. It is to create a regression function that can implement the mapping of applications to score . Experimental data shows that the feature vector prediction model can uniquely be used to represent the actual usage and scheme of a system's specific permissions for the application.

Design and development of an intelligent system based on the Internet of Things for the early detection of forest fires

Automatic environmental monitoring is a field that encompasses several scientific practices for the assessment of risks that may negatively impact a given environment, such as the forest. A forest is a natural environment that hosts various forms of plant and animal life, so preserving the forest is a top priority. To this end, the authors of this paper will focus on the development of an intelligent system for the early detection of forest fires, based on an IoT solution. This latter will thus facilitate the exploitation of the functionalities offered by the Cloud and mobile applications. Detecting and predicting forest fires with accuracy is a difficult task that requires machine learning and an in-depth analysis of environmental conditions. This leads the authors to adopt the forward neural network algorithm by highlighting its contribution through real experiments, performed on the prototype developed in this paper.

Implementation of a Miniaturized Planar Tri-Band Microstrip Patch Antenna for Wireless Sensors in Mobile Applications

Antennas in wireless sensor networks (WSNs) are characterized by the enhanced capacity of the network, longer range of transmission, better spatial reuse, and lower interference. In this paper, we propose a planar patch antenna for mobile communication applications operating at 1.8, 3.5, and 5.4 GHz. A planar microstrip patch antenna (MPA) consists of two F-shaped resonators that enable operations at 1.8 and 3.5 GHz while operation at 5.4 GHz is achieved when the patch is truncated from the middle. The proposed planar patch is printed on a low-cost FR-4 substrate that is 1.6 mm in thickness. The equivalent circuit model is also designed to validate the reflection coefficient of the proposed antenna with the S11 obtained from the circuit model. It contains three RLC (resistor–inductor–capacitor) circuits for generating three frequency bands for the proposed antenna. Thereby, we obtained a good agreement between simulation and measurement results. The proposed antenna has an elliptically shaped radiation pattern at 1.8 and 3.5 GHz, while the broadside directional pattern is obtained at the 5.4 GHz frequency band. At 1.8, 3.5, and 5.4 GHz, the simulated peak realized gains of 2.34, 5.2, and 1.42 dB are obtained and compared to the experimental peak realized gains of 2.22, 5.18, and 1.38 dB at same frequencies. The results indicate that the proposed planar patch antenna can be utilized for mobile applications such as digital communication systems (DCS), worldwide interoperability for microwave access (WiMAX), and wireless local area networks (WLAN).

ENERDGE: Distributed Energy-Aware Resource Allocation at the Edge

Mobile applications are progressively becoming more sophisticated and complex, increasing their computational requirements. Traditional offloading approaches that use exclusively the Cloud infrastructure are now deemed unsuitable due to the inherent associated delay. Edge Computing can address most of the Cloud limitations at the cost of limited available resources. This bottleneck necessitates an efficient allocation of offloaded tasks from the mobile devices to the Edge. In this paper, we consider a task offloading setting with applications of different characteristics and requirements, and propose an optimal resource allocation framework leveraging the amalgamation of the edge resources. To balance the trade-off between retaining low total energy consumption, respecting end-to-end delay requirements and load balancing at the Edge, we additionally introduce a Markov Random Field based mechanism for the distribution of the excess workload. The proposed approach investigates a realistic scenario, including different categories of mobile applications, edge devices with different computational capabilities, and dynamic wireless conditions modeled by the dynamic behavior and mobility of the users. The framework is complemented with a prediction mechanism that facilitates the orchestration of the physical resources. The efficiency of the proposed scheme is evaluated via modeling and simulation and is shown to outperform a well-known task offloading solution, as well as a more recent one.

GPS-based fine-scale mapping surveys for schistosomiasis assessment: a practical introduction and documentation of field implementation

Abstract Background Fine-scale mapping of schistosomiasis to guide micro-targeting of interventions will gain importance in elimination settings, where the heterogeneity of transmission is often pronounced. Novel mobile applications offer new opportunities for disease mapping. We provide a practical introduction and documentation of the strengths and shortcomings of GPS-based household identification and participant recruitment using tablet-based applications for fine-scale schistosomiasis mapping at sub-district level in a remote area in Pemba, Tanzania. Methods A community-based household survey for urogenital schistosomiasis assessment was conducted from November 2020 until February 2021 in 20 small administrative areas in Pemba. For the survey, 1400 housing structures were prospectively and randomly selected from shapefile data. To identify pre-selected structures and collect survey-related data, field enumerators searched for the houses’ geolocation using the mobile applications Open Data Kit (ODK) and MAPS.ME. The number of inhabited and uninhabited structures, the median distance between the pre-selected and recorded locations, and the dropout rates due to non-participation or non-submission of urine samples of sufficient volume for schistosomiasis testing was assessed. Results Among the 1400 randomly selected housing structures, 1396 (99.7%) were identified by the enumerators. The median distance between the pre-selected and recorded structures was 5.4 m. A total of 1098 (78.7%) were residential houses. Among them, 99 (9.0%) were dropped due to continuous absence of residents and 40 (3.6%) households refused to participate. In 797 (83.1%) among the 959 participating households, all eligible household members or all but one provided a urine sample of sufficient volume. Conclusions The fine-scale mapping approach using a combination of ODK and an offline navigation application installed on tablet computers allows a very precise identification of housing structures. Dropouts due to non-residential housing structures, absence, non-participation and lack of urine need to be considered in survey designs. Our findings can guide the planning and implementation of future household-based mapping or longitudinal surveys and thus support micro-targeting and follow-up of interventions for schistosomiasis control and elimination in remote areas. Trial registration ISRCTN, ISCRCTN91431493. Registered 11 February 2020, https://www.isrctn.com/ISRCTN91431493

Survey on Reverse-Engineering Tools for Android Mobile Devices

With the presence of the Internet and the frequent use of mobile devices to send several transactions that involve personal and sensitive information, it becomes of great importance to consider the security aspects of mobile devices. And with the increasing use of mobile applications that are utilized for several purposes such as healthcare or banking, those applications have become an easy and attractive target for attackers who want to get access to mobile devices and obtain users’ sensitive information. Developing a secure application is very important; otherwise, attackers can easily exploit vulnerabilities in mobile applications which lead to serious security issues such as information leakage or injecting applications with malicious programs to access user data. In this paper, we survey the literature on application security on mobile devices, specifically mobile devices running on the Android platform, and exhibit security threats in the Android system. In addition, we study many reverse-engineering tools that are utilized to exploit vulnerabilities in applications. We demonstrate several reverse-engineering tools in terms of methodology, security holes that can be exploited, and how to use these tools to help in developing more secure applications.

Principle of 6G Wireless Networks: Vision, Challenges and Applications

The 5G mobile communication has now become commercially available. Furthermore, research across the globe has begun to improve the system beyond 5G and it is anticipated that 6G will deliver higher quality services and energy efficiency than 5G. The mobile network architecture needs to be redesigned to meet the requirements of the future. In the wake of the commercial rollout of the 5G model, both users and developers have realized the limitations of the system when compared to the system's original premise of being able to support the vast applications of connected devices. The article discusses the related technologies that can contribute to a robust and seamless network service. An upheaval in the use of vast mobile applications, especially those powered and managed by AI, has opened the doors to discussion on how mobile communication will evolve in the future. 6G is expected to go beyond being merely a mobile internet service provider to support the omnipresent AI services that will form the rock bed of end-to-end connected network-based devices. Moreover, the technologies that support 6G services and comprehensive research that enables this level of technical prowess have also been identified here. This paper presents a collective wide-angle vision that will facilitate a better understanding of the features of the 6G system.

Export Citation Format

Share document.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PeerJ Comput Sci

A systematic literature review on the usability of mobile applications for visually impaired users

Muna al-razgan.

1 King Saud University, Riyadh, Saudi Arabia

Sarah Almoaiqel

Nuha alrajhi.

2 Imam Muhammad Ibn Saud University, Riyadh, Saudi Arabia

Alyah Alhumegani

Abeer alshehri, bashayr alnefaie, raghad alkhamiss, shahad rushdi, associated data.

The following information was supplied regarding data availability:

This is a systematic literature review; there is no raw data.

Interacting with mobile applications can often be challenging for people with visual impairments due to the poor usability of some mobile applications. The goal of this paper is to provide an overview of the developments on usability of mobile applications for people with visual impairments based on recent advances in research and application development. This overview is important to guide decision-making for researchers and provide a synthesis of available evidence and indicate in which direction it is worthwhile to prompt further research. We performed a systematic literature review on the usability of mobile applications for people with visual impairments. A deep analysis following the Preferred Reporting Items for SLRs and Meta-Analyses (PRISMA) guidelines was performed to produce a set of relevant papers in the field. We first identified 932 papers published within the last six years. After screening the papers and employing a snowballing technique, we identified 60 studies that were then classified into seven themes: accessibility, daily activities, assistive devices, navigation, screen division layout, and audio guidance. The studies were then analyzed to answer the proposed research questions in order to illustrate the different trends, themes, and evaluation results of various mobile applications developed in the last six years. Using this overview as a foundation, future directions for research in the field of usability for the visually impaired (UVI) are highlighted.

Introduction

The era of mobile devices and applications has begun. With the widespread use of mobile applications, designers and developers need to consider all types of users and develop applications for their different needs. One notable group of users is people with visual impairments. According to the World Health Organization, there are approximately 285 million people with visual impairments worldwide ( World Health Organization, 2020 ). This is a huge number to keep in mind while developing new mobile applications.

People with visual impairments have urged more attention from the tech community to provide them with the assistive technologies they need ( Khan & Khusro, 2021 ). Small tasks that we do daily, such as picking out outfits or even moving from one room to another, could be challenging for such individuals. Thus, leveraging technology to assist with such tasks can be life changing. Besides, increasing the usability of applications and developing dedicated ones tailored to their needs is essential. The usability of an application refers to its efficiency in terms of the time and effort required to perform a task, its effectiveness in performing said tasks, and its users’ satisfaction ( Ferreira et al., 2020 ). Researchers have been studying this field intensively and proposing different solutions to improve the usability of applications for people with visual impairments.

This paper provides a systematic literature review (SLR) on the usability of mobile applications for people with visual impairments. The study aims to find discussions of usability issues related to people with visual impairments in recent studies and how they were solved using mobile applications. By reviewing published works from the last six years, this SLR aims to update readers on the newest trends, limitations of current research, and future directions in the research field of usability for the visually impaired (UVI).

This SLR can be of great benefit to researchers aiming to become involved in UVI research and could provide the basis for new work to be developed, consequently improving the quality of life for the visually impaired. This review differs from previous review studies ( i.e.,   Khan & Khusro, 2021 ) because we classified the studies into themes in order to better evaluate and synthesize the studies and provide clear directions for future work. The following themes were chosen based on the issues addressed in the reviewed papers: “Assistive Devices,” “Navigation,” “Accessibility,” “Daily Activities,” “Audio Guidance,” and “Gestures.” Figure 1 illustrates the percentage of papers classified in each theme.

The remainder of this paper is organized as follows: the next section specifies the methodology, following this, the results section illustrates the results of the data collection, the discussion section consists of the research questions with their answers and the limitations and potential directions for future work, and the final section summarizes this paper’s main findings and contribution.

Survey Methodology

This systematic literature review used the Meta-Analyses (PRISMA, 2009) guidelines to produce a set of relevant papers in the field. This SLR was undertaken to address the research questions described below. A deep analysis was performed based on a group of studies; the most relevant studies were documented, and the research questions were addressed.

A. Research questions

The research questions addressed by this study are presented in Table 1 with descriptions and the motivations behind them.

B. Search strategy

This review analysed and synthesised studies on usability for the visually impaired from a user perspective following a systematic approach. As proposed by Tanfield, Denyer & Smart (2003) , the study followed a three-stage approach to ensure that the findings were both reliable and valid. These stages were planning the review, conducting the review by analysing papers, and reporting emerging themes and recommendations. These stages will be discussed further in the following section.

1. Planning stage

The planning stage of this review included defining data sources and the search string protocol as well as inclusion and exclusion criteria.

Data sources.

We aimed to use two types of data sources: digital libraries and search engines. The search process was manually conducted by searching through databases. The selected databases and digital libraries are as follows:

• • ACM Library
• • IEEE Xplore
• • ScienceDirect
• • SpringerLink
• • ISI Web of Knowledge
• • Scopus.

The selected search engines were as follows:

• • DBLP (Computer Science Bibliography Website)
• • Google Scholar
• • Microsoft Academic

Search string.

The above databases were initially searched using the following keyword protocol: (“Usability” AND (”visual impaired” OR ”visually impaired” OR “blind” OR “impairment”) AND “mobile”). However, in order to generate a more powerful search string, the Network Analysis Interface for Literature Studies (NAILS) project was used. NAILS is an automated tool for literature analysis. Its main function is to perform statistical and social network analysis (SNA) on citation data ( Knutas et al., 2015 ). In this study, it was used to check the most important work in the relevant fields as shown in Fig. 2 .

NAILS produced a report displaying the most important authors, publications, and keywords and listed the references cited most often in the analysed papers ( Knutas et al., 2015 ) . The new search string was generated after using the NAILS project as follows: (“Usability” OR “usability model” OR “usability dimension” OR “Usability evaluation model” OR “Usability evaluation dimension”) AND (“mobile” OR “Smartphone”) AND (“Visually impaired” OR “Visual impairment” OR “Blind” OR “Low vision” OR “Blindness”).

Inclusion and exclusion criteria.

To be included in this systematic review, each study had to meet the following screening criteria:

• • The study must have been published between 2015 and 2020.
• • The study must be relevant to the main topic (Usability of Mobile Applications for Visually Impaired Users).
• • The study must be a full-length paper.
• • The study must be written in English because any to consider any other languages, the research team will need to use the keywords of this language in this topic and deal with search engines using that language to extract all studies related to our topic to form an SLR with a comprehensive view of the selected languages. Therefore, the research team preferred to focus on studies in English to narrow the scope of this SLR.

A research study was excluded if it did not meet one or more items of the criteria.

2. Conducting stage

The conducting stage of the review involved a systematic search based on relevant search terms. This consisted of three substages: exporting citations, importing citations into Mendeley, and importing citations into Rayyan.

Exporting citations.

First, in exporting the citations and conducting the search through the mentioned databases, a total of 932 studies were found. The numbers are illustrated in Fig. 3 below. The highest number of papers was found in Google Scholar, followed by Scopus, ISI Web of Knowledge, ScienceDirect, IEEE Xplore, Microsoft Academic, and DBLP and ACM Library with two studies each. Finally, SpringerLink did not have any studies that met the inclusion criteria.

The chance of encountering duplicate studies was determined to be high. Therefore, importing citations into Mendeley was necessary in order to eliminate the duplicates.

Importing citations into mendeley.

Mendeley is an open-source reference and citation manager. It can highlight paragraphs and sentences, and it can also list automatic references on the end page. Introducing the use of Mendeley is also expected to avoid duplicates in academic writing, especially for systematic literature reviews ( Basri & Patak, 2015 ). Hence, in the next step, the 932 studies were imported into Mendeley, and each study’s title and abstract were screened independently for eligibility. A total of 187 duplicate studies were excluded. 745 total studies remained after the first elimination process. The search stages are shown in Fig. 4 below.

Importing citations into rayyan.

Rayyan QCRI is a free web and mobile application that helps expedite the initial screening of both abstracts and titles through a semi-automated process while incorporating a high level of usability. Its main benefit is to speed up the most tedious part of the systematic literature review process: selecting studies for inclusion in the review ( Ouzzani et al., 2016 ). Therefore, for the last step, another import was done using Rayyan to check for duplications a final time. Using Rayyan, a total of 124 duplicate studies were found, resulting in a total of 621 studies. Using Rayyan, a two-step filtration was conducted to guarantee that the papers have met the inclusion criteria of this SLR. After filtering based on the abstracts, 564 papers did not meet the inclusion criteria. At this stage, 57 studies remained. The second step of filtration eliminated 11 more studies by reading the full papers; two studies were not written in the English language, and nine were inaccessible.

Snowballing.

Snowballing is an emerging technique used to conduct systematic literature reviews that are considered both efficient and reliable using simple procedures. The procedure for snowballing consisted of three phases in each cycle. The first phase is refining the start set, the second phase is backward snowballing, and the third is forward snowballing. The first step, forming the start set, is basically identifying relevant papers that can have a high potential of satisfying the criteria and research question. Backward snowballing was conducted using the reference list to identify new papers to include. It shall start by going through the reference list and excluding papers that do not fulfill the basic criteria; the rest that fulfil criteria shall be added to the SLR. Forward snowballing refers to identifying new papers based on those papers that cited the paper being examined ( Juneja & Kaur, 2019 ). Hence, in order to be sure that we concluded all related studies after we got the 46 papers, a snowballing step was essential. Forward and backward snowballing were conducted. Each of the 46 studies was examined by checking their references to take a look at any possible addition of sources and examining all papers that cited this study. The snowballing activity added some 38 studies, but after full reading, it became 33 that matched the inclusion criteria. A total of 79 studies were identified through this process.

Quality assessment.

A systematic literature review’s quality is determined by the content of the papers included in the review. As a result, it is important to evaluate the papers carefully ( Zhou et al., 2015 ). Many influential scales exist in the software engineering field for evaluating the validity of individual primary studies and grading the overall intensity of the body of proof. Hence, we adapted the comprehensive guidelines specified by Kitchenhand and Charters ( Keele, 2007 ), and the quasi-gold standard (QGS) ( Keele, 2007 ) was used to establish the quest technique, where a robust search strategy for enhancing the validity and reliability of a SLR’s search process is devised using the QGS. By applying this technique, our quality assessment questions were focused and aligned with the research questions mentioned earlier.

In our last step, we had to verify the papers’ eligibility; we conducted a quality check for each of the 79 studies. For quality assessment, we considered whether the paper answered the following questions:

QA1: Is the research aim clearly stated in the research?

QA2: Does the research contain a usability dimension or techniques for mobile applications for people with visual impairments?

QA3: Is there an existing issue with mobile applications for people with visual impairments that the author is trying to solve?

QA4: Is the research focused on mobile application solutions?

After discussing the quality assessment questions and attempting to find an answer in each paper, we agreed to score each study per question. If the study answers a question, it will be given 2 points; if it only partially answers a question, it will be given 1 point; and if there is no answer for a given question in the study, it will have 0 points.

The next step was to calculate the weight of each study. If the total weight was higher or equal to four points, the paper was accepted in the SLR; if not, the paper was discarded since it did not reach the desired quality level. Figure 5 below illustrates the quality assessment process. After applying the quality assessment, 39 papers were rejected since they received less than four points, which resulted in a final tally of 60 papers.

To summarize, this review was conducted according to the Preferred Reporting Items for SLRs and Meta-Analyses (PRISMA) ( Liberati et al., 2009 ). The PRISMA diagram shown in Fig. 6 illustrates all systematic literature processes used in this study.

3. Analysing stage

All researchers involved in this SLR collected the data. The papers were distributed equally between them, and each researcher read each paper completely to determine its topic, extract the paper’s limitations and future work, write a quick summary about it, and record this information in an Excel spreadsheet.

All researchers worked intensively on this systematic literature review. After completing the previously mentioned steps, the papers were divided among all the researchers. Then, each researcher read their assigned papers completely and then classified them into themes according to the topic they covered. The researchers held several meetings to discuss and specify those themes. The themes were identified by the researchers based on the issues addressed in the reviewed papers. In the end, the researchers resulted in seven themes, as shown in Fig. 7 below. The references selected for each theme can be found in the Table A1 . Afterwards, each researcher was assigned one theme to summarize its studies and report the results. In this section, we review the results.

A. Accessibility

Of a total of 60 studies, 10 focused on issues of accessibility. Accessibility is concerned with whether all users are able to have equivalent user experiences, regardless of abilities. Six studies, Darvishy, Hutter & Frei (2019) , Morris et al. (2016) , Qureshi & Hooi-Ten Wong (2020) , Khan, Khusro & Alam (2018) , Paiva et al. (2020) , and Pereda, Murillo & Paz (2020) , gave suggestions for increasing accessibility, ( Darvishy, Hutter & Frei, 2019 ; Morris et al., 2016 ), gave some suggestions for making mobile map applications and Twitter accessible to visually impaired users, and ( Qureshi & Hooi-Ten Wong, 2020 ; Khan, Khusro & Alam, 2018 ) focused on user interfaces and provided accessibility suggestions suitable for blind people. Paiva et al. (2020) and Pereda, Murillo & Paz (2020) proposed a set of heuristics to evaluate the accessibility of mobile applications. Two studies, Khowaja et al. (2019) and Carvalho et al. (2018) , focused on evaluating usability and accessibility issues on some mobile applications, comparing them, and identifying the number and types of problems that visually impaired users faced. Aqle, Khowaja & Al-Thani (2020) proposed a new web search interface designed for visually impaired users. One study, McKay (2017) , focused on accessibility challenges by applying usability tests on a hybrid mobile app with some visually impaired university students.

B. Assistive devices

People with visual impairments have an essential need for assistive technology since they face many challenges when performing activities in daily life. Out of the 60 studies reviewed, 13 were related to assistive technology. The studies Smaradottir, Martinez & Håland (2017) , Skulimowski et al. (2019) , Barbosa, Hayes & Wang, (2016) , Rosner & Perlman (2018) , Csapó et al. (2015) , Khan & Khusro (2020) , Sonth & Kallimani (2017) , Kim et al. (2016) , Vashistha et al. (2015) ; Kameswaran et al. (2020) , Griffin-Shirley et al. (2017) , and Rahman, Anam & Yeasin (2017) were related to screen readers (voiceovers). On the other hand, Bharatia, Ambawane & Rane (2019) , Lewis et al. (2016) were related to proposing an assistant device for the visually impaired. Of the studies related to screening readers, Sonth & Kallimani, (2017) , Vashistha et al. (2015) , Khan & Khusro (2020) Lewis et al. (2016) cited challenges faced by visually impaired users. Barbosa, Hayes & Wang (2016) , Kim et al. (2016) , Rahman, Anam & Yeasin (2017) suggested new applications, while Smaradottir, Martinez & Håland (2017) , Rosner & Perlman (2018) , Csapó et al. (2015) and Griffin-Shirley et al. (2017) evaluated current existing work. The studies Bharatia, Ambawane & Rane (2019) , Lewis et al. (2016) proposed using wearable devices to improve the quality of life for people with visual impairments.

C. Daily activities

In recent years, people with visual impairments have used mobile applications to increase their independence in their daily activities and learning, especially those based on the braille method. We divide the daily activity section into braille-based applications and applications designed to enhance the independence of the visually impaired. Four studies, Nahar, Sulaiman & Jaafar (2020) , Nahar, Jaafar & Sulaiman (2019) , Araújo et al. (2016) and Gokhale et al. (2017) , implemented and evaluated the usability of mobile phone applications that use braille to help visually impaired people in their daily lives. Seven studies, Vitiello et al. (2018) , Kunaratana-Angkul, Wu & Shin-Renn (2020) , Ghidini et al. (2016) , Madrigal-Cadavid et al. (2019) , Marques, Carriço & Guerreiro (2015) , Oliveira et al. (2018) and Rodrigues et al. (2015) , focused on building applications that enhance the independence and autonomy of people with visual impairments in their daily life activities.

D. Screen division layout

People with visual impairments encounter various challenges in identifying and locating non-visual items on touch screen interfaces like phones and tablets. Incidents of accidentally touching a screen element and frequently following an incorrect pattern in attempting to access objects and screen artifacts hinder blind people from performing typical activities on smartphones ( Khusro et al., 2019 ). In this review, 9 out of 60 studies discuss screen division layout: ( Khusro et al., 2019 ; Khan & Khusro, 2019 ; Grussenmeyer & Folmer, 2017 ; Palani et al., 2018 ; Leporini & Palmucci, 2018 ) discuss touch screen (smartwatch tablets, mobile phones, and tablet) usability among people with visual impairments, while ( Cho & Kim, 2017 ; Alnfiai & Sampalli, 2016 ; Niazi et al., 2016 ; Alnfiai & Sampalli, 2019 ) concern text entry methods that increase the usability of apps among visually impaired people. Khusro et al. (2019) provides a novel contribution to the literature regarding considerations that can be used as guidelines for designing a user-friendly and semantically enriched user interface for blind people. An experiment in Cho & Kim (2017) was conducted comparing the two-button mobile interface usability with the one-finger method and voiceover. Leporini & Palmucci (2018) gathered information on the interaction challenges faced by visually impaired people when answering questions on a mobile touch-screen device, investigated possible solutions to overcome the accessibility and usability challenges.

E. Gestures

In total, 3 of 60 studies discuss gestures in usability. Alnfiai & Sampalli (2017) compared the performance of BrailleEnter, a gesture based input method to the Swift Braille keyboard, a method that requires finding the location of six buttons representing braille dot, while Buzzi et al. (2017) and Smaradottir, Martinez & Haland (2017) provide an analysis of gesture performance on touch screens among visually impaired people.

F. Audio guidance

People with visual impairment primarily depend on audio guidance forms in their daily lives; accordingly, audio feedback helps guide them in their interaction with mobile applications.

Four studies discussed the use of audio guidance in different contexts: one in navigation ( Gintner et al., 2017 ), one in games ( Ara’ujo et al., 2017 ), one in reading ( Sabab & Ashmafee, 2016 ), and one in videos ( Façanha et al., 2016 ). These studies were developed and evaluated based on usability and accessibility of the audio guidance for people with visual impairments and aimed to utilize mobile applications to increase the enjoyment and independence of such individuals.

G. Navigation

Navigation is a common issue that visually impaired people face. Indoor navigation is widely discussed in the literature. Nair et al. (2020) , Al-Khalifa & Al-Razgan (2016) and De Borba Campos et al. (2015) discuss how we can develop indoor navigation applications for visually impaired people. Outdoor navigation is also common in the literature, as seen in Darvishy et al. (2020) , Hossain, Qaiduzzaman & Rahman (2020) , Long et al. (2016) , Prerana et al. (2019) and Bandukda et al. (2020) . For example, in Darvishy et al. (2020) , Touch Explorer, an accessible digital map application, was presented to alleviate many of the problems faced by people with visual impairments while using highly visually oriented digital maps. Primarily, it focused on using non-visual output modalities like voice output, everyday sound, and vibration feedback. Issues with navigation applications were also presented in Maly et al. (2015) . Kameswaran et al. (2020) discussed commonly used technologies in navigation applications for blind people and highlighted the importance of using complementary technologies to convey information through different modalities to enhance the navigation experience. Interactive sonification of images for navigation has also been shown in Skulimowski et al. (2019) .

In this section, the research questions are addressed in detail to clearly achieve the research objective. Also, a detailed overview of each theme will be mentioned below.

Answers to the research questions

This section will answer the research question proposed:

RQ1: What existing UVI issues did authors try to solve with mobile devices?

Mobile applications can help people with visual impairments in their daily activities, such as navigation and writing. Additionally, mobile devices may be used for entertainment purposes. However, people with visual impairments face various difficulties while performing text entry operations, text selection, and text manipulation on mobile applications ( Niazi et al., 2016 ). Thus, the authors of the studies tried to increase touch screens’ usability by producing prototypes or simple systems and doing usability testing to understand the UX of people with visual impairments.

RQ2: What is the role of mobile devices in solving those issues?

Mobile phones are widely used in modern society, especially among users with visual impairments; they are considered the most helpful tool for blind users to communicate with people worldwide ( Smaradottir, Martinez & Håland, 2017 ). In addition, the technology of touch screen assistive technology enables speech interaction between blind people and mobile devices and permits the use of gestures to interact with a touch user interface. Assistive technology is vital in helping people living with disabilities perform actions or interact with systems ( Niazi et al., 2016 ).

RQ3: What are the publication trends on the usability of mobile applications among the visually impaired?

As shown in Fig. 8 below, research into mobile applications’ usability for the visually impaired has increased in the last five years, with a slight dip in 2018. Looking at the most frequent themes, we find that “Assistive Devices” peaked in 2017, while “Navigation” and “Accessibility” increased significantly in 2020. On the other hand, we see that the prevalence of “Daily Activities” stayed stable throughout the research years. The term “Audio Guidance” appeared in 2016 and 2017 and has not appeared in the last three years. “Gestures” also appeared only in 2017. “Screen Layout Division” was present in the literature in the last five years and increased in 2019 but did not appear in 2020.

RQ4: What are the current research limitations and future research directions regarding usability among the visually impaired?

We divide the answer to this question into two sections: first, we will discuss limitations; then, we will discuss future work for each proposed theme.

A. Limitations

Studies on the usability of mobile applications for visually impaired users in the literature have various limitations, and most of them were common among the studies. These limitations were divided into two groups. The first group concerns proposed applications; for example, Rahman, Anam & Yeasin (2017) , Oliveira et al. (2018) and Madrigal-Cadavid et al. (2019) faced issues regarding camera applications in mobile devices due to the considerable effort needed for its usage and being heavily dependent on the availability of the internet. The other group of studies, Rodrigues et al. (2015) , Leporini & Palmucci (2018) , Alnfiai & Sampalli (2016) , and Ara’ujo et al. (2017) , have shown limitations in visually impaired users’ inability to comprehend a graphical user interface. Alnfiai & Sampalli (2017) and Alnfiai & Sampalli (2019) evaluated new braille input methods and found that the traditional braille keyboard, where knowing the exact position of letters QWERTY is required, is limited in terms of usability compared to the new input methods. Most studies faced difficulties regarding the sample size and the fact that many of the participants were not actually blind or visually impaired but only blindfolded. This likely led to less accurate results, as blind or visually impaired people can provide more useful feedback as they experience different issues on a daily basis and are more ideal for this type of study. So, the need for a good sample of participants who actually have this disability is clear to allow for better evaluation results and more feedback and recommendations for future research.

B. Future work

A commonly discussed future work in the chosen literature is to increase the sample sizes of people with visual impairment and focus on various ages and geographical areas to generalize the studies. Table 2 summarizes suggestions for future work according to each theme. Those future directions could inspire new research in the field.

RQ5: What is the focus of research on usability for visually impaired people, and what are the research outcomes in the studies reviewed?

There are a total of 60 outcomes in this research. Of these, 40 involve suggestions to improve usability of mobile applications; four of them address problems that are faced by visually impaired people that reduce usability. Additionally, 16 of the outcomes are assessments of the usability of the prototype or model. Two of the results are recommendations to improve usability. Finally, the last two outcomes are hardware solutions that may help the visually impaired perform their daily activities. Figure 9 illustrates these numbers.

Overview of the reviewed studies

In the following subsections, we summarize all the selected studies based on the classified theme: accessibility, assistive devices, daily activities, screen division layout, gestures, audio guidance, and navigation. The essence of the studies will be determined, and their significance in the field will be explored.

For designers dealing with mobile applications, it is critical to determine and fix accessibility issues in the application before it is delivered to the users ( Khowaja et al., 2019 ). Accessibility refers to giving the users the same user experience regardless of ability. In Khowaja et al. (2019) and Carvalho et al. (2018) , the researchers focused on comparing the levels of accessibility and usability in different applications. They had a group of visually impaired users and a group of sighted users test out the applications to compare the number and type of problems they faced and determine which applications contained the most violations. Because people with visual impairments cannot be ignored in the development of mobile applications, many researchers have sought solutions for guaranteeing accessibility. For example, in Qureshi & Hooi-Ten Wong (2020) , the study contributed to producing a new, effective design for mobile applications based on the suggestions of people with visual impairments and with the help of two expert mobile application developers. In Khan, Khusro & Alam (2018) , an adaptive user interface model for visually impaired people was proposed and evaluated in an empirical study with 63 visually impaired people. In Aqle, Khowaja & Al-Thani (2020) , the researchers proposed a new web search interface for users with visual impairments that is based on discovering concepts through formal concept analysis (FCA). Users interact with the interface to collect concepts, which are then used as keywords to narrow the search results and target the web pages containing the desired information with minimal effort and time. The usability of the proposed search interface (InteractSE) was evaluated by experts in the field of HCI and accessibility, with a set of heuristics by Nielsen and a set of WCAG 2.0 guidelines.

In Darvishy, Hutter & Frei (2019) , the researchers proposed a solution for making mobile map applications accessible for people with blindness or visual impairment. They suggested replacing forests in the map with green color and birds’ sound, replacing water with blue color and water sounds, replacing streets with grey color and vibration, and replacing buildings with yellow color and pronouncing the name of the building. The prototype showed that it was possible to explore a simple map through vibrations, sounds, and speech.

In Morris et al. (2016) the researchers utilized a multi-faceted technique to investigate how and why visually impaired individuals use Twitter and the difficulties they face in doing so. They noted that Twitter had become more image-heavy over time and that picture-based tweets are largely inaccessible to people with visual impairments. The researchers then made several suggestions for how Twitter could be amended to continue to be usable for people with visual impairments.

The researchers in Paiva et al. (2020) focused on how to evaluate proposed methods for ensuring the accessibility and usability of mobile applications. Their checklist, Acc-MobileCheck, contains 47 items that correspond to issues related to comprehension (C), operation (O), perception (P), and adaptation (A) in mobile interface interaction. To validate Acc-MobileCheck, it was reviewed by five experts and three developers and determined to be effective. In Pereda, Murillo & Paz (2020) , the authors also suggest a set of heuristics to evaluate the accessibility of mobile e-commerce applications for visually impaired people. Finally, McKay (2017) conducted an accessibility test for hybrid mobile apps and found that students with blindness faced many barriers to access based on how they used hybrid mobile applications. While hybrid apps can allow for increased time for marketing, this comes at the cost of app accessibility for people with disabilities.

A significant number of people with visual impairments use state-of-the-art software to perform tasks in their daily lives. These technologies are made up of electronic devices equipped with sensors and processors that can make intelligent decisions.

One of the most important and challenging tasks in developing such technologies is to create a user interface that is appropriate for the sensorimotor capabilities of users with blindness ( Csapó et al., 2015 ). Several new hardware tools have proposed to improve the quality of life for people with visual impairments. Three tools were presented in this SLR: a smart stick that can notify the user of any obstacle, helping them to perform tasks easily and efficiently ( Bharatia, Ambawane & Rane, 2019 ), and an eye that can allow users to detect colors (medical evaluation is still required) ( Lewis et al., 2016 ).

The purpose of the study in Griffin-Shirley et al. (2017) was to understand how people with blindness use smartphone applications as assistive technology and how they perceive them in terms of accessibility and usability. An online survey with 259 participants was conducted, and most of the participants rated the applications as useful and accessible and were satisfied with them.

The researchers in Rahman, Anam & Yeasin (2017) designed and implemented EmoAssist, which is a smartphone application that assists with natural dyadic conversations and aims to promote user satisfaction by providing options for accessing non-verbal communication that predicts behavioural expressions and contains interactive dimensions to provide valid feedback. The usability of this application was evaluated in a study with ten people with blindness where several tools were applied in the application. The study participants found that the usability of EmoAssist was good, and it was an effective assistive solution.

This theme contains two main categories: braille-based application studies and applications to enhance the independence of VIU. Both are summarized below.

1- Braille-based applications

Braille is still the most popular method for assisting people with visual impairments in reading and studying, and most educational mobile phone applications are limited to sighted people. Recently, however, some researchers have developed assistive education applications for students with visual impairments, especially those in developing countries. For example, in India, the number of children with visual impairments is around 15 million, and only 5% receive an education ( Gokhale et al., 2017 ). Three of the braille studies focused on education: ( Nahar, Sulaiman & Jaafar, 2020 ; Nahar, Jaafar & Sulaiman, 2019 , and Araújo et al., 2016 ). These studies all used smartphone touchscreens and action gestures to gain input from the student, and then output was provided in the form of audio feedback. In Nahar, Sulaiman & Jaafar (2020) , vibrational feedback was added to guide the users. The participants in Nahar, Sulaiman & Jaafar (2020) ; Nahar, Jaafar & Sulaiman (2019) , and Araújo et al. (2016) included students with blindness of visual impairment and their teachers. The authors in Nahar, Sulaiman & Jaafar (2020) , Nahar, Jaafar & Sulaiman (2019) evaluated the usability of their applications following the same criteria (efficiency, learnability, memorability, errors, and satisfaction). The results showed that in Nahar, Sulaiman & Jaafar (2020) , Nahar, Jaafar & Sulaiman (2019) , and Araújo et al. (2016) , the applications met the required usability criteria. The authors in Gokhale et al. (2017) presented a braille-based solution to help people with visual impairments call and save contacts. A braille keypad on the smartphone touchscreen was used to gain input from the user, which was then converted into haptic and auditory feedback to let the user know what action was taken. The usability of this application was considered before it was designed. The participants’ responses were positive because this kind of user-centric design simplifies navigation and learning processes.

2- Applications to enhance the independence of people with visual impairments

The authors in the studies explored in this section focused on building applications that enhance independence and autonomy in daily life activities for users with visual impairments.

In Vitiello et al. (2018) , the authors presented their mobile application, an assistive solution for visually impaired users called “Crania”, which uses machine learning techniques to help users with visual impairments get dressed by recognizing the colour and texture of their clothing and suggesting suitable combinations. The system provides feedback through voice synthesis. The participants in the study were adults and elderly people, some of whom were completely blind and the rest of whom had partial sight. After testing for usability, all the participants with blindness agreed that using the application was better than their original method, and half of the participants with partial sight said the same thing. At the end of the study, the application was determined to be accessible and easy to use.

In Kunaratana-Angkul, Wu & Shin-Renn (2020) , an application which allows elderly people to measure low vision status at home through their smartphones instead of visiting hospitals was tested, and most of the participants considered it to be untrustworthy because the medical information was insufficient. Even when participants were able to learn how to use the application, most of them were still confused while using it and needed further instruction.

In Ghidini et al. (2016) , the authors studied the habits of people with visual impairments when using their smartphones in order to develop an electronic calendar with different interaction formats, such as voice commands, touch, and vibration interaction. The authors presented the lessons learned and categorized them based on usability heuristics such as feedback, design, user freedom and control, and recognition instead of remembering.

In Madrigal-Cadavid et al. (2019) , the authors developed a drug information application for people with visual impairments to help them access the labels of medications. The application was developed based on a user-centered design process. By conducting a usability test, the authors recognized some usability issues for people with visual impairments, such as difficulty in locating the bar code. Given this, a new version will include a search function that is based on pictures. The application is searched by capturing the bar code or text or giving voice commands that allow the user to access medication information. The participants were people with visual impairments, and most of them required assistance using medications before using the application. This application will enhance independence for people with visual impairments in terms of using medications.

In Marques, Carriço & Guerreiro (2015) , an authentication method is proposed for users with visual impairments that allows them to protect their passwords. It is not secure when blind or visually impaired users spell out their passwords or enter the numbers in front of others, and the proposed solution allows the users to enter their password with one hand by tapping the screen. The blind participants in this study demonstrated that this authentication method is usable and supports their security needs.

In Oliveira et al. (2018) , the author noted that people with visual impairments face challenges in reading, thus he proposed an application called LeR otulos. This application was developed and evaluated for the Android operating system and recognizes text from photos taken by the mobile camera and converts them into an audio description. The prototype was designed to follow the guidelines and recommendations of usability and accessibility. The requirements of the application are defined based on the following usability goals: the steps are easy for the user to remember; the application is efficient, safe, useful, and accessible; and user satisfaction is achieved.

Interacting with talkback audio devices is still difficult for people with blindness, and it is unclear how much benefit they provide to people with visual impairments in their daily activities. The author in Rodrigues et al. (2015) investigates the smartphone adoption process of blind users by conducting experiments, observations, and weekly interviews. An eight-week study was conducted with five visually impaired participants using Samsung and an enabled talkback 2 screen reader. Focusing on understanding the experiences of people with visual impairments when using touchscreen smartphones revealed accessibility and usability issues. The results showed that the participants have difficulties using smartphones because they fear that they cannot use them properly, and that impacts their ability to communicate with family. However, they appreciate the benefits of using smartphones in their daily activities, and they have the ability to use them.

People with visual impairments encounter various challenges identifying and locating non-visual items on touch screen interfaces, such as phones and tablets. Various specifications for developing a user interface for people with visual impairments must be met, such as having touch screen division to enable people with blindness to easily and comfortably locate objects and items that are non-visual on the screen ( Khusro et al., 2019 ). Article ( Khusro et al., 2019 ) highlighted the importance of aspects of the usability analysis, such as screen partitioning, to meet specific usability requirements, including orientation, consistency, operation, time consumption, and navigation complexity when users want to locate objects on their touchscreen. The authors of Khan & Khusro (2019) describe the improvements that people with blindness have experienced in using the smartphone while performing their daily tasks. This information was determined through an empirical study with 41 people with blindness who explained their user and interaction experiences operating a smartphone.

The authors in Palani et al. (2018) provide design guidelines governing the accurate display of haptically perceived graphical materials. Determining the usability parameters and the various cognitive abilities required for optimum and accurate use of device interfaces is crucial. Also the authors of Grussenmeyer & Folmer (2017) highlight the importance of usability and accessibility of smartphones and touch screens for people with visual impairments. The primary focus in Leporini & Palmucci (2018) is on interactive tasks used to finish exercises and to answer questionnaires or quizzes. These tools are used for evaluation tests or in games. When using gestures and screen readers to interact on a mobile device, difficulties may arise ( Leporini & Palmucci, 2018 ), The study has various objectives, including gathering information on the difficulties encountered by people with blindness during interactions with mobile touch screen devices to answer questions and investigating practicable solutions to solve the detected accessibility and usability issues. A mobile app with an educational game was used to apply the proposed approach. Moreover, in Alnfiai & Sampalli (2016) and Niazi et al. (2016) , an analysis of the single-tap braille keyboard created to help people with no or low vision while using touch screen smartphones was conducted. The technology used in Alnfiai & Sampalli (2016) was the talkback service, which provides the user with verbal feedback from the application, allowing users with blindness to key in characters according to braille patterns. To evaluate single tap braille, it was compared to the commonly used QWERTY keyboard. In Niazi et al. (2016) , it was found that participants adapted quickly to single-tap Braille and were able to type on the touch screen within 15 to 20 min of being introduced to this system. The main advantage of single tap braille is that it allows users with blindness to enter letters based on braille coding, which they are already familiar with. The average error rate is lower using single-tap Braille than it is on the QWERTY keyboard. The authors of Niazi et al. (2016) found that minimal typing errors were made using the proposed keypad, which made it an easier option for people with blindness ( Niazi et al., 2016 ). In Cho & Kim (2017) , the authors describe new text entry methods for the braille system including a left touch and a double touch scheme that form a two-button interface for braille input so that people with visual impairments are able to type textual characters without having to move their fingers to locate the target buttons.

One of the main problems affecting the visually impaired is limited mobility for some gestures. We need to know what gestures are usable by people with visual impairments. Moreover, the technology of assistive touchscreen-enabled speech interaction between blind people and mobile devices permits the use of gestures to interact with a touch user interface. Assistive technology is vital in helping people living with disabilities to perform actions or interact with systems. Smaradottir, Martinez & Haland (2017) analyses a voiceover screen reader used in Apple Inc.’s products. An assessment of this assistive technology was conducted with six visually impaired test participants. The main objectives were to pinpoint the difficulties related to the performance of gestures applicable in screen interactions and to analyze the system’s response to the gestures. In this study, a user evaluation was completed in three phases. The first phase entailed training users regarding different hand gestures, the second phase was carried out in a usability laboratory where participants were familiarized with technological devices, and the third phase required participants to solve different tasks. In Knutas et al. (2015) , the vital feature of the system is that it enables the user to interactively select a 3D scene region for sonification by merely touching the phone screen. It uses three different modes to increase usability. Alnfiai & Sampalli (2017) explained a study done to compare the use of two data input methods to evaluate their efficiency with completely blind participants who had prior knowledge of braille. The comparison was made between the braille enter input method that uses gestures and the swift braille keyboard, which necessitates finding six buttons representing braille dots. Blind people typically prefer rounded shapes to angular ones when performing complex gestures, as they experience difficulties performing straight gestures with right angles. Participants highlighted that they experienced difficulties particularly with gestures that have steep or right angles. In Buzzi et al. (2017) , 36 visually impaired participants were selected and split into two groups of low-vision and blind people. They examined their touch-based gesture preferences in terms of the number of strokes, multitouch, and shape angles. For this reason, a wireless system was created to record sample gestures from various participants simultaneously while monitoring the capture process.

People with visual impairment typically cannot travel without guidance due to the inaccuracy of current navigation systems in describing roads and especially sidewalks. Thus, the author of Gintner et al. (2017) aims to design a system to guide people with visual impairments based on geographical features and addresses them through a user interface that converts text to audio using a built-in voiceover engine (Apple iOS). The system was evaluated positively in terms of accessibility and usability as tested in a qualitative study involving six participants with visual impairment.

Based on challenges faced by visually impaired game developers, Ara’ujo et al. (2017) provides guidance for developers to provide accessibility in digital games by using audio guidance for players with visual impairments. The interactions of the player can be conveyed through audio and other basic mobile device components with criteria focused on the game level and speed adjustments, high contrast interfaces, accessible menus, and friendly design. Without braille, people with visual impairments cannot read, but braille is expensive and takes effort, and so it is important to propose technology to facilitate reading for them. In Sabab & Ashmafee (2016) , the author proposed developing a mobile application called “Blind Reader” that reads an audio document and allows the user to interact with the application to gain knowledge. This application was evaluated with 11 participants, and the participants were satisfied with the application. Videos are an important form of digital media, and unfortunately people with visual impairment cannot access these videos. Therefore, Façanha et al. (2016) aims to discover sound synthesis techniques to maximize and accelerate the production of audio descriptions with low-cost phonetic description tools. This tool has been evaluated based on usability with eight people and resulted in a high acceptance rate among users.

1- Indoor navigation

Visually impaired people face critical problems when navigating from one place to another. Whether indoors or outdoors, they tend to stay in one place to avoid the risk of injury or seek the help of a sighted person before moving ( Al-Khalifa & Al-Razgan, 2016 ). Thus, aid in navigation is essential for those individuals. In Nair et al. (2020) , Nair developed an application called ASSIST, which leverages Bluetooth low energy (BLE) beacons and augmented reality (AR) to help visually impaired people move around cluttered indoor places ( e.g. , subways) and provide the needed safe guidance, just like having a sighted person lead the way. In the subway example, the beacons will be distributed across the halls of the subway and the application will detect them. Sensors and cameras attached to the individual will detect their exact location and send the data to the application. The application will then give a sequence of audio feedback explaining how to move around the place to reach a specific point ( e.g. , “in 50 ft turn right”, “now turn left”, “you will reach the destination in 20 steps”). The application also has an interface for sighted and low-vision users that shows the next steps and instructions. A usability study was conducted to test different aspects of the proposed solution. The majority of the participants agreed that they could easily reach a specified location using the application without the help of a sighted person. A survey conducted to give suggestions from the participants for future improvements showed that most participants wanted to attach their phones to their bodies and for the application to consider the different walking speeds of users. They were happy with the audio and vibration feedback that was given before each step or turn they had to take.

In Al-Khalifa & Al-Razgan (2016) , the main purpose of the study was to provide an Arabic-language application for guidance inside buildings using Google Glass and an associated mobile application. First, the building plan must be set by a sighted person who configures the different locations needed. Ebsar will ask the map builder to mark each interesting location with a QR code and generate a room number, and the required steps and turns are tracked using the mobile device’s built-in compass and accelerometer features. All of these are recorded in the application for the use of a visually impaired individual, and at the end, a full map is generated for the building. After setting the building map, a user can navigate inside the building with the help of Ebsar, paired with Google Glass, for input and output purposes. The efficiency, effectiveness, and levels of user satisfaction with this solution were evaluated. The results showed that the errors made were few, indicating that Ebsar is highly effective. The time consumed in performing tasks ranged from medium to low depending on the task; this can be improved later. Interviews with participants indicated the application’s ease of use. De Borba Campos et al. (2015) shows an application simulating a museum map for people with visual impairments. It discusses whether mental maps and interactive games can be used by people with visual impairments to recognize the space around them. After multiple usability evaluation sessions, the mobile application showed high efficiency among participants in understanding the museum’s map without repeating the visitation. The authors make a few suggestions based on feedback from the participants regarding enhancing usability, including using audio cues, adding contextual help to realise the activities carried around in a space, and focusing on audio feedback instead of graphics.

2- Outdoor navigation

Outdoor navigation is also commonly discussed in the literature. In Darvishy et al. (2020) , Touch Explorer was presented to alleviate many of the problems faced by visually impaired people in navigation by developing a non-visual mobile digital map. The application relies on three major methods of communication with the user: voice output, vibration feedback, and everyday sounds. The prototype was developed using simple abstract visuals and mostly relies on voice for explanation of the content. Usability tests show the great impact the prototype had on the understanding of the elements of the map. Few suggestions were given by the participants to increase usability, including GPS localization to locate the user on the map, a scale element for measuring the distance between two map elements, and an address search function.

In Hossain, Qaiduzzaman & Rahman (2020) , a navigation application called Sightless Helper was developed to provide a safe navigation method for people with visual impairments. It relies on footstep counting and GPS location to provide the needed guidance. It can also ensure safe navigation by detect objects and unsafe areas and can detect unusual shaking of the user and alert an emergency contact about the problem. The user interaction categories are voice recognition, touchpad, buttons, and shaking sensors. After multiple evaluations, the application was found to be useful in different scenarios and was considered usable by people with visual impairments. The authors in Long et al. (2016) propose an application that uses both updates from users and information about the real world to help visually impaired people navigate outdoor settings. After interviews with participants, some design goals were set, including the ability to tag an obstacle on the map, check the weather, and provide an emergency service. The application was evaluated and was found to be of great benefit; users made few errors and found it easy to use. In Prerana et al. (2019) , a mobile application called STAVI was presented to help visually impaired people navigate from a source to a destination safely and avoid issues of re-routing. The application depends on voice commands and voice output. The application also has additional features, such as calling, messages, and emergency help. The authors in Bandukda et al. (2020) helped people with visual impairments explore parks and natural spaces using a framework called PLACES. Different interviews and surveys were conducted to identify the issues visually impaired people face when they want to do any leisure activity. These were considered in the development of the framework, and some design directions were presented, such as the use of audio to share an experience.

3- General issues

The authors in Maly et al. (2015) discuss implementing an evaluation model to assess the usability of a navigation application and to understand the issues of communication with mobile applications that people with visual impairments face. The evaluation tool was designed using a client–server architecture and was applied to test the usability of an existing navigation application. The tool was successful in capturing many issues related to navigation and user behavior, especially the issue of different timing between the actual voice instruction and the position of the user. The authors in Kameswaran et al. (2020) conducted a study to find out which navigation technologies blind people can use and to understand the complementarity between navigation technologies and their impact on navigation for visually impaired users. The results of the study show that visually impaired people use both assistive technologies and those designed for non-visually impaired users. Improving voice agents in navigation applications was discussed as a design implication for the visually impaired. In Skulimowski et al. (2019) , the authors show how interactive sonification can be used in simple travel aids for the blind. It uses depth images and a histogram called U-depth, which is simple auditory representations for blind users. The vital feature of this system is that it enables the user to interactively select a 3D scene region for sonification by touching the phone screen. This sonic representation of 3D scenes allows users to identify the environment’s general appearance and determine objects’ distance. The prototype structure was tested by three blind individuals who successfully performed the indoor task. Among the test scenes used included walking along an empty corridor, walking along a corridor with obstacles, and locating an opening between obstacles. However, the results showed that it took a long time for the testers to locate narrow spaces between obstacles.

RQ6: What evaluation methods were used in the studies on usability for visually impaired people that were reviewed?

The most prevalent methods to evaluate the usability of applications were surveys and interviews. These were used to determine the usability of the proposed solutions and obtain feedback and suggestions regarding additional features needed to enhance the usability from the participants’ points of view. Focus groups were also used extensively in the literature. Many of the participants selected were blindfolded and were not actually blind or visually impaired. Moreover, the samples selected for the evaluation methods mentioned above considered the age factor depending on the study’s needs.

Limitation and future work

The limitations of this paper are mainly related to the methodology followed. Focusing on just eight online databases and restricting the search with the previously specified keywords and string may have limited the number of search results. Additionally, a large number of papers were excluded because they were written in other languages. Access limitations were also faced due to some libraries asking for fees to access the papers. Therefore, for future works, a study to expand on the SLR results and reveal the current usability models of mobile applications for the visually impaired to verify the SLR results is needed so that this work contributes positively to assessing difficulties and expanding the field of usability of mobile applications for users with visual impairments.

Conclusions

In recent years, the number of applications focused on people with visual impairments has grown, which has led to positive enhancements in those people’s lives, especially if they do not have people around to assist them. In this paper, the research papers focusing on usability for visually impaired users were analyzed and classified into seven themes: accessibility, daily activities, assistive devices, gestures, navigation, screen division layout, and audio guidance. We found that various research studies focus on accessibility of mobile applications to ensure that the same user experience is available to all users, regardless of their abilities. We found many studies that focus on how the design of the applications can assist in performing daily life activities like braille-based application studies and applications to enhance the independence of VI users. We also found papers that discuss the role of assistive devices like screen readers and wearable devices in solving challenges faced by VI users and thus improving their quality of life. We also found that some research papers discuss limited mobility of some gestures for VI users and investigated ways in which we can know what gestures are usable by people with visual impairments. We found many research papers that focus on improving navigation for VI users by incorporating different output modalities like sound and vibration. We also found various studies focusing on screen division layout. By dividing the screen and focusing on visual impairment-related issues while developing user interfaces, visually impaired users can easily locate the objects and items on the screens. Finally, we found papers that focus on audio guidance to improve usability. The proposed applications use voice-over and speech interactions to guide visually impaired users in performing different activities through their mobiles. Most of the researchers focused on usability in different applications and evaluated the usability issues of these applications with visually impaired participants. Some of the studies included sighted participants to compare the number and type of problems they faced. The usability evaluation was generally based on the following criteria: accessibility, efficiency, learnability, memorability, errors, safety, and satisfaction. Many of the studied applications show a good indication of these applications’ usability and follow the participants’ comments to ensure additional enhancements in usability. This paper aims to provide an overview of the developments on usability of mobile applications for people with visual impairments and use this overview to highlight potential future directions.

References selected for each theme.

Funding Statement

This research project was supported by a grant from the Research Center of the Female Scientific and Medical Colleges, Deanship of Scientific Research, King Saud University. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Additional Information and Declarations

The authors declare there are no competing interests.

Muna Al-Razgan , Sarah Almoaiqel , Nuha Alrajhi , Alyah Alhumegani , Abeer Alshehri , Bashayr Alnefaie , Raghad AlKhamiss and Shahad Rushdi conceived and designed the experiments, performed the experiments, analyzed the data, performed the computation work, prepared figures and/or tables, authored or reviewed drafts of the paper, and approved the final draft.

You are using an outdated browser. Please upgrade your browser to improve your experience.

Apple's new AI model could help Siri see how iOS apps work

A ferret in the wild [Pixabay/Michael Sehlmeyer]

Apple has been working on numerous machine learning and AI projects that it could tease at WWDC 2024. In a just-released paper, it now seems that some of that work has the potential for Siri to understand what apps and iOS itself looks like.

The paper, released by Cornell University on Monday , is titled "Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs." It essentially explains a new multimodal large language model (MLLM) that has the potential to understand the user interfaces of mobile displays.

The Ferret name originally came up from an open-source multi-modal LLM released in October , by researchers from Cornell University working with counterparts from Apple. At the time, Ferret was able to detect and understand different regions of an image for complex queries, such as identifying a species of animal in a selected part of a photograph.

An LLM advancement

The new paper for Ferret-UI explains that, while there have been noteworthy advancements in MLLM usage, they still "fall short in their ability to comprehend and interact effectively with user interface (UI) screens." Ferret-UI is described as a new MLLM tailored for understanding mobile UI screens, complete with "referring, grounding, and reasoning capabilities."

Part of the problem that LLMs have in understanding the interface of a mobile display is how it gets used in the first place. Often in a portrait orientation, it often means icons and other details can take up a very compact part of the display, making it difficult for machines to understand.

To help with this, Ferret has a magnification system to upscale images to "any resolution" to make icons and text more readable.

For processing and training, Ferret also divides the screen into two smaller sections, cutting the screen in half. The paper states that other LLMs tend to scan a lower-resolution global image, which reduces the ability to adequately determine what icons look like.

Adding in significant curation of data for training, it's resulted in a model that can sufficiently understand user queries, understand the nature of various on-screen elements, and to offer contextual responses.

For example, a user could ask how to open the Reminders app, and be told to tap the on-screen Open button. A further query asking if a 15-year-old could use an app could check out age guidelines, if they're visible on the display.

An assistive assistant

While we don't know whether it will be incorporated into systems like Siri, Ferret-UI offers the possibility of advanced control over a device like an iPhone . By understanding user interface elements, it offers the possibility of Siri performing actions for users in apps, by selecting graphical elements within the app on its own.

There are also useful applications for the visually impaired. Such an LLM could be more capable of explaining what is on screen in detail, and potentially carry out actions for the user without them needing to do anything else but ask for it to happen.

Top Stories

Lowest price: Apple's M3 MacBook Air 16GB RAM, 512GB SSD, 10C GPU drops to $1,349

When to expect every Mac to get the AI-based M4 processor

All of the specs of the iPhone SE 4 may have just been leaked

App Store's first emulator looks like it's an ad-loaded knockoff

Ugreen DXP8800 Plus network attached storage review: Good hardware, beta software

The best game controllers for iPhone, iPad, Mac, and Apple TV

Featured deals.

Snap up Apple's latest Mac Studio for $1,795 this weekend

Latest comparisons.

M3 15-inch MacBook Air vs M3 14-inch MacBook Pro — Ultimate buyer's guide

M3 MacBook Air vs M1 MacBook Air — Compared

M3 MacBook Air vs M2 MacBook Air — Compared

Latest news.

The developer of Gameboy emulator GBA4iOS says the top-ranking version on the App Store is a fraudulent knock off of his work that Apple should not have approved.

Apple Music execs reveal months of work behind releasing Taylor Swift's new album

Apple Music teams were working extensively with Taylor Swift to ready her new album and its promotion long before "The Tortured Poets Department" is due to be released.

Crash Detection helps police locate teenagers killed in New Zealand

New Zealand authorities say that Apple's iPhone Crash Detection was instrumental in helping them find the site where two teenage women were killed in an off-road crash.

Apple will reportedly update its entire Mac line to the M4 processor, beginning in late 2024 and concluding with the Mac Pro in the second half of 2025.

Crime blotter: Arrests made in Naperville Apple Store theft

In the latest Apple Crime Blotter, NASA iPad thefts were investigated, stolen credit cards were used at an Apple Store, and iPhones were stolen from a Philadelphia AT&T Store.

Game emulators arrive in App Store following rule changes

Just over a week after Apple changed rules to allow game emulators on iPhone and iPad, the first have made their way into the App Store.

Apple urges court to ignore Epic's anti-steering complaint

Apple has responded to Epic's accusation it is in contempt of court for failing to remove anti-steering measures to its liking, by insisting it's really complying with the order.

Packed with upgrades, Apple's latest M3 MacBook Air with 16GB RAM, 512GB of storage and a 10-core GPU is marked down to $1,349. AppleCare is discounted with an exclusive coupon as well, dropping the price down to $159.

A leak that is chiefly rounding up previous rumors about the iPhone SE 4 has added that it will have AI-based camera features, and contrary to expectations, will resemble the iPhone 13 more than the iPhone 14.

Apple will allow activation of used parts in repairs by users & service shops

Apple is expanding its self-repair program procedures, and is adding the ability for consumers to use parts pairing to fully activate used parts in repairs — assuming they haven't been harvested from stolen devices.

JP Morgan drops Apple target price over questions on when AI iPhone will launch

Investment firm JP Morgan predicts that Apple will introduce AI with the 2025 iPhone 17 Pro, and has dropped its current price target to $210 partly as a result.

Latest Videos

How to get the best video capture possible on iPhone 15 Pro with ProRes

How to turn off Apple's Journal 'Discoverable by Others' setting that's enabled by default

Latest reviews.

Espresso 17 Pro review: Magnetic & modular portable Mac monitor

Journey Loc8 MagSafe Finder Wallet review: an all-in-one Find My wallet

{{ title }}

{{ summary }}