Textbook Lending (Course Reserves)

Access to books, articles, and other items on course reserves for specific courses

Technology and Tool Lending

Technology and tools that can be used within the library or checked out

Your Librarian

Contact your Liaison Librarian to request a course-integrated workshop or instruction on a variety of topics

Subject Research Guides

In-depth subject guides to library resources and research techniques

Course Guides

Specialized information created specifically for a course

Collections

Find out more about STEM-specific collections in Marston Science Library

Subject/Area Specialists

Contact a specialist throughout the Libraries

Libraries & Collections

Quick Picks from Your Librarians

(VPN access only)

More details on the quick picks:

AccessMedicine  – Search engine which helps you find matches based on the true meaning of the concepts detailed in the content.

Access Pharmacy – Allows users to explore leading pharmacy references, search curriculum topics, research drugs and supplements. Courtesy of the College of Pharmacy.

BIOSIS Previews  – Find the journal, meetings, books, and patents data you need to prepare research projects, grant proposals and follow trends in the life sciences.

CABI/Global Health – Keep track of every area of agricultural science, veterinary medicine, nutrition and natural resources.

CINAHL  – Nursing and allied health journals, publications from the American Nurses’ Association and the National League for Nursing, dissertations, standards, legislation, videos and books. State nursing journals and nursing practice acts often available in fulltext.

Cochrane Library – Includes Best Evidence, Cochrane Database of Systematic Reviews, and Database of Abstracts of Reviews of Effectiveness (DARE).

Dissertations & Theses Global (ProQuest)  – Collection of dissertations and theses from around the world, spanning 1742 – present, includes both citations and full text.

EBSCOhost Web  – EBSCOhost databases and discovery technologies are the most-used, premium online information resources for tens of thousands of institutions worldwide.

Embase – International biomedical literature from 1947-present.

HaPI  – HaPI provides rating scales, questionnaires, checklists, tests, interview schedules, and coding schemes/manuals for health and social sciences.”

InCites Journal Citation Reports (formerly know as Impact Factor JCR) – InCites Journal Citation Reports Science Edition contains data from over 5,900 journals in science and technology and InCites Journal Citation Reports Social Sciences Edition contains data from over 1,700 journals in the social sciences.

Micromedex – Drug information database including drug interactions, IV compatibility, toxicology, and drug comparison. Courtesy of UF Health.

Natural Medicines – The most authoritative resource available on dietary supplements, natural medicines, and complementary alternative and integrative therapies.

NCBI  – A national resource for molecular biology information, which creates public databases, conducts research in computational biology, develops software tools for analyzing genome data, and disseminates biomedical information.

PsycINFO  ( APA PsycINFO ) – Experimental psychology (human, animal, and comparative), psychosexual behavior, educational psychology, applied psychology, and sports psychology.

PubMed   – Medicine, nursing, dentistry, veterinary medicine, the preclinical sciences, nutrition, delivery of health care, pharmacology, environmental health, pathology, toxicology, biochemistry, genetics, molecular biology, and psychiatry.

Scopus – Interdisciplinary database covering articles from 240 disciplines. Includes citation tracking.

SportDiscus  – Sports medicine, exercise physiology, biomechanics, psychology, training techniques, coaching, physical education, physical fitness, active living, recreation, history, facilities and equipment.

UpToDate – Evidence-based, peer reviewed clinical information. On campus and VPN access only – courtesy of the College of Medicine Office of Graduate Medical Education.

Web of Science  – Interdisciplinary database covering journals in the sciences, social sciences, arts & humanities. Includes citation tracking.

Quick Links




  • Reference Manager
  • Simple TEXT file

People also looked at

Systematic review article, a systematic review of 10 years of augmented reality usability studies: 2005 to 2014.

peer reviewed journals by uf ar researchers

  • 1 Empathic Computing Laboratory, University of South Australia, Mawson Lakes, SA, Australia
  • 2 Human Interface Technology Lab New Zealand (HIT Lab NZ), University of Canterbury, Christchurch, New Zealand
  • 3 Mississippi State University, Starkville, MS, United States

Augmented Reality (AR) interfaces have been studied extensively over the last few decades, with a growing number of user-based experiments. In this paper, we systematically review 10 years of the most influential AR user studies, from 2005 to 2014. A total of 291 papers with 369 individual user studies have been reviewed and classified based on their application areas. The primary contribution of the review is to present the broad landscape of user-based AR research, and to provide a high-level view of how that landscape has changed. We summarize the high-level contributions from each category of papers, and present examples of the most influential user studies. We also identify areas where there have been few user studies, and opportunities for future research. Among other things, we find that there is a growing trend toward handheld AR user studies, and that most studies are conducted in laboratory settings and do not involve pilot testing. This research will be useful for AR researchers who want to follow best practices in designing their own AR user studies.

1. Introduction

Augmented Reality (AR) is a technology field that involves the seamless overlay of computer generated virtual images on the real world, in such a way that the virtual content is aligned with real world objects, and can be viewed and interacted with in real time ( Azuma, 1997 ). AR research and development has made rapid progress in the last few decades, moving from research laboratories to widespread availability on consumer devices. Since the early beginnings in the 1960's, more advanced and portable hardware has become available, and registration accuracy, graphics quality, and device size have been largely addressed to a satisfactory level, which has led to a rapid growth in the adoption of AR technology. AR is now being used in a wide range of application domains, including Education ( Furió et al., 2013 ; Fonseca et al., 2014a ; Ibáñez et al., 2014 ), Engineering ( Henderson and Feiner, 2009 ; Henderson S. J. and Feiner, 2011 ; Irizarry et al., 2013 ), and Entertainment ( Dow et al., 2007 ; Haugstvedt and Krogstie, 2012 ; Vazquez-Alvarez et al., 2012 ). However, to be widely accepted by end users, AR usability and user experience issues still need to be improved.

To help the AR community improve usability, this paper provides an overview of 10 years of AR user studies, from 2005 to 2014. Our work builds on the previous reviews of AR usability research shown in Table 1 . These years were chosen because they cover an important gap in other reviews, and also are far enough from the present to enable the impact of the papers to be measured. Our goals are to provide a broad overview of user-based AR research, to help researchers find example papers that contain related studies, to help identify areas where there have been few user studies conducted, and to highlight exemplary user studies that embody best practices. We therefore hope the scholarship in this paper leads to new research contributions by providing outstanding examples of AR user studies that can help current AR researchers.

www.frontiersin.org

Table 1 . Summary of earlier surveys of AR usability studies.

1.1. Previous User Study Survey Papers

Expanding on the studies shown in Table 1 , Swan and Gabbard (2005) conducted the first comprehensive survey of AR user studies. They reviewed 1,104 AR papers published in four important venues between 1992 and 2004; among these papers they found only 21 that reported formal user studies. They classified these user study papers into three categories: (1) low-level perceptual and cognitive issues such as depth perception, (2) interaction techniques such as virtual object manipulation, and (3) collaborative tasks. The next comprehensive survey was by Dünser et al. (2008) , who used a list of search queries across several common bibliographic databases, and found 165 AR-related publications reporting user studies. In addition to classifying the papers into the same categories as Swan and Gabbard (2005) , they additionally classified the papers based on user study methods such as objective, subjective, qualitative, and informal. In another literature survey, Bai and Blackwell (2012) reviewed 71 AR papers reporting user studies, but they only considered papers published in the International Symposium on Mixed and Augmented Reality (ISMAR) between 2001 and 2010. They also followed the classification of Swan and Gabbard (2005) , but additionally identified a new category of studies that investigated user experience (UX) issues. Their review thoroughly reported the evaluation goals, performance measures, UX factors investigated, and measurement instruments used. Additionally, they also reviewed the demographics of the studies' participants. However there has been no comprehensive study since 2010, and none of these earlier studies used an impact measure to determine the significance of the papers reviewed.

1.1.1. Survey Papers of AR Subsets

Some researchers have also published review papers focused on more specific classes of user studies. For example, Kruijff et al. (2010) reviewed AR papers focusing on the perceptual pipeline, and identified challenges that arise from the environment, capturing, augmentation, display technologies, and user. Similarly, Livingston et al. (2013) published a review of user studies in the AR X-ray vision domain. As such, their review deeply analyzed perceptual studies in a niche AR application area. Finally, Rankohi and Waugh (2013) reviewed AR studies in the construction industry, although their review additionally considers papers without user studies. In addition to these papers, many other AR papers have included literature reviews which may include a few related user studies such as Wang et al. (2013) , Carmigniani et al. (2011) , and Papagiannakis et al. (2008) .

1.2. Novelty and Contribution

These reviews are valued by the research community, as shown by the number of times they have been cited (e.g., 166 Google Scholar citations for Dünser et al., 2008 ). However, due to a numebr of factors there is a need for a more recent review. Firstly, while early research in AR was primarily based on head-mounted displays (HMDs), in the last few years there has been a rapid increase in the use of handheld AR devices, and more advanced hardware and sensors have become available. These new wearable and mobile devices have created new research directions, which have likely impacted the categories and methods used in AR user studies. In addition, in recent years the AR field has expanded, resulting in a dramatic increase in the number of published AR papers, and papers with user studies in them. Therefore, there is a need for a new categorization of current AR user research, as well as the opportunity to consider new classification measures such as paper impact, as reviewing all published papers has become less plausible. Finally, AR papers are now appearing in a wider range of research venues, so it is important to have a survey that covers many different journals and conferences.

1.2.1. New Contributions Over Existing Surveys

Compared to these earlier reviews, there are a number of important differences with the current survey, including:

• we have considered a larger number of publications from a wide range of sources

• our review covers more recent years than earlier surveys

• we have used paper impact to help filter the papers reviewed

• we consider a wider range of classification categories

• we also review issues experienced by the users.

1.2.2. New Aims of This Survey

To capture the latest trends in usability research in AR, we have conducted a thorough, systematic literature review of 10 years of AR papers published between 2005 and 2014 that contain a user study. We classified these papers based on their application areas, methodologies used, and type of display examined. Our aims are to:

1. identify the primary application areas for user research in AR

2. describe the methodologies and environments that are commonly used

3. propose future research opportunities and guidelines for making AR more user friendly.

The rest of the paper is organized as follows: section 2 details the method we followed to select the papers to review, and how we conducted the reviews. Section 3 then provides a high-level overview of the papers and studies, and introduces the classifications. The following sections report on each of the classifications in more detail, highlighting one of the more impactful user studies from each classification type. Section 5 concludes by summarizing the review and identifying opportunities for future research. Finally, in the appendix we have included a list of all papers reviewed in each of the categories with detailed information.

2. Methodology

We followed a systematic review process divided into two phases: the search process and the review process.

2.1. Search Process

One of our goals was to make this review as inclusive as practically possible. We therefore considered all papers published in conferences and journals between 2005 and 2014, which include the term “Augmented Reality,” and involve user studies. We searched the Scopus bibliographic database, using the same search terms that were used by Dünser et al. (2008) (Table 2 ). This initial search resulted in a total of 1,147 unique papers. We then scanned each one to identify whether or not it actually reported on AR research; excluding papers not related to AR reduced the number to 1,063. We next removed any paper that did not actually report on a user study, which reduced our pool to 604 papers. We then examined these 604 papers, and kept only those papers that provided all of the following information: (i) participant demographics (number, age, and gender), (ii) design of the user study, and (iii) the experimental task. Only 396 papers satisfied all three of these criteria. Finally, unlike previous surveys of AR usability studies, we next considered how much impact each paper had, to ensure that we were reviewing papers that others had cited. For each paper we used Google Scholar to find the total citations to date, and calculated its Average Citation Count (ACC):

For example, if a paper was published in 2010 (a 5 year lifetime until 2014) and had a total of 10 citations in Google Scholar in April 2015, its ACC would be 10/5 = 2.0. Based on this formula, we included all papers that had an ACC of at least 1.5, showing that they had at least a moderate impact in the field. This resulted in a final set of 291 papers that we reviewed in detail. We deliberately excluded papers more recent than 2015 because most of these hadn't gather significant citations yet.

www.frontiersin.org

Table 2 . Search terms used in the Scopus database.

2.2. Reviewing Process

In order to review this many papers, we randomly divided them among the authors for individual review. However, we first performed a norming process, where all of the authors first reviewed the same five randomly selected papers. We then met to discuss our reviews, and reached a consensus about what review data would be captured. We determined that our reviews would focus on the following attributes:

• application areas and keywords

• experimental design (within-subjects, between-subjects, or mixed-factorial)

• type of data collected (qualitative or quantitative)

• participant demographics (age, gender, number, etc.)

• experimental tasks and environments

• type of experiment (pilot, formal, field, heuristic, or case study)

• senses augmented (visual, haptic, olfactory, etc.)

• type of display used (handheld, head-mounted display, desktop, etc.).

In order to systematically enter this information for each paper, we developed a Google Form. During the reviews we also flagged certain papers for additional discussion. Overall, this reviewing phase encompassed approximately 2 months. During this time, we regularly met and discussed the flagged papers; we also clarified any concerns and generally strove to maintain consistency. At the end of the review process we had identified the small number of papers where the classification was unclear, so we held a final meeting to arrive at a consensus view.

2.3. Limitations and Validity Concerns

Although we strove to be systematic and thorough as we selected and reviewed these 291 papers, we can identify several limitations and validity concerns with our methods. The first involves using the Scopus bibliographic database. Although using such a database has the advantage of covering a wide range of publication venues and topics, and although it did cover all of the venues where the authors are used to seeing AR research, it remains possible that Scopus missed publication venues and papers that should have been included. Second, although the search terms we used seem intuitive (Table 2 ), there may have been papers that did not use “Augmented Reality” as a keyword when describing an AR experience. For example, some papers may have used the term “Mixed Reality,” or “Artificial Reality.”

Finally, although using the ACC as a selection factor narrowed the initial 604 papers to 291, it is possible that the ACC excluded papers that should have been included. In particular, because citations are accumulated over time, it is quite likely that we missed some papers from the last several years of our 10-year review period that may soon prove influential.

3. High-Level Overview of Reviewed Papers

Overall, the 291 papers report a total of 369 studies. Table 3 gives summary statistics for the papers, and Table 4 gives summary statistics for the studies. These tables contain bar graphs that visually depict the magnitude of the numbers; each color indicates the number of columns are spanned by the bars. For example, in Table 3 the columns Paper, Mean ACC, and Mean Author Count are summarized individually, and the longest bar in each column is scaled according to the largest number in that column. However, Publications spans two columns, and the largest value is 59, and so all of the other bars for Publications are scaled according to 59.

www.frontiersin.org

Table 3 . Summary of the 291 reviewed papers.

www.frontiersin.org

Table 4 . Summary of the 369 user studies reported by the 291 reviewed papers.

Figure 1 further summarizes the 291 papers through four graphs, all of which indicate changes over the 10 year period between 2005 and 2014. Figure 1A shows the fraction of the total number of AR papers that report user studies, Figure 1B analyzes the kind of display used, Figure 1C categorizes the experiments into application areas, and Figure 1D categorizes the papers according to the kind of experiment that was conducted.

www.frontiersin.org

Figure 1 . Throughout the 10 years, less than 10% of all published AR papers had a user study (A) . Out of the 291 reviewed papers, since 2011 most papers have examined handheld displays, rather than HMDs (B) . We filtered the papers based on ACC and categorized them into nine application areas; the largest areas are Perception and Interaction (C) . Most of the experiments were in controlled laboratory environments (D) .

3.1. Fraction of User Studies Over Time

Figure 1A shows the total number of AR papers published between 2005 and 2014, categorized by papers with and without a user study. As the graph shows, the number of AR papers published in 2014 is five times that published in 2005. However, the proportion of user study papers among all AR papers has remained low, less than 10% of all publication for each year.

3.2. Study Design

As shown in Table 4 , most of the papers (213, or 73%) used a within-subjects design, 43 papers (15%) used a between-subjects design, and 12 papers (4%) used a mixed-factorial design. However, there were 23 papers (8%) which used different study designs than the ones mentioned above, such as Baudisch et al. (2013) , Benko et al. (2014) , and Olsson et al. (2009) .

3.3. Study Type

We found that it was relatively rare for researchers to report on conducting pilot studies before their main study. Only 55 papers (19%) reported conducting at least one pilot study in their experimentation process and just 25 of them reported the pilot studies with adequate details such as study design, participants, and results. This shows that the importance of pilot studies is not well recognized. The majority of the papers (221, or 76%) conducted the experiments in controlled laboratory environments, while only 44 papers (15%) conducted the experiments in a natural environment or as a field study (Figure 1D ). This shows a lack of experimentation in real world conditions. Most of the experiments were formal user studies, and there were almost no heuristic studies, which may indicate that the heuristics of AR applications are not fully developed and there exists a need for heuristics and standardization.

3.4. Data Type

In terms of data collection, a total of 139 papers (48%) collected both quantitative and qualitative data, 78 (27%) papers only qualitative, and 74 (25%) only quantitative. For the experimental task, we found that the most popular task involved performance (178, or 61%), followed by filling out questionnaires (146, or 50%), perceptual tasks (53, or 18%), interviews (41, or 14%) and collaborative tasks (21, or 7%). In terms of dependent measures, subjective ratings were the most popular with 167 papers (57%), followed by error/accuracy measures (130, or 45%), and task completion time (123, or 42%). We defined task as any activity that was carried out by the participants to provide data—both quantitative and/or qualitative—about the experimental system(s). Note that many experiments used more than one experimental task or dependent measure, so the percentages sum to more than 100%. Finally, the bulk of the user studies were conducted in an indoor environment (246, or 83%), not outdoors (43, or 15%), or a combination of both settings (6, or 2%).

3.5. Senses

As expected, an overwhelming majority of papers (281, or 96%) augmented the visual sense. Haptic and Auditory senses were augmented in 27 (9%) and 21 (7%) papers respectively. Only six papers (2%) reported augmenting only the auditory sense and five (2%) papers reported augmenting only the haptic sense. This shows that there is an opportunity for conducting more user studies exploring non-visual senses.

3.6. Participants

The demographics of the participants showed that most of the studies were run with young participants, mostly university students. A total of 182 papers (62%) used participants with an approximate mean age of less than 30 years. A total of 227 papers (78%) reported involving female participants in their experiments, but the ratio of female participants to male participants was low (43% of total participants in those 227 papers). When all 291 papers are considered only 36% of participants were females. Many papers (117, or 40%) did not explicitly mention the source of participant recruitment. From those that did, most (102, or 35%) sourced their participants from universities, whereas only 36 papers (12%) mentioned sourcing participants from the general public. This shows that many AR user studies use young male university students as their subjects, rather than a more representative cross section of the population.

3.7. Displays

We also recorded the displays used in these experiments (Table 3 ). Most of the papers used either HMDs (102 papers, or 35%) or handhelds (100 papers, or 34%), including six papers that used both. Since 2009, the number of papers using HMDs started to decrease while the number of papers using handheld displays increased (Figure 1B ). For example, between 2010 and 2014 (204 papers in our review), 50 papers used HMDs and 79 used handhelds, including one paper that used both, and since 2011 papers using handheld displays consistently outnumbered papers using HMDs. This trend—that handheld mobile AR has recently become the primary display for AR user studies—is of course driven by the ubiquity of smartphones.

3.8. Categorization

We categorized the papers into nine different application areas (Tables 3 , 4 ): (i) Perception (51 papers, or 18%), (ii) Medical (43, or 15%), (iii) Education (42, or 14%), (iv) Entertainment and Gaming (14, or 5%), (v) Industrial (30, or 10%), (vi) Navigation and Driving (24, or 9%), (vii) Tourism and Exploration (8, or 2%), (viii) Collaboration (12, or 4%), and (ix) Interaction (67, or 23%). Figure 1C shows the change over time in number of AR papers with user studies in these categories. The Perception and Interaction categories are rather general areas of AR research, and contain work that reports on more low-level experiments, possibly across multiple application areas. Our analysis shows that there are fewer AR user studies published in Collaboration, Tourism and Exploration, and Entertainment and Gaming, identifying future application areas for user studies. There is also a noticeable increase in the number of user studies in educational applications over time. The drop in number of papers in 2014 is due to the selection criteria of papers having at least 1.5 average citations per year, as these papers were too recent to be cited often. Interestingly, although there were relatively few of them, papers in Collaboration, Tourism and Exploration categories received noticeably higher ACC scores than other categories.

3.9. Average Authors

As shown in Table 3 , most categories had a similar average number of authors for each paper, ranging between 3.24 (Education) and 3.87 (Industrial). However papers in the Medical domain had the highest average number of authors (6.02), which indicates the multidisciplinary nature of this research area. In contrast to all other categories, most of the papers in the Medical category were published in journals, compared to the common AR publications venues, which are mostly conferences. Entertainment and Gaming (4.71), and Navigation and Driving (4.58) also had considerably higher numbers of authors per paper on average.

3.10. Individual Studies

While a total of 369 studies were reported in these 291 papers (Table 4 ), the majority of the papers (231, or 80%) reported only one user study. Forty-seven (16.2%), nine (3.1%), two (<1%), and one (<1%) papers reported two, three, four, and five studies respectively, including pilot studies. In terms of the number of participants used (median) in each study, Tourism and Exploration, and Education were the highest among all categories with an average of 28 participants per study. Other categories used between 12 and 18 participants per study, while the overall median stands at 16 participants. Based on this insight, it can be claimed that 12 to 18 participants per study is a typical range in the AR community. Out of the 369 studies 31 (8.4%) were pilot studies, six (1.6%) heuristic evaluation, 54 (14.6%) field studies, and rest of the 278 (75.3%) were formal controlled user studies. Most of the studies (272, or 73.7%) were designed as within-subjects, 52 (14.1%) between-subjects, and 16 (4.3%) as mixed-factors (Table 4 ).

In the following section we review user studies in each of the nine application areas separately. We provide a commentary on each category and also discuss a representative paper with the highest ACCs in each application area, so that readers can understand typical user studies from that domain. We present tables summarizing all of the papers from these areas at the end of the paper.

4. Application Areas

4.1. collaboration.

A total of 15 studies were reported in 12 papers in the Collaboration application area. The majority of the studies investigated some form of remote collaboration (Table 5 ), although Henrysson et al. (2005a) presented a face-to-face collaborative AR game. Interestingly, out of the 15 studies, eight reported using handheld displays, seven used HMDs, and six used some form of desktop display. This makes sense as collaborative interfaces often require at least one collaborator to be stationary and desktop displays can be beneficial in such setups. One noticeable feature was the low number of studies performed in the wild or in natural settings (field studies). Only three out of 15 studies were performed in natural settings and there were no pilot studies reported, which is an area for potential improvement. While 14 out of 15 studies were designed to be within-subjects, only 12 participants were recruited per study. On average, roughly one-third of the participants were females in all studies considered together. All studies were performed in indoor locations except for ( Gauglitz et al., 2014b ), which was performed in outdoors. While a majority of the studies (8) collected both objective (quantitative) and subjective (qualitative) data, five studies were based on only subjective data, and two studies were based on only objective data, both of which were reported in one paper ( Henrysson et al., 2005a ). Besides subjective feedback or ratings, task completion time and error/accuracy were other prominent dependent variables used. Only one study used NASA TLX ( Wang and Dunston, 2011 ).

www.frontiersin.org

Table 5 . Summary of user studies in Collaboration application area.

4.1.1. Representative Paper

As an example of the type of collaborative AR experiments conducted, we discuss the paper of Henrysson et al. (2005a) in more detail. They developed an AR-based face-to-face collaboration tool using a mobile phone and reported on two user studies. This paper received an ACC of 22.9, which is the highest in this category of papers. In the first study, six pairs of participants played a table-top tennis game in three conditions—face to face AR, face to face non-AR, and non-face to face collaboration. In the second experiment, the authors added (and varied) audio and haptic feedback to the games and only evaluated face to face AR. The same six pairs were recruited for this study as well. Authors collected both quantitative and qualitative (survey and interview) data, although they focused more on the latter. They asked questions regarding the usability of system and asked participants to rank the conditions. They explored several usability issues and provided design guidelines for developing face to face collaborative AR applications using handheld displays. For example, designing applications that have a focus on a single shared work space.

4.1.2. Discussion

The work done in this category is mostly directed toward remote collaboration. With the advent of modern head mounted devices such the Microsoft HoloLens, new types of collaborations can be created, including opportunities for enhanced face to face collaboration. Work needs to be done toward making AR-based remote collaboration akin to the real world with not only shared understanding of the task but also shared understanding of the other collaborators emotional and physiological states. New gesture-based and gaze-based interactions and collaboration across multiple platforms (e.g., between AR and virtual reality users) are novel future research directions in this area.

4.2. Education

Fifty-five studies were reported in 42 papers in the Education application area (Table 6 ). As expected, all studies reported some kind of teaching and learning applications, with a few niche areas, such as music training, educational games, and teaching body movements. Out of 55 studies, 24 used handheld displays, 8 used HMDs, 16 used some form of desktop displays, and 11 used spatial or large-scale displays. One study had augmented only sound feedback and used a head-mounted speaker ( Hatala and Wakkary, 2005 ). Again, a trend of using handheld displays is prominent in this application area as well. Among all the studies reported, 13 were pilot studies, 14 field studies, and 28 controlled lab-based experiments. Thirty-one studies were designed as within-subjects studies, and 16 as between-subjects. Six studies had only one condition tested. The median number of participants was 28, jointly highest among all application areas. Almost 43% of participants were females. Forty-nine studies were performed in indoor locations, four in outdoor locations, and two studies were performed in both locations. Twenty-five studies collected only subjective data, 10 objective data, and 20 studies collected both types of data. While subjective rating was the primary dependent measure used in most of the studies, some specific measures were also noticed, such as pre- and post-test scores, number of items remembered, and engagement. From the keywords used in the papers, it appears that learning was the most common keyword and interactivity, users , and environments also received noticeable importance from the authors.

www.frontiersin.org

Table 6 . Summary of user studies in Education application area.

4.2.1. Representative Paper

The paper from Fonseca et al. (2014a) received the highest ACC (22) in the Education application area of AR. They developed a mobile phone-based AR teaching tool for 3D model visualization and architectural projects for classroom learning. They recruited a total of 57 students (29 females) in this study and collected qualitative data through questionnaires and quantitative data through pre- and post-tests. This data was collected over several months of instruction. The primary dependent variable was the academic performance improvement of the students. Authors used five-point Likert-scale questions as the primary instrument. They reported that using the AR tool in the classroom was correlated with increased motivation and academic achievement. This type of longitudinal study is not common in the AR literature, but is helpful in measuring the actual real-world impact of any application or intervention.

4.2.2. Discussion

The papers in this category covered a diverse range of education and training application areas. There are some papers used AR to teach physically or cognitively impaired patients, while a couple more promoted physical activity. This set of papers focused on both objective and subjective outcomes. For example, Anderson and Bischof (2014) reported a system called ARM trainer to train amputees in the use of myoelectric prostheses that provided an improved user experience over the current standard of care. In a similar work, Gama et al. (2012) presented a pilot study for upper body motor movements where users were taught to move body parts in accordance to the instructions of an expert such as physiotherapist and showed that AR-based system was preferred by the participants. Their system can be applied to teach other kinds of upper body movements beyond just rehabilitation purposes. In another paper, Chang et al. (2013) reported a study where AR helped cognitively impaired people to gain vocational job skills and the gained skills were maintained even after the intervention. Hsiao et al. (2012) and Hsiao (2010) presented a couple of studies where physical activity was included in the learning experience to promote “learning while exercising". There are few other papers that gamified the AR learning content and they primarily focused on subjective data. Iwata et al. (2011) presented ARGo an AR version of the GO game to investigate and promote self-learning. Juan et al. (2011b) developed ARGreenet game to create awareness for recycling. Three papers investigated education content themed around tourism and mainly focused on subjective opinion. For example, Hatala and Wakkary (2005) created a museum guide educating users about the objects in the museum and Szymczak et al. (2012) created multi-sensory application for teaching about the historic sites in a city. There were several other papers that proposed and evaluated different pedagogical approaches using AR including two papers that specifically designed for teaching music such as Liarokapis (2005) and Weing et al. (2013) . Overall these papers show that in the education space a variety of evaluation methods can be used, focusing both on educational outcomes and application usability. Integrating methods of intelligent tutoring systems ( Anderson et al., 1985 ) with AR could provide effective tools for education. Another interesting area to explore further is making these educational interfaces adaptive to the users cognitive load.

4.3. Entertainment and Gaming

We reviewed a total of 14 papers in the Entertainment and Gaming area with 18 studies were reported in these papers (Table 7 ). A majority of the papers reported a gaming application while fewer papers reported about other forms of entertainment applications. Out of the 18 studies, nine were carried out using handheld displays and four studies used HMDs. One of the reported studies, interestingly, did not use any display ( Xu et al., 2011 ). Again, the increasing use of handheld displays is expected as this kind of display provides greater mobility than HMDs. Five studies were conducted as field studies and the rest of the 13 studies were controlled lab-based experiments. Fourteen studies were designed as within-subjects and two were between-subjects. The median number of participants in these studies was 17. Roughly 41.5% of participants were females. Thirteen studies were performed in indoor areas, four were in outdoor locations, and one study was conducted in both locations. Eight studies collected only subjective data, another eight collected both subjective and objective data, and the remaining two collected only objective data. Subjective preference was the primary measure of interest. However, task completion time was also another important measure. In this area, error/accuracy was not found to be a measure in the studies used. In terms of the keywords used by the authors, besides games, mobile and handheld were other prominent keywords. These results highlight the utility of handheld displays for AR Entertainment and Gaming studies.

www.frontiersin.org

Table 7 . Summary of user studies in Entertainment and Gaming application area.

4.3.1. Representative Paper

Dow et al. (2007) presented a qualitative user study exploring the impact of immersive technologies on presence and engagement, using interactive drama, where players had to converse with characters and manipulate objects in the scene. This paper received the highest ACC (9.5) in this category of papers. They compared two versions of desktop 3D based interfaces with an immersive AR based interface in a lab-based environment. Participants communicated in the desktop versions using keyboards and voice. The AR version used a video see-though HMD. They recruited 12 participants (six females) in the within-subjects study, each of whom had to experience interactive dramas. This paper is unusual because user data was collected mostly from open-ended interviews and observation of participant behaviors, and not task performance or subjective questions. They reported that immersive AR caused an increased level of user Presence, however, higher presence did not always led to more engagement.

4.3.2. Discussion

It is clear that advances in mobile connectivity, CPU and GPU processing capabilities, wearable form factors, tracking robustness, and accessibility to commercial-grade game creation tools is leading to more interest in AR for entertainment. There is significant evidence from both AR and VR research of the power of immersion to provide a deeper sense of presence, leading to new opportunities for enjoyment in Mixed Reality (a continuum encompassing both AR and VR Milgram et al., 1995 ) spaces. Natural user interaction will be key to sustaining the use of AR in entertainment, as users will shy away from long term use of technologies that induce fatigue. In this sense, wearable AR will probably be more attractive for entertainment AR applications. In these types of entertainment applications, new types of evaluation measures will need to be used, as shown by the work of Dow et al. (2007) .

4.4. Industrial

There was a total of 30 papers reviewed that focused on Industrial applications, and together they reported 36 user studies. A majority of the studies reported maintenance and manufacturing/assembly related tasks (Table 8 ). Eleven studies used handheld displays, 21 used HMDs, four used spatial or large screen displays, and two used desktop displays. The prevalence of HMDs was expected as most of the applications in this area require use of both hands at times, and as such HMDs are more suitable as displays. Twenty-nine studies were executed in a formal lab-based environment and only six studies were executed in their natural setups. We believe performing more industrial AR studies in the natural environment will lead to more-usable results, as controlled environments may not expose the users to the issues that they face in real-world setups. Twenty-eight studies were designed as within-subjects and six as between-subjects. One study was designed to collect exploratory feedback from a focus group ( Olsson et al., 2009 ). The median number of participants used in these studies was 15 and roughly 23% of them were females. Thirty-two studies were performed in indoor locations and four in outdoor locations. Five studies were based on only subjective data, four on only objective data, and rest of the 27 collected both kinds of data. Use of NASA TLX was very common in this application area, which was expected given the nature of the tasks. Time and error/accuracy were other commonly used measurements along with subjective feedback. The keywords used by the authors to describe their papers highlight a strong interest in interaction, interfaces , and users . Guidance and maintenance are other prominent keywords that authors used.

www.frontiersin.org

Table 8 . Summary of user studies in Industrial area.

4.4.1. Representative Paper

As an example of the papers written in this area, Henderson S. and Feiner (2011) published a work exploring AR documentation for maintenance and repair tasks in a military vehicle, which received the highest ACC (26.25) in the Industrial area. They used a video see-though HMD to implement the study application. In the within-subjects study, the authors recruited six male participants who were professional military mechanics and they performed the tasks in the field settings. They had to perform 18 different maintenance tasks using three conditions—AR, LCD, and HUD. Several quantitative and qualitative (questionnaire) data were collected. As dependent variables they used task completion time, task localization time, head movement, and errors. The AR condition resulted in faster locating tasks and fewer head-movements. Qualitatively, AR was also reported to be more intuitive and satisfying. This paper provides an outstanding example of how to collect both qualitative and quantitative measures in an industrial setting, and so get a better indication of the user experience.

4.4.2. Discussion

Majority of the work in this category focused on maintenance and assembly tasks, whereas a few investigated architecture and planning tasks. Another prominent line of work in this category is military applications. Some work also cover surveying and item selection (stock picking). It will be interesting to investigate non-verbal communication cues in collaborative industrial applications where people form multiple cultural background can easily work together. As most of the industrial tasks require specific training and working in a particular environment, we assert that there needs to be more studies that recruit participants from the real users and perform studies in the field when possible.

4.5. Interaction

There were 71 papers in the Interaction design area and 83 user studies reported in these papers (see Table 9 ). Interaction is a very general area in AR, and the topics covered by these papers were diverse. Forty studies used handheld displays, 33 used HMDs, eight used desktop displays, 12 used spatial or large-screen displays, and 10 studies used a combination of multiple display types. Seventy-one studies were conducted in a lab-based environment, five studies were field studies, and six were pilot studies. Jones et al. (2013) were the only authors to conduct a heuristic evaluation. The median number of participants used in these studies was 14, and approximately 32% of participants were females. Seventy-five studies were performed in indoor locations, seven in outdoor locations, and one study used both locations. Sixteen studies collected only subjective data, 14 collected only objective data, and 53 studies collected both types of data. Task completion time and error/accuracy were the most commonly used dependent variables. A few studies used the NASA TLX workload survey ( Robertson et al., 2007 ; Henze and Boll, 2010b ) and most of the studies used different forms of subjective ratings, such as ranking conditions and rating on a Likert scale. The keywords used by authors identify that the papers in general were focused on interaction, interface, user, mobile , and display devices.

www.frontiersin.org

Table 9 . Summary of user studies in Interaction application area.

4.5.1. Representative Paper

Boring et al. (2010) presented a user study for remote manipulation of content on distant displays using their system, which was named Touch Projector and was implemented on an iPhone 3G. This paper received the highest ACC (31) in the Interaction category of papers. They implemented multiple interaction methods on this application, e.g., manual zoom, automatic zoom, and freezing. The user study involved 12 volunteers (four females) and was designed as a within-subjects study. In the experiment, participants selected targets and dragged targets between displays using the different conditions. Both quantitative and qualitative data (informal feedback) were collected. The main dependent variables were task completion time, failed trials, and docking offset. They reported that participants achieved highest performance with automatic zooming and temporary image freezing. This is a typical study in the AR domain based within a controlled laboratory environment. As usual in interaction studies, a significant amount of the study was focused on user performance with different input conditions, and this paper shows the benefit of capturing different types of performance measures, not just task completion time.

4.5.2. Discussion

User interaction is a cross-cutting focus of research, and as such, does not fall neatly within an application category, but deeply influences user experience in all categories. The balance of expressiveness and efficiency is a core concept in general human-computer interaction, but is of even greater importance in AR interaction, because of the desire to interact while on the go, the danger of increased fatigue, and the need to interact seamlessly with both real and virtual content. Both qualitative and quantitative evaluations will continue to be important in assessing usability in AR applications, and we encourage researchers to continue with this approach. It is also important to capture as many different performance measures as possible from the interaction user study to fully understand how a user interacts with the system.

4.6. Medicine

One of the most promising areas for applying AR is in medical sciences. However, most of the medical-related AR papers were published in medical journals rather than the most common AR publication venues. As we considered all venues in our review, we were able to identify 43 medical papers reporting AR studies and they in total reported 54 user studies. The specific topics were diverse, including laparoscopic surgery, rehabilitation and recovery, phobia treatment, and other medical training. This application area was dominated by desktop displays (34 studies), while 16 studies used HMDs, and handheld displays were used in only one study. This is very much expected, as often in medical setups, a clear view is needed along with free hands without adding any physical load. As expected, all studies were performed in indoor locations. Thirty-six studies were within-subjects and 11 were between-subjects. The median number of participants was 13, and approximately only 14.2% of participants were females, which is considerably lower than the gender-ratio in the profession of medicine. Twenty-two studies collected only objective data, 19 collected only subjective data, and 13 studies collected both types of data. Besides time and accuracy, various domain-specific surveys and other instruments were used in these studies as shown in Table 10 .

www.frontiersin.org

Table 10 . Summary of user studies in Medical application areas.

The keywords used by authors suggest that AR-based research was primarily used in training and simulation . Laparoscopy, rehabilitation , and phobia were topics of primary interest. One difference between the keywords used in medical science vs. other AR fields is the omission of the word user , which indicates that the interfaces designed for medical AR were primarily focused on achieving higher precision and not on user experience. This is understandable as the users are highly trained professionals who need to learn to use new complex interfaces. The precision of the interface is of utmost importance, as poor performance can be life threatening.

4.6.1. Representative Paper

Archip et al. (2007) reported on a study that used AR visualization for image-guided neurosurgery, which received the highest ACC (15.6) in this category of papers. Researchers recruited 11 patients (six females) with brain tumors who underwent surgery. Quantitative data about alignment accuracy was collected as a dependent variable. They found that using AR produced a significant improvement in alignment accuracy compared to the non-AR system already in use. An interesting aspect of the paper was that it focused purely on one user performance measure, alignment accuracy, and there was no qualitative data captured from users about how they felt about the system. This appears to be typical for many medical related AR papers.

4.6.2. Discussion

AR medical applications are typically designed for highly trained medical practitioners, which are a specialist set of users compared to other types of user studies. The overwhelming focus is on improving user performance in medical tasks, and so most of the user studies are heavily performance focused. However, there is an opportunity to include more qualitative measures in medical AR studies, especially those that relate to user estimation of their physical and cognitive workload, such as the NASA TLX survey. In many cases medical AR interfaces are aiming to improve user performance in medical tasks compared to traditional medical systems. This means that comparative evaluations will need to be carried out and previous experience with the existing systems will need to be taken into account.

4.7. Navigation and Driving

A total of 24 papers reported 28 user studies in the Navigation and Driving application areas (see Table 11 ). A majority of the studies reported applications for car driving. However, there were also pedestrian navigation applications for both indoors and outdoors. Fifteen studies used handheld displays, five used HMDs, and two used heads-up displays (HUDs). Spatial or large-screen displays were used in four studies. Twenty-three of the studies were performed in controlled setups and the remaining five were executed in the field. Twenty-two studies were designed as within-subjects, three as between-subjects, and the remaining three were mixed-factors studies. Approximately 38% of participants were females in these studies, where the median number of participants used was 18. Seven studies were performed in an outdoor environment and the rest in indoor locations. This indicates an opportunity to design and test hybrid AR navigation applications that can be used in both indoor and outdoor locations. Seven studies collected only objective data, 18 studies collected a combination of both objective and subjective data, whereas only three studies were based only on subjective data. Task completion time and error/accuracy were the most commonly used dependent variables. Other domain specific variables used were headway variation (deviation from intended path), targets found, number of steps, etc.

www.frontiersin.org

Table 11 . Summary of user studies in Navigation and Driving application area.

Analysis of author-specified keywords suggests that mobile received a strong importance, which is also evident by the profuse use of handheld displays in these studies, since these applications are about mobility. Acceptance was one of the noticeable keywords, which indicates that the studies intended to investigate whether or not a navigation interface is acceptable by the users, given the fact that, in many cases, a navigational tool can affect the safety of the user.

4.7.1. Representative Paper

Morrison et al. (2009) published a paper reporting on a field study that compared a mobile augmented reality map (MapLens) and a 2D map in a between-subjects field study, which received the highest ACC (16.3) in this application area of our review. MapLens was implemented on a Nokia N95 mobile phone and use AR to show virtual points of interest overlaid on a real map. The experimental task was to play a location-based treasure hunt type game outdoors using either MapLens or a 2D map. Researchers collected both quantitative and qualitative (photos, videos, field notes, and questionnaires) data. A total of 37 participants (20 female) took part in the study. The authors found that the AR map created more collaborations between players, and argued that AR maps are more useful as a collaboration tool. This work is important, because it provides an outstanding example of an AR Field study evaluation, which is not very common in the AR domain. User testing in the field can uncover several usability issues that normal lab-based testing cannot identify, particularly in the Navigation application area. For example, Morrison et al. (2009) were able to identify the challenges for a person of using a handheld AR device while trying to maintain awareness of the world around themselves.

4.7.2. Discussion

Navigation is an area where AR technology could provide significant benefit, due to the ability to overlay virtual cues on the real world. This will be increasingly important as AR displays become more common in cars (e.g., windscreen heads up displays) and consumers begin to wear head mounted displays outdoors. Most navigation studies have related to vehicle driving, and so there is a significant opportunity for pedestrian navigation studies. However human movement is more complex and erratic than driving, so these types of studies will be more challenging. Navigation studies will need to take into consideration the user's spatial ability, how to convey depth cues, and methods for spatial information display. The current user studies show how important it is to conduct navigation studies outdoors in a realistic testing environment, and the need to capture a variety of qualitative and quantitative data.

4.8. Perception

Similar to Interaction, Perception is another general field of study within AR, and appears in 51 papers in our review. There were a total of 71 studies reported in these papers. The primary focus was on visual perception (see Table 12 ) such as perception of depth/distance, color, and text. A few studies also reported perception of touch (haptic feedback). AR X-ray vision was also a common interface reported in this area. Perception of egocentric distance received significant attention, while exocentric distance was studied less. Also, near- to medium-field distance estimation was studied more than far-field distances. A comprehensive review of depth perception studies in AR can be found in Dey and Sandor (2014) , which also reports similar facts about AR perceptual studies as found in this review.

www.frontiersin.org

Table 12 . Summary of user studies in Perception application area.

Twenty-one studies used handheld displays, 34 studies used HMDs, and 9 studies used desktop displays. The Phantom haptic display was used by two studies where haptic feedback was studied. Sixty studies were performed as controlled lab-based experiments, and only three studies were performed in the field. Seven studies were pilot studies and there was one heuristic study ( Veas et al., 2012 ). Fifty-three studies were within-subjects, 12 between-subjects, and six mixed-factors. Overall, the median number of participants used in these studies was 16, and 27.3% of participants were females. Fifty-two studies were performed in indoor locations, only 17 studies were executed outdoors, and two studies used both locations. This indicates that indoor visual perception is well studied whereas more work is needed to investigate outdoor visual perception. Outdoor locations present additional challenges for visualizations such as brightness, screen-glare, and tracking (when mobile). This is an area to focus on as a research community. Thirty-two studies were based on only objective data, 14 used only subjective data, and 25 studies collected both kinds of data. Time and error/accuracy were most commonly used dependent measures along with subjective feedback.

Keywords used by authors indicate an emphasis on depth and visual perception, which is expected, as most of the AR interfaces augment the visual sense. Other prominent keywords were X-ray and see-through , which are the areas that have received a significant amount of attention from the community over the last decade.

4.8.1. Representative Paper

A recent paper by Suzuki et al. (2013) , reporting on the interaction of exteroceptive and interoceptive signals in virtual cardiac rubber hand perception, received the highest ACC (13.5) in this category of papers. The authors reported on a lab-based within-subjects user study using 21 participants (11 female) who wore a head-mounted display and experienced a tactile feedback simulating cardiac sensation. Both quantitative and qualitative (survey) data were collected. The main dependent variables were proprioceptive drift and virtual hand ownership. Authors reported that ownership of the virtual hand was significantly higher when tactile sensation was presented synchronously with the heart-beat of the participant than when provided asynchronously. This shows the benefit of combing perceptual cues to improve the user experience.

4.8.2. Discussion

A key focus of AR is trying to create a perceptual illusion that the AR content is seamlessly part of the user's real world. In order to measure how well this is occurring it is important to conduct perceptual user studies. Most studies to date have focused on visual perception, but there is a significant opportunity to conduct studies on non-visual cues, such as audio and haptic perception. One of the challenges of such studies is being able to measure the users perception of an AR cue, and also their confidence in how well they can perceive the cue. For example, asking users to estimate the distance on an AR object from them, and how sure they are about that estimation. New experimental methods may need to be developed to do this well.

4.9. Tourism and Exploration

Tourism is one of the relatively less explored areas of AR user studies, represented by only eight papers in our review (Table 13 ). A total of nine studies were reported, and the primary focus of the papers was on museum-based applications (five papers). Three studies used handheld displays, three used large-screen or spatial displays, and the rest head mounted displays. Six studies were conducted in the field, in the environment where the applications were meant to be used, and only three studies were performed in lab-based controlled environments. Six studies were designed to be within-subjects. This area of studies used a markedly higher number of participants compared to other areas, with the median number of participants being 28, with approximately 38% of them female. All studies were performed in indoor locations. While we are aware of studies in this area that have been performed in outdoor locations, these did not meet the inclusion criteria of our review. Seven studies were based completely on subjective data and two others used both subjective and objective data. As the nature of the interfaces were primarily personal experiences, the over reliance on subjective data is understandable. An analysis of keywords in the papers found that the focus was on museums . User was the most prominent keyword among all, which is very much expected for an interface technology such as AR.

www.frontiersin.org

Table 13 . Summary of user studies in Tourism and Exploration application area.

4.9.1. Representative Paper

The highest ACC (19) in this application area was received by an article published by Olsson et al. (2013) about the expectations of user experience of mobile augmented reality (MAR) services in a shopping context. Authors used semi-structured interviews as their research methodology and conducted 16 interview sessions with 28 participants (16 female) in two different shopping centers. Hence, their collected data was purely qualitative. The interviews were conducted individually, in pairs, and in groups. The authors reported on: (1) the characteristics of the expected user experience and, (2) central user requirements related to MAR in a shopping context. Users expected the MAR systems to be playful, inspiring, lively, collective, and surprising, along with providing context-aware and awareness-increasing services. This type of exploratory study is not common in the AR domain. However, it is a good example of how qualitative data can be used to identify user expectations and conceptualize user-centered AR applications. It is also an interesting study because people were asked what they expected of a mobile AR service, without actually seeing or trying the service out.

4.9.2. Discussion

One of the big advantages of studies done in this area is the relatively large sample sizes, as well as the common use of “in the wild” studies, that assess users outside of controlled environments. For these reasons, we see this application area as useful for exploring applied user interface designs, using real end-users in real environments. We also think that this category will continue to be attractive for applications that use handheld devices, as opposed to head-worn AR devices, since these are so common, and get out of the way of the content when someone wants to enjoy the physically beautiful/important works.

5. Conclusion

5.1. overall summary.

In this paper, we reported on 10 years of user studies published in AR papers. We reviewed papers from a wide range of journals and conferences as indexed by Scopus, which included 291 papers and 369 individual studies. Overall, on average, the number of user study papers among all AR papers published was less than 10% over the 10-year period we reviewed. Our exploration shows that although there has been an increase in the number of studies, the relative percentage appears the same. In addition, since 2011 there has been a shift toward more studies using handheld displays. Most studies were formal user studies, with little field testing and even fewer heuristic evaluations. Over the years there was an increase in AR user studies of educational applications, but there were few collaborative user studies. The use of pilot studies was also less than expected. The most popular data collection method involved filling out questionnaires, which led to subjective ratings being the most widely used dependent measure.

5.2. Findings and Suggestions

This analysis suggests opportunities for increased user studies in collaboration, more use of field studies, and a wider range of evaluation methods. We also find that participant populations are dominated by mostly young, educated, male participants, which suggests the field could benefit by incorporating a more diverse selection of participants. On a similar note, except for the Education and Tourism application categories, the median number of participants used in AR studies was between 12 and 18, which appears to be low compared to other fields of human-subject research. We have also noticed that within-subjects designs are dominant in AR, and these require fewer participants to achieve adequate statistical power. This is in contrast to general research in Psychology, where between-subject designs dominate.

Although formal, lab-based experiments dominated overall, the Education and Tourism application areas had higher ratios of field studies to formal lab-based studies, which required more participants. Researchers working in other application areas of AR could take inspiration from Education and Tourism papers and seek to perform more studies in real-world usage scenarios.

Similarly, because the social and environmental impact of outdoor locations differ from indoor locations, results obtained from indoor studies cannot be directly generalized to outdoor environments. Therefore, more user studies conducted outdoors are needed, especially ethnographic observational studies that report on how people naturally use AR applications. Finally, out of our initial 615 papers, 219 papers (35%) did not report either participant demographics, study design, or experimental task, and so could not be included in our survey. Any user study without these details is hard to replicate, and the results cannot be accurately generalized. This suggests a general need to improve the reporting quality of user studies, and education of researchers in the field on how to conduct good AR user studies.

5.3. Final Thoughts and Future Plans

For this survey, our goal has been to provide a comprehensive account of the AR user studies performed over the last decade. We hope that researchers and practitioners in a particular application area can use the respective summaries when planning their own research agendas. In the future, we plan to explore each individual application area in more depth, and create more detailed and focused reviews. We would also like to create a publicly-accessible, open database containing AR user study papers, where new papers can be added and accessed to inform and plan future research.

Author Contributions

All authors contributed significantly to the whole review process and the manuscript. AD initiated the process with Scopus database search, initial data collection, and analysis. AD, MB, RL, and JS all reviewed and collected data for an equal number of papers. All authors contributed almost equally to writing the paper, where AD and MB took the lead.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää, T., Kandemir, M., Kaski, S., et al. (2011). An augmented reality interface to contextual information. Virt. Real. 15, 161–173. doi: 10.1007/s10055-010-0183-5

CrossRef Full Text | Google Scholar

Akinbiyi, T., Reiley, C. E., Saha, S., Burschka, D., Hasser, C. J., Yuh, D. D., et al. (2006). “Dynamic augmented reality for sensory substitution in robot-assisted surgical systems,” in Annual International Conference of the IEEE Engineering in Medicine and Biology - Proceedings , 567–570.

PubMed Abstract | Google Scholar

Albrecht, U.-V., Folta-Schoofs, K., Behrends, M., and Von Jan, U. (2013). Effects of mobile augmented reality learning compared to textbook learning on medical students: randomized controlled pilot study. J. Med. Int. Res. 15. doi: 10.2196/jmir.2497

PubMed Abstract | CrossRef Full Text | Google Scholar

Allen, M., Regenbrecht, H., and Abbott, M. (2011). “Smart-phone augmented reality for public participation in urban planning,” in Proceedings of the 23rd Australian Computer-Human Interaction Conference, OzCHI 2011 , 11–20.

Google Scholar

Almeida, I., Oikawa, M., Carres, J., Miyazaki, J., Kato, H., and Billinghurst, M. (2012). “AR-based video-mediated communication: a social presence enhancing experience,” in Proceedings - 2012 14th Symposium on Virtual and Augmented Reality, SVR 2012 , 125–130.

Alvarez-Santos, V., Iglesias, R., Pardo, X., Regueiro, C., and Canedo-Rodriguez, A. (2014). Gesture-based interaction with voice feedback for a tour-guide robot. J. Vis. Commun. Image Represent. 25, 499–509. doi: 10.1016/j.jvcir.2013.03.017

Anderson, F., and Bischof, W. F. (2014). Augmented reality improves myoelectric prosthesis training. Int. J. Disabil. Hum. Dev. 13, 349–354. doi: 10.1515/ijdhd-2014-0327

Anderson, J. R., Boyle, C. F., and Reiser, B. J. (1985). Intelligent tutoring systems. Science 228, 456–462.

Anderson, F., Grossman, T., Matejka, J., and Fitzmaurice, G. (2013). “YouMove: enhancing movement training with an augmented reality mirror,” in UIST 2013 - Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology , 311–320.

Archip, N., Clatz, O., Whalen, S., Kacher, D., Fedorov, A., Kot, A., et al. (2007). Non-rigid alignment of pre-operative MRI, fMRI, and DT-MRI with intra-operative MRI for enhanced visualization and navigation in image-guided neurosurgery. Neuroimage 35, 609–624. doi: 10.1016/j.neuroimage.2006.11.060

Arning, K., Ziefle, M., Li, M., and Kobbelt, L. (2012). “Insights into user experiences and acceptance of mobile indoor navigation devices,” in Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia, MUM 2012 .

Arvanitis, T., Petrou, A., Knight, J., Savas, S., Sotiriou, S., Gargalakos, M., et al. (2009). Human factors and qualitative pedagogical evaluation of a mobile augmented reality system for science education used by learners with physical disabilities. Pers. Ubiquit. Comput. 13, 243–250. doi: 10.1007/s00779-007-0187-7

Asai, K., Kobayashi, H., and Kondo, T. (2005). “Augmented instructions - A fusion of augmented reality and printed learning materials,” in Proceedings - 5th IEEE International Conference on Advanced Learning Technologies, ICALT 2005 , Vol. 2005, 213–215.

Asai, K., Sugimoto, Y., and Billinghurst, M. (2010). “Exhibition of lunar surface navigation system facilitating collaboration between children and parents in Science Museum,” in Proceedings - VRCAI 2010, ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Application to Industry , 119–124.

Avery, B., Thomas, B. H., and Piekarski, W. (2008). “User evaluation of see-through vision for mobile outdoor augmented reality,” in Proceedings - 7th IEEE International Symposium on Mixed and Augmented Reality 2008, ISMAR 2008 , 69–72.

Axholt, M., Cooper, M., Skoglund, M., Ellis, S., O'Connell, S., and Ynnerman, A. (2011). “Parameter estimation variance of the single point active alignment method in optical see-through head mounted display calibration,” in Proceedings - IEEE Virtual Reality , 27–34.

Azuma, R. T. (1997). A survey of augmented reality. Presence 6, 355–385.

Bai, Z., and Blackwell, A. F. (2012). Analytic review of usability evaluation in ISMAR. Interact. Comput. 24, 450–460. doi: 10.1016/j.intcom.2012.07.004

Bai, H., Lee, G. A., and Billinghurst, M. (2012). “Freeze view touch and finger gesture based interaction methods for handheld augmented reality interfaces,” in ACM International Conference Proceeding Series , 126–131.

Bai, H., Gao, L., El-Sana, J. B. J., and Billinghurst, M. (2013a). “Markerless 3D gesture-based interaction for handheld augmented reality interfaces,” in SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications, SA 2013 .

Bai, Z., Blackwell, A. F., and Coulouris, G. (2013b). “Through the looking glass: pretend play for children with autism,” in 2013 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2013 , 49–58.

Bai, H., Lee, G. A., and Billinghurst, M. (2014). “Using 3D hand gestures and touch input for wearable AR interaction,” in Conference on Human Factors in Computing Systems - Proceedings , 1321–1326.

Baldauf, M., Lasinger, K., and Fröhlich, P. (2012). “Private public screens - Detached multi-user interaction with large displays through mobile augmented reality. in Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia, MUM 2012 .

Baričević, D., Lee, C., Turk, M., Höllerer, T., and Bowman, D. (2012). “A hand-held AR magic lens with user-perspective rendering,” in ISMAR 2012 - 11th IEEE International Symposium on Mixed and Augmented Reality 2012, Science and Technology Papers , 197–206.

Baudisch, P., Pohl, H., Reinicke, S., Wittmers, E., Lühne, P., Knaust, M., et al. (2013). “Imaginary reality gaming: Ball games without a ball,” in UIST 2013 - Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, UK), 405–410.

Benko, H., and Feiner, S. (2007). “Balloon selection: a multi-finger technique for accurate low-fatigue 3D selection,” in IEEE Symposium on 3D User Interfaces 2007 - Proceedings, 3DUI 2007 , 79–86.

Benko, H., Wilson, A., and Zannier, F. (2014). “Dyadic projected spatial augmented reality,” in UIST 2014 - Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, HI), 645–656.

Bichlmeier, C., Heining, S., Rustaee, M., and Navab, N. (2007). “Laparoscopic virtual mirror for understanding vessel structure: evaluation study by twelve surgeons,” in 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR .

Blum, T., Wieczorek, M., Aichert, A., Tibrewal, R., and Navab, N. (2010). “The effect of out-of-focus blur on visual discomfort when using stereo displays,” in 9th IEEE International Symposium on Mixed and Augmented Reality 2010: Science and Technology, ISMAR 2010 - Proceedings , 13–17.

Boring, S., Baur, D., Butz, A., Gustafson, S., and Baudisch, P. (2010). “Touch projector: Mobile interaction through video,” in Conference on Human Factors in Computing Systems - Proceedings , Vol. 4, (Atlanta, GA), 2287–2296.

Boring, S., Gehring, S., Wiethoff, A., Blöckner, M., Schöning, J., and Butz, A. (2011). “Multi-user interaction on media facades through live video on mobile devices,” in Conference on Human Factors in Computing Systems - Proceedings , 2721–2724.

Botden, S., Buzink, S., Schijven, M., and Jakimowicz, J. (2007). Augmented versus virtual reality laparoscopic simulation: what is the difference? A comparison of the ProMIS augmented reality laparoscopic simulator versus LapSim virtual reality laparoscopic simulator. World J. Surg. 31, 764–772. doi: 10.1007/s00268-006-0724-y

Botden, S., Buzink, S., Schijven, M., and Jakimowicz, J. (2008). ProMIS augmented reality training of laparoscopic procedures face validity. Simul. Healthc. 3, 97–102. doi: 10.1097/SIH.0b013e3181659e91

Botella, C., Juan, M., Baños, R., Alcañiz, M., Guillén, V., and Rey, B. (2005). Mixing realities? An application of augmented reality for the treatment of cockroach phobia. Cyberpsychol. Behav. 8, 162–171. doi: 10.1089/cpb.2005.8.162

Botella, C., Bretón-López, J., Quero, S., Baños, R., and García-Palacios, A. (2010). Treating cockroach phobia with augmented reality. Behav. Ther. 41, 401–413. doi: 10.1016/j.beth.2009.07.002

Botella, C., Breton-López, J., Quero, S., Baños, R., García-Palacios, A., Zaragoza, I., et al. (2011). Treating cockroach phobia using a serious game on a mobile phone and augmented reality exposure: a single case study. Comput. Hum. Behav. 27, 217–227. doi: 10.1016/j.chb.2010.07.043

Bretón-López, J., Quero, S., Botella, C., García-Palacios, A., Baños, R., and Alcañiz, M. (2010). An augmented reality system validation for the treatment of cockroach phobia. Cyberpsychol. Behav. Soc. Netw. 13, 705–710. doi: 10.1089/cyber.2009.0170

Brinkman, W., Havermans, S., Buzink, S., Botden, S., Jakimowicz, J. E., and Schoot, B. (2012). Single versus multimodality training basic laparoscopic skills. Surg. Endosc. Other Intervent. Tech. 26, 2172–2178. doi: 10.1007/s00464-012-2184-9

Bruno, F., Cosco, F., Angilica, A., and Muzzupappa, M. (2010). “Mixed prototyping for products usability evaluation,” in Proceedings of the ASME Design Engineering Technical Conference , Vol. 3, 1381–1390.

Bunnun, P., Subramanian, S., and Mayol-Cuevas, W. W. (2013). In Situ interactive image-based model building for Augmented Reality from a handheld device. Virt. Real. 17, 137–146. doi: 10.1007/s10055-011-0206-x

Cai, S., Chiang, F.-K., and Wang, X. (2013). Using the augmented reality 3D technique for a convex imaging experiment in a physics course. Int. J. Eng. Educ. 29, 856–865.

Cai, S., Wang, X., and Chiang, F.-K. (2014). A case study of Augmented Reality simulation system application in a chemistry course. Comput. Hum. Behav. 37, 31–40. doi: 10.1016/j.chb.2014.04.018

Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., and Ivkovic, M. (2011). Augmented reality technologies, systems and applications. Multim. Tools Appl. 51, 341–377. doi: 10.1007/s11042-010-0660-6

Chang, Y.-J., Kang, Y.-S., and Huang, P.-C. (2013). An augmented reality (AR)-based vocational task prompting system for people with cognitive impairments. Res. Dev. Disabil. 34, 3049–3056. doi: 10.1016/j.ridd.2013.06.026

Chastine, J., Nagel, K., Zhu, Y., and Yearsovich, L. (2007). “Understanding the design space of referencing in collaborative augmented reality environments,” in Proceedings - Graphics Interface , 207–214.

Chen, S., Chen, M., Kunz, A., Yantaç, A., Bergmark, M., Sundin, A., et al. (2013). “SEMarbeta: mobile sketch-gesture-video remote support for car drivers,” in ACM International Conference Proceeding Series , 69–76.

Chiang, T., Yang, S., and Hwang, G.-J. (2014). Students' online interactive patterns in augmented reality-based inquiry activities. Comput. Educ. 78, 97–108. doi: 10.1016/j.compedu.2014.05.006

Chintamani, K., Cao, A., Ellis, R., and Pandya, A. (2010). Improved telemanipulator navigation during display-control misalignments using augmented reality cues. IEEE Trans. Syst. Man Cybern. A Syst. Humans 40, 29–39. doi: 10.1109/TSMCA.2009.2030166

Choi, J., and Kim, G. J. (2013). Usability of one-handed interaction methods for handheld projection-based augmented reality. Pers. Ubiquit. Comput. 17, 399–409. doi: 10.1007/s00779-011-0502-1

Choi, J., Jang, B., and Kim, G. J. (2011). Organizing and presenting geospatial tags in location-based augmented reality. Pers. Ubiquit. Comput. 15, 641–647. doi: 10.1007/s00779-010-0343-3

Chun, W. H., and Höllerer, T. (2013). “Real-time hand interaction for augmented reality on mobile phones,” in International Conference on Intelligent User Interfaces, Proceedings IUI , 307–314.

Cocciolo, A., and Rabina, D. (2013). Does place affect user engagement and understanding?: mobile learner perceptions on the streets of New York. J. Document. 69, 98–120. doi: 10.1108/00220411311295342

Datcu, D., and Lukosch, S. (2013). “Free-hands interaction in augmented reality,” in SUI 2013 - Proceedings of the ACM Symposium on Spatial User Interaction , 33–40.

Denning, T., Dehlawi, Z., and Kohno, T. (2014). “ In situ with bystanders of augmented reality glasses: perspectives on recording and privacy-mediating technologies,” in Conference on Human Factors in Computing Systems - Proceedings , 2377–2386.

Dey, A., and Sandor, C. (2014). Lessons learned: evaluating visualizations for occluded objects in handheld augmented reality. Int. J. Hum. Comput. Stud. 72, 704–716. doi: 10.1016/j.ijhcs.2014.04.001

Dey, A., Cunningham, A., and Sandor, C. (2010). “Evaluating depth perception of photorealistic Mixed Reality visualizations for occluded objects in outdoor environments,” in Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST , 211–218.

Dey, A., Jarvis, G., Sandor, C., and Reitmayr, G. (2012). “Tablet versus phone: depth perception in handheld augmented reality,” in ISMAR 2012 - 11th IEEE International Symposium on Mixed and Augmented Reality 2012, Science and Technology Papers , 187–196.

Dierker, A., Mertes, C., Hermann, T., Hanheide, M., and Sagerer, G. (2009). “Mediated attention with multimodal augmented reality,” in ICMI-MLMI'09 - Proceedings of the International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interfaces , 245–252.

Dixon, B., Daly, M., Chan, H., Vescan, A., Witterick, I., and Irish, J. (2011). Augmented image guidance improves skull base navigation and reduces task workload in trainees: a preclinical trial. Laryngoscope 121, 2060–2064. doi: 10.1002/lary.22153

Dow, S., Mehta, M., Harmon, E., MacIntyre, B., and Mateas, M. (2007). “Presence and engagement in an interactive drama,” in Conference on Human Factors in Computing Systems - Proceedings (San Jose, CA), 1475–1484.

Dünser, A., Grasset, R., and Billinghurst, M. (2008). A Survey of Evaluation Techniques Used in Augmented Reality Studies . Technical Report.

Dünser, A., Billinghurst, M., Wen, J., Lehtinen, V., and Nurminen, A. (2012a). Exploring the use of handheld AR for outdoor navigation. Comput. Graph. 36, 1084–1095. doi: 10.1016/j.cag.2012.10.001

Dünser, A., Walker, L., Horner, H., and Bentall, D. (2012b). “Creating interactive physics education books with augmented reality,” in Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI 2012 ,107–114.

Espay, A., Baram, Y., Dwivedi, A., Shukla, R., Gartner, M., Gaines, L., et al. (2010). At-home training with closed-loop augmented-reality cueing device for improving gait in patients with Parkinson disease. J. Rehabil. Res. Dev. 47, 573–582. doi: 10.1682/JRRD.2009.10.0165

Fichtinger, G., Deguet, A., Masamune, K., Balogh, E., Fischer, G., Mathieu, H., et al. (2005). Image overlay guidance for needle insertion in CT scanner. IEEE Trans. Biomed. Eng. 52, 1415–1424. doi: 10.1109/TBME.2005.851493

Fichtinger, G. D., Deguet, A., Fischer, G., Iordachita, I., Balogh, E. B., Masamune, K., et al. (2005). Image overlay for CT-guided needle insertions. Comput. Aided Surg. 10, 241–255. doi: 10.3109/10929080500230486

Fiorentino, M., Debernardis, S., Uva, A. E., and Monno, G. (2013). Augmented reality text style readability with see-through head-mounted displays in industrial context. Presence 22, 171–190. doi: 10.1162/PRES_a_00146

Fiorentino, M., Uva, A. E., Gattullo, M., Debernardis, S., and Monno, G. (2014). Augmented reality on large screen for interactive maintenance instructions. Comput. Indust. 65, 270–278. doi: 10.1016/j.compind.2013.11.004

Fonseca, D., Redondo, E., and Villagrasa, S. (2014b). “Mixed-methods research: a new approach to evaluating the motivation and satisfaction of university students using advanced visual technologies,” in Universal Access in the Information Society .

Fonseca, D., Martí, N., Redondo, E., Navarro, I., and Sánchez, A. (2014a). Relationship between student profile, tool use, participation, and academic performance with the use of Augmented Reality technology for visualized architecture models. Comput. Hum. Behav. 31, 434–445. doi: 10.1016/j.chb.2013.03.006

Freitas, R., and Campos, P. (2008). “SMART: a system of augmented reality for teaching 2nd grade students,” in Proceedings of the 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction, BCS HCI 2008 , Vol. 2, 27–30.

Fröhlich, P., Simon, R., Baillie, L., and Anegg, H. (2006). “Comparing conceptual designs for mobile access to geo-spatial information,” in ACM International Conference Proceeding Series , Vol. 159, 109–112.

Fröhlich, P., Baldauf, M., Hagen, M., Suette, S., Schabus, D., and Kun, A. (2011). “Investigating safety services on the motorway: the role of realistic visualization,” in Proceedings of the 3rd International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2011 , 143–150.

Furió, D., González-Gancedo, S., Juan, M.-C., Seguí, I., and Rando, N. (2013). Evaluation of learning outcomes using an educational iPhone game vs. traditional game. Comput. Educ. 64, 1–23. doi: 10.1016/j.compedu.2012.12.001

Gabbard, J., and Swan II, J. c. (2008). Usability engineering for augmented reality: employing user-based studies to inform design. IEEE Trans. Visual. Comput. Graph. 14, 513–525. doi: 10.1109/TVCG.2008.24

Gabbard, J., Schulman, R., Edward Swan II, J., Lucas, J., Hix, D., and Gupta, D. (2005). “An empirical user-based study of text drawing styles and outdoor background textures for augmented reality,” in Proceedings - IEEE Virtual Reality , 11–18.317.

Gabbard, J., Swan II, J., and Mix, D. (2006). The effects of text drawing styles, background textures, and natural lighting on text legibility in outdoor augmented reality. Presence 15, 16–32. doi: 10.1162/pres.2006.15.1.16

Gabbard, J., Swan II, J., Hix, D., Kim, S.-J., and Fitch, G. (2007). “Active text drawing styles for outdoor augmented reality: a user-based study and design implications,” in Proceedings - IEEE Virtual Reality , 35–42.

Gama, A. D., Chaves, T., Figueiredo, L., and Teichrieb, V. (2012). “Guidance and movement correction based on therapeutics movements for motor rehabilitation support systems,” in Proceedings - 2012 14th Symposium on Virtual and Augmented Reality, SVR 2012 (Rio de Janeiro), 191–200.

Gandy, M., Catrambone, R., MacIntyre, B., Alvarez, C., Eiriksdottir, E., Hilimire, M., et al. (2010). “Experiences with an AR evaluation test bed: presence, performance, and physiological measurement,” in 9th IEEE International Symposium on Mixed and Augmented Reality 2010: Science and Technology, ISMAR 2010 - Proceedings , 127–136.

Gauglitz, S., Lee, C., Turk, M., and Höllerer, T. (2012). “Integrating the physical environment into mobile remote collaboration,” in MobileHCI'12 - Proceedings of the 14th International Conference on Human Computer Interaction with Mobile Devices and Services , 241–250.

Gauglitz, S., Nuernberger, B., Turk, M., and Höllerer, T. (2014a). “In touch with the remote world: remote collaboration with augmented reality drawings and virtual navigation,” in Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST , 197–205.

Gauglitz, S., Nuernberger, B., Turk, M., and Höllerer, T. (2014b). “World-stabilized annotations and virtual scene navigation for remote collaboration,” in UIST 2014 - Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, HI), 449–460.

Gavish, N., Gutiérrez, T., Webel, S., Rodríguez, J., Peveri, M., Bockholt, U., et al. (2013). Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact. Learn. Environ . 23, 778–798. doi: 10.1080/10494820.2013.815221

Gee, A., Webb, M., Escamilla-Ambrosio, J., Mayol-Cuevas, W., and Calway, A. (2011). A topometric system for wide area augmented reality. Comput. Graph. (Pergamon) 35, 854–868. doi: 10.1016/j.cag.2011.04.006

Goldiez, B., Ahmad, A., and Hancock, P. (2007). Effects of augmented reality display settings on human wayfinding performance. IEEE Trans. Syst. Man Cybern. C Appl. Rev. 37, 839–845. doi: 10.1109/TSMCC.2007.900665

Grasset, R., Lamb, P., and Billinghurst, M. (2005). “Evaluation of mixed-space collaboration,” in Proceedings - Fourth IEEE and ACM International Symposium on Symposium on Mixed and Augmented Reality, ISMAR 2005 , Vol. 2005, 90–99.

Grasset, R., Langlotz, T., Kalkofen, D., Tatzgern, M., and Schmalstieg, D. (2012). “Image-driven view management for augmented reality browsers,” in ISMAR 2012 - 11th IEEE International Symposium on Mixed and Augmented Reality 2012, Science and Technology Papers , 177–186.

Grasso, R., Faiella, E., Luppi, G., Schena, E., Giurazza, F., Del Vescovo, R., et al. (2013). Percutaneous lung biopsy: comparison between an augmented reality CT navigation system and standard CT-guided technique. Int. J. Comput. Assist. Radiol. Surg. 8, 837–848. doi: 10.1007/s11548-013-0816-8

Grechkin, T. Y., Nguyen, T. D., Plumert, J. M., Cremer, J. F., and Kearney, J. K. (2010). How does presentation method and measurement protocol affect distance estimation in real and virtual environments? ACM Trans. Appl. Percept. 7:26. doi: 10.1145/1823738.1823744

Grubert, J., Morrison, A., Munz, H., and Reitmayr, G. (2012). “Playing it real: magic lens and static peephole interfaces for games in a public space,” in MobileHCI'12 - Proceedings of the 14th International Conference on Human Computer Interaction with Mobile Devices and Services , 231–240.

Gupta, A., Fox, D., Curless, B., and Cohen, M. (2012). “DuploTrack: a real-time system for authoring and guiding duplo block assembly,” in UIST'12 - Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology , 389–401.

Gustafsson, A., and Gyllenswärd, M. (2005). “The power-aware cord: Energy awareness through ambient information display,” in Conference on Human Factors in Computing Systems - Proceedings , 1423–1426.

Ha, T., and Woo, W. (2010). “An empirical evaluation of virtual hand techniques for 3D object manipulation in a tangible augmented reality environment,” in 3DUI 2010 - IEEE Symposium on 3D User Interfaces 2010, Proceedings , 91–98.

Ha, T., Billinghurst, M., and Woo, W. (2012). An interactive 3D movement path manipulation method in an augmented reality environment. Interact. Comput. 24, 10–24. doi: 10.1016/j.intcom.2011.06.006

Hakkarainen, M., Woodward, C., and Billinghurst, M. (2008). “Augmented assembly using a mobile phone,” in Proceedings - 7th IEEE International Symposium on Mixed and Augmented Reality 2008, ISMAR 2008 , 167–168.

Hartl, A., Grubert, J., Schmalstieg, D., and Reitmayr, G. (2013). “Mobile interactive hologram verification,” in 2013 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2013 , 75–82.

Hatala, M., and Wakkary, R. (2005). Ontology-based user modeling in an augmented audio reality system for museums. User Modell. User Adapt. Interact. 15, 339–380. doi: 10.1007/s11257-005-2304-5

Hatala, M., Wakkary, R., and Kalantari, L. (2005). Rules and ontologies in support of real-time ubiquitous application. Web Semant. 3, 5–22. doi: 10.1016/j.websem.2005.05.004

Haugstvedt, A.-C., and Krogstie, J. (2012). “Mobile augmented reality for cultural heritage: A technology acceptance study,” in ISMAR 2012 - 11th IEEE International Symposium on Mixed and Augmented Reality 2012, Science and Technology Papers (Atlanta, GA), 247–255.

Heller, F., Krämer, A., and Borchers, J. (2014). “Simplifying orientation measurement for mobile audio augmented reality applications,” in Conference on Human Factors in Computing Systems - Proceedings , 615–623.

Henderson, S. J., and Feiner, S. (2008). “Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality,” in Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST , 211–218.

Henderson, S. J., and Feiner, S. (2009). “Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret,” in Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009 (Orlando, FL), 135–144.

Henderson, S., and Feiner, S. (2010). Opportunistic tangible user interfaces for augmented reality. IEEE Trans. Visual. Comput. Graph. 16, 4–16. doi: 10.1109/TVCG.2009.91

Henderson, S., and Feiner, S. (2011). Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Visual. Comput. Graphics 17, 1355–1368. doi: 10.1109/TVCG.2010.245

Henderson, S. J., and Feiner, S. K. (2011). “Augmented reality in the psychomotor phase of a procedural task,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011 (Basel), 191–200.

Henrysson, A., Billinghurst, M., and Ollila, M. (2005a). “Face to face collaborative AR on mobile phones,” in Proceedings - Fourth IEEE and ACM International Symposium on Symposium on Mixed and Augmented Reality, ISMAR 2005 , Vol. 2005, 80–89.

Henrysson, A., Billinghurst, M., and Ollila, M. (2005b). “Virtual object manipulation using a mobile phone,” in ACM International Conference Proceeding Series , Vol. 157, 164–171.

Henrysson, A., Marshall, J., and Billinghurst, M. (2007). “Experiments in 3D interaction for mobile phone AR,” in Proceedings - GRAPHITE 2007, 5th International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia , 187–194.

Henze, N., and Boll, S. (2010a). “Designing a CD augmentation for mobile phones,” in Conference on Human Factors in Computing Systems - Proceedings , 3979–3984.

Henze, N., and Boll, S. (2010b). “Evaluation of an off-screen visualization for magic lens and dynamic peephole interfaces,” in ACM International Conference Proceeding Series (Lisbon), 191–194.

Hincapié-Ramos, J., Roscher, S., Büschel, W., Kister, U., Dachselt, R., and Irani, P. (2014). “CAR: contact augmented reality with transparent-display mobile devices,” in PerDis 2014 - Proceedings: 3rd ACM International Symposium on Pervasive Displays 2014 , 80–85.

Hoang, T. N., and Thomas, B. H. (2010). “Augmented viewport: an action at a distance technique for outdoor AR using distant and zoom lens cameras,” in Proceedings - International Symposium on Wearable Computers, ISWC .

Horeman, T., Rodrigues, S., Van Den Dobbelsteen, J., Jansen, F.-W., and Dankelman, J. (2012). Visual force feedback in laparoscopic training. Surg. Endosc. Other Intervent. Techniq. 26, 242–248. doi: 10.1007/s00464-011-1861-4

Horeman, T., Van Delft, F., Blikkendaal, M., Dankelman, J., Van Den Dobbelsteen, J., and Jansen, F.-W. (2014). Learning from visual force feedback in box trainers: tissue manipulation in laparoscopic surgery. Surg. Endosc. Other Intervent. Techniq. 28, 1961–1970. doi: 10.1007/s00464-014-3425-x

Hou, L., and Wang, X. (2013). A study on the benefits of augmented reality in retaining working memory in assembly tasks: a focus on differences in gender. Automat. Construct. 32, 38–45. doi: 10.1016/j.autcon.2012.12.007

Hsiao, K.-F., Chen, N.-S., and Huang, S.-Y. (2012). Learning while exercising for science education in augmented reality among adolescents. Interact. Learn. Environ. 20, 331–349. doi: 10.1080/10494820.2010.486682

Hsiao, O.-F. (2010). Can we combine learning with augmented reality physical activity? J. Cyber Ther. Rehabil. 3, 51–62.

Hunter, S., Kalanithi, J., and Merrill, D. (2010). “Make a Riddle and TeleStory: designing children's applications for the Siftables platform,” in Proceedings of IDC2010: The 9th International Conference on Interaction Design and Children , 206–209.

Hürst, W., and Van Wezel, C. (2013). Gesture-based interaction via finger tracking for mobile augmented reality. Multimedia Tools Appl. 62, 233–258. doi: 10.1007/s11042-011-0983-y

Ibáñez, M., Di Serio, A., Villarán, D., and Delgado Kloos, C. (2014). Experimenting with electromagnetism using augmented reality: impact on flow student experience and educational effectiveness. Comput. Educ. 71, 1–13. doi: 10.1016/j.compedu.2013.09.004

Irizarry, J., Gheisari, M., Williams, G., and Walker, B. (2013). InfoSPOT: a mobile augmented reality method for accessing building information through a situation awareness approach. Autom. Construct. 33, 11–23. doi: 10.1016/j.autcon.2012.09.002

Iwai, D., Yabiki, T., and Sato, K. (2013). View management of projected labels on nonplanar and textured surfaces. IEEE Trans. Visual. Comput. Graph. 19, 1415–1424. doi: 10.1109/TVCG.2012.321

Iwata, T., Yamabe, T., and Nakajima, T. (2011). “Augmented reality go: extending traditional game play with interactive self-learning support,” in Proceedings - 17th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications, RTCSA 2011 , Vol. 1, (Toyama), 105–114.

Jankowski, J., Samp, K., Irzynska, I., Jozwowicz, M., and Decker, S. (2010). “Integrating text with video and 3D graphics: the effects of text drawing styles on text readability,” in Conference on Human Factors in Computing Systems - Proceedings , Vol. 2, 1321–1330.

Jeon, S., and Choi, S. (2011). Real stiffness augmentation for haptic augmented reality. Presence 20, 337–370. doi: 10.1162/PRES_a_00051

Jeon, S., and Harders, M. (2012). “Extending haptic augmented reality: modulating stiffness during two-point squeezing,” in Haptics Symposium 2012, HAPTICS 2012 - Proceedings , 141–146.

Jeon, S., Choi, S., and Harders, M. (2012). Rendering virtual tumors in real tissue Mock-Ups using haptic augmented reality. IEEE Trans. Hapt. 5, 77–84. doi: 10.1109/TOH.2011.40

Jo, H., Hwang, S., Park, H., and Ryu, J.-H. (2011). Aroundplot: focus+context interface for off-screen objects in 3D environments. Comput. Graph. (Pergamon) 35, 841–853.

Jones, J., Swan, J., Singh, G., Kolstad, E., and Ellis, S. (2008). “The effects of virtual reality, augmented reality, and motion parallax on egocentric depth perception,” in APGV 2008 - Proceedings of the Symposium on Applied Perception in Graphics and Visualization , 9–14.

Jones, J., Swan II, J., Singh, G., and Ellis, S. (2011). “Peripheral visual information and its effect on distance judgments in virtual and augmented environments,” in Proceedings - APGV 2011: ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization , 29–36.

Jones, B., Benko, H., Ofek, E., and Wilson, A. (2013). “IllumiRoom: peripheral projected illusions for interactive experiences,” in Conference on Human Factors in Computing Systems - Proceedings (Paris), 869–878.

Juan, M., and Joele, D. (2011). A comparative study of the sense of presence and anxiety in an invisible marker versus a marker augmented reality system for the treatment of phobia towards small animals. Int. J. Hum. Comput. Stud. 69, 440–453. doi: 10.1016/j.ijhcs.2011.03.002

Juan, M., and Prez, D. (2010). Using augmented and virtual reality for the development of acrophobic scenarios. Comparison of the levels of presence and anxiety. Comput. Graph. (Pergamon) 34, 756–766. doi: 10.1016/j.cag.2010.08.001

Juan, M., Carrizo, M., Abad, F., and Giménez, M. (2011a). “Using an augmented reality game to find matching pairs,” in 19th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, WSCG 2011 - In Co-operation with EUROGRAPHICS, Full Papers Proceedings , 59–66.

Juan, M., Furió, D., Alem, L., Ashworth, P., and Cano, J. (2011b). “ARGreenet and BasicGreenet: Two mobile games for learning how to recycle,” in 19th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, WSCG 2011 - In Co-operation with EUROGRAPHICS, Full Papers Proceedings (Plzen), 25–32.

Kasahara, S., and Rekimoto, J. (2014). “JackIn: integrating first-person view with out-of-body vision generation for human-human augmentation,” in ACM International Conference Proceeding Series .

Kellner, F., Bolte, B., Bruder, G., Rautenberg, U., Steinicke, F., Lappe, M., et al. (2012). Geometric calibration of head-mounted displays and its effects on distance estimation. IEEE Trans. Visual. Comput. Graph. 18, 589–596. doi: 10.1109/TVCG.2012.45

Kerber, F., Lessel, P., Mauderer, M., Daiber, F., Oulasvirta, A., and Krüger, A. (2013). “Is autostereoscopy useful for handheld AR?,” in Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia, MUM 2013 .

Kern, D., Stringer, M., Fitzpatrick, G., and Schmidt, A. (2006). “Curball - A prototype tangible game for inter-generational play,” in Proceedings of the Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises, WETICE , 412–417.

Kerr, S., Rice, M., Teo, Y., Wan, M., Cheong, Y., Ng, J., et al. (2011). “Wearable mobile augmented reality: evaluating outdoor user experience,” in Proceedings of VRCAI 2011: ACM SIGGRAPH Conference on Virtual-Reality Continuum and its Applications to Industry , 209–216.

Kim, M. J. (2013). A framework for context immersion in mobile augmented reality. Automat. Construct. 33, 79–85. doi: 10.1016/j.autcon.2012.10.020

King, M., Hale, L., Pekkari, A., Persson, M., Gregorsson, M., and Nilsson, M. (2010). An affordable, computerised, table-based exercise system for stroke survivors. Disabil. Rehabil. Assist. Technol. 5, 288–293. doi: 10.3109/17483101003718161

Kjeldskov, J., Skov, M. B., Nielsen, G. W., Thorup, S., and Vestergaard, M. (2013). Digital urban ambience: mediating context on mobile devices in a city. Pervasive Mobile Comput. 9, 738–749. doi: 10.1016/j.pmcj.2012.05.002

Knörlein, B., Di Luca, M., and Harders, M. (2009). “Influence of visual and haptic delays on stiffness perception in augmented reality,” in Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009 , 49–52.

Ko, S. M., Chang, W. S., and Ji, Y. G. (2013). Usability principles for augmented reality applications in a smartphone environment. Int. J. Hum. Comput. Interact. 29, 501–515. doi: 10.1080/10447318.2012.722466

Kron, A., and Schmidt, G. (2005). “Haptic telepresent control technology applied to disposal of explosive ordnances: principles and experimental results,” in IEEE International Symposium on Industrial Electronics , Vol. IV, 1505–1510.

Kruijff, E., Swan II, J. E., and Feiner, S. (2010). “Perceptual issues in augmented reality revisited,” in Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International Symposium on (Seoul), 3–12.

Kurt, S. (2010). From information to experience: place-based augmented reality games as a model for learning in a globally networked society. Teach. Coll. Rec. 112, 2565–2602.

Langlotz, T., Regenbrecht, H., Zollmann, S., and Schmalstieg, D. (2013). “Audio stickies: visually-guided spatial audio annotations on a mobile augmented reality platform,” in Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, OzCHI 2013 , 545–554.

Lau, M., Hirose, M., Ohgawara, A., Mitani, J., and Igarashi, T. (2012). “Situated modeling: a shape-stamping interface with tangible primitives,” in Proceedings of the 6th International Conference on Tangible, Embedded and Embodied Interaction, TEI 2012 , 275–282.

Leblanc, F., Senagore, A., Ellis, C., Champagne, B., Augestad, K., Neary, P., et al. (2010). Hand-assisted laparoscopic sigmoid colectomy skills acquisition: augmented reality simulator versus human cadaver training models. J. Surg. Educ. 67, 200–204. doi: 10.1016/j.jsurg.2010.06.004

Lee, M., and Billinghurst, M. (2008). “A Wizard of Oz Study for an AR Multimodal Interface,” in ICMI'08: Proceedings of the 10th International Conference on Multimodal Interfaces , 249–256.

Lee, G. A., and Billinghurst, M. (2011). “A user study on the Snap-To-Feature interaction method,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011 , 245–246.

Lee, C., Bonebrake, S., Höllerer, T., and Bowman, D. (2009). “A replication study testing the validity of AR simulation in VR for controlled experiments,” in Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009 , 203–204.

Lee, C., Bonebrake, S., Höllerer, T., and Bowman, D. (2010). “The role of latency in the validity of AR simulation,” in Proceedings - IEEE Virtual Reality , 11–18.

Lee, J., Kim, Y., and Kim, G. J. (2012). “Funneling and saltation effects for tactile interaction with virtual objectsvirtual,” in Conference on Human Factors in Computing Systems - Proceedings , 3141–3148.

Lee, C., Rincon, G., Meyer, G., Höllerer, T., and Bowman, D. (2013a). The effects of visual realism on search tasks in mixed reality simulation. IEEE Trans. Visual. Comput. Graph. 19, 547–556. doi: 10.1109/TVCG.2013.41

Lee, J., Olwal, A., Ishii, H., and Boulanger, C. (2013b). “SpaceTop: integrating 2D and spatial 3D interactions in a see-through desktop environment,” in Conference on Human Factors in Computing Systems - Proceedings , 189–192.

Lee, M., Billinghurst, M., Baek, W., Green, R., and Woo, W. (2013c). A usability study of multimodal input in an augmented reality environment. Virt. Real. 17, 293–305. doi: 10.1007/s10055-013-0230-0

Lee, S., Lee, J., Lee, A., Park, N., Lee, S., Song, S., et al. (2013d). Augmented reality intravenous injection simulator based 3D medical imaging for veterinary medicine. Veter. J. 196, 197–202. doi: 10.1016/j.tvjl.2012.09.015

Lehtinen, V., Nurminen, A., and Oulasvirta, A. (2012). “Integrating spatial sensing to an interactive mobile 3D map,” in IEEE Symposium on 3D User Interfaces 2012, 3DUI 2012 - Proceedings , 11–14.

Leithinger, D., Follmer, S., Olwal, A., Luescher, S., Hogge, A., Lee, J., et al. (2013). “Sublimate: state-changing virtual and physical rendering to augment interaction with shape displays,” in Conference on Human Factors in Computing Systems - Proceedings , 1441–1450.

Li, N., Gu, Y., Chang, L., and Duh, H.-L. (2011). “Influences of AR-supported simulation on learning effectiveness in face-to-face collaborative learning for physics,” in Proceedings of the 2011 11th IEEE International Conference on Advanced Learning Technologies, ICALT 2011 , 320–322.

Liarokapis, F. (2005). “Augmented reality scenarios for guitar learning,” in Theory and Practice of Computer Graphics 2005, TPCG 2005 - Eurographics UK Chapter Proceedings , 163–170.

Lin, T.-J., Duh, H.-L., Li, N., Wang, H.-Y., and Tsai, C.-C. (2013). An investigation of learners' collaborative knowledge construction performances and behavior patterns in an augmented reality simulation system. Comput. Educ. 68, 314–321. doi: 10.1016/j.compedu.2013.05.011

Lindeman, R., Noma, H., and De Barros, P. (2007). “Hear-through and mic-through augmented reality: using bone conduction to display spatialized audio,” in 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR .

Liu, S., Hua, H., and Cheng, D. (2010). A novel prototype for an optical see-through head-mounted display with addressable focus cues. IEEE Trans. Visual. Comput. Graph. 16, 381–393. doi: 10.1109/TVCG.2009.95

Liu, C., Huot, S., Diehl, J., MacKay, W., and Beaudouin-Lafon, M. (2012). “Evaluating the benefits of real-time feedback in mobile Augmented Reality with hand-held devices,” in Conference on Human Factors in Computing Systems - Proceedings , 2973–2976.

Livingston, M. A., and Ai, Z. (2008). “The effect of registration error on tracking distant augmented objects,” in Proceedings - 7th IEEE International Symposium on Mixed and Augmented Reality 2008, ISMAR 2008 , 77–86.

Livingston, M., Zanbaka, C., Edward Swan II, J., and Smallman, H. (2005). “Objective measures for the effectiveness of augmented reality,” in Proceedings - IEEE Virtual Reality , 287–288.

Livingston, M., Ai, Z., Swan II, J., and Smallman, H. (2009a). “Indoor vs. outdoor depth perception for mobile augmented reality,” in Proceedings - IEEE Virtual Reality , 55–62.

Livingston, M. A., Ai, Z., and Decker, J. W. (2009b). “A user study towards understanding stereo perception in head-worn augmented reality displays,” in Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009 , 53–56.

Livingston, M. A., Barrow, J. H., and Sibley, C. M. (2009c). “Quantification of contrast sensitivity and color perception using Head-Worn augmented reality displays,” in Proceedings - IEEE Virtual Reality , 115–122.

Livingston, M. A., Ai, Z., Karsch, K., and Gibson, G. O. (2011). User interface design for military AR applications. Virt. Real. 15, 175–184. doi: 10.1007/s10055-010-0179-1

Livingston, M. A., Dey, A., Sandor, C., and Thomas, B. H. (2013). Pursuit of “X-Ray Vision” for Augmented Reality . New York, NY: Springer.

Livingston, M. A. (2007). “Quantification of visual capabilities using augmented reality displays,” in Proceedings - ISMAR 2006: Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality , 3–12.

Looser, J., Billinghurst, M., Grasset, R., and Cockburn, A. (2007). “An evaluation of virtual lenses for object selection in augmented reality,” in Proceedings - GRAPHITE 2007, 5th International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia , 203–210.

Lu, W., Duh, B.-L., and Feiner, S. (2012). “Subtle cueing for visual search in augmented reality,” in ISMAR 2012 - 11th IEEE International Symposium on Mixed and Augmented Reality 2012, Science and Technology Papers , 161–166.

Luckin, R., and Fraser, D. (2011). Limitless or pointless? An evaluation of augmented reality technology in the school and home. Int. J. Technol. Enhanced Learn. 3, 510–524. doi: 10.1504/IJTEL.2011.042102

Luo, X., Kline, T., Fischer, H., Stubblefield, K., Kenyon, R., and Kamper, D. (2005a). “Integration of augmented reality and assistive devices for post-stroke hand opening rehabilitation,” in Annual International Conference of the IEEE Engineering in Medicine and Biology - Proceedings , Vol. 7, 6855–6858.

Luo, X. c., Kenyon, R., Kline, T., Waldinger, H., and Kamper, D. (2005b). “An augmented reality training environment for post-stroke finger extension rehabilitation,” in Proceedings of the 2005 IEEE 9th International Conference on Rehabilitation Robotics , Vol. 2005, 329–332.

Lv, Z. (2013). “Wearable smartphone: wearable hybrid framework for hand and foot gesture interaction on smartphone,” in Proceedings of the IEEE International Conference on Computer Vision , 436–443.

Magnusson, C., Molina, M., Rassmus-Gröhn, K., and Szymczak, D. (2010). “Pointing for non-visual orientation and navigation,” in NordiCHI 2010: Extending Boundaries - Proceedings of the 6th Nordic Conference on Human-Computer Interaction , 735–738.

Maier, P., Dey, A., Waechter, C. A. L., Sandor, C., Tšnnis, M., and Klinker, G. (2011). “An empiric evaluation of confirmation methods for optical see-through head-mounted display calibration,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality , 267–268.

Markov-Vetter, D., Moll, E., and Staadt, O. (2012). “Evaluation of 3D selection tasks in parabolic flight conditions: pointing task in augmented reality user interfaces,” in Proceedings - VRCAI 2012: 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry , 287–293.

Markovic, M., Dosen, S., Cipriani, C., Popovic, D., and Farina, D. (2014). Stereovision and augmented reality for closed-loop control of grasping in hand prostheses. J. Neural Eng. 11:046001. doi: 10.1088/1741-2560/11/4/046001

Marner, M. R., Irlitti, A., and Thomas, B. H. (2013). “Improving procedural task performance with Augmented Reality annotations,” in 2013 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2013 , 39–48.

Martin-Gutierrez, J. (2011). “Generic user manual for maintenance of mountain bike brakes based on augmented reality,” in Proceedings of the 28th International Symposium on Automation and Robotics in Construction, ISARC 2011 , 1401–1406.

Mercier-Ganady, J., Lotte, F., Loup-Escande, E., Marchal, M., and Lecuyer, A. (2014). “The Mind-Mirror: see your brain in action in your head using EEG and augmented reality,” in Proceedings - IEEE Virtual Reality , 33–38.

Milgram, P., Takemura, H., Utsumi, A., and Kishino, F. (1995). “Augmented reality: a class of displays on the reality-virtuality continuum,” in Telemanipulator and Telepresence Technologies , Vol. 2351 (International Society for Optics and Photonics), 282–293.

Morrison, A., Oulasvirta, A., Peltonen, P., Lemmelä, S., Jacucci, G., Reitmayr, G., et al. (2009). “Like bees around the hive: a comparative study of a mobile augmented reality map,” in Conference on Human Factors in Computing Systems - Proceedings (Boston, MA), 1889–1898.

Mossel, A., Venditti, B., and Kaufmann, H. (2013a). “3Dtouch and homer-s: intuitive manipulation techniques for one-handed handheld augmented reality,” in ACM International Conference Proceeding Series .

Mossel, A., Venditti, B., and Kaufmann, H. (2013b). “Drillsample: precise selection in dense handheld augmented reality environments,” in ACM International Conference Proceeding Series .

Moussa, G., Radwan, E., and Hussain, K. (2012). Augmented reality vehicle system: left-turn maneuver study. Transport. Res. C Emerging Technol. 21, 1–16. doi: 10.1016/j.trc.2011.08.005

Mulloni, A., Wagner, D., and Schmalstieg, D. (2008). “Mobility and social interaction as core gameplay elements in multi-player augmented reality,” in Proceedings - 3rd International Conference on Digital Interactive Media in Entertainment and Arts, DIMEA 2008 , 472–478.

Mulloni, A., Seichter, H., and Schmalstieg, D. (2011a). “Handheld augmented reality indoor navigation with activity-based instructions,” in Mobile HCI 2011 - 13th International Conference on Human-Computer Interaction with Mobile Devices and Services , 211–220.

Mulloni, A., Seichter, H., and Schmalstieg, D. (2011b). “User experiences with augmented reality aided navigation on phones,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011 , 229–230.

Mulloni, A., Ramachandran, M., Reitmayr, G., Wagner, D., Grasset, R., and Diaz, S. (2013). “User friendly SLAM initialization,” in 2013 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2013 , 153–162.

Möller, A., Kranz, M., Huitl, R., Diewald, S., and Roalter, L. (2012). “A mobile indoor navigation system interface adapted to vision-based localization,” in Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia, MUM 2012 .

Möller, A., Kranz, M., Diewald, S., Roalter, L., Huitl, R., Stockinger, T., et al. (2014). “Experimental evaluation of user interfaces for visual indoor navigation,” in Conference on Human Factors in Computing Systems - Proceedings , 3607–3616.

Ng-Thow-Hing, V., Bark, K., Beckwith, L., Tran, C., Bhandari, R., and Sridhar, S. (2013). “User-centered perspectives for automotive augmented reality,” in 2013 IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities, ISMAR-AMH 2013 , 13–22.

Nicolau, S., Garcia, A., Pennec, X., Soler, L., and Ayache, N. (2005). An augmented reality system to guide radio-frequency tumour ablation. Comput. Animat. Virt. Worlds 16, 1–10. doi: 10.1002/cav.52

Nilsson, S., and Johansson, B. (2007). “Fun and usable: augmented Reality instructions in a hospital setting,” in Australasian Computer-Human Interaction Conference, OZCHI'07 , 123–130.

Oda, O., and Feiner, S. (2009). “Interference avoidance in multi-user hand-held augmented reality,” in Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009 , 13–22.

Ofek, E., Iqbal, S. T., and Strauss, K. (2013). “Reducing disruption from subtle information delivery during a conversation: mode and bandwidth investigation,” in Conference on Human Factors in Computing Systems - Proceedings , 3111–3120.

Oh, S., and Byun, Y. (2012). “The design and implementation of augmented reality learning systems,” in Proceedings - 2012 IEEE/ACIS 11th International Conference on Computer and Information Science, ICIS 2012 , 651–654.

Oh, J.-Y., and Hua, H. (2007). “User evaluations on form factors of tangible magic lenses,” in Proceedings - ISMAR 2006: Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality , 23–32.

Olsson, T., and Salo, M. (2011). “Online user survey on current mobile augmented reality applications,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011 , 75–84.

Olsson, T., and Salo, M. (2012). “Narratives of satisfying and unsatisfying experiences of current mobile Augmented Reality applications,” in Conference on Human Factors in Computing Systems - Proceedings , 2779–2788.

Olsson, T., Ihamäki, P., Lagerstam, E., Ventä-Olkkonen, L., and Väänänen-Vainio-Mattila, K. (2009). “User expectations for mobile mixed reality services: an initial user study,” in VTT Symp. (Valtion Teknillinen Tutkimuskeskus) (Helsinki), 177–184.

Olsson, T., Kärkkäinen, T., Lagerstam, E., and Ventä-Olkkonen, L. (2012). User evaluation of mobile augmented reality scenarios. J. Ambient Intell. Smart Environ. 4, 29–47. doi: 10.3233/AIS-2011-0127

Olsson, T., Lagerstam, E., Kärkkäinen, T., and Väänänen-Vainio-Mattila, K. (2013). Expected user experience of mobile augmented reality services: a user study in the context of shopping centres. Pers. Ubiquit. Comput. 17, 287–304. doi: 10.1007/s00779-011-0494-x

Papagiannakis, G., Singh, G., and Magnenat-Thalmann, N. (2008). A survey of mobile and wireless technologies for augmented reality systems. Comput. Anim. Virt. Worlds 19, 3–22. doi: 10.1002/cav.v19:1

Pescarin, S., Pagano, A., Wallergård, M., Hupperetz, W., and Ray, C. (2012). “Archeovirtual 2011: an evaluation approach to virtual museums,” in Proceedings of the 2012 18th International Conference on Virtual Systems and Multimedia, VSMM 2012: Virtual Systems in the Information Society , 25–32.

Petersen, N., and Stricker, D. (2009). “Continuous natural user interface: reducing the gap between real and digital world,” in Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009 , 23–26.

Peterson, S., Axholt, M., Cooper, M., and Ellis, S. (2009). “Visual clutter management in augmented reality: effects of three label separation methods on spatial judgments,” in 3DUI - IEEE Symposium on 3D User Interfaces 2009 - Proceedings , 111–118.

Poelman, R., Akman, O., Lukosch, S., and Jonker, P. (2012). “As if being there: mediated reality for crime scene investigation,” in Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW , 1267–1276.

Porter, S. R., Marner, M. R., Smith, R. T., Zucco, J. E., and Thomas, B. H. (2010). “Validating spatial augmented reality for interactive rapid prototyping,” in 9th IEEE International Symposium on Mixed and Augmented Reality 2010: Science and Technology, ISMAR 2010 - Proceedings , 265–266.

Pucihar, K., Coulton, P., and Alexander, J. (2014). “The use of surrounding visual context in handheld AR: device vs. user perspective rendering,” in Conference on Human Factors in Computing Systems - Proceedings , 197–206.

Pusch, A., Martin, O., and Coquillart, S. (2008). “HEMP - Hand-displacement-based Pseudo-haptics: a study of a force field application,” in 3DUI - IEEE Symposium on 3D User Interfaces 2008 , 59–66.

Pusch, A., Martin, O., and Coquillart, S. (2009). HEMP-hand-displacement-based pseudo-haptics: a study of a force field application and a behavioural analysis. Int. J. Hum. Comput. Stud. 67, 256–268. doi: 10.1016/j.ijhcs.2008.09.015

Rankohi, S., and Waugh, L. (2013). Review and analysis of augmented reality literature for construction industry. Visual. Eng. 1, 1–18. doi: 10.1186/2213-7459-1-9

Rauhala, M., Gunnarsson, A.-S., Henrysson, A., and Ynnerman, A. (2006). “A novel interface to sensor networks using handheld augmented reality,” in ACM International Conference Proceeding Series , Vol. 159, 145–148.

Regenbrecht, H., McGregor, G., Ott, C., Hoermann, S., Schubert, T., Hale, L., et al. (2011). “Out of reach? - A novel AR interface approach for motor rehabilitation,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011 , 219–228.

Regenbrecht, H., Hoermann, S., McGregor, G., Dixon, B., Franz, E., Ott, C., et al. (2012). Visual manipulations for motor rehabilitation. Comput. Graph. (Pergamon) 36, 819–834.

Regenbrecht, H., Hoermann, S., Ott, C., Muller, L., and Franz, E. (2014). Manipulating the experience of reality for rehabilitation applications. Proc. IEEE 102, 170–184. doi: 10.1109/JPROC.2013.2294178

Reif, R., and Günthner, W. A. (2009). Pick-by-vision: augmented reality supported order picking. Vis. Comput. 25, 461–467. doi: 10.1007/s00371-009-0348-y

Ritter, E., Kindelan, T., Michael, C., Pimentel, E., and Bowyer, M. (2007). Concurrent validity of augmented reality metrics applied to the fundamentals of laparoscopic surgery (FLS). Surg. Endosc. Other Intervent. Techniq. 21, 1441–1445. doi: 10.1007/s00464-007-9261-5

Robertson, C., MacIntyre, B., and Walker, B. N. (2007). “An evaluation of graphical context as a means for ameliorating the effects of registration error,” in IEEE Transactions on Visualization and Computer Graphics , Vol. 15, 179–192.

Robertson, C., Maclntyre, B., and Walker, B. (2008). “An evaluation of graphical context when the graphics are outside of the task area,” in Proceedings - 7th IEEE International Symposium on Mixed and Augmented Reality 2008, ISMAR 2008 , 73–76.

Rohs, M., Schöning, J., Raubal, M., Essl, G., and Krüger, A. (2007). “Map navigation with mobile devices: virtual versus physical movement with and without visual context,” in Proceedings of the 9th International Conference on Multimodal Interfaces, ICMI'07 , 146–153.

Rohs, M., Schleicher, R., Schöning, J., Essl, G., Naumann, A., and Krüger, A. (2009a). Impact of item density on the utility of visual context in magic lens interactions. Pers. Ubiquit. Comput. 13, 633–646. doi: 10.1007/s00779-009-0247-2

Rohs, M., Schöning, J., Schleicher, R., Essl, G., Naumann, A., and Krüger, A. (2009b). “Impact of item density on magic lens interactions,” in MobileHCI09 - The 11th International Conference on Human-Computer Interaction with Mobile Devices and Services .

Rohs, M., Oulasvirta, A., and Suomalainen, T. (2011). “Interaction with magic lenses: Real-world validation of a Fitts' law model,” in Conference on Human Factors in Computing Systems - Proceedings , 2725–2728.

Rosenthal, S., Kane, S., Wobbrock, J., and Avrahami, D. (2010). “Augmenting on-screen instructions with micro-projected guides: when it works, and when it fails,” in UbiComp'10 - Proceedings of the 2010 ACM Conference on Ubiquitous Computing , 203–212.

Rusch, M., Schall, M. Jr., Gavin, P., Lee, J., Dawson, J., Vecera, S., et al. (2013). Directing driver attention with augmented reality cues. Transport. Res. Part F Traf. Psychol. Behav. 16, 127–137. doi: 10.1016/j.trf.2012.08.007

Salamin, P., Thalmann, D., and Vexo, F. (2006). “The benefits of third-person perspective in virtual and augmented reality,” in Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST , 27–30.

Salvador-Herranz, G., Pérez-López, D., Ortega, M., Soto, E., Alcañiz, M., and Contero, M. (2013). “Manipulating virtual objects with your hands: a case study on applying desktop Augmented Reality at the primary school,” in Proceedings of the Annual Hawaii International Conference on System Sciences , 31–39.

Sandor, C., Cunningham, A., Dey, A., and Mattila, V.-V. (2010). “An augmented reality X-ray system based on visual saliency,” in 9th IEEE International Symposium on Mixed and Augmented Reality 2010: Science and Technology, ISMAR 2010 - Proceedings , 27–36.

Santos, M. E. C., Chen, A., Terawaki, M., Yamamoto, G., Taketomi, T., Miyazaki, J., et al. (2013). “Augmented reality x-ray interaction in k-12 education: Theory, student perception and teacher evaluation,” in Proceedings - 2013 IEEE 13th International Conference on Advanced Learning Technologies, ICALT 2013 , 141–145.

Schall, G., Zollmann, S., and Reitmayr, G. (2013a). Smart Vidente: advances in mobile augmented reality for interactive visualization of underground infrastructure. Pers. Ubiquit. Comput. 17, 1533–1549. doi: 10.1007/s00779-012-0599-x

Schall, M., Rusch, M., Lee, J., Dawson, J., Thomas, G., Aksan, N., et al. (2013b). Augmented reality cues and elderly driver hazard perception. Hum. Fact. 55, 643–658. doi: 10.1177/0018720812462029

Schinke, T., Henze, N., and Boll, S. (2010). “Visualization of off-screen objects in mobile augmented reality,” in ACM International Conference Proceeding Series , 313–316.

Schoenfelder, R., and Schmalstieg, D. (2008). “Augmented reality for industrial building acceptance,” in Proceedings - IEEE Virtual Reality , 83–90.

Schwerdtfeger, B., and Klinker, G. (2008). “Supporting order picking with augmented reality,” in Proceedings - 7th IEEE International Symposium on Mixed and Augmented Reality 2008, ISMAR 2008 , 91–94.

Schwerdtfeger, B., Reif, R., Günthner, W., Klinker, G., Hamacher, D., Schega, L., et al. (2009). “Pick-by-vision: a first stress test,” in Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009 , 115–124.

Schwerdtfeger, B., Reif, R., Günthner, W. A., and Klinker, G. (2011). Pick-by-vision: there is something to pick at the end of the augmented tunnel. Virt. Real. 15, 213–223. doi: 10.1007/s10055-011-0187-9

Shatte, A., Holdsworth, J., and Lee, I. (2014). Mobile augmented reality based context-aware library management system. Exp. Syst. Appl. 41, 2174–2185. doi: 10.1016/j.eswa.2013.09.016

Singh, G., Swan II, J., Jones, J., and Ellis, S. (2010). “Depth judgment measures and occluding surfaces in near-field augmented reality,” in Proceedings - APGV 2010: Symposium on Applied Perception in Graphics and Visualization , 149–156.

Singh, G., Swan II, J., Jones, J., and Ellis, S. (2012). “Depth judgments by reaching and matching in near-field augmented reality,” in Proceedings - IEEE Virtual Reality , 165–166.

Sodhi, R. B., Benko, H., and Wilson, A. (2012). “LightGuide: projected visualizations for hand movement guidance,” in Conference on Human Factors in Computing Systems - Proceedings , 179–188.

Sodhi, R., Jones, B., Forsyth, D., Bailey, B., and Maciocci, G. (2013). “BeThere: 3D mobile collaboration with spatial input,” in Conference on Human Factors in Computing Systems - Proceedings , 179–188.

Sommerauer, P., and Müller, O. (2014). Augmented reality in informal learning environments: a field experiment in a mathematics exhibition. Comput. Educ. 79, 59–68. doi: 10.1016/j.compedu.2014.07.013

Sukan, M., Feiner, S., Tversky, B., and Energin, S. (2012). “Quick viewpoint switching for manipulating virtual objects in hand-held augmented reality using stored snapshots,” in ISMAR 2012 - 11th IEEE International Symposium on Mixed and Augmented Reality 2012, Science and Technology Papers , 217–226.

Sumadio, D. D., and Rambli, D. R. A. (2010). “Preliminary evaluation on user acceptance of the augmented reality use for education,” in 2010 2nd International Conference on Computer Engineering and Applications, ICCEA 2010 , Vol. 2, 461–465.

Suzuki, K., Garfinkel, S., Critchley, H., and Seth, A. (2013). Multisensory integration across exteroceptive and interoceptive domains modulates self-experience in the rubber-hand illusion. Neuropsychologia 51, 2909–2917. doi: 10.1016/j.neuropsychologia.2013.08.014

Swan, J. E. II., and Gabbard, J. L. (2005). “Survey of user-based experimentation in augmented reality,” in Proceedings of 1st International Conference on Virtual Reality, HCI International 2005 , 1–9.

Sylaiou, S., Mania, K., Karoulis, A., and White, M. (2010). Exploring the relationship between presence and enjoyment in a virtual museum. Int. J. Hum. Comput. Stud. 68, 243–253. doi: 10.1016/j.ijhcs.2009.11.002

Szymczak, D., Rassmus-G?ohn, K., Magnusson, C., and Hedvall, P.-O. (2012). “A real-world study of an audio-tactile tourist guide,” in MobileHCI'12 - Proceedings of the 14th International Conference on Human Computer Interaction with Mobile Devices and Services (San Francisco, CA), 335–344.

Takano, K., Hata, N., and Kansaku, K. (2011). Towards intelligent environments: an augmented reality-brain-machine interface operated with a see-through head-mount display. Front. Neurosci. 5:60. doi: 10.3389/fnins.2011.00060

Tangmanee, K., and Teeravarunyou, S. (2012). “Effects of guided arrows on head-up display towards the vehicle windshield,” in 2012 Southeast Asian Network of Ergonomics Societies Conference: Ergonomics Innovations Leveraging User Experience and Sustainability, SEANES 2012 .

Teber, D., Guven, S., Simpfendörfer, T., Baumhauer, M., Güven, E., Yencilek, F., et al. (2009). Augmented reality: a new tool To improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur. Urol. 56, 332–338. doi: 10.1016/j.eururo.2009.05.017

Thomas, R., John, N., and Delieu, J. (2010). Augmented reality for anatomical education. J. Vis. Commun. Med. 33, 6–15. doi: 10.3109/17453050903557359

Thomas, B. H. (2007). “Evaluation of three input techniques for selection and annotation of physical objects through an augmented reality view,” in Proceedings - ISMAR 2006: Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality , 33–36.

Tillon, A. B., Marchal, I., and Houlier, P. (2011). “Mobile augmented reality in the museum: Can a lace-like technology take you closer to works of art?,” in 2011 IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities, ISMAR-AMH 2011 , 41–47.

Tomioka, M., Ikeda, S., and Sato, K. (2013). “Approximated user-perspective rendering in tablet-based augmented reality,” in 2013 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2013 , 21–28.

Toyama, T., Dengel, A., Suzuki, W., and Kise, K. (2013). “Wearable reading assist system: augmented reality document combining document retrieval and eye tracking,” in Proceedings of the International Conference on Document Analysis and Recognition, ICDAR , 30–34.

Toyama, T., Sonntag, D., Dengel, A., Matsuda, T., Iwamura, M., and Kise, K. (2014a). “A mixed reality head-mounted text translation system using eye gaze input,” in International Conference on Intelligent User Interfaces, Proceedings IUI , 329–334.

Toyama, T., Sonntag, D., Orlosky, J., and Kiyokawa, K. (2014b). “A natural interface for multi-focal plane head mounted displays using 3D gaze,” in Proceedings of the Workshop on Advanced Visual Interfaces AVI , 25–32.

Tsuda, T., Yamamoto, H., Kameda, Y., and Ohta, Y. (2005). “Visualization methods for outdoor see-through vision,” in ACM International Conference Proceeding Series , Vol. 157, 62–69.

Tumler, J., Doil, F., Mecke, R., Paul, G., Schenk, M., Pfister, E., et al. (2008). “Mobile augmented reality in industrial applications: approaches for solution of user-related issues,” in Proceedings - 7th IEEE International Symposium on Mixed and Augmented Reality 2008, ISMAR 2008 , 87–90.

Tönnis, M., and Klinker, G. (2007). “Effective control of a car driver's attention for visual and acoustic guidance towards the direction of imminent dangers,” in Proceedings - ISMAR 2006: Fifth IEEE and ACM International Symposium on Mixed and Augmented Reality , 13–22.

Tönnis, M., Sandor, C., Klinker, G., Lange, C., and Bubb, H. (2005). “Experimental evaluation of an augmented reality visualization for directing a car driver's attention,” in Proceedings - Fourth IEEE and ACM International Symposium on Symposium on Mixed and Augmented Reality, ISMAR 2005 , Vol. 2005, 56–59.

Vazquez-Alvarez, Y., Oakley, I., and Brewster, S. (2012). Auditory display design for exploration in mobile audio-augmented reality. Pers. Ubiquit. Comput. 16, 987–999. doi: 10.1007/s00779-011-0459-0

Veas, E., Mendez, E., Feiner, S., and Schmalstieg, D. (2011). “Directing attention and influencing memory with visual saliency modulation,” in Conference on Human Factors in Computing Systems - Proceedings , 1471–1480.

Veas, E., Grasset, R., Kruijff, E., and Schmalstieg, D. (2012). Extended overview techniques for outdoor augmented reality. IEEE Trans. Visual. Comput. Graphics 18, 565–572. doi: 10.1109/TVCG.2012.44

Vignais, N., Miezal, M., Bleser, G., Mura, K., Gorecky, D., and Marin, F. (2013). Innovative system for real-time ergonomic feedback in industrial manufacturing. Appl. Ergon. 44, 566–574. doi: 10.1016/j.apergo.2012.11.008

Voida, S., Podlaseck, M., Kjeldsen, R., and Pinhanez, C. (2005). “A study on the manipulation of 2D objects in a projector/camera-based augmented reality environment,” in CHI 2005: Technology, Safety, Community: Conference Proceedings - Conference on Human Factors in Computing Systems , 611–620.

Wacker, F., Vogt, S., Khamene, A., Jesberger, J., Nour, S., Elgort, D., et al. (2006). An augmented reality system for MR image-guided needle biopsy: initial results in a swine model. Radiology 238, 497–504. doi: 10.1148/radiol.2382041441

Wagner, D., Billinghurst, M., and Schmalstieg, D. (2006). “How real should virtual characters be?,” in International Conference on Advances in Computer Entertainment Technology 2006 .

Wang, X., and Dunston, P. (2011). Comparative effectiveness of mixed reality-based virtual environments in collaborative design. IEEE Trans. Syst. Man Cybernet. C Appl. Rev. 41, 284–296. doi: 10.1109/TSMCC.2010.2093573

Wang, X., Kim, M. J., Love, P. E., and Kang, S.-C. (2013). Augmented reality in built environment: classification and implications for future research. Autom. Construct. 32, 1–13. doi: 10.1016/j.autcon.2012.11.021

Weichel, C., Lau, M., Kim, D., Villar, N., and Gellersen, H. (2014). “MixFab: a mixed-reality environment for personal fabrication,” in Conference on Human Factors in Computing Systems - Proceedings , 3855–3864.

Weing, M., Schaub, F., Röhlig, A., Könings, B., Rogers, K., Rukzio, E., et al. (2013). “P.I.A.N.O.: enhancing instrument learning via interactive projected augmentation,” in UbiComp 2013 Adjunct - Adjunct Publication of the 2013 ACM Conference on Ubiquitous Computing (Zurich), 75–78.

White, S., Lister, L., and Feiner, S. (2007). “Visual hints for tangible gestures in augmented reality,” in 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR .

White, S., Feng, D., and Feiner, S. (2009). “Interaction and presentation techniques for shake menus in tangible augmented reality,” in Science and Technology Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality, ISMAR 2009 , 39–48.

Wilson, K., Doswell, J., Fashola, O., Debeatham, W., Darko, N., Walker, T., et al. (2013). Using augmented reality as a clinical support tool to assist combat medics in the treatment of tension pneumothoraces. Milit. Med. 178, 981–985. doi: 10.7205/MILMED-D-13-00074

Wither, J., and Höllerer, T. (2005). “Pictorial depth cues for outdoor augmented reality,” in Proceedings - International Symposium on Wearable Computers, ISWC , Vol. 2005, 92–99.

Wither, J., DiVerdi, S., and Höllerer, T. (2007). “Evaluating display types for AR selection and annotation,” in 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR .

Wither, J., Allen, R., Samanta, V., Hemanus, J., Tsai, Y.-T., Azuma, R., et al. (2010). “The Westwood experience: connecting story to locations via mixed reality,” in 9th IEEE International Symposium on Mixed and Augmented Reality 2010: Arts, Media, and Humanities, ISMAR-AMH 2010 - Proceedings , 39–46.

Wither, J., Tsai, Y.-T., and Azuma, R. (2011). Indirect augmented reality. Comput. Graph. (Pergamon) 35, 810–822. doi: 10.1016/j.cag.2011.04.010

Wojciechowski, R., and Cellary, W. (2013). Evaluation of learners' attitude toward learning in ARIES augmented reality environments. Comput. Educ. 68, 570–585. doi: 10.1016/j.compedu.2013.02.014

Wrzesien, M., Bretón-López, J., Botella, C., Burkhardt, J.-M., Alcañiz, M., Pérez-Ara, M., et al. (2013). How technology influences the therapeutic process: evaluation of the patient-therapist relationship in augmented reality exposure therapy and in vivo exposure therapy. Behav. Cogn. Psychother. 41, 505–509. doi: 10.1017/S1352465813000088

Xu, Y., Gandy, M., Deen, S., Schrank, B., Spreen, K., Gorbsky, M., et al. (2008). “BragFish: exploring physical and social interaction in co-located handheld augmented reality games,” in Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology, ACE 2008 , 276–283.

Xu, Y., Barba, E., Radu, I., Gandy, M., and MacIntyre, B. (2011). “Chores are fun: understanding social play in board games for digital tabletop game design,” in Proceedings of DiGRA 2011 Conference: Think Design Play (Urtecht).

Yamabe, T., and Nakajima, T. (2013). Playful training with augmented reality games: case studies towards reality-oriented system design. Multimedia Tools Appl. 62, 259–286. doi: 10.1007/s11042-011-0979-7

Yeh, K.-C., Tsai, M.-H., and Kang, S.-C. (2012). On-site building information retrieval by using projection-based augmented reality. J. Comput. Civil Eng. 26, 342–355. doi: 10.1061/(ASCE)CP.1943-5487.0000156

Yoo, H.-N., Chung, E., and Lee, B.-H. (2013). The effects of augmented reality-based otago exercise on balance, gait, and falls efficacy of elderly women. J. Phys. Ther. Sci. 25, 797–801. doi: 10.1589/jpts.25.797

Yuan, M. L., Ong, S. K., and Nee, A. Y. C. (2008). Augmented reality for assembly guidance using a virtual interactive tool. Int. J. Product. Res. 46, 1745–1767. doi: 10.1080/00207540600972935

Yudkowsky, R., Luciano, C., Banerjee, P., Schwartz, A., Alaraj, A., Lemole, G., et al. (2013). Practice on an augmented reality/haptic simulator and library of virtual brains improves residents' ability to perform a ventriculostomy. Simul. Healthcare 8, 25–31. doi: 10.1097/SIH.0b013e3182662c69

Zhang, R., Nordman, A., Walker, J., and Kuhl, S. A. (2012). Minification affects verbal- and action-based distance judgments differently in head-mounted displays. ACM Trans. Appl. Percept. 9:14. doi: 10.1145/2325722.2325727

Zhang, J., Sung, Y.-T., Hou, H.-T., and Chang, K.-E. (2014). The development and evaluation of an augmented reality-based armillary sphere for astronomical observation instruction. Comput. Educ. 73, 178–188. doi: 10.1016/j.compedu.2014.01.003

Zhou, Z., Cheok, A., Qiu, Y., and Yang, X. (2007). The role of 3-D sound in human reaction and performance in augmented reality environments. IEEE Trans. Syst. Man Cybern. A Syst. Hum. 37, 262–272. doi: 10.1109/TSMCA.2006.886376

Keywords: augmented reality, systematic review, user studies, usability, experimentation, classifications

Citation: Dey A, Billinghurst M, Lindeman RW and Swan JE II (2018) A Systematic Review of 10 Years of Augmented Reality Usability Studies: 2005 to 2014. Front. Robot. AI 5:37. doi: 10.3389/frobt.2018.00037

Received: 19 December 2017; Accepted: 19 March 2018; Published: 17 April 2018.

Reviewed by:

Copyright © 2018 Dey, Billinghurst, Lindeman and Swan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Arindam Dey, [email protected]

UF Law Scholarship Repository

Browse journals and peer-reviewed series, florida entertainment and sports law review, florida journal of international law.

ISSN 1556-2670

Florida Law Review

ISSN 1045-4241

Florida Tax Review

ISSN 2476-1699

Journal of Technology Law & Policy

ISSN 1087-6995

University of Florida Journal of Law & Public Policy

ISSN 1942-8626

Advanced Search

  • Notify me via email or RSS
  • Collections
  • Disciplines

Author Corner

  • Submit Research
  • Repository Home
  • Faculty Profiles

University of Florida

Home | About | FAQ | My Account | Accessibility Statement

Privacy Copyright

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • My Bibliography
  • Collections
  • Citation manager

Save citation to file

Email citation, add to collections.

  • Create a new collection
  • Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

  • Search in PubMed
  • Search in NLM Catalog
  • Add to Search

Effectiveness of accelerated reader on children's reading outcomes: A meta-analytic review

Affiliations.

  • 1 University of Kentucky, Lexington, Kentucky, USA.
  • 2 University of Wisconsin-Madison, Madison, Wisconsin, USA.
  • PMID: 36401356
  • DOI: 10.1002/dys.1730

Accelerated reader (AR) is a computerized reading program commonly used in schools. The program aims to enhance students' reading achievement and encourage students to read more through goal setting and frequent reading practice. A meta-analytic review of the AR was conducted to analyse its effectiveness as an evidence-based intervention for improving student reading achievement, attitude, and motivation. This study investigated potential moderating variables, including publication type, participant, and study characteristics that impact student reading outcomes. A total of 44 studies from peer-reviewed journal articles and dissertations met the inclusion criteria. Participants included 16,653 students enrolled in elementary, middle, and high school. Hedges' g effect sizes measures suggest pretest-posttest one-group AR studies have moderate effects (g = 0.541) while comparison group AR studies have marginal effects (g = 0.278). A meta-regression model of six potential categorical moderators of comparison group studies indicted no significant moderators. Implications and the need for further research regarding evidence-based and culturally appropriate reading interventions are discussed.

Keywords: accelerated reader; evidence-based intervention; meta-analysis; reading.

© 2022 John Wiley & Sons Ltd.

PubMed Disclaimer

Similar articles

  • Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
  • Physical activity and academic achievement: an analysis of potential student- and school-level moderators. Behringer HK, Saksvig ER, Boedeker PJ, Elish PN, Kay CM, Calvert HG, Meyer AM, Gazmararian JA. Behringer HK, et al. Int J Behav Nutr Phys Act. 2022 Aug 30;19(1):110. doi: 10.1186/s12966-022-01348-3. Int J Behav Nutr Phys Act. 2022. PMID: 36042515 Free PMC article.
  • Effects of a Comprehensive School Health Program on Elementary Student Academic Achievement. Centeio EE, Somers C, Moore EWG, Kulik N, Garn A, McCaughtry N. Centeio EE, et al. J Sch Health. 2021 Mar;91(3):239-249. doi: 10.1111/josh.12994. Epub 2021 Jan 6. J Sch Health. 2021. PMID: 33404083
  • Effect of classroom-based physical activity interventions on academic and physical activity outcomes: a systematic review and meta-analysis. Watson A, Timperio A, Brown H, Best K, Hesketh KD. Watson A, et al. Int J Behav Nutr Phys Act. 2017 Aug 25;14(1):114. doi: 10.1186/s12966-017-0569-9. Int J Behav Nutr Phys Act. 2017. PMID: 28841890 Free PMC article. Review.
  • The WHO Health Promoting School framework for improving the health and well-being of students and their academic achievement. Langford R, Bonell CP, Jones HE, Pouliou T, Murphy SM, Waters E, Komro KA, Gibbs LF, Magnus D, Campbell R. Langford R, et al. Cochrane Database Syst Rev. 2014 Apr 16;(4):CD008958. doi: 10.1002/14651858.CD008958.pub2. Cochrane Database Syst Rev. 2014. PMID: 24737131 Review.
  • Alinaghi, N., & Reed, W. R. (2018). Meta-analysis and publication bias: How well does the FAT-PET-PEESE procedure work? Research Synthesis Methods, 9(2), 285-311. https://doi.org/10.1002/jrsm.1298
  • *Barton, J. O. (2000). A comparison of the effect of basal reading with Accelerated Reader to basal reading without Accelerated Reader on fifth-grade reading comprehension achievement scores (9984524) [Doctoral dissertation, University of Mississippi]. ProQuest Dissertations and Theses.
  • Barton-Arwood, S. M., Wehby, J. H., & Falk, K. B. (2005). Reading instruction for elementary-age students with emotional and behavioral disorders: Academic and behavioral outcomes. Exceptional Children, 72(1), 7-27. https://doi.org/10.1177/001440290507200101
  • Biggers, D. (2001). The argument against accelerated reader. Journal of Adolescent & Adult Literacy, 45, 72-75.
  • *Boone, L. S. (2017). Agony or ecstasy: A mixed methods study of the Accelerated Reader program and students' attitudes toward reading [Doctoral dissertation, University of Tennessee]. Retrieved from https://trace.tennessee.edu/utk_graddiss/4736

Publication types

  • Search in MeSH

Related information

Linkout - more resources, full text sources, research materials.

  • NCI CPTC Antibody Characterization Program

Miscellaneous

  • NCI CPTAC Assay Portal

full text provider logo

  • Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

About the Journal

Focus and scope.

UFJUR is published each year in the fall. UF JUR gives scholars a voice for their research in the arts and humanities, social, behavioral, education, STEM and medicine. Students submit their work with faculty/mentor approval.

Peer Review Process

Each paper goes through a double blind peer-review (UF) process with experts in the discipline of the article topic.  If the article is selected, the author will be asked to submit a revised version of the work before being accepted for publication. For this reason, it is important that authors provide up-to-date contact information.

Publication Frequency

This journal is published in the fall. Submissions open on Februrary 15th and are due on April 1st of any given year.

Open Access Policy

This journal provides immediate open access to its content on the principle that making research freely available to the public supports a greater global exchange of knowledge.

http://creativecommons.org/licenses/by-nc-nd/4.0

  UF JUR  is published by the Center for Undergraduate Research .

  • University of Florida

Journal History

UF JUR  publishes one issue in the fall.  Current undergraduate students from all majors who work with a faculty mentor are welcome to submit their article for publication.

ISSN 2638-0668

UF Center for Undergraduate Research

Published by the UF Center for Undergraduate Research with support from the LibraryPress@UF . | The Florida OJ service is provided through the  FLVC Library Services and the George A. Smathers Libraries at the University of Florida . |  FLVC Privacy Policy .

E-ISSN 2638-0668

More information about the publishing system, Platform and Workflow by OJS/PKP.

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

  • Advanced Search
  • Journal List
  • Front Psychol

The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature

Pietro cipresso.

1 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy

2 Department of Psychology, Catholic University of the Sacred Heart, Milan, Italy

Irene Alice Chicchi Giglioli

3 Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, Valencia, Spain

Mariano Alcañiz Raya

Giuseppe riva, associated data.

The recent appearance of low cost virtual reality (VR) technologies – like the Oculus Rift, the HTC Vive and the Sony PlayStation VR – and Mixed Reality Interfaces (MRITF) – like the Hololens – is attracting the attention of users and researchers suggesting it may be the next largest stepping stone in technological innovation. However, the history of VR technology is longer than it may seem: the concept of VR was formulated in the 1960s and the first commercial VR tools appeared in the late 1980s. For this reason, during the last 20 years, 100s of researchers explored the processes, effects, and applications of this technology producing 1000s of scientific papers. What is the outcome of this significant research work? This paper wants to provide an answer to this question by exploring, using advanced scientometric techniques, the existing research corpus in the field. We collected all the existent articles about VR in the Web of Science Core Collection scientific database, and the resultant dataset contained 21,667 records for VR and 9,944 for augmented reality (AR). The bibliographic record contained various fields, such as author, title, abstract, country, and all the references (needed for the citation analysis). The network and cluster analysis of the literature showed a composite panorama characterized by changes and evolutions over the time. Indeed, whether until 5 years ago, the main publication media on VR concerned both conference proceeding and journals, more recently journals constitute the main medium of communication. Similarly, if at first computer science was the leading research field, nowadays clinical areas have increased, as well as the number of countries involved in VR research. The present work discusses the evolution and changes over the time of the use of VR in the main areas of application with an emphasis on the future expected VR’s capacities, increases and challenges. We conclude considering the disruptive contribution that VR/AR/MRITF will be able to get in scientific fields, as well in human communication and interaction, as already happened with the advent of mobile phones by increasing the use and the development of scientific applications (e.g., in clinical areas) and by modifying the social communication and interaction among people.

Introduction

In the last 5 years, virtual reality (VR) and augmented reality (AR) have attracted the interest of investors and the general public, especially after Mark Zuckerberg bought Oculus for two billion dollars ( Luckerson, 2014 ; Castelvecchi, 2016 ). Currently, many other companies, such as Sony, Samsung, HTC, and Google are making huge investments in VR and AR ( Korolov, 2014 ; Ebert, 2015 ; Castelvecchi, 2016 ). However, if VR has been used in research for more than 25 years, and now there are 1000s of papers and many researchers in the field, comprising a strong, interdisciplinary community, AR has a more recent application history ( Burdea and Coiffet, 2003 ; Kim, 2005 ; Bohil et al., 2011 ; Cipresso and Serino, 2014 ; Wexelblat, 2014 ). The study of VR was initiated in the computer graphics field and has been extended to several disciplines ( Sutherland, 1965 , 1968 ; Mazuryk and Gervautz, 1996 ; Choi et al., 2015 ). Currently, videogames supported by VR tools are more popular than the past, and they represent valuables, work-related tools for neuroscientists, psychologists, biologists, and other researchers as well. Indeed, for example, one of the main research purposes lies from navigation studies that include complex experiments that could be done in a laboratory by using VR, whereas, without VR, the researchers would have to go directly into the field, possibly with limited use of intervention. The importance of navigation studies for the functional understanding of human memory in dementia has been a topic of significant interest for a long time, and, in 2014, the Nobel Prize in “Physiology or Medicine” was awarded to John M. O’Keefe, May-Britt Moser, and Edvard I. Moser for their discoveries of nerve cells in the brain that enable a sense of place and navigation. Journals and magazines have extended this knowledge by writing about “the brain GPS,” which gives a clear idea of the mechanism. A huge number of studies have been conducted in clinical settings by using VR ( Bohil et al., 2011 ; Serino et al., 2014 ), and Nobel Prize winner, Edvard I. Moser commented about the use of VR ( Minderer et al., 2016 ), highlighting its importance for research and clinical practice. Moreover, the availability of free tools for VR experimental and computational use has made it easy to access any field ( Riva et al., 2011 ; Cipresso, 2015 ; Brown and Green, 2016 ; Cipresso et al., 2016 ).

Augmented reality is a more recent technology than VR and shows an interdisciplinary application framework, in which, nowadays, education and learning seem to be the most field of research. Indeed, AR allows supporting learning, for example increasing-on content understanding and memory preservation, as well as on learning motivation. However, if VR benefits from clear and more definite fields of application and research areas, AR is still emerging in the scientific scenarios.

In this article, we present a systematic and computational analysis of the emerging interdisciplinary VR and AR fields in terms of various co-citation networks in order to explore the evolution of the intellectual structure of this knowledge domain over time.

Virtual Reality Concepts and Features

The concept of VR could be traced at the mid of 1960 when Ivan Sutherland in a pivotal manuscript attempted to describe VR as a window through which a user perceives the virtual world as if looked, felt, sounded real and in which the user could act realistically ( Sutherland, 1965 ).

Since that time and in accordance with the application area, several definitions have been formulated: for example, Fuchs and Bishop (1992) defined VR as “real-time interactive graphics with 3D models, combined with a display technology that gives the user the immersion in the model world and direct manipulation” ( Fuchs and Bishop, 1992 ); Gigante (1993) described VR as “The illusion of participation in a synthetic environment rather than external observation of such an environment. VR relies on a 3D, stereoscopic head-tracker displays, hand/body tracking and binaural sound. VR is an immersive, multi-sensory experience” ( Gigante, 1993 ); and “Virtual reality refers to immersive, interactive, multi-sensory, viewer-centered, 3D computer generated environments and the combination of technologies required building environments” ( Cruz-Neira, 1993 ).

As we can notice, these definitions, although different, highlight three common features of VR systems: immersion, perception to be present in an environment, and interaction with that environment ( Biocca, 1997 ; Lombard and Ditton, 1997 ; Loomis et al., 1999 ; Heeter, 2000 ; Biocca et al., 2001 ; Bailenson et al., 2006 ; Skalski and Tamborini, 2007 ; Andersen and Thorpe, 2009 ; Slater, 2009 ; Sundar et al., 2010 ). Specifically, immersion concerns the amount of senses stimulated, interactions, and the reality’s similarity of the stimuli used to simulate environments. This feature can depend on the properties of the technological system used to isolate user from reality ( Slater, 2009 ).

Higher or lower degrees of immersion can depend by three types of VR systems provided to the user:

  • simple • Non-immersive systems are the simplest and cheapest type of VR applications that use desktops to reproduce images of the world.
  • simple • Immersive systems provide a complete simulated experience due to the support of several sensory outputs devices such as head mounted displays (HMDs) for enhancing the stereoscopic view of the environment through the movement of the user’s head, as well as audio and haptic devices.
  • simple • Semi-immersive systems such as Fish Tank VR are between the two above. They provide a stereo image of a three dimensional (3D) scene viewed on a monitor using a perspective projection coupled to the head position of the observer ( Ware et al., 1993 ). Higher technological immersive systems have showed a closest experience to reality, giving to the user the illusion of technological non-mediation and feeling him or her of “being in” or present in the virtual environment ( Lombard and Ditton, 1997 ). Furthermore, higher immersive systems, than the other two systems, can give the possibility to add several sensory outputs allowing that the interaction and actions were perceived as real ( Loomis et al., 1999 ; Heeter, 2000 ; Biocca et al., 2001 ).

Finally, the user’s VR experience could be disclosed by measuring presence, realism, and reality’s levels. Presence is a complex psychological feeling of “being there” in VR that involves the sensation and perception of physical presence, as well as the possibility to interact and react as if the user was in the real world ( Heeter, 1992 ). Similarly, the realism’s level corresponds to the degree of expectation that the user has about of the stimuli and experience ( Baños et al., 2000 , 2009 ). If the presented stimuli are similar to reality, VR user’s expectation will be congruent with reality expectation, enhancing VR experience. In the same way, higher is the degree of reality in interaction with the virtual stimuli, higher would be the level of realism of the user’s behaviors ( Baños et al., 2000 , 2009 ).

From Virtual to Augmented Reality

Looking chronologically on VR and AR developments, we can trace the first 3D immersive simulator in 1962, when Morton Heilig created Sensorama, a simulated experience of a motorcycle running through Brooklyn characterized by several sensory impressions, such as audio, olfactory, and haptic stimuli, including also wind to provide a realist experience ( Heilig, 1962 ). In the same years, Ivan Sutherland developed The Ultimate Display that, more than sound, smell, and haptic feedback, included interactive graphics that Sensorama didn’t provide. Furthermore, Philco developed the first HMD that together with The Sword of Damocles of Sutherland was able to update the virtual images by tracking user’s head position and orientation ( Sutherland, 1965 ). In the 70s, the University of North Carolina realized GROPE, the first system of force-feedback and Myron Krueger created VIDEOPLACE an Artificial Reality in which the users’ body figures were captured by cameras and projected on a screen ( Krueger et al., 1985 ). In this way two or more users could interact in the 2D-virtual space. In 1982, the US’ Air Force created the first flight simulator [Visually Coupled Airbone System Simulator (VCASS)] in which the pilot through an HMD could control the pathway and the targets. Generally, the 80’s were the years in which the first commercial devices began to emerge: for example, in 1985 the VPL company commercialized the DataGlove, glove sensors’ equipped able to measure the flexion of fingers, orientation and position, and identify hand gestures. Another example is the Eyephone, created in 1988 by the VPL Company, an HMD system for completely immerging the user in a virtual world. At the end of 80’s, Fake Space Labs created a Binocular-Omni-Orientational Monitor (BOOM), a complex system composed by a stereoscopic-displaying device, providing a moving and broad virtual environment, and a mechanical arm tracking. Furthermore, BOOM offered a more stable image and giving more quickly responses to movements than the HMD devices. Thanks to BOOM and DataGlove, the NASA Ames Research Center developed the Virtual Wind Tunnel in order to research and manipulate airflow in a virtual airplane or space ship. In 1992, the Electronic Visualization Laboratory of the University of Illinois created the CAVE Automatic Virtual Environment, an immersive VR system composed by projectors directed on three or more walls of a room.

More recently, many videogames companies have improved the development and quality of VR devices, like Oculus Rift, or HTC Vive that provide a wider field of view and lower latency. In addition, the actual HMD’s devices can be now combined with other tracker system as eye-tracking systems (FOVE), and motion and orientation sensors (e.g., Razer Hydra, Oculus Touch, or HTC Vive).

Simultaneously, at the beginning of 90’, the Boing Corporation created the first prototype of AR system for showing to employees how set up a wiring tool ( Carmigniani et al., 2011 ). At the same time, Rosenberg and Feiner developed an AR fixture for maintenance assistance, showing that the operator performance enhanced by added virtual information on the fixture to repair ( Rosenberg, 1993 ). In 1993 Loomis and colleagues produced an AR GPS-based system for helping the blind in the assisted navigation through adding spatial audio information ( Loomis et al., 1998 ). Always in the 1993 Julie Martin developed “Dancing in Cyberspace,” an AR theater in which actors interacted with virtual object in real time ( Cathy, 2011 ). Few years later, Feiner et al. (1997) developed the first Mobile AR System (MARS) able to add virtual information about touristic buildings ( Feiner et al., 1997 ). Since then, several applications have been developed: in Thomas et al. (2000) , created ARQuake, a mobile AR video game; in 2008 was created Wikitude that through the mobile camera, internet, and GPS could add information about the user’s environments ( Perry, 2008 ). In 2009 others AR applications, like AR Toolkit and SiteLens have been developed in order to add virtual information to the physical user’s surroundings. In 2011, Total Immersion developed D’Fusion, and AR system for designing projects ( Maurugeon, 2011 ). Finally, in 2013 and 2015, Google developed Google Glass and Google HoloLens, and their usability have begun to test in several field of application.

Virtual Reality Technologies

Technologically, the devices used in the virtual environments play an important role in the creation of successful virtual experiences. According to the literature, can be distinguished input and output devices ( Burdea et al., 1996 ; Burdea and Coiffet, 2003 ). Input devices are the ones that allow the user to communicate with the virtual environment, which can range from a simple joystick or keyboard to a glove allowing capturing finger movements or a tracker able to capture postures. More in detail, keyboard, mouse, trackball, and joystick represent the desktop input devices easy to use, which allow the user to launch continuous and discrete commands or movements to the environment. Other input devices can be represented by tracking devices as bend-sensing gloves that capture hand movements, postures and gestures, or pinch gloves that detect the fingers movements, and trackers able to follow the user’s movements in the physical world and translate them in the virtual environment.

On the contrary, the output devices allow the user to see, hear, smell, or touch everything that happens in the virtual environment. As mentioned above, among the visual devices can be found a wide range of possibilities, from the simplest or least immersive (monitor of a computer) to the most immersive one such as VR glasses or helmets or HMD or CAVE systems.

Furthermore, auditory, speakers, as well as haptic output devices are able to stimulate body senses providing a more real virtual experience. For example, haptic devices can stimulate the touch feeling and force models in the user.

Virtual Reality Applications

Since its appearance, VR has been used in different fields, as for gaming ( Zyda, 2005 ; Meldrum et al., 2012 ), military training ( Alexander et al., 2017 ), architectural design ( Song et al., 2017 ), education ( Englund et al., 2017 ), learning and social skills training ( Schmidt et al., 2017 ), simulations of surgical procedures ( Gallagher et al., 2005 ), assistance to the elderly or psychological treatments are other fields in which VR is bursting strongly ( Freeman et al., 2017 ; Neri et al., 2017 ). A recent and extensive review of Slater and Sanchez-Vives (2016) reported the main VR application evidences, including weakness and advantages, in several research areas, such as science, education, training, physical training, as well as social phenomena, moral behaviors, and could be used in other fields, like travel, meetings, collaboration, industry, news, and entertainment. Furthermore, another review published this year by Freeman et al. (2017) focused on VR in mental health, showing the efficacy of VR in assessing and treating different psychological disorders as anxiety, schizophrenia, depression, and eating disorders.

There are many possibilities that allow the use of VR as a stimulus, replacing real stimuli, recreating experiences, which in the real world would be impossible, with a high realism. This is why VR is widely used in research on new ways of applying psychological treatment or training, for example, to problems arising from phobias (agoraphobia, phobia to fly, etc.) ( Botella et al., 2017 ). Or, simply, it is used like improvement of the traditional systems of motor rehabilitation ( Llorens et al., 2014 ; Borrego et al., 2016 ), developing games that ameliorate the tasks. More in detail, in psychological treatment, Virtual Reality Exposure Therapy (VRET) has showed its efficacy, allowing to patients to gradually face fear stimuli or stressed situations in a safe environment where the psychological and physiological reactions can be controlled by the therapist ( Botella et al., 2017 ).

Augmented Reality Concept

Milgram and Kishino (1994) , conceptualized the Virtual-Reality Continuum that takes into consideration four systems: real environment, augmented reality (AR), augmented virtuality, and virtual environment. AR can be defined a newer technological system in which virtual objects are added to the real world in real-time during the user’s experience. Per Azuma et al. (2001) an AR system should: (1) combine real and virtual objects in a real environment; (2) run interactively and in real-time; (3) register real and virtual objects with each other. Furthermore, even if the AR experiences could seem different from VRs, the quality of AR experience could be considered similarly. Indeed, like in VR, feeling of presence, level of realism, and the degree of reality represent the main features that can be considered the indicators of the quality of AR experiences. Higher the experience is perceived as realistic, and there is congruence between the user’s expectation and the interaction inside the AR environments, higher would be the perception of “being there” physically, and at cognitive and emotional level. The feeling of presence, both in AR and VR environments, is important in acting behaviors like the real ones ( Botella et al., 2005 ; Juan et al., 2005 ; Bretón-López et al., 2010 ; Wrzesien et al., 2013 ).

Augmented Reality Technologies

Technologically, the AR systems, however various, present three common components, such as a geospatial datum for the virtual object, like a visual marker, a surface to project virtual elements to the user, and an adequate processing power for graphics, animation, and merging of images, like a pc and a monitor ( Carmigniani et al., 2011 ). To run, an AR system must also include a camera able to track the user movement for merging the virtual objects, and a visual display, like glasses through that the user can see the virtual objects overlaying to the physical world. To date, two-display systems exist, a video see-through (VST) and an optical see-though (OST) AR systems ( Botella et al., 2005 ; Juan et al., 2005 , 2007 ). The first one, disclosures virtual objects to the user by capturing the real objects/scenes with a camera and overlaying virtual objects, projecting them on a video or a monitor, while the second one, merges the virtual object on a transparent surface, like glasses, through the user see the added elements. The main difference between the two systems is the latency: an OST system could require more time to display the virtual objects than a VST system, generating a time lag between user’s action and performance and the detection of them by the system.

Augmented Reality Applications

Although AR is a more recent technology than VR, it has been investigated and used in several research areas such as architecture ( Lin and Hsu, 2017 ), maintenance ( Schwald and De Laval, 2003 ), entertainment ( Ozbek et al., 2004 ), education ( Nincarean et al., 2013 ; Bacca et al., 2014 ; Akçayır and Akçayır, 2017 ), medicine ( De Buck et al., 2005 ), and psychological treatments ( Juan et al., 2005 ; Botella et al., 2005 , 2010 ; Bretón-López et al., 2010 ; Wrzesien et al., 2011a , b , 2013 ; see the review Chicchi Giglioli et al., 2015 ). More in detail, in education several AR applications have been developed in the last few years showing the positive effects of this technology in supporting learning, such as an increased-on content understanding and memory preservation, as well as on learning motivation ( Radu, 2012 , 2014 ). For example, Ibáñez et al. (2014) developed a AR application on electromagnetism concepts’ learning, in which students could use AR batteries, magnets, cables on real superficies, and the system gave a real-time feedback to students about the correctness of the performance, improving in this way the academic success and motivation ( Di Serio et al., 2013 ). Deeply, AR system allows the possibility to learn visualizing and acting on composite phenomena that traditionally students study theoretically, without the possibility to see and test in real world ( Chien et al., 2010 ; Chen et al., 2011 ).

As well in psychological health, the number of research about AR is increasing, showing its efficacy above all in the treatment of psychological disorder (see the reviews Baus and Bouchard, 2014 ; Chicchi Giglioli et al., 2015 ). For example, in the treatment of anxiety disorders, like phobias, AR exposure therapy (ARET) showed its efficacy in one-session treatment, maintaining the positive impact in a follow-up at 1 or 3 month after. As VRET, ARET provides a safety and an ecological environment where any kind of stimulus is possible, allowing to keep control over the situation experienced by the patients, gradually generating situations of fear or stress. Indeed, in situations of fear, like the phobias for small animals, AR applications allow, in accordance with the patient’s anxiety, to gradually expose patient to fear animals, adding new animals during the session or enlarging their or increasing the speed. The various studies showed that AR is able, at the beginning of the session, to activate patient’s anxiety, for reducing after 1 h of exposition. After the session, patients even more than to better manage animal’s fear and anxiety, ware able to approach, interact, and kill real feared animals.

Materials and Methods

Data collection.

The input data for the analyses were retrieved from the scientific database Web of Science Core Collection ( Falagas et al., 2008 ) and the search terms used were “Virtual Reality” and “Augmented Reality” regarding papers published during the whole timespan covered.

Web of science core collection is composed of: Citation Indexes, Science Citation Index Expanded (SCI-EXPANDED) –1970-present, Social Sciences Citation Index (SSCI) –1970-present, Arts and Humanities Citation Index (A&HCI) –1975-present, Conference Proceedings Citation Index- Science (CPCI-S) –1990-present, Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) –1990-present, Book Citation Index– Science (BKCI-S) –2009-present, Book Citation Index– Social Sciences & Humanities (BKCI-SSH) –2009-present, Emerging Sources Citation Index (ESCI) –2015-present, Chemical Indexes, Current Chemical Reactions (CCR-EXPANDED) –2009-present (Includes Institut National de la Propriete Industrielle structure data back to 1840), Index Chemicus (IC) –2009-present.

The resultant dataset contained a total of 21,667 records for VR and 9,944 records for AR. The bibliographic record contained various fields, such as author, title, abstract, and all of the references (needed for the citation analysis). The research tool to visualize the networks was Cite space v.4.0.R5 SE (32 bit) ( Chen, 2006 ) under Java Runtime v.8 update 91 (build 1.8.0_91-b15). Statistical analyses were conducted using Stata MP-Parallel Edition, Release 14.0, StataCorp LP. Additional information can be found in Supplementary Data Sheet 1 .

The betweenness centrality of a node in a network measures the extent to which the node is part of paths that connect an arbitrary pair of nodes in the network ( Freeman, 1977 ; Brandes, 2001 ; Chen, 2006 ).

Structural metrics include betweenness centrality, modularity, and silhouette. Temporal and hybrid metrics include citation burstness and novelty. All the algorithms are detailed ( Chen et al., 2010 ).

The analysis of the literature on VR shows a complex panorama. At first sight, according to the document-type statistics from the Web of Science (WoS), proceedings papers were used extensively as outcomes of research, comprising almost 48% of the total (10,392 proceedings), with a similar number of articles on the subject amounting to about 47% of the total of 10, 199 articles. However, if we consider only the last 5 years (7,755 articles representing about 36% of the total), the situation changes with about 57% for articles (4,445) and about 33% for proceedings (2,578). Thus, it is clear that VR field has changed in areas other than at the technological level.

About the subject category, nodes and edges are computed as co-occurring subject categories from the Web of Science “Category” field in all the articles.

According to the subject category statistics from the WoS, computer science is the leading category, followed by engineering, and, together, they account for 15,341 articles, which make up about 71% of the total production. However, if we consider just the last 5 years, these categories reach only about 55%, with a total of 4,284 articles (Table ​ (Table1 1 and Figure ​ Figure1 1 ).

Category statistics from the WoS for the entire period and the last 5 years.

%FrequencySubject category (for all the period)
42,159131Computer Science, 1990–2016
28,666210Engineering, 1990–2016
8,211779Psychology, 1990–2016
7,151548Neurosciences and Neurology, 1992–2016
6,551418Surgery, 1992–2016
5,851267Automation and Control Systems, 1993–2016
4,801040Neurosciences, 1992–2016
4,741027Imaging Science and Photographic Technology, 1992–2016
4,30931Education and Educational Research, 1993–2016
3,92849Robotics, 1992–2016
29,802311Computer Science, 2011–2016
25,441973Engineering, 2011–2016
11,10861Neurosciences and Neurology, 2011–2016
9,32723Psychology, 2011–2016
7,70597Surgery, 2011–2016
7,53584Neurosciences, 2011–2016
6,02467Education and Educational Research, 2011–2016
5,54430Rehabilitation, 2011–2016
4,42343Clinical Neurology, 2011–2016
3,92304Materials Science, 2011–2016

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g001.jpg

Category from the WoS: network for the last 5 years.

The evidence is very interesting since it highlights that VR is doing very well as new technology with huge interest in hardware and software components. However, with respect to the past, we are witnessing increasing numbers of applications, especially in the medical area. In particular, note its inclusion in the top 10 list of rehabilitation and clinical neurology categories (about 10% of the total production in the last 5 years). It also is interesting that neuroscience and neurology, considered together, have shown an increase from about 12% to about 18.6% over the last 5 years. However, historic areas, such as automation and control systems, imaging science and photographic technology, and robotics, which had accounted for about 14.5% of the total articles ever produced were not even in the top 10 for the last 5 years, with each one accounting for less than 4%.

About the countries, nodes and edges are computed as networks of co-authors countries. Multiple occurrency of a country in the same paper are counted once.

The countries that were very involved in VR research have published for about 47% of the total (10,200 articles altogether). Of the 10,200 articles, the United States, China, England, and Germany published 4921, 2384, 1497, and 1398, respectively. The situation remains the same if we look at the articles published over the last 5 years. However, VR contributions also came from all over the globe, with Japan, Canada, Italy, France, Spain, South Korea, and Netherlands taking positions of prominence, as shown in Figure ​ Figure2 2 .

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g002.jpg

Country network (node dimension represents centrality).

Network analysis was conducted to calculate and to represent the centrality index ( Freeman, 1977 ; Brandes, 2001 ), i.e., the dimension of the node in Figure ​ Figure2. 2 . The top-ranked country, with a centrality index of 0.26, was the United States (2011), and England was second, with a centrality index of 0.25. The third, fourth, and fifth countries were Germany, Italy, and Australia, with centrality indices of 0.15, 0.15, and 0.14, respectively.

About the Institutions, nodes and edges are computed as networks of co-authors Institutions (Figure ​ (Figure3 3 ).

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g003.jpg

Network of institutions: the dimensions of the nodes represent centrality.

The top-level institutions in VR were in the United States, where three universities were ranked as the top three in the world for published articles; these universities were the University of Illinois (159), the University of South California (147), and the University of Washington (146). The United States also had the eighth-ranked university, which was Iowa State University (116). The second country in the ranking was Canada, with the University of Toronto, which was ranked fifth with 125 articles and McGill University, ranked 10 th with 103 articles.

Other countries in the top-ten list were Netherlands, with the Delft University of Technology ranked fourth with 129 articles; Italy, with IRCCS Istituto Auxologico Italiano, ranked sixth (with the same number of publication of the institution ranked fifth) with 125 published articles; England, which was ranked seventh with 125 articles from the University of London’s Imperial College of Science, Technology, and Medicine; and China with 104 publications, with the Chinese Academy of Science, ranked ninth. Italy’s Istituto Auxologico Italiano, which was ranked fifth, was the only non-university institution ranked in the top-10 list for VR research (Figure ​ (Figure3 3 ).

About the Journals, nodes, and edges are computed as journal co-citation networks among each journals in the corresponding field.

The top-ranked Journals for citations in VR are Presence: Teleoperators & Virtual Environments with 2689 citations and CyberPsychology & Behavior (Cyberpsychol BEHAV) with 1884 citations; however, looking at the last 5 years, the former had increased the citations, but the latter had a far more significant increase, from about 70% to about 90%, i.e., an increase from 1029 to 1147.

Following the top two journals, IEEE Computer Graphics and Applications ( IEEE Comput Graph) and Advanced Health Telematics and Telemedicine ( St HEAL T) were both left out of the top-10 list based on the last 5 years. The data for the last 5 years also resulted in the inclusion of Experimental Brain Research ( Exp BRAIN RES) (625 citations), Archives of Physical Medicine and Rehabilitation ( Arch PHYS MED REHAB) (622 citations), and Plos ONE (619 citations) in the top-10 list of three journals, which highlighted the categories of rehabilitation and clinical neurology and neuroscience and neurology. Journal co-citation analysis is reported in Figure ​ Figure4, 4 , which clearly shows four distinct clusters.

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g004.jpg

Co-citation network of journals: the dimensions of the nodes represent centrality. Full list of official abbreviations of WoS journals can be found here: https://images.webofknowledge.com/images/help/WOS/A_abrvjt.html .

Network analysis was conducted to calculate and to represent the centrality index, i.e., the dimensions of the nodes in Figure ​ Figure4. 4 . The top-ranked item by centrality was Cyberpsychol BEHAV, with a centrality index of 0.29. The second-ranked item was Arch PHYS MED REHAB, with a centrality index of 0.23. The third was Behaviour Research and Therapy (Behav RES THER), with a centrality index of 0.15. The fourth was BRAIN, with a centrality index of 0.14. The fifth was Exp BRAIN RES, with a centrality index of 0.11.

Who’s Who in VR Research

Authors are the heart and brain of research, and their roles in a field are to define the past, present, and future of disciplines and to make significant breakthroughs to make new ideas arise (Figure ​ (Figure5 5 ).

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g005.jpg

Network of authors’ numbers of publications: the dimensions of the nodes represent the centrality index, and the dimensions of the characters represent the author’s rank.

Virtual reality research is very young and changing with time, but the top-10 authors in this field have made fundamentally significant contributions as pioneers in VR and taking it beyond a mere technological development. The purpose of the following highlights is not to rank researchers; rather, the purpose is to identify the most active researchers in order to understand where the field is going and how they plan for it to get there.

The top-ranked author is Riva G, with 180 publications. The second-ranked author is Rizzo A, with 101 publications. The third is Darzi A, with 97 publications. The forth is Aggarwal R, with 94 publications. The six authors following these three are Slater M, Alcaniz M, Botella C, Wiederhold BK, Kim SI, and Gutierrez-Maldonado J with 90, 90, 85, 75, 59, and 54 publications, respectively (Figure ​ (Figure6 6 ).

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g006.jpg

Authors’ co-citation network: the dimensions of the nodes represent centrality index, and the dimensions of the characters represent the author’s rank. The 10 authors that appear on the top-10 list are considered to be the pioneers of VR research.

Considering the last 5 years, the situation remains similar, with three new entries in the top-10 list, i.e., Muhlberger A, Cipresso P, and Ahmed K ranked 7th, 8th, and 10th, respectively.

The authors’ publications number network shows the most active authors in VR research. Another relevant analysis for our focus on VR research is to identify the most cited authors in the field.

For this purpose, the authors’ co-citation analysis highlights the authors in term of their impact on the literature considering the entire time span of the field ( White and Griffith, 1981 ; González-Teruel et al., 2015 ; Bu et al., 2016 ). The idea is to focus on the dynamic nature of the community of authors who contribute to the research.

Normally, authors with higher numbers of citations tend to be the scholars who drive the fundamental research and who make the most meaningful impacts on the evolution and development of the field. In the following, we identified the most-cited pioneers in the field of VR Research.

The top-ranked author by citation count is Gallagher (2001), with 694 citations. Second is Seymour (2004), with 668 citations. Third is Slater (1999), with 649 citations. Fourth is Grantcharov (2003), with 563 citations. Fifth is Riva (1999), with 546 citations. Sixth is Aggarwal (2006), with 505 citations. Seventh is Satava (1994), with 477 citations. Eighth is Witmer (2002), with 454 citations. Ninth is Rothbaum (1996), with 448 citations. Tenth is Cruz-neira (1995), with 416 citations.

Citation Network and Cluster Analyses for VR

Another analysis that can be used is the analysis of document co-citation, which allows us to focus on the highly-cited documents that generally are also the most influential in the domain ( Small, 1973 ; González-Teruel et al., 2015 ; Orosz et al., 2016 ).

The top-ranked article by citation counts is Seymour (2002) in Cluster #0, with 317 citations. The second article is Grantcharov (2004) in Cluster #0, with 286 citations. The third is Holden (2005) in Cluster #2, with 179 citations. The 4th is Gallagher et al. (2005) in Cluster #0, with 171 citations. The 5th is Ahlberg (2007) in Cluster #0, with 142 citations. The 6th is Parsons (2008) in Cluster #4, with 136 citations. The 7th is Powers (2008) in Cluster #4, with 134 citations. The 8th is Aggarwal (2007) in Cluster #0, with 121 citations. The 9th is Reznick (2006) in Cluster #0, with 121 citations. The 10th is Munz (2004) in Cluster #0, with 117 citations.

The network of document co-citations is visually complex (Figure ​ (Figure7) 7 ) because it includes 1000s of articles and the links among them. However, this analysis is very important because can be used to identify the possible conglomerate of knowledge in the area, and this is essential for a deep understanding of the area. Thus, for this purpose, a cluster analysis was conducted ( Chen et al., 2010 ; González-Teruel et al., 2015 ; Klavans and Boyack, 2015 ). Figure ​ Figure8 8 shows the clusters, which are identified with the two algorithms in Table ​ Table2 2 .

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g007.jpg

Network of document co-citations: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank, and the numbers represent the strengths of the links. It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past VR research to the current research.

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g008.jpg

Document co-citation network by cluster: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank and the red writing reports the name of the cluster with a short description that was produced with the mutual information algorithm; the clusters are identified with colored polygons.

Cluster ID and silhouettes as identified with two algorithms ( Chen et al., 2010 ).

IDSizeSilho-uetteMean (Citee Year)Label (TFIDF, tf idf weighting algorithm)Label (LLR, log-likelihood ratio, p-level)
0840.8122005(25.82) laparoscopic skill; (25.01) proficiency; (24.5) basic laparoscopic skill; (24.14) trainer; (23.79) establishing validityTraining (143.21, 1.0E-4); performance (73.38, 1.0E-4); laparoscopic skill (72.93, 1.0E-4)
1770.7581992(17.76) ergonomic; (17.66) reality; (16.83) virtual reality; (16.04) virtual environment; (15.76) assemblyErgonomic (54.1, 1.0E-4); virtual reality interface (34.63, 1.0E-4); developing virtual environment (34.48, 1.0E-4)
2620.9922007(24.5) gaming; (24.5) wii; (24.47) stroke; (23.07) rehabilitation; (22.38) cerebral palsyStroke (82.9, 1.0E-4); children (75.13, 1.0E-4); stroke rehabilitation (57.95, 1.0E-4)
3610.7581994(15) reality; (14.66) virtual reality; (14.25) surgery; (14.1) telemedical information society; (13.73) chemistryTelemedical information society (34.85, 1.0E-4); gaining insight (23.21, 1.0E-4); next decade (18.32, 1.0E-4)
4560.9342008(25.4) therapy; (23.55) exposure therapy; (22.41) disorder; (21.63) virtual reality exposure therapy; (20.99) post-traumatic stressTreatment (109.92, 1.0E-4); post-traumatic stress disorder (78.95, 1.0E-4); virtual reality exposure therapy (66.15, 1.0E-4)
5490.8851992(16.03) reality; (15.31) virtual reality; (15.01) autistic children; (12.79) child; (12.79) childrenAutistic children (29.81, 1.0E-4); possibilities (23.84, 1.0E-4); communication (22.08, 1.0E-4)
6410.8551998(17.6) laparoscopic skill; (16.95) direct observation; (16.95) measuring operative performance; (16.95) videotape; (16.15) measuringLaparoscopic skills training (52.73, 1.0E-4); measuring operative performance (40.97, 1.0E-4); videotape (40.97, 1.0E-4)
7410.9461998(20.71) therapy; (18.76) exposure therapy; (17.85) exposure; (17.35) anxiety; (17.2) virtual reality exposure therapyVirtual reality exposure therapy (32.01, 1.0E-4); spider phobia (27.67, 1.0E-4); ptsd vietnam veteran (22.12, 1.0E-4)
83811989(30.67) Japanese institutional mechanism; (30.67) systems perspective; (20.88) mechanism; (19.25) perspective; (17.97) systemJapanese institutional mechanism (615.45, 1.0E-4); systems perspective (615.45, 1.0E-4); virtual reality (16.28, 1.0E-4)
92111987(23.27) routine use; (23.27) current application; (23.27) behavioral-assessment; (23.27) obstacle; (23.27) future possibilitiesFuture possibilities (168.77, 1.0E-4); routine use (168.77, 1.0E-4); current application (168.77, 1.0E-4)
10180.9341991(12.45) reality; (12.26) virtual-reality; (9.73) medicine; (9.07) virtual reality; (5.71) technologyVirtual-reality (88.95, 1.0E-4); medicine (34.87, 1.0E-4); pretty interface (9.63, 0.005)
11160.9371990(13.37) tutorial; (12.45) reality; (11.98) virtual reality; (11.12) virtual reality technology; (10.78) technologyTutorial (51.15, 1.0E-4); virtual reality technology (44.66, 1.0E-4); space (16.78, 1.0E-4)
121211988(20.05) special effect; (20.05) cyberspace; (13.65) space; (11.38) effect; (10.73) realitySpecial effect (128.6, 1.0E-4); cyberspace (128.6, 1.0E-4); virtual reality (27.79, 1.0E-4)
1380.9951997(14.88) neural substrate; (14.88) human spatial navigation; (14.88) cognitive map; (11.56) navigation; (10.64) cognitiveNeural substrate (72.6, 1.0E-4); human spatial navigation (66.58, 1.0E-4); cognitive map (66.58, 1.0E-4)
1460.9932008(12.06) neurosurgery; (9.74) computer technology; (9.74) surgical application; (9.43) surgery; (8.55) teachingNeurosurgery (28.72, 1.0E-4); computer technology (18.1, 1.0E-4); surgical application (18.1, 1.0E-4)

The identified clusters highlight clear parts of the literature of VR research, making clear and visible the interdisciplinary nature of this field. However, the dynamics to identify the past, present, and future of VR research cannot be clear yet. We analysed the relationships between these clusters and the temporal dimensions of each article. The results are synthesized in Figure ​ Figure9. 9 . It is clear that cluster #0 (laparoscopic skill), cluster #2 (gaming and rehabilitation), cluster #4 (therapy), and cluster #14 (surgery) are the most popular areas of VR research. (See Figure ​ Figure9 9 and Table ​ Table2 2 to identify the clusters.) From Figure ​ Figure9, 9 , it also is possible to identify the first phase of laparoscopic skill (cluster #6) and therapy (cluster #7). More generally, it is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past VR research to the current research.

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g009.jpg

Network of document co-citation: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank and the red writing on the right hand side reports the number of the cluster, such as in Table ​ Table2, 2 , with a short description that was extracted accordingly.

We were able to identify the top 486 references that had the most citations by using burst citations algorithm. Citation burst is an indicator of a most active area of research. Citation burst is a detection of a burst event, which can last for multiple years as well as a single year. A citation burst provides evidence that a particular publication is associated with a surge of citations. The burst detection was based on Kleinberg’s algorithm ( Kleinberg, 2002 , 2003 ). The top-ranked document by bursts is Seymour (2002) in Cluster #0, with bursts of 88.93. The second is Grantcharov (2004) in Cluster #0, with bursts of 51.40. The third is Saposnik (2010) in Cluster #2, with bursts of 40.84. The fourth is Rothbaum (1995) in Cluster #7, with bursts of 38.94. The fifth is Holden (2005) in Cluster #2, with bursts of 37.52. The sixth is Scott (2000) in Cluster #0, with bursts of 33.39. The seventh is Saposnik (2011) in Cluster #2, with bursts of 33.33. The eighth is Burdea et al. (1996) in Cluster #3, with bursts of 32.42. The ninth is Burdea and Coiffet (2003) in Cluster #22, with bursts of 31.30. The 10th is Taffinder (1998) in Cluster #6, with bursts of 30.96 (Table ​ (Table3 3 ).

Cluster ID and references of burst article.

ClusterReferenceYearStrengthBeginEnd1990–2016
7Rothbaum, 1995, Am J Psychiat, V152, P626199538,9419962003
3 , Force Touch Feedback, V, P199632,4219972004
6Taffinder, 1998, St Heal T, V50, P124199830,9620002006
0Scott, 2000, J Am Coll Surgeons, V191, P272, Doi200033,3920032008
0Seymour, 2002, Ann Surg, V236, P458, Doi200288,9320042010
22 , Virtual Reality Tech, V, P200331,3020042010
0Grantcharov, 2004, Brit J Surg, V91, P146, Doi200451,4020052012
2Holden, 2005, Cyberpsychol Behav, V8, P187, Doi200537,5220072013
2Saposnik, 2010, Stroke, V41, P1477, Doi201040,8420122016
2Saposnik, 2011, Stroke, V42, P1380, Doi201133,3320122016

Citation Network and Cluster Analyses for AR

Looking at Augmented Reality scenario, the top ranked item by citation counts is Azuma (1997) in Cluster #0, with citation counts of 231. The second one is Azuma et al. (2001) in Cluster #0, with citation counts of 220. The third is Van Krevelen (2010) in Cluster #5, with citation counts of 207. The 4th is Lowe (2004) in Cluster #1, with citation counts of 157. The 5th is Wu (2013) in Cluster #4, with citation counts of 144. The 6th is Dunleavy (2009) in Cluster #4, with citation counts of 122. The 7th is Zhou (2008) in Cluster #5, with citation counts of 118. The 8th is Bay (2008) in Cluster #1, with citation counts of 117. The 9th is Newcombe (2011) in Cluster #1, with citation counts of 109. The 10th is Carmigniani et al. (2011) in Cluster #5, with citation counts of 104.

The network of document co-citations is visually complex (Figure ​ (Figure10) 10 ) because it includes 1000s of articles and the links among them. However, this analysis is very important because can be used to identify the possible conglomerate of knowledge in the area, and this is essential for a deep understanding of the area. Thus, for this purpose, a cluster analysis was conducted ( Chen et al., 2010 ; González-Teruel et al., 2015 ; Klavans and Boyack, 2015 ). Figure ​ Figure11 11 shows the clusters, which are identified with the two algorithms in Table ​ Table3 3 .

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g010.jpg

Network of document co-citations: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank, and the numbers represent the strengths of the links. It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past AR research to the current research.

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g011.jpg

The identified clusters highlight clear parts of the literature of AR research, making clear and visible the interdisciplinary nature of this field. However, the dynamics to identify the past, present, and future of AR research cannot be clear yet. We analysed the relationships between these clusters and the temporal dimensions of each article. The results are synthesized in Figure ​ Figure12. 12 . It is clear that cluster #1 (tracking), cluster #4 (education), and cluster #5 (virtual city environment) are the current areas of AR research. (See Figure ​ Figure12 12 and Table ​ Table3 3 to identify the clusters.) It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past AR research to the current research.

An external file that holds a picture, illustration, etc.
Object name is fpsyg-09-02086-g012.jpg

We were able to identify the top 394 references that had the most citations by using burst citations algorithm. Citation burst is an indicator of a most active area of research. Citation burst is a detection of a burst event, which can last for multiple years as well as a single year. A citation burst provides evidence that a particular publication is associated with a surge of citations. The burst detection was based on Kleinberg’s algorithm ( Kleinberg, 2002 , 2003 ). The top ranked document by bursts is Azuma (1997) in Cluster #0, with bursts of 101.64. The second one is Azuma et al. (2001) in Cluster #0, with bursts of 84.23. The third is Lowe (2004) in Cluster #1, with bursts of 64.07. The 4th is Van Krevelen (2010) in Cluster #5, with bursts of 50.99. The 5th is Wu (2013) in Cluster #4, with bursts of 47.23. The 6th is Hartley (2000) in Cluster #0, with bursts of 37.71. The 7th is Dunleavy (2009) in Cluster #4, with bursts of 33.22. The 8th is Kato (1999) in Cluster #0, with bursts of 32.16. The 9th is Newcombe (2011) in Cluster #1, with bursts of 29.72. The 10th is Feiner (1993) in Cluster #8, with bursts of 29.46 (Table ​ (Table4 4 ).

IDSizeSilho-uetteMean (Citee Year)Label (TFIDF, tf idf weighting algorithm)Label (LLR, log-likelihood ratio, p-level)
01220.6691999(18.41) internetInternet (39.96, 1.0E-4)
1660.8062007(16.67) trackingMobile phone (47.52, 1.0E-4)
2650.8271994(17.48) natural environmentNatural feature tracking (57.72, 1.0E-4)
3560.892004(17.33) liverLaparoscopic surgery (30.43, 1.0E-4)
4500.9432011(19.32) educationEducation (64.26, 1.0E-4)
5480.862007(15.96) virtual city environmentVirtual city environment (32.68, 1.0E-4)
6200.9971989(21.65) knowledge-based augmented realityKnowledge-based augmented reality (250.67, 1.0E-4)
7190.9261992(19.32) hand-eye calibrationHand–eye calibration (104.98, 1.0E-4)

Our findings have profound implications for two reasons. At first the present work highlighted the evolution and development of VR and AR research and provided a clear perspective based on solid data and computational analyses. Secondly our findings on VR made it profoundly clear that the clinical dimension is one of the most investigated ever and seems to increase in quantitative and qualitative aspects, but also include technological development and article in computer science, engineer, and allied sciences.

Figure ​ Figure9 9 clarifies the past, present, and future of VR research. The outset of VR research brought a clearly-identifiable development in interfaces for children and medicine, routine use and behavioral-assessment, special effects, systems perspectives, and tutorials. This pioneering era evolved in the period that we can identify as the development era, because it was the period in which VR was used in experiments associated with new technological impulses. Not surprisingly, this was exactly concomitant with the new economy era in which significant investments were made in information technology, and it also was the era of the so-called ‘dot-com bubble’ in the late 1990s. The confluence of pioneering techniques into ergonomic studies within this development era was used to develop the first effective clinical systems for surgery, telemedicine, human spatial navigation, and the first phase of the development of therapy and laparoscopic skills. With the new millennium, VR research switched strongly toward what we can call the clinical-VR era, with its strong emphasis on rehabilitation, neurosurgery, and a new phase of therapy and laparoscopic skills. The number of applications and articles that have been published in the last 5 years are in line with the new technological development that we are experiencing at the hardware level, for example, with so many new, HMDs, and at the software level with an increasing number of independent programmers and VR communities.

Finally, Figure ​ Figure12 12 identifies clusters of the literature of AR research, making clear and visible the interdisciplinary nature of this field. The dynamics to identify the past, present, and future of AR research cannot be clear yet, but analyzing the relationships between these clusters and the temporal dimensions of each article tracking, education, and virtual city environment are the current areas of AR research. AR is a new technology that is showing its efficacy in different research fields, and providing a novel way to gather behavioral data and support learning, training, and clinical treatments.

Looking at scientific literature conducted in the last few years, it might appear that most developments in VR and AR studies have focused on clinical aspects. However, the reality is more complex; thus, this perception should be clarified. Although researchers publish studies on the use of VR in clinical settings, each study depends on the technologies available. Industrial development in VR and AR changed a lot in the last 10 years. In the past, the development involved mainly hardware solutions while nowadays, the main efforts pertain to the software when developing virtual solutions. Hardware became a commodity that is often available at low cost. On the other hand, software needs to be customized each time, per each experiment, and this requires huge efforts in term of development. Researchers in AR and VR today need to be able to adapt software in their labs.

Virtual reality and AR developments in this new clinical era rely on computer science and vice versa. The future of VR and AR is becoming more technological than before, and each day, new solutions and products are coming to the market. Both from software and hardware perspectives, the future of AR and VR depends on huge innovations in all fields. The gap between the past and the future of AR and VR research is about the “realism” that was the key aspect in the past versus the “interaction” that is the key aspect now. First 30 years of VR and AR consisted of a continuous research on better resolution and improved perception. Now, researchers already achieved a great resolution and need to focus on making the VR as realistic as possible, which is not simple. In fact, a real experience implies a realistic interaction and not just great resolution. Interactions can be improved in infinite ways through new developments at hardware and software levels.

Interaction in AR and VR is going to be “embodied,” with implication for neuroscientists that are thinking about new solutions to be implemented into the current systems ( Blanke et al., 2015 ; Riva, 2018 ; Riva et al., 2018 ). For example, the use of hands with contactless device (i.e., without gloves) makes the interaction in virtual environments more natural. The Leap Motion device 1 allows one to use of hands in VR without the use of gloves or markers. This simple and low-cost device allows the VR users to interact with virtual objects and related environments in a naturalistic way. When technology is able to be transparent, users can experience increased sense of being in the virtual environments (the so-called sense of presence).

Other forms of interactions are possible and have been developing continuously. For example, tactile and haptic device able to provide a continuous feedback to the users, intensifying their experience also by adding components, such as the feeling of touch and the physical weight of virtual objects, by using force feedback. Another technology available at low cost that facilitates interaction is the motion tracking system, such as Microsoft Kinect, for example. Such technology allows one to track the users’ bodies, allowing them to interact with the virtual environments using body movements, gestures, and interactions. Most HMDs use an embedded system to track HMD position and rotation as well as controllers that are generally placed into the user’s hands. This tracking allows a great degree of interaction and improves the overall virtual experience.

A final emerging approach is the use of digital technologies to simulate not only the external world but also the internal bodily signals ( Azevedo et al., 2017 ; Riva et al., 2017 ): interoception, proprioception and vestibular input. For example, Riva et al. (2017) recently introduced the concept of “sonoception” ( www.sonoception.com ), a novel non-invasive technological paradigm based on wearable acoustic and vibrotactile transducers able to alter internal bodily signals. This approach allowed the development of an interoceptive stimulator that is both able to assess interoceptive time perception in clinical patients ( Di Lernia et al., 2018b ) and to enhance heart rate variability (the short-term vagally mediated component—rMSSD) through the modulation of the subjects’ parasympathetic system ( Di Lernia et al., 2018a ).

In this scenario, it is clear that the future of VR and AR research is not just in clinical applications, although the implications for the patients are huge. The continuous development of VR and AR technologies is the result of research in computer science, engineering, and allied sciences. The reasons for which from our analyses emerged a “clinical era” are threefold. First, all clinical research on VR and AR includes also technological developments, and new technological discoveries are being published in clinical or technological journals but with clinical samples as main subject. As noted in our research, main journals that publish numerous articles on technological developments tested with both healthy and patients include Presence: Teleoperators & Virtual Environments, Cyberpsychology & Behavior (Cyberpsychol BEHAV), and IEEE Computer Graphics and Applications (IEEE Comput Graph). It is clear that researchers in psychology, neuroscience, medicine, and behavioral sciences in general have been investigating whether the technological developments of VR and AR are effective for users, indicating that clinical behavioral research has been incorporating large parts of computer science and engineering. A second aspect to consider is the industrial development. In fact, once a new technology is envisioned and created it goes for a patent application. Once the patent is sent for registration the new technology may be made available for the market, and eventually for journal submission and publication. Moreover, most VR and AR research that that proposes the development of a technology moves directly from the presenting prototype to receiving the patent and introducing it to the market without publishing the findings in scientific paper. Hence, it is clear that if a new technology has been developed for industrial market or consumer, but not for clinical purpose, the research conducted to develop such technology may never be published in a scientific paper. Although our manuscript considered published researches, we have to acknowledge the existence of several researches that have not been published at all. The third reason for which our analyses highlighted a “clinical era” is that several articles on VR and AR have been considered within the Web of Knowledge database, that is our source of references. In this article, we referred to “research” as the one in the database considered. Of course, this is a limitation of our study, since there are several other databases that are of big value in the scientific community, such as IEEE Xplore Digital Library, ACM Digital Library, and many others. Generally, the most important articles in journals published in these databases are also included in the Web of Knowledge database; hence, we are convinced that our study considered the top-level publications in computer science or engineering. Accordingly, we believe that this limitation can be overcome by considering the large number of articles referenced in our research.

Considering all these aspects, it is clear that clinical applications, behavioral aspects, and technological developments in VR and AR research are parts of a more complex situation compared to the old platforms used before the huge diffusion of HMD and solutions. We think that this work might provide a clearer vision for stakeholders, providing evidence of the current research frontiers and the challenges that are expected in the future, highlighting all the connections and implications of the research in several fields, such as clinical, behavioral, industrial, entertainment, educational, and many others.

Author Contributions

PC and GR conceived the idea. PC made data extraction and the computational analyses and wrote the first draft of the article. IG revised the introduction adding important information for the article. PC, IG, MR, and GR revised the article and approved the last version of the article after important input to the article rationale.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer GC declared a shared affiliation, with no collaboration, with the authors PC and GR to the handling Editor at the time of the review.

1 https://www.leapmotion.com/

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2018.02086/full#supplementary-material

  • Akçayır M., Akçayır G. (2017). Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ. Res. Rev. 20 1–11. 10.1016/j.edurev.2016.11.002 [ CrossRef ] [ Google Scholar ]
  • Alexander T., Westhoven M., Conradi J. (2017). “Virtual environments for competency-oriented education and training,” in Advances in Human Factors, Business Management, Training and Education , (Berlin: Springer International Publishing; ), 23–29. 10.1007/978-3-319-42070-7_3 [ CrossRef ] [ Google Scholar ]
  • Andersen S. M., Thorpe J. S. (2009). An if–thEN theory of personality: significant others and the relational self. J. Res. Pers. 43 163–170. 10.1016/j.jrp.2008.12.040 [ CrossRef ] [ Google Scholar ]
  • Azevedo R. T., Bennett N., Bilicki A., Hooper J., Markopoulou F., Tsakiris M. (2017). The calming effect of a new wearable device during the anticipation of public speech. Sci. Rep. 7 : 2285 . 10.1038/s41598-017-02274-2 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Azuma R., Baillot Y., Behringer R., Feiner S., Julier S., MacIntyre B. (2001). Recent advances in augmented reality. IEEE Comp. Graph. Appl. 21 34–47. 10.1109/38.963459 [ CrossRef ] [ Google Scholar ]
  • Bacca J., Baldiris S., Fabregat R., Graf S. (2014). Augmented reality trends in education: a systematic review of research and applications. J. Educ. Technol. Soc. 17 133 . [ Google Scholar ]
  • Bailenson J. N., Yee N., Merget D., Schroeder R. (2006). The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction. Presence 15 359–372. 10.1162/pres.15.4.359 [ CrossRef ] [ Google Scholar ]
  • Baños R. M., Botella C., Garcia-Palacios A., Villa H., Perpiñá C., Alcaniz M. (2000). Presence and reality judgment in virtual environments: a unitary construct? Cyberpsychol. Behav. 3 327–335. 10.1089/10949310050078760 [ CrossRef ] [ Google Scholar ]
  • Baños R., Botella C., García-Palacios A., Villa H., Perpiñá C., Gallardo M. (2009). Psychological variables and reality judgment in virtual environments: the roles of absorption and dissociation. Cyberpsychol. Behav. 2 143–148. 10.1089/cpb.1999.2.143 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Baus O., Bouchard S. (2014). Moving from virtual reality exposure-based therapy to augmented reality exposure-based therapy: a review. Front. Hum. Neurosci. 8 : 112 . 10.3389/fnhum.2014.00112 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Biocca F. (1997). The cyborg’s dilemma: progressive embodiment in virtual environments. J. Comput. Mediat. Commun. 3 10.1111/j.1083-6101.1997 [ CrossRef ] [ Google Scholar ]
  • Biocca F., Harms C., Gregg J. (2001). “The networked minds measure of social presence: pilot test of the factor structure and concurrent validity,” in 4th Annual International Workshop on Presence , Philadelphia, PA, 1–9. [ Google Scholar ]
  • Blanke O., Slater M., Serino A. (2015). Behavioral, neural, and computational principles of bodily self-consciousness. Neuron 88 145–166. 10.1016/j.neuron.2015.09.029 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Bohil C. J., Alicea B., Biocca F. A. (2011). Virtual reality in neuroscience research and therapy. Nat. Rev. Neurosci. 12 : 752 . 10.1038/nrn3122 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Borrego A., Latorre J., Llorens R., Alcañiz M., Noé E. (2016). Feasibility of a walking virtual reality system for rehabilitation: objective and subjective parameters. J. Neuroeng. Rehabil. 13 : 68 . 10.1186/s12984-016-0174-171 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Botella C., Bretón-López J., Quero S., Baños R. M., García-Palacios A. (2010). Treating cockroach phobia with augmented reality. Behav. Ther. 41 401–413. 10.1016/j.beth.2009.07.002 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Botella C., Fernández-Álvarez J., Guillén V., García-Palacios A., Baños R. (2017). Recent progress in virtual reality exposure therapy for phobias: a systematic review. Curr. Psychiatry Rep. 19 : 42 . 10.1007/s11920-017-0788-4 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Botella C. M., Juan M. C., Baños R. M., Alcañiz M., Guillén V., Rey B. (2005). Mixing realities? An application of augmented reality for the treatment of cockroach phobia. Cyberpsychol. Behav. 8 162–171. 10.1089/cpb.2005.8.162 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Brandes U. (2001). A faster algorithm for betweenness centrality. J. Math. Sociol. 25 163–177. 10.1080/0022250X.2001.9990249 [ CrossRef ] [ Google Scholar ]
  • Bretón-López J., Quero S., Botella C., García-Palacios A., Baños R. M., Alcañiz M. (2010). An augmented reality system validation for the treatment of cockroach phobia. Cyberpsychol. Behav. Soc. Netw. 13 705–710. 10.1089/cyber.2009.0170 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Brown A., Green T. (2016). Virtual reality: low-cost tools and resources for the classroom. TechTrends 60 517–519. 10.1007/s11528-016-0102-z [ CrossRef ] [ Google Scholar ]
  • Bu Y., Liu T. Y., Huang W. B. (2016). MACA: a modified author co-citation analysis method combined with general descriptive metadata of citations. Scientometrics 108 143–166. 10.1007/s11192-016-1959-5 [ CrossRef ] [ Google Scholar ]
  • Burdea G., Richard P., Coiffet P. (1996). Multimodal virtual reality: input-output devices, system integration, and human factors. Int. J. Hum. Compu. Interact. 8 5–24. 10.1080/10447319609526138 [ CrossRef ] [ Google Scholar ]
  • Burdea G. C., Coiffet P. (2003). Virtual Reality Technology Vol. 1 Hoboken, NJ: John Wiley & Sons. [ Google Scholar ]
  • Carmigniani J., Furht B., Anisetti M., Ceravolo P., Damiani E., Ivkovic M. (2011). Augmented reality technologies, systems and applications. Multimed. Tools Appl. 51 341–377. 10.1007/s11042-010-0660-6 [ CrossRef ] [ Google Scholar ]
  • Castelvecchi D. (2016). Low-cost headsets boost virtual reality’s lab appeal. Nature 533 153–154. 10.1038/533153a [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Cathy (2011). The History of Augmented Reality. The Optical Vision Site. Available at: http://www.theopticalvisionsite.com/history-of-eyewear/the-history-of-augmented-reality/#.UelAUmeAOyA [ Google Scholar ]
  • Chen C. (2006). CiteSpace II: detecting and visualizing emerging trends and transient patterns in scientific literature. J. Assoc. Inform. Sci. Technol. 57 359–377. 10.1002/asi.20317 [ CrossRef ] [ Google Scholar ]
  • Chen C., Ibekwe-SanJuan F., Hou J. (2010). The structure and dynamics of cocitation clusters: a multipleperspective cocitation analysis. J. Assoc. Inform. Sci. Technol. 61 1386–1409. 10.1002/jez.b.22741 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Chen Y. C., Chi H. L., Hung W. H., Kang S. C. (2011). Use of tangible and augmented reality models in engineering graphics courses. J. Prof. Issues Eng. Educ. Pract. 137 267–276. 10.1061/(ASCE)EI.1943-5541.0000078 [ CrossRef ] [ Google Scholar ]
  • Chicchi Giglioli I. A., Pallavicini F., Pedroli E., Serino S., Riva G. (2015). Augmented reality: a brand new challenge for the assessment and treatment of psychological disorders. Comput. Math. Methods Med. 2015 : 862942 . 10.1155/2015/862942 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Chien C. H., Chen C. H., Jeng T. S. (2010). “An interactive augmented reality system for learning anatomy structure,” in Proceedings of the International Multiconference of Engineers and Computer Scientists , Vol. 1 (Hong Kong: International Association of Engineers; ), 17–19. [ Google Scholar ]
  • Choi S., Jung K., Noh S. D. (2015). Virtual reality applications in manufacturing industries: past research, present findings, and future directions. Concurr. Eng. 23 40–63. 10.1177/1063293X14568814 [ CrossRef ] [ Google Scholar ]
  • Cipresso P. (2015). Modeling behavior dynamics using computational psychometrics within virtual worlds. Front. Psychol. 6 : 1725 . 10.3389/fpsyg.2015.01725 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Cipresso P., Serino S. (2014). Virtual Reality: Technologies, Medical Applications and Challenges. Hauppauge, NY: Nova Science Publishers, Inc. [ Google Scholar ]
  • Cipresso P., Serino S., Riva G. (2016). Psychometric assessment and behavioral experiments using a free virtual reality platform and computational science. BMC Med. Inform. Decis. Mak. 16 : 37 . 10.1186/s12911-016-0276-5 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Cruz-Neira C. (1993). “Virtual reality overview,” in SIGGRAPH 93 Course Notes 21st International Conference on Computer Graphics and Interactive Techniques, Orange County Convention Center , Orlando, FL. [ Google Scholar ]
  • De Buck S., Maes F., Ector J., Bogaert J., Dymarkowski S., Heidbuchel H., et al. (2005). An augmented reality system for patient-specific guidance of cardiac catheter ablation procedures. IEEE Trans. Med. Imaging 24 1512–1524. 10.1109/TMI.2005.857661 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Di Lernia D., Cipresso P., Pedroli E., Riva G. (2018a). Toward an embodied medicine: a portable device with programmable interoceptive stimulation for heart rate variability enhancement. Sensors (Basel) 18 : 2469 . 10.3390/s18082469 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Di Lernia D., Serino S., Pezzulo G., Pedroli E., Cipresso P., Riva G. (2018b). Feel the time. Time perception as a function of interoceptive processing. Front. Hum. Neurosci. 12 : 74 . 10.3389/fnhum.2018.00074 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Di Serio Á., Ibáñez M. B., Kloos C. D. (2013). Impact of an augmented reality system on students’ motivation for a visual art course. Comput. Educ. 68 586–596. 10.1016/j.compedu.2012.03.002 [ CrossRef ] [ Google Scholar ]
  • Ebert C. (2015). Looking into the future. IEEE Softw. 32 92–97. 10.1109/MS.2015.142 [ CrossRef ] [ Google Scholar ]
  • Englund C., Olofsson A. D., Price L. (2017). Teaching with technology in higher education: understanding conceptual change and development in practice. High. Educ. Res. Dev. 36 73–87. 10.1080/07294360.2016.1171300 [ CrossRef ] [ Google Scholar ]
  • Falagas M. E., Pitsouni E. I., Malietzis G. A., Pappas G. (2008). Comparison of pubmed, scopus, web of science, and Google scholar: strengths and weaknesses. FASEB J. 22 338–342. 10.1096/fj.07-9492LSF [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Feiner S., MacIntyre B., Hollerer T., Webster A. (1997). “A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment,” in Digest of Papers. First International Symposium on Wearable Computers , (Cambridge, MA: IEEE; ), 74–81. 10.1109/ISWC.1997.629922 [ CrossRef ] [ Google Scholar ]
  • Freeman D., Reeve S., Robinson A., Ehlers A., Clark D., Spanlang B., et al. (2017). Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol. Med. 47 2393–2400. 10.1017/S003329171700040X [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Freeman L. C. (1977). A set of measures of centrality based on betweenness. Sociometry 40 35–41. 10.2307/3033543 [ CrossRef ] [ Google Scholar ]
  • Fuchs H., Bishop G. (1992). Research Directions in Virtual Environments. Chapel Hill, NC: University of North Carolina at Chapel Hill. [ Google Scholar ]
  • Gallagher A. G., Ritter E. M., Champion H., Higgins G., Fried M. P., Moses G., et al. (2005). Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann. Surg. 241 : 364 . 10.1097/01.sla.0000151982.85062.80 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Gigante M. A. (1993). Virtual reality: definitions, history and applications. Virtual Real. Syst. 3–14. 10.1016/B978-0-12-227748-1.50009-3 [ CrossRef ] [ Google Scholar ]
  • González-Teruel A., González-Alcaide G., Barrios M., Abad-García M. F. (2015). Mapping recent information behavior research: an analysis of co-authorship and co-citation networks. Scientometrics 103 687–705. 10.1007/s11192-015-1548-z [ CrossRef ] [ Google Scholar ]
  • Heeter C. (1992). Being there: the subjective experience of presence. Presence 1 262–271. 10.1162/pres.1992.1.2.262 [ CrossRef ] [ Google Scholar ]
  • Heeter C. (2000). Interactivity in the context of designed experiences. J. Interact. Adv. 1 3–14. 10.1080/15252019.2000.10722040 [ CrossRef ] [ Google Scholar ]
  • Heilig M. (1962). Sensorama simulator . U.S. Patent No - 3, 870. Virginia: United States Patent and Trade Office. [ Google Scholar ]
  • Ibáñez M. B., Di Serio Á., Villarán D., Kloos C. D. (2014). Experimenting with electromagnetism using augmented reality: impact on flow student experience and educational effectiveness. Comput. Educ. 71 1–13. 10.1016/j.compedu.2013.09.004 [ CrossRef ] [ Google Scholar ]
  • Juan M. C., Alcañiz M., Calatrava J., Zaragozá I., Baños R., Botella C. (2007). “An optical see-through augmented reality system for the treatment of phobia to small animals,” in Virtual Reality, HCII 2007 Lecture Notes in Computer Science Vol. 4563 ed. Schumaker R. (Berlin: Springer; ), 651–659. [ Google Scholar ]
  • Juan M. C., Alcaniz M., Monserrat C., Botella C., Baños R. M., Guerrero B. (2005). Using augmented reality to treat phobias. IEEE Comput. Graph. Appl. 25 31–37. 10.1109/MCG.2005.143 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Kim G. J. (2005). A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence 14 119–146. 10.1162/1054746053967094 [ CrossRef ] [ Google Scholar ]
  • Klavans R., Boyack K. W. (2015). Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge? J. Assoc. Inform. Sci. Technol. 68 984–998. 10.1002/asi.23734 [ CrossRef ] [ Google Scholar ]
  • Kleinberg J. (2002). “Bursty and hierarchical structure in streams,” in Paper Presented at the Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2002; Edmonton , Alberta, NT: 10.1145/775047.775061 [ CrossRef ] [ Google Scholar ]
  • Kleinberg J. (2003). Bursty and hierarchical structure in streams. Data Min. Knowl. Discov. 7 373–397. 10.1023/A:1024940629314 [ CrossRef ] [ Google Scholar ]
  • Korolov M. (2014). The real risks of virtual reality. Risk Manag. 61 20–24. [ Google Scholar ]
  • Krueger M. W., Gionfriddo T., Hinrichsen K. (1985). “Videoplace—an artificial reality,” in Proceedings of the ACM SIGCHI Bulletin Vol. 16 New York, NY: ACM, 35–40. 10.1145/317456.317463 [ CrossRef ] [ Google Scholar ]
  • Lin C. H., Hsu P. H. (2017). “Integrating procedural modelling process and immersive VR environment for architectural design education,” in MATEC Web of Conferences Vol. 104 Les Ulis: EDP Sciences; 10.1051/matecconf/201710403007 [ CrossRef ] [ Google Scholar ]
  • Llorens R., Noé E., Ferri J., Alcañiz M. (2014). Virtual reality-based telerehabilitation program for balance recovery. A pilot study in hemiparetic individuals with acquired brain injury. Brain Inj. 28 : 169 . [ Google Scholar ]
  • Lombard M., Ditton T. (1997). At the heart of it all: the concept of presence. J. Comput. Mediat. Commun. 3 10.1111/j.1083-6101.1997.tb00072.x [ CrossRef ] [ Google Scholar ]
  • Loomis J. M., Blascovich J. J., Beall A. C. (1999). Immersive virtual environment technology as a basic research tool in psychology. Behav. Res. Methods Instr. Comput. 31 557–564. 10.3758/BF03200735 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Loomis J. M., Golledge R. G., Klatzky R. L. (1998). Navigation system for the blind: auditory display modes and guidance. Presence 7 193–203. 10.1162/105474698565677 [ CrossRef ] [ Google Scholar ]
  • Luckerson V. (2014). Facebook Buying Oculus Virtual-Reality Company for $2 Billion. Available at: http://time.com/37842/facebook-oculus-rift [ Google Scholar ]
  • Maurugeon G. (2011). New D’Fusion Supports iPhone4S and 3DSMax 2012. Available at: http://www.t-immersion.com/blog/2011-12-07/augmented-reality-dfusion-iphone-3dsmax [ Google Scholar ]
  • Mazuryk T., Gervautz M. (1996). Virtual Reality-History, Applications, Technology and Future. Vienna: Institute of Computer Graphics Vienna University of Technology. [ Google Scholar ]
  • Meldrum D., Glennon A., Herdman S., Murray D., McConn-Walsh R. (2012). Virtual reality rehabilitation of balance: assessment of the usability of the nintendo Wii ® fit plus. Disabil. Rehabil. 7 205–210. 10.3109/17483107.2011.616922 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Milgram P., Kishino F. (1994). A taxonomy of mixed reality visual displays. IEICE Trans. Inform. Syst. 77 1321–1329. [ Google Scholar ]
  • Minderer M., Harvey C. D., Donato F., Moser E. I. (2016). Neuroscience: virtual reality explored. Nature 533 324–325. 10.1038/nature17899 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Neri S. G., Cardoso J. R., Cruz L., Lima R. M., de Oliveira R. J., Iversen M. D., et al. (2017). Do virtual reality games improve mobility skills and balance measurements in community-dwelling older adults? Systematic review and meta-analysis. Clin. Rehabil. 31 1292–1304. 10.1177/0269215517694677 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Nincarean D., Alia M. B., Halim N. D. A., Rahman M. H. A. (2013). Mobile augmented reality: the potential for education. Procedia Soc. Behav. Sci. 103 657–664. 10.1016/j.sbspro.2013.10.385 [ CrossRef ] [ Google Scholar ]
  • Orosz K., Farkas I. J., Pollner P. (2016). Quantifying the changing role of past publications. Scientometrics 108 829–853. 10.1007/s11192-016-1971-9 [ CrossRef ] [ Google Scholar ]
  • Ozbek C. S., Giesler B., Dillmann R. (2004). “Jedi training: playful evaluation of head-mounted augmented reality display systems,” in Proceedings of SPIE. The International Society for Optical Engineering Vol. 5291 eds Norwood R. A., Eich M., Kuzyk M. G. (Denver, CO: ), 454–463. [ Google Scholar ]
  • Perry S. (2008). Wikitude: Android App with Augmented Reality: Mind Blow-Ing. Digital Lifestyles. [ Google Scholar ]
  • Radu I. (2012). “Why should my students use AR? A comparative review of the educational impacts of augmented-reality,” in Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on , (IEEE) , 313–314. 10.1109/ISMAR.2012.6402590 [ CrossRef ] [ Google Scholar ]
  • Radu I. (2014). Augmented reality in education: a meta-review and cross-media analysis. Pers. Ubiquitous Comput. 18 1533–1543. 10.1007/s00779-013-0747-y [ CrossRef ] [ Google Scholar ]
  • Riva G. (2018). The neuroscience of body memory: From the self through the space to the others. Cortex 104 241–260. 10.1016/j.cortex.2017.07.013 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Riva G., Gaggioli A., Grassi A., Raspelli S., Cipresso P., Pallavicini F., et al. (2011). NeuroVR 2-A free virtual reality platform for the assessment and treatment in behavioral health care. Stud. Health Technol. Inform. 163 493–495. [ PubMed ] [ Google Scholar ]
  • Riva G., Serino S., Di Lernia D., Pavone E. F., Dakanalis A. (2017). Embodied medicine: mens sana in corpore virtuale sano. Front. Hum. Neurosci. 11 : 120 . 10.3389/fnhum.2017.00120 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Riva G., Wiederhold B. K., Mantovani F. (2018). Neuroscience of virtual reality: from virtual exposure to embodied medicine. Cyberpsychol. Behav. Soc. Netw. 10.1089/cyber.2017.29099.gri [Epub ahead of print]. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Rosenberg L. (1993). “The use of virtual fixtures to enhance telemanipulation with time delay,” in Proceedings of the ASME Winter Anual Meeting on Advances in Robotics, Mechatronics, and Haptic Interfaces Vol. 49 (New Orleans, LA: ), 29–36. [ Google Scholar ]
  • Schmidt M., Beck D., Glaser N., Schmidt C. (2017). “A prototype immersive, multi-user 3D virtual learning environment for individuals with autism to learn social and life skills: a virtuoso DBR update,” in International Conference on Immersive Learning , Cham: Springer, 185–188. 10.1007/978-3-319-60633-0_15 [ CrossRef ] [ Google Scholar ]
  • Schwald B., De Laval B. (2003). An augmented reality system for training and assistance to maintenance in the industrial context. J. WSCG 11 . [ Google Scholar ]
  • Serino S., Cipresso P., Morganti F., Riva G. (2014). The role of egocentric and allocentric abilities in Alzheimer’s disease: a systematic review. Ageing Res. Rev. 16 32–44. 10.1016/j.arr.2014.04.004 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Skalski P., Tamborini R. (2007). The role of social presence in interactive agent-based persuasion. Media Psychol. 10 385–413. 10.1080/15213260701533102 [ CrossRef ] [ Google Scholar ]
  • Slater M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364 3549–3557. 10.1098/rstb.2009.0138 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Slater M., Sanchez-Vives M. V. (2016). Enhancing our lives with immersive virtual reality. Front. Robot. AI 3 : 74 10.3389/frobt.2016.00074 [ CrossRef ] [ Google Scholar ]
  • Small H. (1973). Co-citation in the scientific literature: a new measure of the relationship between two documents. J. Assoc. Inform. Sci. Technol. 24 265–269. 10.1002/asi.4630240406 [ CrossRef ] [ Google Scholar ]
  • Song H., Chen F., Peng Q., Zhang J., Gu P. (2017). Improvement of user experience using virtual reality in open-architecture product design. Proc. Inst. Mech. Eng. B J. Eng. Manufact. 232 . [ Google Scholar ]
  • Sundar S. S., Xu Q., Bellur S. (2010). “Designing interactivity in media interfaces: a communications perspective,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , (Boston, MA: ACM; ), 2247–2256. 10.1145/1753326.1753666 [ CrossRef ] [ Google Scholar ]
  • Sutherland I. E. (1965). The Ultimate Display. Multimedia: From Wagner to Virtual Reality. New York, NY: Norton. [ Google Scholar ]
  • Sutherland I. E. (1968). “A head-mounted three dimensional display,” in Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I , (ACM) , 757–764. 10.1145/1476589.1476686 [ CrossRef ] [ Google Scholar ]
  • Thomas B., Close B., Donoghue J., Squires J., De Bondi P., Morris M., et al. (2000). “ARQuake: an outdoor/indoor augmented reality first person application,” in Digest of Papers. Fourth International Symposium on Wearable Computers , (Atlanta, GA: IEEE; ), 139–146. 10.1109/ISWC.2000.888480 [ CrossRef ] [ Google Scholar ]
  • Ware C., Arthur K., Booth K. S. (1993). “Fish tank virtual reality,” in Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems , (Amsterdam: ACM; ), 37–42. 10.1145/169059.169066 [ CrossRef ] [ Google Scholar ]
  • Wexelblat A. (ed.) (2014). Virtual Reality: Applications and Explorations. Cambridge, MA: Academic Press. [ Google Scholar ]
  • White H. D., Griffith B. C. (1981). Author cocitation: a literature measure of intellectual structure. J. Assoc. Inform. Sci. Technol. 32 163–171. 10.1002/asi.4630320302 [ CrossRef ] [ Google Scholar ]
  • Wrzesien M., Alcañiz M., Botella C., Burkhardt J. M., Bretón-López J., Ortega M., et al. (2013). The therapeutic lamp: treating small-animal phobias. IEEE Comput. Graph. Appl. 33 80–86. 10.1109/MCG.2013.12 [ PubMed ] [ CrossRef ] [ Google Scholar ]
  • Wrzesien M., Burkhardt J. M., Alcañiz M., Botella C. (2011a). How technology influences the therapeutic process: a comparative field evaluation of augmented reality and in vivo exposure therapy for phobia of small animals. Hum. Comput. Interact. 2011 523–540. [ Google Scholar ]
  • Wrzesien M., Burkhardt J. M., Alcañiz Raya M., Botella C. (2011b). “Mixing psychology and HCI in evaluation of augmented reality mental health technology,” in CHI’11 Extended Abstracts on Human Factors in Computing Systems , (Vancouver, BC: ACM; ), 2119–2124. [ Google Scholar ]
  • Zyda M. (2005). From visual simulation to virtual reality to games. Computer 38 25–32. 10.1109/MC.2005.297 [ CrossRef ] [ Google Scholar ]

Accessibility Links

  • Skip to content
  • Skip to search IOPscience
  • Skip to Journals list
  • Accessibility help
  • Accessibility Help

Click here to close this panel.

ERL graphic iopscience_header.png

Purpose-led Publishing is a coalition of three not-for-profit publishers in the field of physical sciences: AIP Publishing, the American Physical Society and IOP Publishing.

Together, as publishers that will always put purpose above profit, we have defined a set of industry standards that underpin high-quality, ethical scholarly communications.

We are proudly declaring that science is our only shareholder.

Quantifying the consensus on anthropogenic global warming in the scientific literature

John Cook 1,2,3 , Dana Nuccitelli 2,4 , Sarah A Green 5 , Mark Richardson 6 , Bärbel Winkler 2 , Rob Painting 2 , Robert Way 7 , Peter Jacobs 8 and Andrew Skuce 2,9

Published 15 May 2013 • © 2013 IOP Publishing Ltd Environmental Research Letters , Volume 8 , Number 2 Citation John Cook et al 2013 Environ. Res. Lett. 8 024024 DOI 10.1088/1748-9326/8/2/024024

Article metrics

1454837 Total downloads

Share this article

Author e-mails.

[email protected]

Author affiliations

1 Global Change Institute, University of Queensland, Australia

2 Skeptical Science, Brisbane, Queensland, Australia

3 School of Psychology, University of Western Australia, Australia

4 Tetra Tech, Incorporated, McClellan, CA, USA

5 Department of Chemistry, Michigan Technological University, USA

6 Department of Meteorology, University of Reading, UK

7 Department of Geography, Memorial University of Newfoundland, Canada

8 Department of Environmental Science and Policy, George Mason University, USA

9 Salt Spring Consulting Ltd, Salt Spring Island, BC, Canada

  • Received 18 January 2013
  • Accepted 22 April 2013
  • Published 15 May 2013

Buy this article in print

We analyze the evolution of the scientific consensus on anthropogenic global warming (AGW) in the peer-reviewed scientific literature, examining 11 944 climate abstracts from 1991–2011 matching the topics 'global climate change' or 'global warming'. We find that 66.4% of abstracts expressed no position on AGW, 32.6% endorsed AGW, 0.7% rejected AGW and 0.3% were uncertain about the cause of global warming. Among abstracts expressing a position on AGW, 97.1% endorsed the consensus position that humans are causing global warming. In a second phase of this study, we invited authors to rate their own papers. Compared to abstract ratings, a smaller percentage of self-rated papers expressed no position on AGW (35.5%). Among self-rated papers expressing a position on AGW, 97.2% endorsed the consensus. For both abstract ratings and authors' self-ratings, the percentage of endorsements among papers expressing a position on AGW marginally increased over time. Our analysis indicates that the number of papers rejecting the consensus on AGW is a vanishingly small proportion of the published research.

Export citation and abstract BibTeX RIS

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence . Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

1. Introduction

An accurate perception of the degree of scientific consensus is an essential element to public support for climate policy (Ding et al 2011 ). Communicating the scientific consensus also increases people's acceptance that climate change (CC) is happening (Lewandowsky et al 2012 ). Despite numerous indicators of a consensus, there is wide public perception that climate scientists disagree over the fundamental cause of global warming (GW; Leiserowitz et al 2012 , Pew 2012 ). In the most comprehensive analysis performed to date, we have extended the analysis of peer-reviewed climate papers in Oreskes ( 2004 ). We examined a large sample of the scientific literature on global CC, published over a 21 year period, in order to determine the level of scientific consensus that human activity is very likely causing most of the current GW (anthropogenic global warming, or AGW).

Surveys of climate scientists have found strong agreement (97–98%) regarding AGW amongst publishing climate experts (Doran and Zimmerman 2009 , Anderegg et al 2010 ). Repeated surveys of scientists found that scientific agreement about AGW steadily increased from 1996 to 2009 (Bray 2010 ). This is reflected in the increasingly definitive statements issued by the Intergovernmental Panel on Climate Change on the attribution of recent GW (Houghton et al 1996 ,  2001 , Solomon et al 2007 ).

The peer-reviewed scientific literature provides a ground-level assessment of the degree of consensus among publishing scientists. An analysis of abstracts published from 1993–2003 matching the search 'global climate change' found that none of 928 papers disagreed with the consensus position on AGW (Oreskes 2004 ). This is consistent with an analysis of citation networks that found a consensus on AGW forming in the early 1990s (Shwed and Bearman 2010 ).

Despite these independent indicators of a scientific consensus, the perception of the US public is that the scientific community still disagrees over the fundamental cause of GW. From 1997 to 2007, public opinion polls have indicated around 60% of the US public believes there is significant disagreement among scientists about whether GW was happening (Nisbet and Myers 2007 ). Similarly, 57% of the US public either disagreed or were unaware that scientists agree that the earth is very likely warming due to human activity (Pew 2012 ).

Through analysis of climate-related papers published from 1991 to 2011, this study provides the most comprehensive analysis of its kind to date in order to quantify and evaluate the level and evolution of consensus over the last two decades.

2. Methodology

This letter was conceived as a 'citizen science' project by volunteers contributing to the Skeptical Science website ( www.skepticalscience.com ). In March 2012, we searched the ISI Web of Science for papers published from 1991–2011 using topic searches for 'global warming' or 'global climate change'. Article type was restricted to 'article', excluding books, discussions, proceedings papers and other document types. The search was updated in May 2012 with papers added to the Web of Science up to that date.

We classified each abstract according to the type of research (category) and degree of endorsement. Written criteria were provided to raters for category (table  1 ) and level of endorsement of AGW (table  2 ). Explicit endorsements were divided into non-quantified (e.g., humans are contributing to global warming without quantifying the contribution) and quantified (e.g., humans are contributing more than 50% of global warming, consistent with the 2007 IPCC statement that most of the global warming since the mid-20th century is very likely due to the observed increase in anthropogenic greenhouse gas concentrations).

Table 1.  Definitions of each type of research category.

CategoryDescriptionExample
(1) ImpactsEffects and impacts of climate change on the environment, ecosystems or humanity'...global climate change together with increasing direct impacts of human activities, such as fisheries, are affecting the population dynamics of marine top predators'
(2) MethodsFocus on measurements and modeling methods, or basic climate science not included in the other categories'This paper focuses on automating the task of estimating Polar ice thickness from airborne radar data...'
(3) MitigationResearch into lowering CO emissions or atmospheric CO levels'This paper presents a new approach for a nationally appropriate mitigation actions framework that can unlock the huge potential for greenhouse gas mitigation in dispersed energy end-use sectors in developing countries'
(4) Not climate-relatedSocial science, education, research about people's views on climate'This paper discusses the use of multimedia techniques and augmented reality tools to bring across the risks of global climate change'
(5) OpinionNot peer-reviewed articles'While the world argues about reducing global warming, chemical engineers are getting on with the technology. Charles Butcher has been finding out how to remove carbon dioxide from flue gas'
(6) PaleoclimateExamining climate during pre-industrial times'Here, we present a pollen-based quantitative temperature reconstruction from the midlatitudes of Australia that spans the last 135 000 years...'

Table 2.  Definitions of each level of endorsement of AGW.

Level of endorsementDescriptionExample
(1) Explicit endorsement with quantificationExplicitly states that humans are the primary cause of recent global warming'The global warming during the 20th century is caused mainly by increasing greenhouse gas concentration especially since the late 1980s'
(2) Explicit endorsement without quantificationExplicitly states humans are causing global warming or refers to anthropogenic global warming/climate change as a known fact'Emissions of a broad range of greenhouse gases of varying lifetimes contribute to global climate change'
(3) Implicit endorsementImplies humans are causing global warming. E.g., research assumes greenhouse gas emissions cause warming without explicitly stating humans are the cause'...carbon sequestration in soil is important for mitigating global climate change'
(4a) No positionDoes not address or mention the cause of global warming 
(4b) UncertainExpresses position that human's role on recent global warming is uncertain/undefined'While the extent of human-induced global warming is inconclusive...'
(5) Implicit rejectionImplies humans have had a minimal impact on global warming without saying so explicitly E.g., proposing a natural mechanism is the main cause of global warming'...anywhere from a major portion to all of the warming of the 20th century could plausibly result from natural causes according to these results'
(6) Explicit rejection without quantificationExplicitly minimizes or rejects that humans are causing global warming'...the global temperature record provides little support for the catastrophic view of the greenhouse effect'
(7) Explicit rejection with quantificationExplicitly states that humans are causing less than half of global warming'The human contribution to the CO content in the atmosphere and the increase in temperature is negligible in comparison with other sources of carbon dioxide emission'

Abstracts were randomly distributed via a web-based system to raters with only the title and abstract visible. All other information such as author names and affiliations, journal and publishing date were hidden. Each abstract was categorized by two independent, anonymized raters. A team of 12 individuals completed 97.4% (23 061) of the ratings; an additional 12 contributed the remaining 2.6% (607). Initially, 27% of category ratings and 33% of endorsement ratings disagreed. Raters were then allowed to compare and justify or update their rating through the web system, while maintaining anonymity. Following this, 11% of category ratings and 16% of endorsement ratings disagreed; these were then resolved by a third party.

Upon completion of the final ratings, a random sample of 1000 'No Position' category abstracts were re-examined to differentiate those that did not express an opinion from those that take the position that the cause of GW is uncertain. An 'Uncertain' abstract explicitly states that the cause of global warming is not yet determined (e.g., '...the extent of human-induced global warming is inconclusive...') while a 'No Position' abstract makes no statement on AGW.

To complement the abstract analysis, email addresses for 8547 authors were collected, typically from the corresponding author and/or first author. For each year, email addresses were obtained for at least 60% of papers. Authors were emailed an invitation to participate in a survey in which they rated their own published papers (the entire content of the article, not just the abstract) with the same criteria as used by the independent rating team. Details of the survey text are provided in the supplementary information (available at stacks.iop.org/ERL/8/024024/mmedia ).

The ISI search generated 12 465 papers. Eliminating papers that were not peer-reviewed (186), not climate-related (288) or without an abstract (47) reduced the analysis to 11 944 papers written by 29 083 authors and published in 1980 journals. To simplify the analysis, ratings were consolidated into three groups: endorsements (including implicit and explicit; categories 1–3 in table  2 ), no position (category 4) and rejections (including implicit and explicit; categories 5–7).

We examined four metrics to quantify the level of endorsement:

  • (1)   The percentage of endorsements/rejections/undecideds among all abstracts.
  • (2)   The percentage of endorsements/rejections/undecideds among only those abstracts expressing a position on AGW.
  • (3)   The percentage of scientists authoring endorsement/ rejection abstracts among all scientists.
  • (4)   The same percentage among only those scientists who expressed a position on AGW (table  3 ).

Table 3.  Abstract ratings for each level of endorsement, shown as percentage and total number of papers.

Position% of all abstracts% among abstracts with AGW position (%)% of all authors% among authors with AGW position (%)
Endorse AGW32.6% (3896)97.134.8% (10 188)98.4
No AGW position66.4% (7930)64.6% (18 930)
Reject AGW0.7% (78)1.90.4% (124)1.2
Uncertain on AGW0.3% (40)1.00.2% (44)0.4

3.1. Endorsement percentages from abstract ratings

Among abstracts that expressed a position on AGW, 97.1% endorsed the scientific consensus. Among scientists who expressed a position on AGW in their abstract, 98.4% endorsed the consensus.

The time series of each level of endorsement of the consensus on AGW was analyzed in terms of the number of abstracts (figure  1 (a)) and the percentage of abstracts (figure  1 (b)). Over time, the no position percentage has increased (simple linear regression trend 0.87% ± 0.28% yr −1 , 95% CI, R 2  = 0.66, p  < 0.001) and the percentage of papers taking a position on AGW has equally decreased.

Figure 1.

Figure 1.  (a) Total number of abstracts categorized into endorsement, rejection and no position. (b) Percentage of endorsement, rejection and no position/undecided abstracts. Uncertain comprise 0.5% of no position abstracts.

Download figure:

The average numbers of authors per endorsement abstract (3.4) and per no position abstract (3.6) are both significantly larger than the average number of authors per rejection abstract (2.0). The scientists originated from 91 countries (identified by email address) with the highest representation from the USA ( N  = 2548) followed by the United Kingdom ( N  = 546), Germany ( N  = 404) and Japan ( N  = 379) (see supplementary table S1 for full list, available at stacks.iop.org/ERL/8/024024/mmedia ).

3.2. Endorsement percentages from self-ratings

We emailed 8547 authors an invitation to rate their own papers and received 1200 responses (a 14% response rate). After excluding papers that were not peer-reviewed, not climate-related or had no abstract, 2142 papers received self-ratings from 1189 authors. The self-rated levels of endorsement are shown in table  4 . Among self-rated papers that stated a position on AGW, 97.2% endorsed the consensus. Among self-rated papers not expressing a position on AGW in the abstract, 53.8% were self-rated as endorsing the consensus. Among respondents who authored a paper expressing a view on AGW, 96.4% endorsed the consensus.

Table 4.  Self-ratings for each level of endorsement, shown as percentage and total number of papers.

Position% of all papers% among papers with AGW position (%)% of respondents% among respondents with AGW position (%)
Endorse AGW 62.7% (1342)97.262.7% (746)96.4
No AGW position 35.5% (761)34.9% (415)
Reject AGW 1.8% (39)2.82.4% (28)3.6

a Self-rated papers that endorse AGW have an average endorsement rating less than 4 (1  =explicit endorsement with quantification, 7  =  explicit rejection with quantification). b Undecided self-rated papers have an average rating equal to 4. c Rejection self-rated papers have an average rating greater than 4.

Figure  2 (a) shows the level of self-rated endorsement in terms of number of abstracts (the corollary to figure  1 (a)) and figure  2 (b) shows the percentage of abstracts (the corollary to figure  1 (b)). The percentage of self-rated rejection papers decreased (simple linear regression trend −0.25% ± 0.18% yr −1 , 95% CI, R 2  = 0.28, p  = 0.01, figure  2 (b)). The time series of self-rated no position and consensus endorsement papers both show no clear trend over time.

Figure 2.

Figure 2.  (a) Total number of endorsement, rejection and no position papers as self-rated by authors. Year is the published year of each self-rated paper. (b) Percentage of self-rated endorsement, rejection and no position papers.

A direct comparison of abstract rating versus self-rating endorsement levels for the 2142 papers that received a self-rating is shown in table  5 . More than half of the abstracts that we rated as 'No Position' or 'Undecided' were rated 'Endorse AGW' by the paper's authors.

Table 5.  Comparison of our abstract rating to self-rating for papers that received self-ratings.

PositionAbstract ratingSelf-rating
Endorse AGW791 (36.9%)1342 (62.7%)
No AGW position or undecided1339 (62.5%)761 (35.5%)
Reject AGW12 (0.6%)39 (1.8%)

Figure  3 compares the percentage of papers endorsing the scientific consensus among all papers that express a position endorsing or rejecting the consensus. The year-to-year variability is larger in the self-ratings than in the abstract ratings due to the smaller sample sizes in the early 1990s. The percentage of AGW endorsements for both self-rating and abstract-rated papers increase marginally over time (simple linear regression trends 0.10 ± 0.09% yr −1 , 95% CI, R 2  = 0.20, p  = 0.04 for abstracts, 0.35 ± 0.26% yr −1 , 95% CI, R 2  = 0.26, p  = 0.02 for self-ratings), with both series approaching approximately 98% endorsements in 2011.

Figure 3.

Figure 3.  Percentage of papers endorsing the consensus among only papers that express a position endorsing or rejecting the consensus.

4. Discussion

Of note is the large proportion of abstracts that state no position on AGW. This result is expected in consensus situations where scientists ' ...generally focus their discussions on questions that are still disputed or unanswered rather than on matters about which everyone agrees ' (Oreskes 2007 , p 72). This explanation is also consistent with a description of consensus as a 'spiral trajectory' in which ' initially intense contestation generates rapid settlement and induces a spiral of new questions ' (Shwed and Bearman 2010 ); the fundamental science of AGW is no longer controversial among the publishing science community and the remaining debate in the field has moved to other topics. This is supported by the fact that more than half of the self-rated endorsement papers did not express a position on AGW in their abstracts.

The self-ratings by the papers' authors provide insight into the nature of the scientific consensus amongst publishing scientists. For both self-ratings and our abstract ratings, the percentage of endorsements among papers expressing a position on AGW marginally increased over time, consistent with Bray ( 2010 ) in finding a strengthening consensus.

4.1. Sources of uncertainty

The process of determining the level of consensus in the peer-reviewed literature contains several sources of uncertainty, including the representativeness of the sample, lack of clarity in the abstracts and subjectivity in rating the abstracts.

We address the issue of representativeness by selecting the largest sample to date for this type of literature analysis. Nevertheless, 11 944 papers is only a fraction of the climate literature. A Web of Science search for 'climate change' over the same period yields 43 548 papers, while a search for 'climate' yields 128 440 papers. The crowd-sourcing techniques employed in this analysis could be expanded to include more papers. This could facilitate an approach approximating the methods of Doran and Zimmerman ( 2009 ), which measured the level of scientific consensus for varying degrees of expertise in climate science. A similar approach could analyze the level of consensus among climate papers depending on their relevance to the attribution of GW.

Another potential area of uncertainty involved the text of the abstracts themselves. In some cases, ambiguous language made it difficult to ascertain the intended meaning of the authors. Naturally, a short abstract could not be expected to communicate all the details of the full paper. The implementation of the author self-rating process allowed us to look beyond the abstract. A comparison between self-ratings and abstract ratings revealed that categorization based on the abstract alone underestimates the percentage of papers taking a position on AGW.

Lastly, some subjectivity is inherent in the abstract rating process. While criteria for determining ratings were defined prior to the rating period, some clarifications and amendments were required as specific situations presented themselves. Two sources of rating bias can be cited: first, given that the raters themselves endorsed the scientific consensus on AGW, they may have been more likely to classify papers as sharing that endorsement. Second, scientific reticence (Hansen 2007 ) or 'erring on the side of least drama' (ESLD; Brysse et al 2012 ) may have exerted an opposite effect by biasing raters towards a 'no position' classification. These sources of bias were partially addressed by the use of multiple independent raters and by comparing abstract rating results to author self-ratings. A comparison of author ratings of the full papers and abstract ratings reveals a bias toward an under-counting of endorsement papers in the abstract ratings (mean difference 0.6 in units of endorsement level). This mitigated concerns about rater subjectivity, but suggests that scientific reticence and ESLD remain possible biases in the abstract ratings process. The potential impact of initial rating disagreements was also calculated and found to have minimal impact on the level of consensus (see supplemental information, section S1 available at stacks.iop.org/ERL/8/024024/mmedia ).

4.2. Comparisons with previous studies

Our sample encompasses those surveyed by Oreskes ( 2004 ) and Schulte ( 2008 ) and we can therefore directly compare the results. Oreskes ( 2004 ) analyzed 928 papers from 1993 to 2003. Over the same period, we found 932 papers matching the search phrase 'global climate change' (papers continue to be added to the ISI database). From that subset we eliminated 38 papers that were not peer-reviewed, climate-related or had no abstract. Of the remaining 894, none rejected the consensus, consistent with Oreskes' result. Oreskes determined that 75% of papers endorsed the consensus, based on the assumption that mitigation and impact papers implicitly endorse the consensus. By comparison, we found that 28% of the 894 abstracts endorsed AGW while 72% expressed no position. Among the 71 papers that received self-ratings from authors, 69% endorse AGW, comparable to Oreskes' estimate of 75% endorsements.

An analysis of 539 'global climate change' abstracts from the Web of Science database over January 2004 to mid-February 2007 found 45% endorsement and 6% rejection (Schulte 2008 ). Our analysis over a similar period (including all of February 2007) produced 529 papers—the reason for this discrepancy is unclear as Schulte's exact methodology is not provided. Schulte estimated a higher percentage of endorsements and rejections, possibly because the strict methodology we adopted led to a greater number of 'No Position' abstracts. Schulte also found a significantly greater number of rejection papers, including 6 explicit rejections compared to our 0 explicit rejections. See the supplementary information (available at stacks.iop.org/ERL/8/024024/mmedia ) for a tabulated comparison of results. Among 58 self-rated papers, only one (1.7%) rejected AGW in this sample. Over the period of January 2004 to February 2007, among 'global climate change' papers that state a position on AGW, we found 97% endorsements.

5. Conclusion

The public perception of a scientific consensus on AGW is a necessary element in public support for climate policy (Ding et al 2011 ). However, there is a significant gap between public perception and reality, with 57% of the US public either disagreeing or unaware that scientists overwhelmingly agree that the earth is warming due to human activity (Pew 2012 ).

Contributing to this 'consensus gap' are campaigns designed to confuse the public about the level of agreement among climate scientists. In 1991, Western Fuels Association conducted a $510 000 campaign whose primary goal was to ' reposition global warming as theory (not fact) '. A key strategy involved constructing the impression of active scientific debate using dissenting scientists as spokesmen (Oreskes 2010 ). The situation is exacerbated by media treatment of the climate issue, where the normative practice of providing opposing sides with equal attention has allowed a vocal minority to have their views amplified (Boykoff and Boykoff 2004 ). While there are indications that the situation has improved in the UK and USA prestige press (Boykoff 2007 ), the UK tabloid press showed no indication of improvement from 2000 to 2006 (Boykoff and Mansfield 2008 ).

The narrative presented by some dissenters is that the scientific consensus is ' ...on the point of collapse ' (Oddie 2012 ) while ' ...the number of scientific "heretics" is growing with each passing year ' (Allègre et al 2012 ). A systematic, comprehensive review of the literature provides quantitative evidence countering this assertion. The number of papers rejecting AGW is a miniscule proportion of the published research, with the percentage slightly decreasing over time. Among papers expressing a position on AGW, an overwhelming percentage (97.2% based on self-ratings, 97.1% based on abstract ratings) endorses the scientific consensus on AGW.

Acknowledgments

Thanks to James Powell for his invaluable contribution to this analysis, Stephan Lewandowsky for his comments and to those who assisted with collecting email addresses and rating abstracts: Ari Jokimäki, Riccardo Reitano, Rob Honeycutt, Wendy Cook, Phil Scadden, Glenn Tamblyn, Anne-Marie Blackburn, John Hartz, Steve Brown, George Morrison, Alexander C Coulter, Martin B Stolpe (to name just those who are not listed as (co-)author to this paper).

Supplementary data

College of Journalism and Communications - University of Florida

  • Refereed Journals and Law Reviews
  • CJC at Major Academic Conferences
  • Ph.D. Dissertations
  • Master’s Theses
  • Master’s Projects

Research and Insights

  • Research Master’s Theses
  • Master’s Projects in Lieu of Thesis
  • Professional Master’s Projects

peer reviewed journals by uf ar researchers

Research Papers Research Papers

To the left you can access all of the great research coming out of the college. Read through published peer-reviewed journal articles, or browse our graduate student dissertations and theses.

Looking for an expert in a particular area? Click the expert button on the top menu.

Connect with the College of Journalism and Communications

Keep up with the latest news about faculty, alumni, friends, and current students. follow us on social media..

A general passion

May 30, 2024

A general passion

Image credit: pgen.1011291

research article

Conserved signalling functions for Mps1, Mad1 and Mad2 in the Cryptococcus neoformans spindle checkpoint

Mps1-dependent phosphorylation of C-terminal Mad1 residues is a critical step in Cryptococcus spindle checkpoint signalling. 

Image credit: pgen.1011302

Conserved signalling functions for Mps1, Mad1 and Mad2 in the Cryptococcus neoformans spindle checkpoint

Recently Published Articles

  • Streptococcus mitis and Streptococcus pneumoniae leads to higher genetic diversity within rather than between human populations">Long-term evolution of Streptococcus mitis and Streptococcus pneumoniae leads to higher genetic diversity within rather than between human populations
  • LINC03045 regulating glioblastoma invasion">CRISPRi screen of long non-coding RNAs identifies LINC03045 regulating glioblastoma invasion
  • Anopheles gambiae : Insight into the cellular dynamics of a novel synthetic sex ratio distorter">Y chromosome shredding in Anopheles gambiae : Insight into the cellular dynamics of a novel synthetic sex ratio distorter

Current Issue

Current Issue May 2024

Adaptations to nitrogen availability drive ecological divergence of chemosynthetic symbionts

The importance of nitrogen availability in driving the ecological diversification of chemosynthetic symbiont species and the role that bacterial symbionts may play in the adaptation of marine organisms to changing environmental conditions.

Image credit: pgen.1011295

Adaptations to nitrogen availability drive ecological divergence of chemosynthetic symbionts

Paramutation at the maize pl1 locus is associated with RdDM activity at distal tandem repeats

pl1 paramutation depends on trans-chromosomal RNA-directed DNA methylation operating at a discrete cis-linked and copy-number-dependent transcriptional regulatory element.

Image credit: pgen.1011296

Paramutation at the maize pl1 locus is associated with RdDM activity at distal tandem repeats

Research Article

Genomic analyses of Symbiomonas scintillans show no evidence for endosymbiotic bacteria but does reveal the presence of giant viruses

A multi-gene tree showed the three SsV genome types branched within highly supported clades with each of BpV2, OlVs, and MpVs, respectively.

Genomic analyses of Symbiomonas scintillans show no evidence for endosymbiotic bacteria but does reveal the presence of giant viruses

Image credit: pgen.1011218

A natural bacterial pathogen of C . elegans uses a small RNA to induce transgenerational inheritance of learned avoidance

A mechanism of learning and remembering pathogen avoidance likely happens in the wild. 

A natural bacterial pathogen of C. elegans uses a small RNA to induce transgenerational inheritance of learned avoidance

Image credit: pgen.1011178

Spoink , a LTR retrotransposon, invaded D. melanogaster populations in the 1990s

Evidence of Spoink retrotransposon's horizontal transfer into D. melanogaster populations post-1993, suggesting its origin from D.willistoni .

Spoink, a LTR retrotransposon, invaded D. melanogaster populations in the 1990s

Image credit: pgen.1011201

Comparison of clinical geneticist and computer visual attention in assessing genetic conditions

Understanding AI, specifically Deep Learning, in facial diagnostics for genetic conditions can enhance the design and utilization of AI tools.

Comparison of clinical geneticist and computer visual attention in assessing genetic conditions

Image credit: pgen.1011168

Maintenance of proteostasis by Drosophila Rer1 is essential for competitive cell survival and Myc-driven overgrowth

Loss of Rer1 induces proteotoxic stress, leading to cell competition and elimination ...

Maintenance of proteostasis by Drosophila Rer1 is essential for competitive cell survival and Myc-driven overgrowth

Image credit: pgen.1011171

Anthracyclines induce cardiotoxicity through a shared gene expression response signature

TOP2i induce thousands of shared gene expression changes in cardiomyocytes.

Anthracyclines induce cardiotoxicity through a shared gene expression response signature

Image credit: pgen.1011164

New PLOS journals accepting submissions

Five new journals unified in addressing global health and environmental challenges are now ready to receive submissions: PLOS Climate , PLOS Sustainability and Transformation , PLOS Water , PLOS Digital Health , and PLOS Global Public Health

COVID-19 Collection

The COVID-19 Collection highlights all content published across the PLOS journals relating to the COVID-19 pandemic.

Submit your Lab and Study Protocols to PLOS ONE !

PLOS ONE is now accepting submissions of Lab Protocols, a peer-reviewed article collaboration with protocols.io, and Study Protocols, an article that credits the work done prior to producing and publishing results.

PLOS Reviewer Center

A collection of free training and resources for peer reviewers of PLOS journals—and for the peer review community more broadly—drawn from research and interviews with staff editors, editorial board members, and experienced reviewers.

Ten Simple Rules

PLOS Computational Biology 's "Ten Simple Rules" articles provide quick, concentrated guides for mastering some of the professional challenges research scientists face in their careers.

Welcome New Associate Editors!

PLOS Genetics welcomes several new Associate Editors to our board: Nicolas Bierne, Julie Simpson, Yun Li, Hongbin Ji, Hongbing Zhang, Bertrand Servin, & Benjamin Schwessinger

Expanding human variation at PLOS Genetics

The former Natural Variation section at PLOS Genetics relaunches as Human Genetic Variation and Disease. Read the editors' reasoning behind this change.

PLOS Genetics welcomes new Section Editors

Quanjiang Ji (ShanghaiTech University) joined the editorial board and Xiaofeng Zhu (Case Western Reserve University) was promoted as new Section Editors for the PLOS Genetics Methods section.

PLOS Genetics editors elected to National Academy of Sciences

Congratulations to Associate Editor Michael Lichten and Consulting Editor Nicole King, who are newly elected members of the National Academy of Sciences.

Harmit Malik receives Novitski Prize

Congratulations to Associate Editor Harmit Malik, who was awarded the Edward Novitski Prize by the Genetics Society of America for his work on genetic conflict. Harmit has also been elected as a new member of the American Academy of Arts & Sciences.

Publish with PLOS

  • Submission Instructions
  • Submit Your Manuscript

Connect with Us

  • PLOS Genetics on Twitter
  • PLOS on Facebook

Get new content from PLOS Genetics in your inbox

Thank you you have successfully subscribed to the plos genetics newsletter., sorry, an error occurred while sending your subscription. please try again later..

Banner

  • UF Libraries
  • Guides @ UF
  • Education Library Guides
  • Dissertations

Licensed for UF students, faculty, and staff.

Requires logon for using off campus. 

  • Academic Search Premier This link opens in a new window This multidisciplinary database provides full text for more than 4,600 journals, including full text for nearly 3,900 peer-reviewed titles.
  • Education Source This link opens in a new window Provides full text of articles from over 350 journals as far back as 1996, in addition to indexing of more than 770 periodicals dating back to 1983.
  • Educator's Reference Complete This link opens in a new window Covers multiple levels of education from preschool to college, and every educational specialty - such as technology, bilingual education, health education, and testing.
  • ERIC This link opens in a new window Provides access to more than 1.4 million bibliographic records of journal articles and other education-related materials.
  • ERIC (ProQuest) This link opens in a new window Provides access to more than 1.4 million bibliographic records of journal articles and other education-related materials.
  • Google for Education Has many resources available for teachers, students, and parents including videos and lesson plans.
  • Health & Psychosocial Instruments (HAPI)( Ovid ) This link opens in a new window Offers comprehensive bibliographic coverage of a wide variety of evaluation and measurement tools for health and psychosocial studies, for practitioners, educators, researchers, and students.
  • JSTOR This link opens in a new window Is an archive of academic journals with issues older than 3-5 years available in full text. It covers many disciplines, including sociology, psychology, population studies, and more.
  • Mental Measurements Yearbook with Tests in Print This link opens in a new window Provides a comprehensive guide to over 2,700 contemporary testing instruments. The series contains information for a complete evaluation of test products within such diverse areas as psychology, education, business, and leadership.
  • PsycINFO (on APA PsycNET online platform) This link opens in a new window Is the only search platform designed specifically to deliver APA content. Search and discover vital behavioral science content efficiently with APA PsycNET's uniquely engineered cross-database searching.
  • PsycINFO (on EBSCOHost online platform) 1887-Current This link opens in a new window The American Psychological Association’s (APA) resource for abstracts of scholarly journal articles, book chapters, books, and dissertations, is the largest resource devoted to peer-reviewed literature in behavioral science and mental health.
  • Professional Development Collection This link opens in a new window Provides a highly specialized collection of nearly 520 high quality education journals, including more than 350 peer-reviewed titles.

Open Access

  • Teaching Channel For Common Core resources, teacher videos, strategies and lesson plans.
  • Web of Science This link opens in a new window Covers the sciences, social sciences, art and humanities. Also includes the Journal Citation Reports (JCR) which contains data on more than 2,600 journals across the social sciences.
  • All UF Databases An A-Z listing of all UF library databases.

Find Dissertations

Use the links below to search for dissertations and masters theses. Many will be available in the UF Libraries collections or online. You can obtain dissertations that are not held at UF by using Interlibrary Loan , or even by purchasing them. If you need help, talk to a Reference Librarian .

University of Florida Dissertations and Theses

  • UF Libraries Catalog
  • UF Electronic Theses & Dissertations

Dissertations and Theses from Other Institutions

UF Subscription

  • Dissertations and Theses (WorldCat Discovery) This link opens in a new window Worldwide search for doctoral dissertations and masters' theses.
  • Center for Research Libraries (CRL) This link opens in a new window
  • << Previous: Education Library Guides
  • Next: Dissertations >>
  • Last Updated: May 3, 2024 11:59 AM
  • URL: https://guides.uflib.ufl.edu/educ

Creative Commons License

IMAGES

  1. Peer reviewed journals list PDF: all subjects

    peer reviewed journals by uf ar researchers

  2. Definition Essay: Peer reviewed journal article example

    peer reviewed journals by uf ar researchers

  3. Tips for Publishing in Peer-Reviewed Journals: Building Your Research Career

    peer reviewed journals by uf ar researchers

  4. How to Identify a Scholarly, Peer-Reviewed Journal Article

    peer reviewed journals by uf ar researchers

  5. Peer-Reviewed Journals

    peer reviewed journals by uf ar researchers

  6. Peer Review process

    peer reviewed journals by uf ar researchers

VIDEO

  1. Luva ya brookeee @Brookemonk #viral #preppyyyy #capcut #edit

  2. DUNE PART 2: (UPTOWN FUNK x NOIR) EDIT

  3. Everyone let’s hit 400 and 500 subscribers in may and I will do this

  4. Prayer to Glorify God in everything we do

  5. #new #reels #uploads #likesharesubscribe 🎶📸🙏

  6. wow

COMMENTS

  1. Articles » George A. Smathers Libraries

    Primo Search Search for materials from the Library Catalog as well as electronic resources such as ebooks, ejournals, and full text articles Databases Find and access articles in our Project Starter databases, General and Multidisciplinary databases, or just browse the UF Libraries' A-Z database list Journals Search or browse online and print journals UF has…

  2. Databases » George A. Smathers Libraries

    Following are other databases that will help you get started in your research - especially when looking for articles. Each database searches thousands of journals and provides links to peer-reviewed and scholarly articles. These do not search UF Library Catalog records. Academic Search Premier. More than 13,600 indexed and abstracted journals.

  3. Journals » George A. Smathers Libraries

    Find Journals Search the library for online journals Citation Linker Useful if you already have an article citation Ulrich's Periodical Directory A listing of over 250,000 periodical titles Journals » George A. Smathers Libraries » UF Libraries » University of Florida

  4. Google Scholar

    Google Scholar provides a simple way to broadly search for scholarly literature. Search across a wide variety of disciplines and sources: articles, theses, books, abstracts and court opinions.

  5. Guides @ UF: Mechanical & Aerospace Engineering: Articles

    engrXiv. engrXiv (engineering archive) is dedicated to the dissemination of engineering knowledge quickly and efficiently. engrXiv was launched in beta in July 2016 under the direction of Devin R. Berg, associate professor of mechanical engineering at the University of Wisconsin-Stout, and in partnership with the Center for Open Science.

  6. Guides @ UF: Introduction to Library Research: Finding Articles

    Most academic journal articles cannot be found using Google and require access through the libraries. The following multidisciplinary library databases cover the humanities, social sciences, and sciences: Includes more than 4,000 peer-reviewed, full-text journals. Access to academic journal articles, books, and primary sources in 75 disciplines ...

  7. Finding Journal Articles

    Includes more than 4,000 peer-reviewed, full-text journals. Access to academic journal articles, books, and primary sources in 75 disciplines. Includes humanities, social science, and science citations. Our Databases page provides even more general databases to begin your research. For strategic searching within a subject field use the library ...

  8. Find » Marston Science Library » UF Libraries

    Search for articles in one of our databases. Articles. Search for articles in journals, newspapers, magazines, & trade publications, subject database for in-depth searching, identify scholarly, peer-reviewed articles . Journals. Browse or search for journals by title, publisher, document type, and subject. Textbook Lending (Course Reserves)

  9. Search

    Published by the UF Center for Undergraduate Research with support from the LibraryPress@UF. | The Florida OJ service is provided through the FLVC Library Services and the George A. Smathers Libraries at the University of Florida. | ...

  10. The Past, Present, and Future of Virtual and Augmented Reality Research

    The dynamics to identify the past, present, and future of AR research cannot be clear yet, but analyzing the relationships between these clusters and the temporal dimensions of each article tracking, education, and virtual city environment are the current areas of AR research. AR is a new technology that is showing its efficacy in different ...

  11. Databases » Health Science Center Libraries

    On campus and VPN access only - courtesy of the College of Medicine Office of Graduate Medical Education. Web of Science - Interdisciplinary database covering journals in the sciences, social sciences, arts & humanities. Includes citation tracking. More details on the quick picks: AccessMedicine - Search engine which helps you find ...

  12. Frontiers

    Augmented Reality (AR) interfaces have been studied extensively over the last few decades, with a growing number of user-based experiments. In this paper, we systematically review 10 years of the most influential AR user studies, from 2005 to 2014. A total of 291 papers with 369 individual user studies have been reviewed and classified based on their application areas. The primary contribution ...

  13. Journal Articles

    Not every article in a peer-reviewed journal is peer-reviewed. Many academic or research journals also include editorials, comments, conference summaries, and reviews that may not be peer-reviewed. Version of Record is the official article after peer-review and layout are completed. Author-accepted manuscript is a peer-reviewed article that is ...

  14. PDF 2023 UF/IFAS Annual Report of Peer-reviewed Journal Articles

    To determine if a specific journal is peer-reviewed, follow these steps: 1. Search: a. From the Marston Science Library webpage https://marston.uflib.ufl.edu/ select "Search the Catalog." b. In the search box, type the title of the journal and click the magnifying glass. 2. Results List: Matching journal titles will appear in a results list ...

  15. Browse Journals and Peer-Reviewed Series

    Browse Journals and Peer-Reviewed Series. Florida Entertainment and Sports Law Review. Florida Journal of International Law. ISSN 1556-2670. ... ISSN 2476-1699. Journal of Technology Law & Policy. ISSN 1087-6995. University of Florida Journal of Law & Public Policy. ISSN 1942-8626 . Enter search terms: Select context to search: Advanced Search ...

  16. Effectiveness of accelerated reader on children's reading ...

    A total of 44 studies from peer-reviewed journal articles and dissertations met the inclusion criteria. Participants included 16,653 students enrolled in elementary, middle, and high school. Hedges' g effect sizes measures suggest pretest-posttest one-group AR studies have moderate effects (g = 0.541) while comparison group AR studies have ...

  17. About the Journal

    Peer Review Process. Each paper goes through a double blind peer-review (UF) process with experts in the discipline of the article topic. ... UF JUR is published by the Center for Undergraduate Research. University of Florida; Journal History. UF JUR publishes one issue in the fall. Current undergraduate students from all majors who work with a ...

  18. The Past, Present, and Future of Virtual and Augmented Reality Research

    The dynamics to identify the past, present, and future of AR research cannot be clear yet, but analyzing the relationships between these clusters and the temporal dimensions of each article tracking, education, and virtual city environment are the current areas of AR research. AR is a new technology that is showing its efficacy in different ...

  19. Quantifying the consensus on anthropogenic global warming in the

    The peer-reviewed scientific literature provides a ground-level assessment of the degree of consensus among publishing scientists. An analysis of abstracts published from 1993â 2003 matching the search 'global climate change' found that none of 928 papers disagreed with the consensus position on AGW (Oreskes 2004 ).

  20. Guides @ UF: First Year Florida: Finding Articles

    Search Tips Search Filters . Use search filters to narrow your search. Each database is different, but look for: Full Text: Limits results to sources that are available in full.(You can read it now). Scholarly / Peer Reviewed: Also called refereed, limits results to articles from peer reviewed journals.These articles are reviewed by experts in the field and considered scholarly sources.

  21. Research Papers

    CJC at Major Academic Conferences. Research Master's Theses. Master's Projects in Lieu of Thesis. Professional Master's Projects. Research Papers Research Papers. To the left you can access all of the great research coming out of the college. Read through published peer-reviewed journal articles, or browse our graduate student ...

  22. PLOS Genetics

    PLOS ONE is now accepting submissions of Lab Protocols, a peer-reviewed article collaboration with protocols.io, ... training and resources for peer reviewers of PLOS journals—and for the peer review community more broadly—drawn from research and interviews with staff editors, editorial board members, and experienced reviewers.

  23. Journal Articles

    The official journal of the Florida Entomological Society, and the first long-published, referreed, natural science journal on the Internet. Open Entomology Journal. A peer-reviewed journal which publishes research articles, reviews, and letters in all areas of experimental and applied entomology including pest management science.

  24. COVID-19 vaccines and adverse events of special interest: A

    1.Introduction. Since declaration of the COVID-19 pandemic by the World Health Organization (WHO) on March 11, 2020 [1] more than 13.5 billion doses of COVID-19 vaccines have been administered worldwide [2].As of November 2023, at least 70.5 % of the world's population had received at least one dose of a COVID-19 vaccine [2].This unparalleled scenario underscores the pressing need for ...

  25. Databases

    This multidisciplinary database provides full text for more than 4,600 journals, including full text for nearly 3,900 peer-reviewed titles. Education Source This link opens in a new window Provides full text of articles from over 350 journals as far back as 1996, in addition to indexing of more than 770 periodicals dating back to 1983.