android security research papers

The Android Platform Security Model (2023)

Research areas.

Security, Privacy and Abuse Prevention

We believe open collaboration is essential for progress

We're proud to work with academic and research institutions to push the boundaries of AI and computer science. Learn more about our student and faculty programs, as well as our global outreach initiatives.

android security Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Android Security Analysis using Mobile Sandbox

Today, smartphones and Android devices are effectively in development like never before and have become the easiest cybercrime forum. It is necessary for security experts to investigate the vengeful programming composed for these frameworks if we closely observe the danger to security and defence. The main objective of this paper was to describe Mobile Sandbox, which is said to be a platform intended to periodically examine Android applications in new ways. First of all in the essence of the after-effects of static analysis that is used to handle the dynamic investigation, it incorporates static and dynamic examination and attempts to justify the introduction of executed code. On the other hand, to log calls to native APIs, it uses those techniques, and in the end, it combines the end results with machine learning techniques to collect the samples analysed into dangerous ones. We reviewed the platform for more than 69, 000 applications from multi-talented Asian international businesses sectors and found that about 21% of them officially use the local calls in their code

Today, smartphones and Android devices are effectively in development like never before and have become the easiest cybercrime forum. It is necessary for security experts to investigate the vengeful programming composed for these frameworks if we closely observe the danger to security and defence. The main objective of this paper was to describe Mobile Sandbox, which is said to be a platform intended to periodically examine Android applications in new ways. First of all in the essence of the after-effects of static analysis that is used to handle the dynamic investigation, it incorporates static and dynamic examination and attempts to justify the introduction of executed code. On the other hand, to log calls to native APIs, it uses those techniques, and in the end, it combines the end results with machine learning techniques to collect the samples analysed into dangerous ones. We reviewed the platform for more than 69, 000 applications from multi-talented Asian international businesses sectors and found that about 21% of them officially use the local calls in their code.

Mitigating Remote Code Execution Vulnerabilities: A Study on Tomcat and Android Security Updates

The android platform security model.

Android is the most widely deployed end-user focused operating system. With its growing set of use cases encompassing communication, navigation, media consumption, entertainment, finance, health, and access to sensors, actuators, cameras, or microphones, its underlying security model needs to address a host of practical threats in a wide variety of scenarios while being useful to non-security experts. The model needs to strike a difficult balance between security, privacy, and usability for end users, assurances for app developers, and system performance under tight hardware constraints. While many of the underlying design principles have implicitly informed the overall system architecture, access control mechanisms, and mitigation techniques, the Android security model has previously not been formally published. This article aims to both document the abstract model and discuss its implications. Based on a definition of the threat model and Android ecosystem context in which it operates, we analyze how the different security measures in past and current Android implementations work together to mitigate these threats. There are some special cases in applying the security model, and we discuss such deliberate deviations from the abstract model.

Android security: State of art and challenges

Android security assessment: a review, taxonomy and research gap study, security apps under the looking glass: an empirical analysis of android security apps, deploying android security updates: an extensive study involving manufacturers, carriers, and end users, modeling android security using an extension of knowledge discovery metamodel, an empirical study of android security bulletins in different vendors, export citation format, share document.

Android enterprise

Read this year’s Android Security Paper for the latest on mobile protections

Oct 17, 2023

As cyberthreats continue to grow, we’ve updated the Android Security Paper to share Android’s latest, proactive security measures.

The world is facing a growing number of cybersecurity challenges, and the cost of cybercrime for organizations can range from tens of thousands of dollars to multiple millions. In fact, a recent FBI report shared that more than 800,000 cybercrime-related complaints were filed in 2022, with losses totaling over $10 billion.

Mobile devices are popular targets for cybercriminals, so it’s essential to put strong mobile security measures in place. Collaboration is also critical to improving mobile security. Developers, device manufacturers, security researchers, vendors, academics and the wider Android community constantly work together to discover and mitigate platform vulnerabilities as part of the Android Open Source Project .

To share and document the latest Android security capabilities, we’ve published an update to the Android Security Paper . The paper provides a comprehensive overview of the platform’s built-in, proactive security across hardware, anti-exploitation, Google Security Services and the range of management APIs available for businesses and governments alike.

For teams implementing a Zero Trust security model , the paper lists over 100 unique signals across more than 30 APIs that are available to use. It also details the many policy controls engineered to prevent data leaks and help improve compliance.

You’ll also find the latest improvements to Work Profile , a unique Android management and isolation capability that creates a dedicated work and personal profile on a single Android device. Regardless of whether an employee or company owns the device, corporate data housed in the work profile stays secure while data in the personal profile stays private.

This year’s update highlights our approach to multi-layered security, such as hardware, operating system, network and application security. We also share more about data protection, industry standards and Android’s security certifications.

Read the entire paper to learn more about our ongoing commitment to provide best-in-class mobile security and further our mission of being the most open, trusted and helpful mobile platform for work. You can also register for our webinar on Wednesday, October 25 for a deep dive on Android 14’s new, advanced security and compliance features for businesses .

Related stories

Browse safely with real-time protection on Chrome

Tackling cybersecurity vulnerabilities through Secure by Design

9 new Android features to help you stay productive

6 Android experiences to see at MWC Barcelona

How AI can strengthen digital security

Working together to address AI risks and opportunities at MSC

Let’s stay in touch. Get the latest news from Google in your inbox.

Android vs iOS Security: A Comparative Study

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• International Journal of Engineering Research & Technology (IJERT)

• Mission & Scope
• Editorial Board
• Peer-Review Policy
• Publication Ethics Policy
• Journal Policies
• Join as Reviewer
• Conference Partners
• Call for Papers
• Journal Statistics – 2023-2024
• Submit Manuscript
• Journal Charges (APC)
• Register as Volunteer
• Upcoming Conferences
• CONFERENCE PROCEEDINGS
• Thesis Archive
• Thesis Publication FAQs
• Thesis Publication Charges
• Author Login
• Reviewer Login

NSDARM – 2020 (Volume 8 - Issue 04)

Analysis and research of system security based on android.

• Article Download / Views: 573
• Authors : Loshima Lohi
• Paper ID : IJERTCONV8IS04003
• Volume & Issue : NSDARM – 2020 (Volume 8 – Issue 04)
• Published (First Online): 17-03-2020
• ISSN (Online) : 2278-0181
• Publisher Name : IJERT

Loshima Lohi

Asst. Professor Carmel College, Mala

Abstract: Android may be a smart mobile terminal operating platform core on Linux. But thanks to its open-source software and programmable framework character, it leads the Android system susceptible to get virus attacks. This paper has deeply researched from the Linux system security mechanism, Android-specific security mechanisms and other protection mechanisms. And on this basis, Android devices have achieved closely guarded on normal state. So that attackers cannot use the kernel module or core library to get highest access permission and be attacked. Meanwhile, to further strengthen the security of Android devices, it enables them to properly handle the high-risk threat. This paper also strengthened intrusion detection system (HIDS) based on the host in order to detect malicious software and strengthen the Android system-level access control.

Keywords Android, System Security

INTRODUCTION

Android is a software stack for mobile devices that has an OS , middleware and key applications. Android SDK is used to develop android applications. It uses Java programming language. It is planned to run on differing types of devices. Android platform is based on Linux technology. It is composed of OS, interface and application components. Its issuance breaks the monopoly status of Microsoft windows mobile OS and Nokias Symbian OS. It allows anyone to develop him own applications. So there's an opportunity that a user is probably going to download and install malicious software's written by software hackers.

ANDROID PLATFORM ARCHITECTURE

Android has built in tools. Android platform composed of Linux kernel, system libraries, android run time, and application framework then on five parts. Android relies on Linux 2.6 version. It provides core system services security, memory management, process management, network group, driven model. The core part is similar to an abstract level between the hardware layer and other software within the systems. Android includes a set of C/C++ libraries. Androids core libraries provide

most of the function to the Java class libraries.

Fig 1: Android Architecture

ANDROID RUNTIME

Android runtime consists of two components. First, a set of core libraries. Second, the Virtual machine Dalvik. Java programs are received and translated by the VM Dalvik. Applications will be encapsulated in Dalvik. A VM is available for every and each program even though some programs are running in parallel.

APPLICATION FRAMEWORK

An application framework is a software framework that's used to implement a typical structure of an application for a selected OS. Any application can publish its own features. These functions can be used by any other application.

Now like most of the main software and operating platforms on the world Android also comes with a software development kit which is termed commonly as Android SDK. Android SDK provides you the API libraries and tools for building and developing new applications on Android operating environment using the java programming language. This procedure of developing the applications on Android platform in java programming language using the tools and API libraries provided by Android SDK is named as Android Application Framework.

BASIC FEATURES SUPPORTED ANDROID APPLICATION FRAMEWORK

Android Application Framework supports the features that made us use and luxuriate in the wide selection of applications for kind of uses. Here are some of the important features:

WebKit engine based integrated browser.

Optimized graphics powered by the advanced graphics library.

SQL for storage of structured data.

For various types of video, audio and image formats media support.

Device emulator, tools for debugging, etc.

In the above mentioned list we did not mention some of the hardware dependant features as these tend to largely vary as per the device, though nevertheless android application framework support them. Some of the device dependant features supported by android include GSM telephony, network connection profiles such as Bluetooth, Edge, 3G, WiFi, utility features such as camera, compass, GPS, etc.

APPLICATIONS

Applications are written in Java programming language. The Android SDK tools compile the code into an android package, an archive file with a .apk suffix. The android software platform comes with a set of basic applications. These applications can run simultaneously.

Android initially came into existence with the sure fire concept that developments are given the ability and freedom to make enthralling Mobile applications while taking advantage of everything that the mobile handset has to offer.

Android is built on open Linux Kernel. This particular software for Mobile Application is formed to be open source, thereby giving the chance to the developers to introduce and incorporate any technological advancement. Build on custom virtual machine android gives its users the addition usage and application power, to initiate an interactive and efficient application and operational Software for your phone.

Googles mobile operating device, the android is its awesome creation within the definitive creation of Software Applications for the mobile arena it also facilitates the g- juice in your mobile thus initiating an entire new world of Mobile Technology experience by its customers.

We at Arokia IT are technically equipped to initiate any level of those amazing software applications using the android genius from Google. Around within the year 2007, Google announced its Android OS and Open Handset Alliance with these two major contributions to the mobile industry that ultimately changed our experience with mobile interface.

OPEN HANDSET ALLIANCE

Open Handset Alliance is an amalgamation of Tech Companies with common and particular interest within the mobile user enhancement experience. Companies like Google, HTC, Motorola, Samsung, Telecom Italia, T Mobile, LG, Texas Instruments also as Sony Ericsson, Vodafone, Toshiba and Hawaii are Tech giant supported their core abilities and strengths, while keeping and pursuing the characters and goals of every company, their basic idea of this joining of hands was the feature-rich mobile experience for the end user. This alliance meant the sharing of ideas and innovation, to bring out these ideas into reality. This provided the millions and millions of Mobile users the experience that they never had.

Like the Apple iphone, Android OS allows third party developers to innovate and build Applications and software for mobile devices. Android is an open, flexible and stable enough to associate itself with newer and newer evolving Technologies. Androids vast range of easy to use tools and wide selection of libraries provides Mobile Application developers with the means of a tremendous mobile operating software to come up with the foremost efficient and rich Mobile Applications changing the world of many mobile users.

A service is a component that runs within the background to perform long-running operations. For example, a service might play music in the background while the user is during a different application, or it'd fetch data over the network without blocking user interaction with an activity.

ANDROID SECURITY

Android's Five Key Security Features:

Security at the OS level through the Linux kernel

Madatory application sandbox

Secure inter process communication

Application signing

Application-defined and user-granted permissions

Android System Security

In the default settings, no application has permission to perform any operations that might adversely impact other applications, the OS, or the user. Androids security mechanism is especially reflected in 2 aspects – Android system security and data security.

Android Security: System-Level Security Features

The Linux kernel provides Android with a group of security measures. It grants the OS a user-based permissions model, process isolation, a secure mechanism for IPC, and the ability to get rid of any unnecessary or potentially insecure parts of the kernel. It further works to stop multiple system users from accessing each others resources and exhausting them.

ANDROID APPLICATION SECURITY FEATURES

This user-based protection allows Android to make an Application Sandbox. Each Android app is assigned a unique user ID, and every runs as a separate process. Therefore, each application is enforced at the method level through the Linux kernel, which doesn't allow applications to interact with each other , and provides them only limited access to the Android operating system. This gives the user permission-based access control, and he/she is presented with an inventory of the activities the Android application will perform and what it'll require to try to to them, before the app is even downloaded. The same goes for file system permissions each application (or user) has its own files, and unless a developer explicitly exposes files to a different Android application, files created by one application can't be read or altered by another.

Android Application Security Scans

When building and testing the safety of Android apps, developers should follow Android security best practices

and keep the following in mind when performing security tests:

Inbound SMS listeners (command and control)

Unsafe file creation

Improper database storage

Unsafe use of shared preferences

Storage of sensitive data on mass storage device

Content provider SQL injection

APN or proxy modification

Android Security: Geared Towards User-Friendly Security

All of Androids more technical security measures are designed to be simply presented to the user, meaning that they will be easily controlled through the interface. Straightforward methods of improving your Android devices security can include: using a password or pin, setting your phone to lock after a period of inactivity, only enabling wireless connections that you use, and only installing Android apps you trust and have personally vetted.

Google also only allows tested and proven secure Android applications into its marketplace, meaning that the user has less of an opportunity of putting in a malicious app. Furthermore, the Android security system prompts the user to permit the installation of an application, meaning that it's impossible to remotely install and run an application. Users can further make sure that their Android device is secure by regularly installing system updates.

Android system security protection

Android system safety inherited the planning of Linux within the design ideology. In practice, each Android application runs in its own process. In the OS, each application runs with a singular system identity. Most of the security functions are provided by the permission mechanism. Permission are often restricted to particular specific process operations. Android is privilege separated. Data security mainly relies on software signature mechanism. It uses AndroidManifest.xml file. When specified software services are called, the system first checks this file. To make use of protected features of the device, one must include in Android Manifest.xml, one or more tags declaring the permissions.

ANDROID ANTI THEFT SECURITY

The ultimate security for Android device just in case it's ever lost or stolen. Advantages of this feature are accurate tracking, encoding , Spy camera activation and Device lock down. It also validates permissions for send SMS messages, hardware controls, take pictures and videos, your location, fine (GPS) location, receive SMS , read SMS or MMS, edit SMS or MMS, full internet access, read contact data and write contact data.

Android Open Source Project. "Security Overview." Tech Info. N.p., 2012. Web. 18 June 2012. http://source.android.com/tech/security/index.html

http://www.arokiait.com/whatis-android.htm

Intelligent Computation Technology and Automation (ICICTA), 2012 Fifth International Conference on 13 February 2012

Transcript of Analysis And Research Of System Security Based On Android Analysis and Research Of System Security Based On Android By Raghunath

Leave a Reply

You must be logged in to post a comment.

MobSF: Open-source security research platform for mobile apps

The Mobile Security Framework (MobSF) is an open-source research platform for mobile application security, encompassing Android, iOS, and Windows Mobile.

MobSF can be used for mobile app security assessment, penetration testing, malware analysis, and privacy evaluation. The Static Analyzer is adept at handling popular mobile app binaries such as APK, IPA, APPX, and source code. The Dynamic Analyzer is compatible with Android and iOS applications, providing a platform for instrumented testing that includes real-time data and network traffic analysis.

MobSF integrates into DevSecOps or CI/CD pipelines facilitated by REST APIs and CLI tools, enhancing your security workflow.

MobSF possesses the capability to conduct static analysis simply through the upload of mobile app binaries. This feature enables even individuals without specialized knowledge to generate security reports for mobile applications.

Additionally, for more experienced mobile security researchers, MobSF offers an interactive dynamic analysis environment. This environment allows for the operation and instrumentation of Android and iOS applications, facilitating real-time security analysis.

“Before MobSF, there were multiple scattered tools available to security engineers. One needs to get experience in these to perform a successful security assessment. MobSF drastically automated many of the tools/processes in this pipeline, making them transparent to the analyst,” Ajin Abraham , Security Researcher and creator of the Mobile Security Framework told Help Net Security.

“In the case of dynamic analysis, it’s always time-consuming to create a VM/device and configure it properly to perform dynamic analysis. We were able to automate all of the environment creation work. To perform dynamic analysis, point a supported VM to MobSF, which will set up the environment, install the agents, configure HTTPs proxy, bypass generic app protections, etc. You can now focus more on testing than spending time on setup and troubleshooting an environment,” Abraham added.

The Mobile Security Framework (MobSF) is available for free on GitHub .

Must read: 15 open-source cybersecurity tools you’ll wish you’d known earlier

More open-source tools to consider:

• CloudGrappler: Open-source tool detects activity in cloud environments
• RiskInDroid: Open-source risk analysis of Android apps
• PyRIT: Open-source framework to find risks in generative AI systems
• BobTheSmuggler: Open-source tool for undetectable payload delivery
• Web Check: Open-source intelligence for any website
• TruffleHog: Open-source solution for scanning secrets
• CVE Prioritizer: Open-source tool to prioritize vulnerability patching
• Fabric: Open-source framework for augmenting humans using AI
• SiCat: Open-source exploit finder
• SOAPHound: Open-source tool to collect Active Directory data via ADWS
• Prowler: Open-source security tool for AWS, Google Cloud Platform, Azure
• Latio Application Security Tester: Use AI to scan your code
• CVEMap: Open-source tool to query, browse and search CVEs
• Faction: Open-source pentesting report generation and collaboration framework
• Adalanche: Open-source Active Directory ACL visualizer, explorer
• AuthLogParser: Open-source tool for analyzing Linux authentication logs
• DriveFS Sleuth: Open-source tool for investigating Google Drive File Stream’s disk forensic artifacts
• Subdominator: Open-source tool for detecting subdomain takeovers
• EMBA: Open-source security analyzer for embedded devices
• code analysis
• cybersecurity
• mobile apps
• mobile devices
• open source

Featured news

• US organizations targeted with emails delivering NetSupport RAT
• Attackers are targeting financial departments with SmokeLoader malware
• Shadow AI is the latest cybersecurity threat you need to prepare for
• Whitepaper: Why Microsoft’s password protection is not enough
• eBook: Defending the Infostealer Threat
• Guide: SaaS Offboarding Checklist
• CISA: Here’s how you can foil DDoS attacks
• Inside the book – See Yourself in Cyber: Security Careers Beyond Hacking

Security Blog

Secure by design: google’s perspective on memory safety.

Google’s Project Zero reports that memory safety vulnerabilities —security defects caused by subtle coding errors related to how a program accesses memory—have been "the standard for attacking software for the last few decades and it’s still how attackers are having success". Their analysis shows two thirds of 0-day exploits detected in the wild used memory corruption vulnerabilities. Despite substantial investments to improve memory-unsafe languages, those vulnerabilities continue to top the most commonly exploited vulnerability classes .

In this post, we share our perspective on memory safety in a comprehensive whitepaper . This paper delves into the data, challenges of tackling memory unsafety, and discusses possible approaches for achieving memory safety and their tradeoffs. We'll also highlight our commitments towards implementing several of the solutions outlined in the whitepaper, most recently with a $1,000,000 grant to the Rust Foundation , thereby advancing the development of a robust memory-safe ecosystem.

Why we’re publishing this now

2022 marked the 50th anniversary of memory safety vulnerabilities. Since then, memory safety risks have grown more obvious. Like others', Google's internal vulnerability data and research show that memory safety bugs are widespread and one of the leading causes of vulnerabilities in memory-unsafe codebases. Those vulnerabilities endanger end users, our industry, and the broader society. We're encouraged to see governments also taking this issue seriously, as with the U.S. Office of the National Cyber Director publication of a paper on the topic last week.

By sharing our insights and experiences, we hope to inspire the broader community and industry to adopt memory-safe practices and technologies, ultimately making technology safer.

Our perspective

At Google, we have decades of experience addressing, at scale, large classes of vulnerabilities that were once similarly prevalent as memory safety issues. Our approach, which we call “ Safe Coding ”, treats vulnerability-prone coding constructs  themselves as hazards (i.e., independently of, and in addition to, the vulnerability they might cause), and is centered around ensuring developers do not encounter such hazards during regular coding practice.

Based on this experience, we expect that high assurance memory safety can only be achieved via a Secure-by-Design approach centered around comprehensive adoption of languages with rigorous memory safety guarantees. As a consequence, we are considering a gradual transition towards memory-safe languages like Java, Go, and Rust.

Over the past decades, in addition to large Java and Go memory-safe codebases, Google has developed and accumulated hundreds of millions of lines of C++ code that is in active use and under active, ongoing development. This very large existing codebase results in significant challenges for a transition to memory safety:

We see no realistic path for an evolution of C++ into a language with rigorous memory safety guarantees that include temporal safety.

A large-scale rewrite of all existing C++ code into a different, memory-safe language appears very difficult and will likely remain impractical.

We consider it important to complement a transition to memory safe languages for new code and particularly at-risk components with safety improvements for existing C++ code, to the extent practicable. We believe that substantial improvements can be achieved through an incremental transition to a partially-memory-safe C++ language subset, augmented with hardware security features when available. For instance, see our work improving spatial safety in GCP's networking stack .

Our investments in memory-safe languages

We are actively investing in many of the solutions outlined in our whitepaper and in our response to the US Federal Government’s RFI on Open Source Software Security .

Android has written several components in Rust over the last few years, leading to compelling security improvements . In Android’s Ultra-wideband (UWB) module, this has improved the security of the module while also reducing the memory usage and inter-procedural calls. 

Chrome has started shipping some features in Rust ; in one case, Chrome was able to move its QR code generator out of a sandbox by adopting a new memory-safe library written in Rust, leading to both better security and better performance.

Google recently announced a $1,000,000 grant to the Rust foundation to enhance interoperability with C++ code. This will facilitate incremental adoption of Rust in existing memory-unsafe code bases, which will be key to enabling even more new development to occur in a memory-safe language. Relatedly, we are also working on addressing cross-language attacks that can occur when mixing Rust and C++ in the same binary.

Google is investing in building the memory-safe open-source ecosystem through ISRG Prossimo and OpenSSF’s Alpha-Omega project . Back in 2021, we funded efforts to bring Rust to the Linux Kernel , which is now enabling us to write memory-safe drivers . This funding is also going towards providing alternatives or upgrades to key open-source libraries in a memory-safe language, such as providing a memory safe TLS implementation .

We know that memory safe languages will not address every security bug, but just as our efforts to eliminate XSS attacks through tooling showed , removing large classes of exploits both directly benefits consumers of software and allows us to move our focus to addressing further classes of security vulnerabilities.

To access the full whitepaper and learn more about Google's perspective on memory safety, visit https://research.google/pubs/secure-by-design-googles-perspective-on-memory-safety/

No comments :

Post a Comment

• #sharethemicincyber
• #supplychain #security #opensource
• android security
• app security
• chrome enterprise
• chrome security
• connected devices
• federated learning
• google play
• google play protect
• interoperability
• iot security
• linux kernel
• memory safety
• Open Source
• pha family highlights
• private compute core
• security rewards program
• supply chain
• targeted spyware
• vulnerabilities

You are viewing this page in an unauthorized frame window.

This is a potential security issue, you are being redirected to https://nvd.nist.gov

You have JavaScript disabled. This site requires JavaScript to be enabled for complete site functionality.

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Information Technology Laboratory

National vulnerability database.

• Vulnerabilities

NIST is currently working to establish a consortium to address challenges in the NVD program and develop improved tools and methods. You will temporarily see delays in analysis efforts during this transition. We apologize for the inconvenience and ask for your patience as we work to improve the NVD program.

Weakness Enumeration

Change history, cve modified by cert/cc 3/22/2024 11:15:12 pm, cve modified by cert/cc 3/16/2024 12:15:07 pm, new cve received by nist 3/15/2024 2:15:08 pm.

March 18, 2024

Too Much Trust in AI Poses Unexpected Threats to the Scientific Process

It’s vital to “keep humans in the loop” to avoid humanizing machine-learning models in research

By Lauren Leffer

Moor Studio/Getty Images

Machine-learning models are quickly becoming common tools in scientific research. These artificial intelligence systems are helping bioengineers discover new potential antibiotics , veterinarians interpret animals’ facial expressions , papyrologists read words on ancient scrolls , mathematicians solve baffling problems and climatologists predict sea-ice movements . Some scientists are even probing large language models’ potential as proxies or replacements for human participants in psychology and behavioral research. In one recent example, computer scientists ran ChatGPT through the conditions of the Milgram shock experiment —the famous study on obedience in which people gave what they believed were increasingly painful electric shocks to an unseen person when told to do so by an authority figure—and other well-known psychology studies. The artificial intelligence model responded in a similar way as humans did —75 percent of simulated participants administered shocks of 300 volts and above.

But relying on these machine-learning algorithms also carries risks. Some of those risks are commonly acknowledged, such as generative AI’s tendency to spit out occasional “hallucinations” (factual inaccuracies or nonsense). Artificial intelligence tools can also replicate and even amplify human biases about characteristics such as race and gender. And the AI boom, which has given rise to complex, trillion-variable models, requires water- and energy-hungry data centers that likely have high environmental costs.

One big risk is less obvious, though potentially very consequential: humans tend to automatically attribute a great deal of authority and trust to machines. This misplaced faith could cause serious problems when AI systems are used for research , according to a paper published in early March in Nature .

On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing . By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

“These tools are being anthropomorphized and framed as humanlike and superhuman. We risk inappropriately extending trust to the information produced by AI,” says the new paper’s co-author Molly Crockett , a cognitive psychologist and neuroscientist at Princeton University. AI models are human-made products, and they “represent the views and positions of the people who developed them,” says Lisa Messeri , a Yale University sociocultural anthropologist who worked with Crockett on the paper. Scientific American spoke with both researchers to learn more about the ways scientists use AI—and the potential effects of trusting this technology too much.

[ An edited transcript of the interview follows. ]

Why did you write this paper?

LISA MESSERI: [Crockett] and I started seeing and sharing all sorts of large, lofty promises of what AI could offer the scientific pipeline and scientific community. When we really started to think we needed to write something was when we saw claims that large language models could become substitutions for human subjects in research. These claims, given our years of conversation, seemed wrong-footed.

MOLLY CROCKETT: I have been using machine learning in my own research for several years, [and] advances in AI are enabling scientists to ask questions we couldn’t ask before. But, as I’ve been doing this research and observing that excitement among colleagues, I have developed a sense of uneasiness that’s been difficult to shake.

Beyond using large language models to replace human participants, how are scientists thinking about deploying AI?

CROCKETT: Previously we helped write a response to a study in [ Proceedings of the National Academy of Sciences USA ] that claimed machine learning could be used to predict whether research would [be replicable] just from the words in a paper.... That struck us as technically implausible. But more broadly, we’ve discovered that scientists are talking about using AI tools to make their work more objective and to be more productive.

We found that both of those goals are quite risky and open up scientists to producing more while understanding less. The worry is that we’re going to think that these tools are helping us to understand the world better, when in reality they might actually be distorting our view.

MESSERI: We categorize the AI uses we observed in our review into four categories: the Surrogate, the Oracle, the Quant and the Arbiter. The Surrogate is what we’ve already discussed—it replaces human subjects. The Oracle is an AI tool that is asked to synthesize the existing corpus of research and produce something, such as a review or new hypotheses. The Quant is AI that is used by scientists to process the intense amount of data out there—maybe produced by those machine surrogates. AI Arbiters are like [the tools described] in the [ PNAS ] replication study [Crockett] mentioned, tools for evaluating and adducting research. We call these visions for AI because they’re not necessarily being executed today in a successful or clean way, but they’re all being explored and proposed.

For each of these uses, you’ve pointed out that even if AI’s hallucinations and other technical problems are solved, risks remain. What are those risks?

CROCKETT: The overarching metaphor we use is this idea of monoculture, which comes from agriculture. Monocultures are very efficient. They improve productivity. But they’re vulnerable to being invaded by pests or disease; you’re more likely to lose the whole crop when you have a monoculture versus a diversity of what you’re growing. Scientific monocultures, too, are vulnerable to risks such as errors propagating throughout the whole system. This is especially the case with the foundation models in AI research, where one infrastructure is being used and applied across many domains. If there’s some error in that system, it can have widespread effects.

We identify two kinds of scientific monocultures that can arise with widespread AI adoption. The first is the monoculture of knowing. AI tools are only suited to answer certain kinds of questions. Because these tools boost productivity, the overall set of research questions being explored could become tailored to what AI is good at.

Then there’s the monoculture of the knower , where AI tools come to replace human thinkers. And because AI tools have a specific standpoint, this eliminates the diversity of different human perspectives from research production. When you have many different kinds of minds working on a scientific problem, you’re more likely to spot false assumptions or missed opportunities.

Both monocultures could lead to cognitive illusions.

What do you mean by illusions?

MESSERI: One example that’s already out there in psychology is the illusion of explanatory depth. Basically, when someone in your community claims they know something, you tend to assume you know that thing as well.

In your paper you cite research demonstrating that using a search engine can trick someone into believing they know something—when really they only have online access to that knowledge. And students who use AI assistant tools to respond to test questions end up thinking they understand a topic better than they do.

MESSERI: Exactly. Building off that one illusion of explanatory depth, we also identify two others. First, the illusion of exploratory breadth, where someone thinks they’re examining more than they are: There are an infinite number of questions we could ask about science and about the world. We worry that with the expansion of AI, the questions that AI is well suited to answer will be mistaken for the entire field of questions one could ask. Then there’s the risk of an illusion of objectivity. Either there’s an assumption that AI represents all standpoints or there’s an assumption that AI has no standpoint at all. But at the end of the day, AI tools are created by humans coming from a particular perspective.

How can scientists avoid falling into these traps? How can we mitigate these risks?

MESSERI: There’s the institutional level where universities and publishers dictate research. These institutions are developing partnerships with AI companies. We have to be very circumspect about the motivations behind that.... One mitigation strategy is just to be incredibly forthright about where the funding for AI is coming from and who benefits from the work being done on it.

CROCKETT: At the institutional level, funders, journal editors and universities can be mindful of developing a diverse portfolio of research to ensure that they’re not putting all the resources into research that uses a single-AI approach. In the future, it might be necessary to consciously protect resources for the kinds of research that can’t be addressed with AI tools.

And what sort of research is that?

CROCKETT: Well, as of right now, AI cannot think like a human. Any research about human thought and behavior, and also qualitative research, is not addressable with AI tools.

Would you say that in the worst-case scenario, AI poses an existential threat to human scientific knowledge production? Or is that an overstatement?

CROCKETT: I don’t think that it’s an overstatement. I think we are at a crossroads around how we decide what knowledge is and how we proceed in the endeavor of knowledge production.

Is there anything else you think is important for the public to really understand about what’s happening with AI and scientific research?

MESSERI: From the perspective of reading media coverage of AI, it seems as though this is some preordained, inevitable “evolution” of scientific and technical development. But as an anthropologist of science and technology, I would really like to emphasize that science and tech don’t proceed in an inevitable direction. It is always human-driven. These narratives of inevitability are themselves a product of human imagination and come from mistaking the desire by some to be a prophecy for all. Everyone, even nonscientists, can be part of questioning this narrative of inevitability by imagining the different futures that might come true instead.

CROCKETT: Being skeptical about AI in science doesn’t require being a hater of AI in science and technology. We love science. I’m excited about AI and its potential for science. But just because an AI tool is being used in science does not mean that it is automatically better science.

As scientists, we are trained to deny our humanness. We’re trained that human experience, bias and opinion have no place in the scientific method. The future of autonomous, AI “self-driving” labs is the pinnacle of realizing that sort of training. But increasingly we are seeing evidence that diversity of thought, experience and training in humans that do the science is vital for producing robust, innovative and creative knowledge. We don’t want to lose that. To keep the vitality of scientific knowledge production, we need to keep humans in the loop.