a quantitative risk assessment methodology for construction project

The Infona portal uses cookies, i.e. strings of text saved by a browser on the user's device. The portal can access those files and use them to remember the user's data, such as their chosen settings (screen view, interface language, etc.), or their login data. By using the Infona portal the user accepts automatic saving and using this information for portal operation purposes. More information on the subject can be found in the Privacy Policy and Terms of Service. By closing this window the user confirms that they have read the information on cookie usage, and they accept the privacy policy and the way cookies are used by the portal. You can change the cookie settings in your browser.

• Login or register account

INFONA - science communication portal

• advanced search
• conferences
• Collections

A quantitative risk assessment methodology for construction project $("#expandableTitles").expandable();

• Contributors

Fields of science

• Bibliography

Sādhanā > 2018 > 43 > 7 > 1-16

It is observed that most of the infrastructure projects fail to meet their cost and time constraints, which will lead to a low return on investment. The paper highlights that the present risk management tools and techniques do not provide an adequate basis for response selection in managing critical risks specific to infrastructure projects. This paper proposes a risk quantification methodology and demonstrates its application for an industrial construction project. A case study is used to present an application of the proposed risk management methodology to help organisations efficiently choose risk response strategy and allocate limited resources. The research adopts an integrated approach to prioritize risks using Group Technique for Order Preference by Similarity to Ideal Solution (GTOPSIS) and to quantify risks in terms of overall project delays using Judgemental Risk Analysis Process (JRAP), and Monte Carlo Simulation (MCS). A comparison between the results of qualitative risk analysis using GTOPSIS and quantitative risk analysis i.e., JRAP and MCS is presented. It is found that JRAP along with MCS could provide some powerful results which could help the management control project risks. The crux of this paper is that the risks are highly dependent on project schedule and the proposed methodology could give a better risk priority list because it considers slackness associated with the project activities. The analysis can help improve the understanding of implications of specific risk factors on project completion time and cost, while it attempts to quantify risks. In turn, this enables the project manager to devise a suitable strategy for risk response and mitigation.

Identifiers

User assignment, assignment remove confirmation, you're going to remove this assignment. are you sure.

Vishal Kumar Gupta

• Indian Institute of Technology, RCG School of Infrastructure Design and Management, Kharagpur, India

Jitesh J Thakkar

• Indian Institute of Technology, Department of Industrial and Systems Engineering, Kharagpur, India

Construction project project risk management judgmental risk analysis process (JRAP) group technique for order preference by similarity to ideal solution and sensitivity analysis (GTOPSIS)

Additional information

• Read online
• Add to read later
• Add to collection
• Add to followed

Export to bibliography

• Terms of service

Accessibility options

• Report an error / abuse

Reporting an error / abuse

Sending the report failed.

Submitting the report failed. Please, try again. If the error persists, contact the administrator by writing to [email protected].

You can adjust the font size by pressing a combination of keys:

• CONTROL + + increase font size
• CONTROL + – decrease font

You can change the active elements on the page (buttons and links) by pressing a combination of keys:

• TAB go to the next element
• SHIFT + TAB go to the previous element

Register for our Life Sciences Capital Benchmarking Webinar here

05 December 2022

The value of scheduling and quantitative risk analysis for your construction project

Navigating the multiple, and often complex, moving parts of any construction project is essential to its timely completion. In a previous article relating to scheduling and project controls fundamentals , it was established that project planning is key to managing uncertainties and risks, especially those resulting from global, geo-political events, such as border closures and labour shortages as a result of the pandemic, or the volatility of commodity prices and construction materials off the back of the Russia-Ukraine conflict.  

In this insight article, John Carleton (Director – ANZ) and Dean Nicholson (Senior Planner) put further emphasis on the value of in-depth scheduling and how this helps to mitigate and minimise risks in your construction project.  

Project lifecycle constraints 

Linesight’s latest commodity report (Q3 2022) highlights that supply chain disruptions resulting from COVID and geopolitical instability issues, combined with high interest rates, inflation and increasing energy costs, continue to impact construction costs and overall viability of some projects. As an example, procurement of certain equipment or materials may now take up to 24 months, which is a huge leap from the usual 12 months or less.  

Similarly, the shortage of skilled labour, particularly of construction workers, continue to result in slower delivery times, thus leading to increased costs for major builds.[1] In fact, skills shortages have been referred to as the ‘Challenge of the Decade’ for the construction industry in Australia, with an anticipated more than 100,000 unfilled roles by next year.[2]  

In addition, construction clients, developers, or general contractors are oftentimes involved in multiple construction projects with simultaneous deadlines, and delays in project delivery timelines will ultimately result in unnecessary costs. 

In-depth scheduling and quantitative risk analysis 

An effective way to ensure a construction project stays on track is through in-depth scheduling through quantitative risk analysis (QRA). The type of information and outputs generated from a QRA can be used to inform key stakeholders of the threats to timelines and enables close and regular monitoring of project risks throughout the project life cycle. [3] However, it is important to note that the QRA will be most effective if the appropriate pre-requisite steps are completed in collaboration with all stakeholders involved in the construction project (clients, contractors, suppliers, consultants, etc.) to ensure all potential risks are accounted for.  

One of the most effective ways to do this is to hold QRA workshops involving all stakeholders to determine and agree the risks, the likelihood of risk occurrence, the potential impact to schedule, cost and performance, and the required strategies to manage these risks. This type of approach allows for quantitative analysis with a probabilistic outcome of the project, based on the qualitative inputs of the wider project team.   

The first step involved is creating a schedule for all stages and aspects of the project, from the procurement of materials and equipment to obtaining the necessary construction permits and approvals, some of which may be subject to the site location. The objective is to create a schedule that provides transparency in terms of the project status, actions required and timeline to achieve completion. The schedule should be developed with the project team through integrated planning sessions, ensuring the steps involved in each stage are fully understood and timelines and durations are validated and agreed by all parties. This allows the identification of high-level pinch points for all stakeholders. 

Once the schedule and timeframes are set, the third step is to capture all potential risks that can impact a project into a risk register. The goal is to identify as many risks as possible, regardless of how critical these are. The best way to achieve this is by running a risk workshop with all project stakeholders in attendance to ensure local expertise, subject matter expert insight, and lessons learned are captured as part of the process. Each risk identified is assigned an owner with responsibility for monitoring the risk and its potential impact on schedule and cost and developing risk mitigation and management solutions. 

Finally, metrics are applied to quantify the risk. Each risk is assessed in terms of the likelihood of it occurring and the level of impact on the project. For example, 5% chance of occurrence but with high impact to delay timeline by X months. Risk ranging is an integral part of the qualitative process within the workshop. It is imperative that the estimated cost and time impact is discussed and agreed with all stakeholders ensuring that the project team ‘owns’ the risk ranging.  

Once all the workshop results are collated, a quantitative analysis on both cost and schedule is conducted using the Monte Carlo simulation using industry-standard risk software applications, to provide a probable statistical outcome of project success against the input dates and costs. Here, we can establish the likelihood of when the project will finish, and how much it is likely to cost when taking account of all the input data from the workshop. 

This process delivers a quantifiable analysis that enables clients to make more informed decisions on how to proceed with the project. Clients can consider whether a different methodology needs to be put in place, whether the team needs to be expanded or indeed whether alternative materials should be considered for the construction.   

QRA workshops should be run at the end of each execution phase or at least on a quarterly basis, to effectively monitor and assess existing and emerging risks and in conjunction with any re-baselining of the project. This type of consistent monitoring and realignment throughout the project life cycle creates a better and more comprehensive understanding of the impact and probability of risks enabling the necessary adjustments and contingencies and overall delivery of certainty in project outcome for the client. 

References: 

[1] Infrastructure Magazine: Skilled construction worker shortage to reach critical levels - https://infrastructuremagazine.com.au/2022/08/19/skilled-construction-worker-shortage-to-reach-critical-levels/  

[2] The Urban Developer: Skills Shortages ‘Challenge of the Decade’ for Australia’s Construction Industry - https://www.theurbandeveloper.com/articles/skills-shortages-challenge-of-the-decade-for-australia-s-construction-industry  

[3] Project Management Institute: Quantifying risk - https://www.pmi.org/learning/library/quantitative-risk-assessment-methods-9929  

Having Field Link issues? Access the temporary work-around link to the right.

• Estimating Link
• Billing Link
• Markets Served
• Try our solutions for free Free Trial ➔
• Knowledge Link
• Try TCLI solution for free? Free Trial ➔
• Leadership Team
• Awards & Recognition

Construction Risk Assessment: What is it and How is it Done?

The roadmap to successful construction risk analysis.

While completing a heavy construction project requires a high level of technical proficiency, it also demands acute awareness of potential risks and the expertise to manage them effectively. This is where construction risk assessment plays an important role. But how exactly do you fully grasp this concept and apply it in real-world scenarios?

At its heart, risk assessment in construction is about proactively exploring ‘what-ifs.’ It involves identifying potential pitfalls before they occur, understanding who could be impacted, and formulating strategic preventive measures. This approach transforms unforeseen challenges into manageable tasks, establishing a framework that not only protects your project’s integrity and financial viability but, most importantly, ensures your team’s safety.

Our guide on construction risk assessment aims to untangle the nuances of this subject in an approachable, easy-to-understand manner. We believe that regardless of your experience level in the construction industry, understanding the fundamentals of risk assessment is key to dealing with the challenges you will face in this sector.

Join us as we break down these concepts, equipping you with the knowledge to turn potential risks into well-managed opportunities.

What is construction risk assessment?

Construction risk assessment is a vital process integral to the safety and success of any project. As outlined by Arizona State University , risk analysis in construction has four iterative stages: identification, assessment, analysis, and mitigation.

The process begins with the identification of risks, which involves a thorough examination of every aspect of a project. It includes evaluating various factors such as site conditions, materials, and work methods. The aim is to pinpoint potential safety issues, environmental impacts, or factors that could cause delays or budget overruns.

Following identification, the process moves to assessment. This step focuses on determining how likely each risk is to occur and the potential severity of its impact. Here, the emphasis is on recognizing risks and understanding their possible effects on the project.

The next stage is analysis. This involves a deeper dive into the identified risks, examining their root causes, interdependencies, and overall context within the project.

Finally, the process concludes with strategizing how to mitigate these risks. This might entail modifying project plans, enhancing safety protocols, or adjusting timelines. The ultimate goal is to either reduce the likelihood of these risks or mitigate their impact if they do materialize.

Identifying risks in heavy construction projects

In the specialized field of heavy construction, which involves large-scale and complex projects, risk identification is essential. The risks faced in heavy construction are varied, typically encompassing safety concerns, environmental impacts, and challenges in project execution.

Safety risks

A top priority in any construction project is the physical safety of workers. Regrettably, the construction industry is notorious for its high rate of work-related injuries and fatalities. According to the Associated Builders and Contractors of Southern California , the industry accounted for 20% of all work-related fatalities in the United States in 2019. Common safety risks include accidents from improper equipment use, falls, structural failures, and exposure to hazardous materials.

Environmental risks

The environmental impact of construction activities is another area of concern. These risks include pollution, harm to ecosystems, and unintended disruption to wildlife habitats. For example, a road construction project can pose threats to nearby sources of water or wildlife. Neglecting to address these risks could not only harm the environment but also lead to legal sanctions.

Project execution risks

These are risks that can derail the project’s timeline or budget. Common issues include delays due to adverse weather conditions, unexpected geological challenges, or logistical hurdles in transporting materials. A notable risk in heavy construction is encountering subsurface obstacles during infrastructure projects. This is especially common in underground utility installations where crews might unearth pre-existing infrastructure or natural obstacles, leading to unanticipated delays and additional costs.

Evaluating and analyzing construction risks

Once the risks a construction project could face are identified, the next steps are evaluating and analyzing these risks. This is where the complexities of each identified risk are thoroughly examined to determine their potential impact on the project.

Qualitative risk analysis

This method evaluates risks based on their probability and impact, typically using a risk matrix. The matrix helps in categorizing risks as high, medium, or low priority, according to their likelihood of occurrence and the potential effect on the project. This approach allows project managers to focus their attention and resources on the most significant risks.

Quantitative risk analysis

Quantitative risk analysis involves numerical methods to estimate the probability and potential impact of risks. This could include techniques like Monte Carlo simulations, which use probability distributions to predict the likelihood of different outcomes, or cost-benefit analysis to weigh the potential impacts against the costs of mitigation strategies.

Risk analysis tools and methods

Several tools and methods are employed in the risk analysis process:

• Risk assessment workshops: These sessions involve team discussions to assess and prioritize risks, drawing on the collective knowledge and experience of the project’s stakeholders.
• SWOT (strengths, weaknesses, opportunities, threats) analysis: This tool is used to assess internal and external factors that could impact the project.
• Checklists and templates: Standardized checklists and templates, such as those from the U.S. Department of Homeland Security , assist in evaluating risks and determining their probability and potential impact.
• Software: Advanced software tools can aid in both qualitative and quantitative risk analysis, streamlining the evaluation and analysis process and allowing project managers to maintain smooth operations.

Implementing risk mitigation strategies

After identifying, evaluating, and analyzing the risks a project faces, the final stage is implementing strategies to mitigate these risks. This phase is about taking proactive steps to address the identified risks, ensuring both legal and contractual compliance, and drawing from real-world examples in the industry.

Developing a risk mitigation plan

A well-structured risk mitigation plan includes specific actions designed to manage the identified risks. Potential measures could range from introducing new safety protocols and revising project schedules to implementing more stringent quality control processes. Importantly, this plan should be dynamic, allowing for adjustments as the project progresses and new risks emerge.

Ensuring legal and contractual compliance

All risk mitigation strategies must comply with legal regulations and contractual agreements. This compliance ensures the safety of workers, the protection of the environment, and adherence to industry standards. Regular audits and reviews can help in maintaining this compliance throughout the project’s lifecycle.

Reviewing case studies and examples

Case studies and hypothetical scenarios provide valuable insights into effective risk mitigation. For instance, the “ Guide to Risk Assessment and Allocation for Highway Construction Management ” by the U.S. Department of Transportation’s Federal Highway Administration presents several hypothetical scenarios. involving the construction of a fictitious highway interchange. These examples offer a comprehensive view of how to apply risk assessment and mitigation strategies effectively in a real-world setting.

Closing remarks on effective construction risk assessment

As we’ve explored the role of risk assessment in construction, its importance in ensuring the safety and success of projects cannot be overstated. Each step, from identifying potential hazards to evaluating and analyzing their impacts, is vital in developing a strong risk mitigation plan. Effectively implementing these strategies ensures compliance with legal and contractual obligations while simultaneously safeguarding the project’s integrity and the well-being of everyone involved.

If you’re seeking a deeper dive into risk assessment in heavy construction projects, the “ Guide to Risk Assessment and Allocation for Highway Construction Management ” by the U.S. Department of Transportation’s Federal Highway Administration is an invaluable resource.

Should you require personalized assistance or have any inquiries, our team at TCLI is always available to help .

With the right approach and resources, assessing the risks in heavy construction can become a manageable, even rewarding, endeavor.

Elevate your Business with Software from the Construction Link, Inc.

Fill out the form to learn how TCLI's software solutions can take your business to the next level.

Related Articles

25 key risks in construction projects

Breaking down the common risks in construction projects.

Achieving success in the competitive...

What is Takeoff in Construction?

Enhancing project planning with construction takeoffs.

Construction takeoff is a fundamental...

Will Construction Costs go Down in 2024?

Predicting the financial climate of the construction industry.

According to the Associated General...

Address: 3394 Sutton Rd. Geneva, NY 14456

© 2010 — 2023, All rights reserved

• Privacy Policy

• Artificial Intelligence & Machine Learning
• Brand Management
• Career Guidance
• Cyber Security
• Data Science and Business Intelligence
• Digital Marketing
• Engineering
• Finance & Accounting
• Internet & Web Development
• Cloud Computing
• Leadership and Coaching
• Learning and Development & Team Training
• Manufacturing
• Communication Management
• Cost Management
• HR Management
• Integration Management
• Procurement Management
• Quality Management
• Risk Management
• Scope Management
• Stakeholder Management
• Time Management
• Agile and Scrum
• Construction Management
• Salesforce Training
• Autocad Tutorials
• Building Information Modeling (BIM)
• Excel Tutorials
• Microsoft Project
• Primavera P6
• Product Reviews
• Other Segments

Quantitative Risk Analysis Tools, Definition, Examples

During the life cycle of a project, team members encounter different types of risks. Those risks often have negative impacts on the project objectives. However, some of them have positive impacts. Risk analysis is an important concept in project management because as a project manager if you don’t know how to identify and evaluate factors that could affect the success of your project, you can not deal with them adequately. Therefore, project managers should have enough knowledge of conducting different types of risk analysis. Most of the time, qualitative risk analysis provides you enough understanding of the characteristics of a given risk. However, in some cases, you need to conduct a perform quantitative risk analysis process to evaluate the risk. In this article, we will discuss both quantitative and qualitative risk analysis definitions, tools, and examples to help you to understand their key differences. Note that quantitative and qualitative risk analysis is a significant concept for PMP or CAPM Certification Exams.

Table of Contents

Definition of Risk

A risk can be defined in various aspects. In project management, risk is any unexpected event that has the potential to affect the project goals – positive or negative. Risks can influence the resources, deliverables, processes, and objectives of a project. Risks can be classified as positive or negative risks according to their impacts. The risk management process includes identification, evaluation, prioritization, and response.

What is Quantitative Risk Analysis?

Quantitative risk analysis focuses on numerically evaluating the influence of project risks on project elements. It is performed to understand the probability and impact of risks on project objectives. For instance, in a construction project, risk analysis can help us to calculate the impact of a project delay such as obtaining a permit late causes a twenty-day delay with a cost of $80,000.

Quantitative risk analysis is an objective tool, that quantifies project risks which are usually prioritized during qualitative risk analysis . It is difficult to perform this risk analysis for all project risks because of time constraints. Therefore it is often performed for the risks that have the highest probability and impact. The main purpose of conducting a risk analysis is to determine the most appropriate strategies to deal with both positive and negative risks.

What is the Purpose of Quantitative Risk Analysis

As discussed above, quantitative risk analysis is a numeric and objective risk analysis tool. It provides detailed information regarding the probability and impact of a given risk. From this aspect, it supports decision-making. Considering critical business decisions, this tool provides more data and information than qualitative analysis.

It helps project managers and business owners to make better duration and cost estimates. This is an important point because, if the estimates are not correct, they will lead to wrong decisions.

When to Use Quantitative Risk Analysis

The process often flows as the following;

• Identifying project risks
• Scaling project risks by using qualitative risk analysis
• Performing quantitative risk analysis for the critical risks

So, when to use this risk analysis

• If your project requires Contingency Reserve to manage schedule and budget risks
• If your project is large and sophisticated and you need better decisions
• If you need more accurate and objective risk analysis for a given risk

Difference Between Qualitative and Quantitative Risk Analysis

The below table summarizes the difference between these two risk analysis

Quantitative Risk Analysis Tools

There are many tools and techniques available to perform quantitative risk analysis. Below are a few of them;

• Decision Tree Analysis: A visual technique that supports decision-making.
• Expected Monetary Value: A statistical technique that helps to compare and evaluate risks. It can be used along with decision tree analysis.
• Three-Point Estimate: Three-point estimate is applicable to both duration and cost estimates. It relies on three different estimates that are Optimistic, Pessimistic, and Most Likely estimates.
• Sensitivity Analysis: Sensitivity analysis is also referred to as what-if or simulation analysis. It is a financial model that identifies how target variables are affected based on changes in other variables.
• Failure Mode and Effects Analysis (FMEA): is the process of analyzing as many components as possible to determine potential failure modes in a system and their causes and effects.
• Monte Carlo Analysis: Monte Carlo Analysis is performed to model the probability of various outcomes in a process that can’t handily be estimated because of the intervention of random variables. It is used along with three-point estimates.

Quantitative Risk Analysis Example

In this risk analysis example, we will use the Expected Monetary Value technique to calculate the project risk exposure and the amount of Contingency Reserve.

If you are preparing for your PMI PMP Certification Exam, you need to understand the basics of risk analysis definitions, tools, and examples. Because PMP aspirants often get confused about how quantitative and qualitative risk analysis differs. In this article, we explained their differences by focusing on quantitative risk analysis definition, tools, and examples. If you want to add or share anything regarding the concept, please use the comments section below.

Cost Benefit Analysis Example

Further Reading

A vital key to effective project management

Francois Simosa is the head of training for the Gragados Training Associates, which provides special project management and risk management training programs.

Risk Management in Binary Options Trading – projectcubicle

Risk Management in Binary Options Trading: How to Minimize Your Losses If you’re interested in binary options trading, managing...

Contingency Reserve vs Management Reserve – projectcubicle

Contingency Reserve vs Management Reserve Have you ever heard contingency budget in project management? Actually, Contingency Reserve vs Management...

Common Risks In Supply Chain Management – projectcubicle

What Are The Most Common Risks In Supply Chain Management? Supply chain management is an essential component of any...

How Cybersecurity and Risk Management Are Related – projectcubicle

How Cybersecurity and Risk Management Are Related ? While the rapid digital transformation has brought forth several benefits for...

SWOT vs PEST Analysis: Which Should You Use? – projectcubicle

SWOT vs PEST Analysis: Which Should You Use? Today we will explore what is PEST analysis definition, SWOT and...

Monte Carlo Simulation Example and Solution – projectcubicle

Monte Carlo Simulation Example and Solution The Monte Carlo Simulation is a quantitative risk analysis technique which is used...

Navigating Investment Risks: Deviation Risk Measures – What is, How is – projectcubicle

Navigating Investment Risks: Deviation Risk Measures Introduction: Why Understanding Risk is Crucial for Investors Whether you’re a beginner investor...

Risk Management Terms and Short Definitions – projectcubicle

Risk Management Terms and Short Definitions Risks are the unforeseen events that may have positive or negative effects on...

Identifying the monetary risks for a project means to know your strengths and weaknesses.

Leave a reply Click here to cancel the reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

This site uses Akismet to reduce spam. Learn how your comment data is processed .

Latest Articles

• 6 Steps to Build Your Employee Value Proposition (EVP) and Attract Top Talent 
• VBA Sleep Function: Pause Your Macro Code
• VBA CDBL Function: How to use the CDBL Function (VBA)
• Advantages of CBAP Certification for Business Professionals – projectcubicle
•  Top 10 Skills Employers Value Most In MBA Graduates – projectcubicle
• Individual Development Plan Examples & Template for Managers – projectcubicle December 24, 2020
• Lessons Learned Template Example& Questions – projectcubicle October 22, 2019
• Project Risk Management Plan Template and Example – projectcubicle July 26, 2019
• A Sample Kickoff Meeting Agenda Template for Projects – projectcubicle July 25, 2019
• Statement of Work Template and Example – projectcubicle May 27, 2019

I want to learn…

Microsoft Excel

• Project Management
• Software Development

Terms of Use

Privacy Policy

The main goal of this site is to provide high quality tutorials and other training resources to help professionals learn project management and improve themselves.

• Business & Management
• IT Service Management
• Learning and Development / Enterprise Team Training
• PMP Certification Training
• Real Estate
• Software Tools

Log in with your credentials

Forgot your details.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Advanced Search
• Journal List
• Environ Health
• PMC10821313

The methodology of quantitative risk assessment studies

Maxime rigaud.

1 Inserm, University of Grenoble Alpes, CNRS, IAB, Team of Environmental Epidemiology Applied to Reproduction and Respiratory Health, Grenoble, France

Jurgen Buekers

2 VITO, Flemish Institute for Technological Research, Unit Health, Mol, Belgium

Jos Bessems

Xavier basagaña.

3 ISGlobal, Barcelona, 08003 Spain

4 Universitat Pompeu Fabra (UPF), Barcelona, 08003 Spain

5 CIBER Epidemiología y Salud Pública (CIBERESP), Madrid, 28029 Spain

Sandrine Mathy

6 CNRS, University Grenoble Alpes, INRAe, Grenoble INP, GAEL, Grenoble, France

Mark Nieuwenhuijsen

Rémy slama, associated data.

Not applicable.

Once an external factor has been deemed likely to influence human health and a dose response function is available, an assessment of its health impact or that of policies aimed at influencing this and possibly other factors in a specific population can be obtained through a quantitative risk assessment, or health impact assessment (HIA) study. The health impact is usually expressed as a number of disease cases or disability-adjusted life-years (DALYs) attributable to or expected from the exposure or policy. We review the methodology of quantitative risk assessment studies based on human data. The main steps of such studies include definition of counterfactual scenarios related to the exposure or policy, exposure(s) assessment, quantification of risks (usually relying on literature-based dose response functions), possibly economic assessment, followed by uncertainty analyses. We discuss issues and make recommendations relative to the accuracy and geographic scale at which factors are assessed, which can strongly influence the study results. If several factors are considered simultaneously, then correlation, mutual influences and possibly synergy between them should be taken into account. Gaps or issues in the methodology of quantitative risk assessment studies include 1) proposing a formal approach to the quantitative handling of the level of evidence regarding each exposure-health pair (essential to consider emerging factors); 2) contrasting risk assessment based on human dose–response functions with that relying on toxicological data; 3) clarification of terminology of health impact assessment and human-based risk assessment studies, which are actually very similar, and 4) other technical issues related to the simultaneous consideration of several factors, in particular when they are causally linked.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12940-023-01039-x.

Introduction

The main aims of environmental health research are i) to identify positive or negative determinants of health-related states (environmental factors in a broad sense encompassing physical, chemical, social, behavioral, systemic factors); ii) to understand the mechanisms underlying the effects of these factors; iii) to quantify the corresponding population impact, which can either be a burden or benefit. This quantification can be done e.g., in terms of number of deaths or disease cases or of healthy years of life lost attributable to the factor or set of factors; and iv) to identify interventions (which can be of all natures and act e.g., on the body, behaviors, knowledge, representations, values, the social, physical and chemical environments, the economy) allowing to limit this impact and preserve or improve the health of populations and limit health inequalities.

In the classical view of the 1983 Redbook of the US National Research Council on risk assessment [ 1 , 2 ], aim i) corresponds to hazard identification and aim iii) to risk assessment. Aim iv) is usually seen as being tackled by health impact assessment (HIA) studies, or analytical HIA studies [ 3 ] but as we will discuss (see "Issues related to terminology" below, last section), from a methodological point of view, the approaches used to tackle aims iii) and iv) are essentially similar. We will therefore use (quantitative) risk assessment to point to studies filling specifically the aim of steps iii) (quantification of population impact of existing factors) and iv) (which corresponds to the quantification of the expected impact of hypothetical policies or interventions). The overall aim of quantitative risk assessment as broadly defined can be described as the quantification of the population impact of any type of factor, exposure, policy or program, hypothesized or already present.

Risk assessment studies typically allow to answer questions such as “How many cases of these diseases are attributable (yesterday/today/tomorrow) to this (exposure) factor or policy?”, or “How many disease cases would be avoided today/in the future if this (exposure) factor was/had been brought to a certain level, or if this policy was/had been implemented, all other things not influenced by this factor or policy being kept identical?”. These questions relate to the consequences of interventions (more precisely about the comparison of counterfactual situations), and not about associations or effects (i.e., hazards, for example: can exposure to this factor cause liver cancer?), as is typical for epidemiological study designs such as cohorts and case–control studies. Such measures of associations, or dose–response functions, are essential but not sufficient to assess the risk. Indeed, dose–response functions alone generally do not allow providing a relevant hierarchy of the disease burden, or impact, associated with each factor: the impact can be higher for an exposure with a mild dose–response function curve than for an exposure with a steep dose–response function, if the first exposure is more frequent than the latter exposure.

For many, if not all risk factors that influence the occurrence of a health-related event, it is not possible to identify if the occurrence of this event in a given subject has been caused by the risk factor; indeed, disease causes generally do not leave a specific (unambiguous) signature in the body, even for strong associations such as the induction of lung cancer by active tobacco smoking. Therefore, one cannot add up cases identified from death certificates or other routine medical data to estimate the disease or mortality burden attributable to an exposure or policy. Similarly, before-after observational studies can be used to document quantitatively health changes in a given population, following e.g., a heat wave, a major air pollution episode [ 4 ], a decrease in air pollution following the temporary closure of an industrial site, an abrupt change in regulation [ 5 ] or an event such as the Olympic Games. However, they are no general solution here; indeed, they may allow to identify a hazard but they are limited to documenting observed (factual) changes and not to studying other (counterfactual) scenarios.

Consequently, one has to rely on more indirect – modelling – approaches. This can be done combining knowledge about the overall frequency of the health parameter considered in the specific population under study, about the distribution of exposure and about dose–response function(s) associated with the factor(s), typically stemming from long-term studies such as cohorts (or animal studies in the case of animal-based risk assessment approaches). Risk assessment studies are related to three research and activity streams: that of the epidemiological concept of population attributable fraction, or etiologic fraction, dating back from the mid-twentieth century [ 6 – 8 ], that of chemical risk assessment derived from toxicological studies [ 9 ], and the practice of environmental impact assessment in relation to a planned policy or project, in which the consideration of the health impacts possibly induced by the planned policy or project has become more frequent, in addition to the consideration of its environmental impacts [ 3 , 10 ].

These quantitative risk assessment studies contribute to integrating and translating knowledge generated from environmental health research in a form more relevant for policy making. They can be used to define or compare risk management strategies or projects, policies, infrastructures of various kinds with possible health consequences (Fig.  1 ). The risk assessment step can be followed by (or include) an economic assessment step, in which the estimated health impact is translated into an economic cost, providing an economic assessment of the impact of the factor or policy considered.

Position of quantitative risk assessment in the process of risk characterization and management. Risk assessment can be used to assess the impacts of the “factors” considered (leftmost box) and of policies aiming at managing and limiting their impacts. Adapted from [ 11 ]

Many risk assessment studies have been conducted in relation to atmospheric pollutants, urban policies, metals, tobacco smoke, alcohol consumption, dietary factors, access to clean water. Although many chemicals are likely to influence health, the consideration of chemical exposures in (human-based) risk assessment studies appears relatively limited [ 12 , 13 ]. The most recent worldwide environmental burden of disease assessment coordinated by the IHME (Institute for Health Metrics and Evaluation, Seattle, USA) considered 87 risk factors and combinations of risk factors, including air pollutants, non-optimal temperatures, lead, unsafe water, sanitation and handwashing, but no other chemical factors except those considered in an occupational setting [ 14 ]. However, the production volume of synthetic chemicals is still increasing and is expected to triple in 2050 compared to 2010 [ 15 ], and widespread human exposure is documented by biomonitoring studies [ 16 , 17 ].

Several reviews about risk assessment and HIA studies have been published [ 10 , 18 – 23 ]. For example, Harris-Roxas et al. provided a narrative of the historical origins of health impact assessment, strengths and opportunities [ 20 ], Nieuwenhuijsen [ 21 ] reviewed issues related to the participatory nature of HIA studies, while Briggs [ 19 ] provided a conceptual framework for integrated environmental health impact assessment. With the exception of issues related to the challenging concepts of etiologic fraction/excess cases [ 7 , 8 ], very few reviews focused on methodological issues related to the technicalities of the execution of the assessment itself, while, with the development of more refined approaches to assess exposures, the identification of a growing number of hazards through toxicological and biomarker-based epidemiological studies or epidemiological studies based on fine-scale (e.g., air pollution) modeling, there is a need to review options and strategies related to input data and handling of uncertainties at each step of risk assessment studies.

We therefore aimed to perform a literature review (see [ 24 ]) of the methodology of quantitative risk assessment studies, discussing sequentially each step posterior to issue framing [ 19 ]. Qualitative health impact assessment approaches and those based on animal (toxicological) dose–response functions were not considered, with the exception of a few points illustrating in which respect the latter diverge from assessments based on human dose–response functions. We conclude by summarizing the identified methodological gaps, in particular related to the handling of emerging factors with partial data, and issues related to terminology.

Key issues and options at each step

Overall methodology of quantitative risk assessment studies.

The main technical steps of quantitative risk assessment, include:

• Definition/identification of the factor(s) (environmental factors/infrastructure/plan/policy) considered;
• Definition of the study area and study population;
• Description of the counterfactual situations compared and of the study period (may be merged with step 1);
• Assessment/description of “exposures” in the study population under each counterfactual scenario;
• Identification of the hazards (health outcomes) induced by the factors considered and of the corresponding dose–response functions and level of evidence;
• Assessment of “baseline” (usually, current) disease frequency or of the DALYs attributable to the health outcomes considered, if needed;
• Quantification of health risk or impact (e.g., in number of disease cases or DALYs);
• Quantification of the social and economic impacts;
• Uncertainty analysis;
• Reporting/communication.

Note that some reviews [ 19 , 25 ] include preparatory or organizational steps, which are not detailed here. Public involvement (not discussed here) can be present at virtually each of these steps. The steps of identification of the exposures, outcomes, area and population considered (numbered 1–3 above) and protocol definition are sometimes referred to as “scoping” step. Note also that the order of some steps is somewhat arbitrary. For example step 2 may come first.

Not all quantitative risk assessment studies follow all of these steps in practice, depending on their scope and the chosen approach. In practice, some studies may stop before completing the risk assessment step – for example many “risk assessment” exercises conducted by national agencies actually only correspond to the identification of hazards associated with the exposure or policy considered without quantification of the corresponding risk, e.g., because robust dose–response functions are lacking (see Sect. "  Issues related to terminology " for further discussion). In case a policy is indeed implemented, a “policy implementation phase”, monitoring the implementation of the policy and possibly its actual impact, is sometimes added.

We will review steps 2 to 9, corresponding to the “appraisal steps” in the WHO terminology [ 25 ].

These steps are depicted in Fig.  2 . We first consider the simple case of a single environmental factor for which exposure levels are available or can be assessed (exposure being here understood in its strict meaning of the contact of an environmental factor with the human body). From the knowledge about exposure in the population under study and various other essential pieces of information detailed below, an estimation of the corresponding health impact will be provided, and generally compared to the impact under an alternative (or counterfactual) scenario (e.g., corresponding to the same population in which exposure is set to zero or another reference value, sometimes called TMREL, or Theoretical Minimum Risk Exposure Level, see Selection of the target scenario(s) – exposure levels). This health impact may be positive or negative, restricted to a specific symptom or disease, or consider several diseases or integrated measures of health such as Disability Adjusted Life Years (DALYs), which integrate years lost due to both death and disability.

Overview of the main steps of risk assessment studies. The starting point of the study (or the counterfactual scenarios) can be formulated in terms of policy, program, project (1), environmental emissions (2), environmental level (3A), behavior (3B), human exposure (4)

Numerous variations around this basic scheme exist: one can start upstream of human exposure (box 4 in Fig.  2 ), for example from the environmental level of the factor (e.g., the amount of contamination of food by metals or the average atmospheric concentration of particulate matter [ 26 ], box 3A), further upstream, considering a source of potentially harmful or beneficial factors (e.g., a factory that emits several pollutants, the presence of green space [ 27 ]; box 2), or a behavior, possibly influencing exposures, such as burning incense, use of tanning cabins, use of electronic screens or cigarette smoking [ 28 ] (box 3B), or a policy/infrastructure or foreseen societal or environmental change (box 1), such as the ban of coal burning or of inefficient woodstoves in a specific area [ 29 ] or of flavored tobacco [ 30 ], the building of a school, or the temperature changes expected from climate change a few decades ahead [ 31 ], this policy, infrastructure or environmental or societal change being either real or hypothetical. In this latter case, some assessment (e.g., though modelling) of the variations of all the chemical, physical, psychosocial factors that may change as a result of the policy may be required, if it is not already available, e.g. as a result of a pre-existing environmental impact assessment study. One can also start downstream of exposure(s), in particular from a body dose or from the organ dose or the excreted level of an exposure biomarker. Depending on the starting point, specific data and modelling may be required, typically to translate the information about this starting point (say, a behavior or the presence of a factory) into an exposure metric that can be translated into a health risk, which will generally be the exposure metric for which reliable dose–response functions exist. Downstream of the health risk assessment, one may wish to also consider economic and other societal impacts.

Compared situations

Principle of counterfactual comparisons.

To estimate the impact of a given factor or policy, one needs to subtract the number of disease cases expected under the hypothesis that the policy is present or the factor(s) has a given distribution in the study population from the number of cases expected in the situation assuming that the policy is absent or different, or that the factor(s) has a different distribution. Either of these situations may correspond to reality as observed at a given time point. For this estimation to be relevant (i.e., for it to correspond to an estimation of the causal effect of the “change” implemented), the comparison has to be done in the same population and at the same time point. Otherwise, trends in the health outcome unrelated to the considered policy or factor may exist and bias the comparison between both situations. As an illustration, comparing the number of air pollution-related diseases in 10 years, after a road infrastructure has been built, with the situation today, in the absence of the infrastructure, would not distinguish the impact on health of changes in the traffic fleet in time from the impact of the infrastructure per se. In the terminology of causal inference theory, this corresponds to considering counterfactual situations. The reliance on counterfactual thinking is a basis for causal inference: the causal effect of a factor A corresponds to the difference between the situation in the presence of A and a (counterfactual) situation in which everything is similar with the exception that A has been removed [ 32 , 33 ]. Note that the epidemiological concept of population attributable fraction developed before the application of counterfactual thinking to the health field, with the possible consequence that the counterfactual scenarios are not always explicit when estimating population attributable fractions.

The comparison can apply to the past, the current period, or to another period in the future. Therefore, the two main options are:

• Counterfactual approach at the current period or in the past: The number of disease cases (or healthy years lost because of specific diseases or another health metric) at the current time t 0 in a hypothetical (“counterfactual”) world similar to the current world but for the fact that the factor is absent, or altered, or the policy has been implemented, is compared to the number of disease cases (or healthy years lost or another health metric) in the real world considered at the same time t 0 ; t 0 needs not to be to the time when the study is performed but can correspond to some time point in the past. We group the situations corresponding to t 0 corresponding to current or past situations because in principle real data on exposures and health can be accessed for the default scenario. See e.g., examples in [ 34 , 35 ].
• Counterfactual approach with future scenarios: Here, the number of disease cases (or healthy years lost because of specific diseases or another health metric) at a specific time t 1 in the future in the study population in which the factor has been removed or altered, or the policy has been implemented, is compared to what would be observed at the same time assuming a specific reference scenario [ 29 , 36 , 37 ]. Time t 1 might correspond to the time when the planned policy is expected to have been implemented, or some time later when a new stationary state is assumed to have been reached.

t 0 and t 1 are usually not time points but time periods over which impacts are summed; they may typically correspond to a one-year period, but may also correspond to a long duration, which may correspond to the life expectancy of the planned infrastructure, in which case the risk assessment may be repeated during each year of the study period to allow integration of the impact and possibly infrastructure or policy costs over this period. This is in particular relevant if the costs and benefits vary over time (the costs being possibly borne in the beginning and the positive impacts reaped after a longer duration). Such an integration allows to provide a relevant average of the yearly impacts and costs. An example is the assessment of the health, economic impact and cost of measures to limit air pollution over the 2017–2045 period, assuming that the measures have been implemented at the start of the study period [ 29 ].

The two options above may actually be combined, by providing estimates of the situation at time t 1 in the future under various scenarios, together with an estimate providing to another earlier time t 0 such as the current time. As an illustration, Martinez-Solanas et al. estimated the impact of non-optimal (both cold and warm) temperatures at the end of the twenty-first century under three greenhouse gas emission scenarios, but also at the historical period (1971–2005), allowing both to compare two possible futures with different levels of action against greenhouse gas emissions, as well as to compare the current situations with some possible futures [ 31 ].

Note that the first situation typically corresponds to attributable fraction calculations [ 7 ] done on the basis of the measure of association (e.g., a relative risk) estimated in an epidemiological study, using the exposure in the population from which the relative risk has been estimated. Also note that the observational study equivalent to the second situation corresponds to what is called the difference-in-differences approach [ 38 ]; in this observational approach, in order to estimate the impact of a real intervention, a community or group experiencing an intervention is compared to itself before the intervention, while another group that did not experience the intervention to correct this before-after comparison for temporal trends in the health outcome of interest).

A specificity of the second (“future”) option relates to the temporal evolutions of the study area. These possibly include changes in the demography (age structure and hence also possibly raw disease risk, as the incidence of most diseases varies with age), in specific disease risk factors besides the one in focus and also possibly regarding the dose response function, in particular for health outcomes very sensitive to societal changes, such as mortality. Such evolutions may be difficult to predict, in particular over periods of several decades or more. Illustrations include studies of long-term effects of ozone depletion [ 37 ] or of climate change, in which sociodemographic changes as well as societal adaptation to high temperatures [ 39 ] are expected.

Selection of the target scenario(s) – exposure levels

The scenarios correspond to the counterfactual situations that one wishes to compare to answer the study aim. One scenario will typically correspond to the current or baseline situation (if one is interested in the effect of a factor present today), or to the extension over time of the current situation, the so-called “business as usual” scenario, if the question asked pertains to a policy or infrastructure that would be implemented or built in the future. The alternative scenario(s) will correspond to the hypothetical situation in which the policy considered has been implemented (the policy being e.g., the construction of an infrastructure, a change in urban design, or a lowering of the pollution levels, if the study aims at quantifying the impact of a specific exposure). Of course, several counterfactual scenarios can be considered and compared (see examples in Table  1 or [ 29 ]).

A series of 10 scenarios compared in a risk assessment study of the impact of atmospheric pollution (fine particulate matter, or PM 2.5 ). From [ 40 ]

a Corresponding to the 5th percentile of PM 2.5 concentration distribution among French rural towns

b The 10th percentile of PM 2.5 exposure by Housing Block Regrouped for Statistical Information (IRIS) in the study area (corresponding to 10.3 and 12.4 µg/m 3 in Grenoble and Lyon conurbations, respectively)

c Baseline corresponds to the PM 2.5 exposure average for the 2015–2017 period, taken as a reference in the present study

d Mortality reduction targets expressed as a proportion of the non-accidental death cases attributable to PM 2.5 exposure that can be prevented under the scenario S2: “No anthropogenic PM 2.5 emissions”

e S6: -2.9 and -3.3 µg/m 3 in Grenoble and Lyon conurbations, respectively; S7: -4.4 and -5.1 µg/m 3 ; S8: -6.0 and -6.9 µg/m3

f Inspired by the 2008/50/EU Directive, which targets relative PM 2.5 yearly average decreases to obtain by 2020. The decrease value depends on the exposure average for the last three years (2015–2017): -15% in the case of Grenoble and Lyon conurbations

g The 90th percentile corresponded to 16.0 and 17.4 µg/m 3 in Grenoble and Lyon conurbations, respectively

An essential question here, for actions on one or several specific factors, relates to the targeted levels (or distribution) of the factor. In the case of a study aiming to quantify the current impact of a factor that has monotonic effects on all diseases considered, the counterfactual situation in which no one is exposed to the factor can be considered. This would correspond to the ban of the substance or the behavior considered, assuming that the substance is not persistent in the body or the environment, and that the compliance to the regulation is perfect. Other scenarios are however worth considering in specific situations; in particular, one may wish to consider levels strictly higher than zero as an alternative to the current situation if the factor corresponds to a substance persistent in the body or the environment (so that a ban would not lead to the immediate disappearance of the pollutant, as is the case for DDT [dichlorodiphenyltrichloroethane] or PCBs [polychlorinated biphenyls]), if it has both human and natural sources (as is the case for particulate matter, which are also emitted by volcanic eruptions) or if exposure to the factor does not have a monotonic association with disease occurrence (as is the case for outdoor temperature, exposure to sun radiations or level of essential elements in the body). The methodology of the Global burden of disease (GBD) project coordinated by IHME refers to a Theoretical Minimum Risk Exposure Level (TMREL) defined as the exposure level that “minimizes risk at the population level, or […] captures the maximum attributable burden” [ 14 ]. Alternatives exist, such as considering a feasible minimum (which may however require a specific approach in order to be rigorously identified), or specific existing guideline levels (e.g., WHO air quality guidelines [ 41 ]).

In the case of particulate matter (PM), studies in the early 2000s typically used WHO PM guideline value (then 10 µg/m 3 for PM 2.5 ) as the target, which then seemed a remote target – although this value did not correspond to a “no-effect level”, which has not been evidenced for PM 2.5 . Today, as exposures in many Western cities decreased close or below 10 µg/m 3 , lower reference values are often chosen, such as the 2021 WHO guideline value of 5 µg/m 3 , the lowest observed value, or the fifth percentile of the values observed across the cities of the country (which, in the case of PM, will lead to very different reference values in India and Canada, making between-country comparisons difficult), or an estimate of the levels that would be observed in the absence of anthropogenic source of PM. Since the “target” value has of course generally a very large impact on estimates, it is crucial for it to be explicitly quoted when summarizing the study results. Mueller et al. [ 36 ] give the example of a policy that would simultaneously target noise, air pollution, green space and heat exposure, as well as physical activity, taking for each exposure factor levels internationally recommended (in general by WHO) as counterfactual scenario (see Figure S 1 ).

Considering environmental and societal side effects of policies

Ideally, the counterfactual scenarios should consider possible side effects and remote consequences of the considered intervention. For example, a risk assessment of a ban of bisphenol A (totally or for specific uses) could consider several counterfactual scenarios, one in which the consumer products containing bisphenol A are simply banned (possibly trying to consider the societal cost of such a ban if the products product a benefit to society), and others in which bisphenol A is replaced by other compounds, including some that also have possible adverse effects, such as bisphenol S. Studies on the expected impacts of climate change mitigation strategies, such as the limitation of fossil fuels use, may consider health effects due to the expected long-term improvement in climate, but also those related to changes in air pollution levels possibly resulting from the limitation of fossil fuels use, and also the possible consequences of increases in physical activity if this limitation of fossil fuels is expected to be followed by shifts from individual cars to other more active modes of transportation.

Study area and population

The study area should be coherent with the policy or factor considered, trying to include the whole population likely to undergo positive or negative impacts from the factor or policy. Its choice should also take into account the entity (institution, community, administration, population…) able to take a decision regarding this policy. Choosing an area larger than that targeted by a policy makes sense, as it may allow to consider unplanned effects on the surrounding areas (for example, in the case of a policy planning to ban the most polluting vehicles from a city, an increase in the traffic of these vehicles in the surrounding cities), and to provide estimates specific to various sub-areas, which are also relevant because sometimes the exact area concerned with a possible policy is not always decided a priori – in which case the study may help making this decision. However, one should also keep in mind that considering a study area larger than that in which the policy will be implemented may entail possible dilution effects – i.e., the impact may appear lower than it is actually in the population targeted by the policy, if expressed on a multiplicative scale, that is, as a change in the proportion of deaths or DALYs in the area. When considering a policy decided at the city level, estimating the health impact in the city and possibly the surrounding ones is for example relevant; for a European policy, one may consider the whole Europe, or a (possibly random) group of regions if limiting the size of the study population limits bias, costs or uncertainties, which is not always the case for risk assessment studies contrarily to field studies.

In addition to factors related to health (see Assessment of disease frequency) and to exposure to the factor or policy considered (such as possibly fine-scale data on population density), it is usually relevant to collect information on population size, sociodemographic (documenting e.g., age and social-category distribution) and behavioral factors. Although seldom done in practice in the context of risk assessment studies, it is worth considering conducting a specific survey to document specific characteristics of the study population not available in administrative or health databases. For example, if the intervention may as a side effect impact physical activity (which has a non-linear relation to health), it is useful to document the distribution of physical activity in the population.

Exposure (risk factors) assessment

Identification of exposures.

In the simple case of a risk assessment study considering a single pre-identified factor A, before assessing the impact (Health impact), one has to:

• assess the expected exposure to factor A in the study population under the considered counterfactual scenarios (see Exposure assessment tools – general considerations to Reliance on exposure biomarkers);
• identify all health endpoints and possibly biological parameters H 1 , H 2 … H i that can or could be influenced by A (see Identifying all health-related events possibly influenced by the considered exposures);
• assess the level of evidence regarding the possible effect of A on H 1 , of A on H 2 … H i (see Estimating the strength of evidence about the effect of factors on health);
• assess the incidence of the health endpoints in the considered population, and the dose response function of all exposure-outcome pairs (A, H 1 ), (A, H 2 )… (A, H i ) (see Exposure-response functions).

As described in Fig.  2 , the level of intervention can correspond to either the emission of factor A (e.g., what a chemical plant is allowed to emit on a yearly basis, box 1), its concentration in a milieu (e.g., air, water, food, box 3A), human exposure to A (referring, strictly speaking, to the contact of humans with the factor, integrating both the duration of the contact and the level of the compound, box 4); A may also correspond to a behavior (e.g., having sexual intercourse without using condoms, box 3B). We will here refer to all of these situations with the same simplifying terminology of exposure in the loose sense, keeping in mind that this differs from its strict definition given above, and that depending on the situation, one may have to do some modelling to translate the intervention into a metric compatible with the exposure response function to be used (see Exposure assessment tools – general considerations below).

If the starting point of the study is now a family of factors (e.g., endocrine disruptors, or environmental factors, as in the case of Environmental burden of disease assessments), then one may first have to list/identify which factors fall under this definition. This step can in practice be challenging and may require a specific technical study.

If the study aims at assessing the impact of a policy, one has to first analyze if and how the policy may translate in terms of environmental (understood in the broad sense) impacts – that is, identify which chemical, physical and social factors A 1 , A 2 … A j may be influenced by the policy, and quantify the expected amplitude of the variations in these factors once the policy has been implemented. For example, if one aims at estimating the impact of banning a fraction of gasoline- and diesel-powered cars or trucks from an area, then one will generally have to rely on atmospheric dispersion models to estimate which changes in specific atmospheric pollutants will be induced by this ban, in addition to consider other consequences of the ban, e.g. related to physical activity. This actually corresponds to a study in itself (an environmental and social impact assessment), which may be already available, possibly because of legal requirements. One would then have to perform the three steps listed above (see end of Study area and population) for each of the factors A 1 , A 2 … A j , influenced by the policy, which may imply to assess and synthesize the evidence regarding a large number of exposure-outcome pairs (A i , H j ).

The consideration of several risk factors has implications in the way the health impact is estimated, which are discussed in Consideration of multiple risks factors below.

Exposure assessment tools – general considerations

Whatever the starting point of the study (i.e., the targeted intervention or factor), the estimation of the health impact should ideally rely on some estimate of the exposure metric coherent with the dose–response function considered (see Exposure-response functions below), which should itself be chosen to minimize the uncertainties and bias in the final risk estimate. If, for example, the evaluated intervention corresponds to the closing of a plant emitting hazardous gases, one could attempt estimating the spatial distribution of the air concentration of the corresponding gases in the area surrounding the plant, averaged over a relevant time period, to convert this spatial distribution into an estimate of population exposure taking into account the spatial distribution of density of the target population (e.g., general population or any specific subgroup) in the study area, and possibly any relevant information on the time–space activity budget of the local populations, and then estimate the population risk from this estimated distribution of population exposure and a dose–response function chosen in coherence with this exposure metric. Similarly, if the intervention aims at changing a behavior such as having unprotected sexual intercourse, one would ideally need to obtain estimates of such behaviors before and after the hypothetical intervention, to provide an estimate of the incidence of a sexually transmitted disease.

All the tools of exposure science and etiological epidemiological studies can in principle be used to assess exposures, from environmental models to questionnaires, dosimeters and biomarkers, keeping in mind that they differ in terms of resolution, accuracy, cost, potential for bias…

As risk assessment studies are expected to provide an estimate relevant in a given population, representativeness of the group in which exposure is assessed with respect to this target population is a desired feature, contrarily to etiological epidemiological studies, for which representativeness is usually not required [ 42 ]. For this reason, “simpler” rather than more accurate and cumbersome approaches to exposure assessment than those used in etiological studies may be preferred, since the later, although more accurate at the individual level, may entail selection bias and thus have in some instances more limited validity at the population level. Consequently, environmental models or data, which may be developed without always requiring contact with the population, are very frequently used in risk assessment studies of environmental factors; we discuss below issues related to the spatial resolution of such models without entering into details of models development and validity not specific to risk assessment studies [ 43 ]; for behaviors, questionnaires may be used, while for many chemical exposures, data from biomarkers or exposure models may be preferred.

In addition to issues related to the validity of the exposure metric itself, errors may arise because of participation (selection) bias induced by the selection of the population for which this metric is available (e.g., answering a questionnaire or providing biospecimens to assess exposure biomarkers). Simply identifying a representative sampling base may be challenging in some areas if no relevant public database exists or if access cannot be granted; even if such a relevant sampling base exists, a participation rate of 100% cannot be expected, and refusing to participate can a priori be expected to be associated with sociodemographic characteristics and possibly with the exposure of interest. Tools such as reweighing approaches may be used to limit the impact of such selection bias on the estimated exposure distribution.

Again, whatever the approach used, the estimate of exposure provided should be coherent with the hazard identified and the dose response function to be used in the following steps (i.e., a dose–response function based on yearly exposure averages should not be combined with a weekly estimate of exposure).

Issues related to the assessment of environmental factors through questionnaires

Questionnaires may be used to assess behaviors, such as the frequency of use of specific modes of transportation (which would be relevant if one aims to quantify the impact of a policy entailing a shift in the share of specific modes of transportation), diet, smoking or physical activity and also psychosocial factors. Just like for environmental models, validation studies are particularly relevant to discuss and possibly quantify (see below, sensitivity analyses, Sensitivity and uncertainty analyses) any bias possibly induced by the questionnaire used, keeping in mind that the validity of questionnaires may be population-specific, as the validity of replies will depend on the social desirability of the considered behavior (for example, replies to questionnaires on alcohol consumption may be dependent on the social perception of alcohol consumption in the given cultural background), as well as evolve over time in a given society, limiting the validity of temporal comparisons.

When it comes to assessing exposure to specific chemical or physical factors, questionnaires may be very limited. Indeed, one is generally not aware of one’s own exposure to a chemical, in particular if exposure occurs through several routes (e.g., in the case of bisphenol A, PCBs, or specific pesticides), and cannot provide any quantitative estimate of her/his own exposure; additionally, one’s perception of exposures may be strongly influenced by social or psychological factors (e.g., one’s perception of the noxiousness of the factor, or the existence of a disease that one considers to be imputable to the considered exposure), which may bias the estimated impact (assuming that perception is not of interest by itself). As an illustration, a European study showed limited agreement of one’s self-declared exposure to traffic with an objective assessment of exposure, to extents that varied between countries [ 44 ]. The fact that questionnaires alone are typically limited to assess exposures to chemical and physical factors does not imply that they cannot be used in combination with other sources of information to provide a relevant exposure estimate (see Reliance on environmental models and surveys below). Moreover, questionnaires (including those relying on smartphone-based applications) are essential to assess behavioral factors such as dietary patterns, alcohol or tobacco consumption, physical exercise, transportation patterns, sexual activity… These often rely on data collected independently from the health impact assessment study, but there is no reason for investigators planning such a study not to envision an ad-hoc questionnaire survey.

Reliance on environmental models and surveys

In some areas, public authorities or other institutions developed “databases” regarding specific factors that may be a relevant source of information about the exposure of interest. These databases may be built from various sources and approaches. These include data on sources of specific hazards (e.g., maps on the location of pollution sources or on the chemical composition of cosmetics or other consumers’ products), environmental monitoring (e.g., databases on water contamination (typically based on measurements in the water distribution network, job-exposure matrices in an occupational setting or models), corresponding to boxes 2, 3A and 3B of Fig.  2 (the case of human biomonitoring is discussed specifically below). When available, databases on sources and environmental models and measurements have the advantage of being possibly representative of a specific milieu, in particular if they are based on environmental measurements or models, which can more easily be developed on a representative basis than measurements implying the participation of human subjects; they also have the advantage to rely on information often not known to the inhabitants of the study area (e.g., the level of contamination of the air or drinking water or food by specific compounds) and to possibly cover large spatial areas and extended temporal windows. They may provide source-specific information, which may be relevant if the health impact assessment study considers a possible intervention limited to a specific source of exposure; for example, atmospheric pollution dispersion models typically combine emissions from urban heating, traffic, industry… and may be used to predict environmental levels that would be observed assuming that emissions from one or several specific sources are decreased [ 29 ]. They possibly allow avoiding directly contacting individuals to collect information on their exposure, although in many situations it is actually relevant to combine such environmental models with questionnaire data: for example, questionnaires are essential to combine models or data on food contamination or dietary characteristics with individual information on food consumption patterns [ 45 ], models or measurements in drinking water benefit from information on water use (sources and amount of water drunk, frequency and temperature of baths and showers… [ 46 ], and atmospheric pollution models may be relevantly combined with individual information on time space activity, to integrate individual exposures across the places where each individual spends time [ 47 , 48 ]. Indeed, environmental models typically provide an estimate of an environmental level and not an estimate of personal exposure in the strict meaning. Other issues related to their use is that such models do not exist for all environmental factors and areas and that their spatial resolution may be limited.

Environmental models—Issues related to spatial scale

Indeed, the spatial resolutions of such models vary between models. In the case of atmospheric pollutants, early risk assessment studies typically relied on data from background monitoring stations, which cover a generally small fraction of territories and are sometimes distant by several kilometers one from another. In addition, background monitoring stations are by definition located in “background” sites, which are located far (typically, a couple hundred meters or more) from air pollution “hot spots” (see Fig.  3 ). On the one hand, relying on data from background monitoring stations may underestimate health impacts, as these background stations are not meant to represent people living or spending time close to air pollution hot spots such as industrial sources or high-traffic roads. On the other hand, stations located close to specific sources to monitor their activity are not meant to provide an estimate valid for a large area.

Cross-sectional variations of fine particulate matter (PM 2.5 ) throughout the urban area of Lyon, as estimated from a fine-scale dispersion model, and typical locations of background permanent monitoring stations (black circles). Adapted from [ 40 ]

In the last decades, models providing a finer-scale resolution, such as geostatistical models based on measurement campaigns in a large number of points, land-use regression or dispersion models were developed [ 49 ]. These have a much finer spatial resolution (Fig.  4 ). In a study in two French urban areas, assessing exposures from background monitoring stations entailed an underestimation of the mortality attributable to fine particulate matter by 10 to 20%, compared to fine-scale models with spatial resolutions taking into account variations in the pollutants’ concentration with a spatial scale of about 10 m [ 34 ]. Identifying the most relevant approach for risk assessment purposes is not straightforward. Even some fine-scale approaches may entail some error, as these may represent a “smoothed” version of the actual spatial contrasts in air pollution, and as smoothing typically entails a poor representation of the extreme values. These models with a very fine spatial resolution may be limited in terms of temporal resolution, which may be an issue for some health outcomes. Moreover and maybe counter-intuitively, relying on spatially very fine models may not be desirable in risk assessment studies in which there is no available data on the time–space activity of individuals. Indeed, if, in the absence of such individual time–space activity data, one has to assume that individuals are exposed at the concentration assessed at the location of their home address, then models that tend to smooth concentration over rather large areas may be more accurate for the purpose of risk assessment than models very fine spatially used ignoring the other places where individuals spend time.

Spatial resolutions of various air pollution (nitrogen dioxide) exposure models developed in a middle size city. a  Estimates based on permanent background monitoring stations; b  geostatistical model relying on a fine-scale measurement campaign; c  dispersion model taking into account emission and meteorological conditions; d  Land-use regression model relying on the same measurement points as geostatistical model ( b ) [ 50 ]

Integrating environmental models with data on population density

As already mentioned, environmental models do not provide an estimate of the population exposure in the strict sense, if only because population density varies with space so that simply averaging the environmental levels over the study area, which gives the same weight to each location (and is equivalent to assume that the population is homogeneously distributed spatially across the study area) may poorly approximate population exposure. Getting closer to population exposure may imply to combine the estimated environmental level with data on population density (i.e., weighting concentrations with population density), which will allow considering the fact that the population is not evenly distributed in a given area. Kulhánová et al. [ 51 ] have illustrated these issues in a study of the lung cancer risk attributable to fine particulate matter exposure in France. Compared to a model that took into account PM 2.5 exposure at a 2 km resolution and population density, a model ignoring the spatial distribution of homes within each département (geographical units of 200,000 to one million inhabitants) underestimated the population attributable fraction by about one third; when the variations in population sizes between départements was ignored, so that one assumed that everyone in the country was exposed to the median level observed at the country level, the estimated population attributable fraction was divided by 3.6, compared to the original one taking into account population density and fine-scale air pollution data (see Table  2 ). A large part of this bias was due to ignoring population density.

Illustration of the influence of the spatial resolution of the exposure model and of the consideration of data on population density in health impact assessment studies (adapted from [ 51 ])

The table gives the estimated population attributable fraction (PAF) of lung cancer cases attributable to fine particulate matter (PM 2.5 ) exposure in France among subjects aged 30 years and more, for the year 2015 [ 51 ]

In approach 1 (main model), the PAF is estimated using a fine scale PM 2.5 dispersion model (2 km grid) at the country level, averaged at the “IRIS” (neighborhood) scale and weighted by population density. In approach 2, exposure is smoothed by assuming that all IRIS of each département have the same PM 2.5 concentration (corresponding to the median population-weighted value in each département), or that all départements in the country have the same PM 2.5 concentration value (“country scale”). In approach 3, values also correspond to the median value at the département (respectively, country) levels, with the only difference compared to approach 2 that median value are estimated without weighting with population density

Approach 4 differs from approach 1 in that an alternative RR of 1.40 per 10 µg/m 3 increase, obtained from a meta-analysis from ESCAPE project including 14 cohorts from eight European countries [ 52 ] is used, while a RR of 1.09 is used in model 1 [ 53 ]

CI Confidence interval, PAF Population attributable fraction, RR Relative risk

Note that at this step, the environmental levels and population density data can be combined with other spatially referenced data, such as information on sociodemographic characteristics, as a way to provide an estimate of how exposure distribution varies across these sociodemographic characteristics.

Reliance on personal dosimeters

Exposure assessment may also rely on personal sensors and dosimeters [ 54 ]. Generally, these have the advantages to provide an estimate of exposure that does not rely on detailed data on the sources of the factor, which are not always available (for many chemicals whose sources are not always monitored, such as benzene and other volatile compounds or pesticides), not on a modeling of the dispersion of the factor from its sources to the human environment, contrarily to the approaches discussed above in 2.4.4 to 2.4.6. Since dosimeters are carried by individuals, they efficiently allow taking into account the variability in exposure due to people moving between different environment [ 47 ]. They also allow to capture indoor levels of the factor of interest, which is of importance (for factors whose levels indoors and outdoors differ, such as ozone, benzene, radiations, temperature, noise…) given that people spend the vast majority of time indoors, at least in Northern countries. This increased “spatial” resolution compared to the above-mentioned environmental models (which typically capture outdoor levels, and generally at one location only if the time space activity of the population is not assessed) generally comes at the cost of possible limitations in terms of temporal resolution. In particular, it may be cumbersome to assess long-term exposure (which may be toxicologically relevant for specific outcomes) using personal dosimeters, which are typically carried over short-term (typically, from a day to a week) periods; these measurement periods may be repeated over time to improve the accuracy of the assessment as a proxy of long-term exposure [ 55 ], as discussed below for exposure biomarkers. Dosimeters are particularly relevant for media-specific exposures or factors, such as atmospheric pollutants including particulate matter [ 56 , 57 ] or nitrogen oxides [ 47 , 58 ], benzene [ 59 , 60 ] and other volatile organic compounds [ 61 ], non-ionizing radiation such as ultra-violet [ 62 ] or ionizing radiation [ 63 ], temperature, noise [ 64 ]… Contrarily to environmental models, their use in the context of a health impact assessment study implies to recruit a population sample as representative of the target population as possible. Their use in risk assessment studies appears quite limited so far outside the occupational setting [ 61 ].

Reliance on exposure biomarkers

In the case of chemicals with multiple routes of exposure such as specific pesticides, which may be present in food, water and air, exposure biomarkers (the assessment of the compound or of its metabolite(s) in a tissue or fluid of the organism) may be a relevant approach. With the development of biomonitoring studies [ 65 – 67 ] and of cohorts collecting biospecimens [ 68 ], exposure biomarkers may be expected to be increasingly used in quantitative risk assessment studies related to chemicals.

Biomarkers study typically provide an estimate of the circulating or excreted level of the compound, which is not exposure in the strict sense but is related to it, while also depending on toxicokinetic factors, that typically vary between subjects [ 69 ]. Biomarkers integrate multiple routes of exposure, in that the level of a compound or its metabolites in a given body compartment will generally depend on the doses entering the body by ingestion, inhalation, dermal contact… (with many specificities according to the compound and tissue in which the metabolites are assessed). This may or may not be seen as an advantage, depending on the study aim, which may relate to exposure as a whole or to route-specific exposure (e.g., that due to food contamination). Considering these different routes is in principle possible via environmental models and measurements, but may be very cumbersome in terms of data collection and modeling work (consider for example a study on the impact of pesticides, which would have to estimate pesticide levels in water, possibly in the air, in food and to assess eating behaviors to try to reconstruct an individual’s exposure). A limitation of exposure biomarkers is related to the short half-life of many chemicals in the body, which implies that a spot biospecimen will generally be a poor proxy of long-term internal dose [ 70 ]. This is an issue for etiological studies, in which dose response functions assessed using a single biomarker in each subject are expected to suffer from bias towards the null [ 70 ]. This issue may also impact risk assessment studies. Indeed, even in the context of classical-type error, which may be hypothesized for biomarkers, although the population average exposure may not be biased when a spot measurement is done in each subject, the estimation of other features of the exposure distribution, such as variance and hence the estimation of specific exposure percentiles, is expected to be biased. Like questionnaire-based approaches, and generally to a larger extent, biomarker-based approach are dependent on individual’s participation and are therefore prone to selection biases; these may, again, be partly corrected if information on the factors associated with participation is available. Just like for all other approaches to assess exposures, behaviors and other health drivers, although rarely done, there is no reason beyond logistics and funding not to consider an ad-hoc biomonitoring survey as part of the HIA, on the contrary. We are not aware of specific quantitative evaluation of the bias associated with all the possible exposure assessment tools to a priori justify the choice of the exposure metric used in a given risk assessment study.

Exposure–response functions

Identifying all health-related events possibly influenced by the considered exposures.

For each factor (in the broad sense of chemical, physical, psychosocial, behavioral factor) primarily considered or identified at the previous stages as possibly influenced by the considered intervention, all health-related events that this factor might influence (the “health effects”, or hazards, a qualitative notion not to be mistaken with the quantitative health impacts) need to be identified. This identification should cover proximal and more remote effects, and positive (beneficial) as well as negative (detrimental) effects. For several environmental factors, the list of possible health effects may be long; for example, lead is a cause of neurological, nephrotoxic, cardiac, reproductive… effects, while particulate matter can affect cardiac, respiratory, metabolic and possibly reproductive and neurodevelopmental function [ 71 ]. Complex policies may entail numerous health consequences; for example, acting on traffic will affect air pollutants, but also noise, traffic accidents, greenhouse gas emissions, that may have long-term health effects (even if the corresponding impact may be limited, depending on the considered spatial scale). Even if the study does not provide a quantitative assessment of all the effects of a given exposure, identifying all of these effects is important. This identification of possible health effects should in principle rely on a systematic review that should encompass the human literature but also toxicology and possibly in vitro or in silico studies that may inform on mechanisms and point to specific health effects. Such an identification of all likely effects of the factor, change or intervention considered may rely on a recent well-conducted published review.

Estimating the strength of evidence about the effect of factors on health

The identification of each health effect possibly influenced by each considered factor should come with some assessment of the corresponding level of evidence. The assessment of the level of evidence evolved in the last decades from experts’ opinion to more formalized systematic reviews, possibly followed by meta-analyses and evidence integration approaches combining streams of evidence from various disciplines such as in silico data, in vivo and in vitro toxicology, environmental sciences, epidemiology… (see e.g., [ 72 , 73 ] or the chapter 6 of [ 74 ] for a presentation of these approaches). Given the sometimes very large effort required by the implementation of such approaches, in particular for factors about which a vast literature exists, it is relevant to rely on existing assessments of the level of evidence, whenever a recent one with a transparent and relevant methodology is available. If not, the time and effort required for this step should not be underestimated, so that a review of the literature from all relevant disciplines, experts from these disciplines can be gathered to synthesize and weight the evidence and provide an assessment on a pre-specified grading scale (e.g., in terms of probability of causation). In case several factors are considered, then the number of exposure outcome pairs considered can be very large. An example of the assessment of the strength of evidence about endocrine disruptors is provided in Trasande et al. [ 75 ] and in Table  3 .

Estimated strength of evidence regarding the effect of endocrine disruptors on health

The overall probability of causation (last column) was based on the toxicological and epidemiological evidence. From Trasande et al. [ 75 ] (extract)

Handling of the strength of evidence about the effect of environmental factors on health

In the past, a common practice was to only consider exposure-outcome pairs (A i , H j ) for which the strength of the evidence regarding the effect of A i on H j was very strong or deemed causal. Another common option is to focus on a specific a priori chosen health outcome induced by the exposure, acknowledging that other effects are ignored; for example, many studies quantified the impact of tobacco smoke on lung cancer only, while other effects, e.g., on cardiovascular diseases, are certain.

The obvious consequence of these practices is to bias the estimated impact of the exposure or policy, generally in the direction of an underestimation (assuming that all associations go in the same direction, e.g., a negative effect of exposures on health). This is obvious for the second option above, but is also true for the first one. This is because in some cases, the discarded exposure-outcome associations will eventually turn out to correspond to very likely effects, as research continues, while the symmetrical situation of an effect deemed very likely or certain becoming unlikely as research unfolds is arguably much rarer in practice [ 76 ].

Possible alternatives to only considering very likely exposure–response pairs include:

• Considering all exposure-outcome pairs for which the estimated level of evidence is above a certain level (e.g., “likely association” or above, keeping in mind that diverse approaches are used to obtain these causality gradings and that various scales of evidence are used by various institutions), and estimating the corresponding impacts of these likely effects just like for the exposure-outcome pairs with a very high level of evidence. A special case consists in considering all exposure-outcome pairs for which there is at least “some evidence” of an association; in this case, there is potential for an overestimation of the overall impact (again, assuming that all associations go in the same direction);
• Performing sensitivity (additional) analyses considering exposure-outcome pairs with decreasing levels of evidence, and report the estimated impact of the exposure or policy considering only very likely effects, as well as the impact estimated considering also likely effects and suspected effects;
• Considering all exposure-outcome pairs for which the estimated level of evidence is above a certain level in the risk assessment and summing their impacts, weighing the sum by a weight increasing with the level of evidence of the corresponding effect, so that in the overall impact more weight is given to impacts from exposure-outcome pairs with a high level of evidence and less to the less likely ones.

Many studies more or less explicitly correspond to approach a). For example, the GBD methodology currently focuses on exposure-outcome pairs for which there is “convincing or probable evidence” [ 14 ]. An example of approach c), which is an intermediate one between the two first alternatives above, is a study of the cost of exposure to endocrine disruptors in the European Union [ 12 , 75 ]. In this study, the health impact (and corresponding cost) attributable to exposure to each considered endocrine disruptors has been assessed using a Monte-Carlo approach in which, at each simulation run, the impact of a given exposure-outcome pair was possibly set to zero, according to the estimated probability of causation (with more likely effects being less often set to zero), and the overall impact was estimated averaging over many simulation runs and summed across exposure-outcome pairs. For example, in the case of a dose–response function of an exposure-outcome pair for which the strength of evidence had been rated to be about 50%, then the corresponding attributable cases were taken into consideration in only half of the simulation runs [ 75 ]. This approach seems relevant if a large number of factors is considered and if one assumes that the literature is not systematically biased. Indeed, if twenty factors are considered and the evidence is properly evaluated, then, assuming that the weight of evidence is estimated to be 50% for all factors, one may expect that eventually 10 of these factors turn out to really have an impact, so that counting half of the effect of the twenty factors may fall closer to the true impact (corresponding to that of ten factors) than if all twenty factors are ignored because the strength of evidence is not high enough. Note that the assumption regarding the fact that the literature is not biased can be debated for environmental factors, as for many factors, the weight of evidence typically tends to increase over time, rather than vary up or down randomly, and as a literature review of environmental alerts concluded that “false alarms” tended to be very rare in environmental health [ 76 ]. This would rather support not restricting the risk assessment to “certain” exposure-outcome associations, and also consider those with a lower level of evidence, possibly taking the level of evidence in the estimation as described above in c). Considering associations with less than certain level of evidence also allows to quantify the possible impact of suspected hazards, which is relevant to prioritize environmental factors for which research efforts should be dedicated [ 77 ]. In practice, the ability to implement the approaches depends on the availability of relevant data on exposures (which can be collected in the context of the risk assessment if not already available), exposure response functions (which may be very long and cumbersome to obtain if they are not already available) and baseline population health; this means that whatever the option chosen to handle the weight of evidence regarding each exposure-health outcome pair, the list of effectively considered pairs may be further restricted because of these issues related to data availability; this is expected to bias the health impact estimate (see Assessing the impact of policies versus the impact of exposures for further discussion). Transparency on all possibly affected outcomes is anyway warranted, even if not all of them can eventually be incorporated in the estimated overall impact.

Exposure to chemical, physical and behavioral factors is now generally assessed on quantitative scales, and the expected impact of policies or plans on environmental and societal factor can often be translated in terms of quantitative variations in drivers of health. The ERF provides an estimate of the effect (positive or detrimental) associated with the exposure at each level (the terms of dose, concentration or exposure response function (DRF, CRF, ERF) or curve are here used synonymously). Issues related to the ERF relate to the validity of its assessment, including the slope and shape and to its applicability to the study population.

In the context of a risk assessment study, contrarily to the estimation of exposure which may in some cases be done ad hoc, it is generally not realistic to expect to generate from scratch a new ERF (whose estimation may require the follow-up of large populations over periods of years or decades) so that one has to rely on external ERFs. If no ERF is available, then one may either 1) try to derive an ERF from existing toxicological (animal) studies [ 78 ], if any or 2) perform a qualitative HIA. If several ERFs are available, then performing a meta-analysis of these ERFs to obtain a more robust one should be considered. This step may be done on the basis of the systematic review possibly performed to assess the level of evidence (see Estimating the strength of evidence about the effect of factors on health above).

We will here assume that some ERFs (or relative risks or any equivalent measure of association) are available. The choice of the exposure response function may have large influences, as illustrated by a review of air pollution HIAs illustrating that the estimated health impact of fine particulate matter in Europe varies by a ratio of two when switching between exposure–response functions based on two large studies [ 79 ] (see also Table  2 ). Three dimensions to consider are those of the sample size from which the ERF has been estimated, the potential for bias of the ERF, e.g., in relation to the adjustment factors considered, and that of its applicability to the study population. Researchers are typically left with the choice between an ERF estimated on a local (or nearby) population, which possibly relies on a small population (hence a large variance), or more global estimates, such as those from meta-analyses, which may be more robust (based on larger populations) but ignore possible differences in sensitivity between the local population and the rest of the world (and therefore are possibly biased). This can be seen as an illustration of the classical bias-variance trade-off. In an ideal situation in which many studies providing an ERF are available, one would characterize the potential for bias of each individual study (evidence evaluation, see e.g., chapter 5 in [ 74 ]) and then perform meta-analyses, in which the poor-quality studies would be discarded if they tend to produce different ERF estimates than the studies with better quality. The potential for heterogeneity in the ERF across populations and geographical areas should also be characterized (see Fig.  5 ), allowing to decide whether it appears more relevant to derive the ERF from a small group of studies in settings very similar to the setting of the HIA, or from a larger group of studies covering a wider diversity of settings.

Meta-analysis of the relative-risk (RR) of lung cancer associated with PM 2.5 exposure, by region [ 53 ]

It may be relevant to also consider other factors, such as the range of exposures in the studies from which the ERF is based, trying to base the ERF on studies with an exposure range similar to that of the population in which the risk assessment is conducted, and avoiding to extrapolate ERFs outside the exposure levels where the bulk of the original data lie. In the case of risk assessment studies focusing on factors of other nature, such as social or behavioral factors, for which the hypothesis of heterogeneity in sensitivity across large areas is more likely, the meta-analysis may not be the preferred option.

The concept underlying the exposure assessment in the study from which the exposure response function is based should be similar to that used in the risk assessment study. For example, if “exposure” in the study from which the exposure–response curve originates corresponds to lead environmental level, then it is generally not advised to rely on lead biomarkers (which assess the levels circulating in the body and not environmental levels) in the risk assessment study; the length of the exposure window should also be similar, as this length generally influences the estimated exposure variability. If both entities differ, then in some cases it may be possible to convert one into the other, using a formula derived from a study in which both entities have been assessed in the same population, or using toxicokinetic modelling. Contrarily to what is sometimes said, this requirement of a similarity of exposure concepts does not imply that the specific approaches used to assess exposures need to be identical. The measurement approaches can differ, provided one is not biased with respect to the other. For example, if the exposure considered is fine particulate matter and the exposure–response function stems from a cohort study in which exposure was assessed relying on permanent monitoring station, then a dispersion model could in principle be used to assess fine particulate matter levels in the risk assessment study. In this example, both the permanent monitoring stations and the dispersion models provide an estimate of the same entity (the environmental level), and since etiologic studies relying on permanent monitoring stations are not expected to be strongly biased compared to studies using environmental models with a finer scale such as a dispersion model (assuming that the concept of Berkson error [ 80 ] applies), the exposure–response function stemming from the study using monitoring stations is in expectation similar to the one that would have been obtained if a dispersion model had been used instead to assess exposure.

Non-linear exposure–response functions

The studied factor may have non-linear associations with the health outcome considered on a given (e.g., additive or multiplicative) scale. Note that such deviations from linearity are not always investigated in etiological studies (possibly for reasons related to limited statistical power). As an illustration, a 2001 review on the ERF of physical activity effects on mortality indicates that only 17 of the 44 studies conducted a test of linear trend [ 81 ]. A more recent and robust review does a meta-analysis on studies with larger population samples and finds a better fit for the curve y =—x  0.25 , with steeper effect of moderate, as opposed to higher, physical activity on mortality [ 82 ].

Non-linear dose–response functions are all the more likely when the underlying mechanisms of action are complex and as the range of exposure values increases. Ignoring this non-linear relation can significantly impact the estimated risk [ 83 ], hence potentially misestimating the risks or benefits of the change, depending on the distribution of the exposure in the population.

Non-linear ERFs have been exhibited for several risk factors. This is the case for temperature and air pollution effects on mortality in particular, and also effects of physical activity on health (Fig.  6 ); they may also be expected e.g., for endocrine disruptors [ 84 ].

Illustration of non-linear exposure response functions: A ) Fine particulate matter and mortality [ 85 ]; B ) Temperature and mortality in Rome [ 86 ], C ) Physical activity and cardiovascular events [ 87 ]. MET: Metabolic equivalents: RR: Relative risk

Note that many risk assessment studies, at least from a certain period in the past, used to assume the existence of thresholds (hence, a non-linear dose–response) in the non-carcinogenic health effects of chemicals, and a lack of threshold of carcinogenic effects. There is to our knowledge no toxicological justification to such general claims. The “threshold” model can be seen as being related to the misconception that the NOAEL (no observed adverse effect level) estimated in some regulatory toxicology studies corresponds to a true “no effect” exposure level. In fact, a NOAEL generally corresponds to a level with an effect, whose size depends in particular on the number of animals used in the experiment aiming to estimate the NOAEL [ 88 ].

Assessment of disease frequency

The ideal situation corresponds to that of an area where a register (or any other system allowing an exhaustive assessment of new disease cases on a representative basis) exists. Such registers exist in many countries for cancers but, outside Scandinavian countries, are rarer for other diseases. Just like for the case of the assessment of exposures, tools typically used in etiologic cohort studies may provide a relevant estimate of the disease frequency, with the same caveat as above, namely that etiologic studies are rarely representative of a given area, which would be a limitation if the disease frequency obtained in such a study is to be used in a risk assessment exercise. Alternatively, one can rely directly on estimates of the disease burden, such as those provided by the Global burden of disease project ( https://www.healthdata.org/results/gbd_summaries/2019 ).

The disease frequency can correspond to different entities, generally related to incidence (the number of cases appearing during a given time period in a population) or prevalence (the number of cases present at a given time point, whatever the time when the disease started). In principle, incidence should be targeted. The entity used to assess disease frequency should be coherent with the measure of association (the exposure–response function) chosen. For example, a hazard rate stemming from a cohort study (or an incident case control study) assesses the change in the disease hazard (the strength of apparition of new cases) and needs to be combined with a measure of incidence and not prevalence.

Health impact

Concepts of impact.

The health impact (or risk) is the core estimate of a risk assessment study. It is a challenging notion, both from a conceptual and estimation perspective, not to mention issues related to the use of this expression with possibly different meanings across scientific and public health communities. When it comes to the human-derived risk assessment studies discussed here, the core product corresponds to notions close to the epidemiologic notion of attributable fraction. Following Greenland [ 8 ], we shall remind that this expression covers different concepts: the etiologic fraction, the excess fraction, the incidence density fraction, to which one shall add the expected (healthy) years of life lost.

The excess fraction corresponds to the proportion of cases that would not have become a case during the study period in the absence of exposure, while the etiologic fraction includes these excess cases as well as the cases due to exposure that would also have occurred during the study period in the absence of exposure, but at a later time point during this period. These two fractions can strongly differ because the etiologic fraction includes, in addition to the “excess cases”, the cases which would have happened also in the absence of exposure, but for which exposure made the incidence happen earlier than if the subject had not been exposed. These cases for which exposure simply made the case happen earlier in the study period may correspond to a large fraction of cases for complex diseases (as opposed to diseases with a simpler etiology such as infectious diseases), and their number increases with the duration of the study period. This is illustrated with the extreme example of a study of a factor increasing mortality or any other inevitable outcome: if the study period is very long, so that all members of the considered population are dead at the end of this period, then the excess fraction will become zero (because everyone eventually dies, even in the absence of the considered exposure)(see for example Fig. 2.6c in [ 89 ]), while the etiologic fraction may be non-null if the exposure does influence mortality [ 8 ]. For this reason, some advise not to use the excess fraction as a metrics, or the similar yearly number of avoided cases in a population [ 89 ]. Although this metric is indeed limited when it comes to estimating a meaningful health impact (that may be used to quantify an economic impact), when comparing the impact of various factors, it is possible that in many common situations the ranking of risk factors is preserved across metrics. In any case, it is of course crucial to only compare exposures in terms of impact assessed using exactly the same metric. Although in principle more relevant, the estimation of the etiologic fraction requires specific biological knowledge or hypotheses [ 8 ].

The incidence density fraction, defined as (ID E+ -ID E- )/ID E+ , where ID E+ (respectively, ID E- ) is the incidence in the exposed (respectively, unexposed) group, has different interpretations, depending on whether one relies on instantaneous or average incidence densities [ 8 ]. Estimating attributable life-years (or healthy life-years) associated with the exposure or policy may appear as a relevant option for many public health purposes [ 8 ] and should be attempted. One reason is that this metric does not suffer from the above-mentioned limitation related to the fact that, since everyone eventually dies (the deaths are postponed, not avoided), the long-term gain expressed as a total number of deaths avoided from the reduction of exposure to a harmful environmental factor will appear much smaller than could be thought at first sight when considering the number of deaths avoided during a given year. The number of avoided life-years, by depending both on the number of deaths postponed each year and on the delay in their occurrence induced by the environmental change, takes into account the two dimensions of the problem. The same goes for the change in life expectancy [ 89 ]. Note that a given number of (healthy) life-years lost may translate into very different impacts on life expectancy, depending on how the life-years lost are distributed in the population (something that cannot be determined without strong assumptions). As an illustration, the UK committee on the medical effects of air pollution (COMEAP) concluded that anthropogenic fine particulate matter (PM 2.5 ) at the level observed in 2008 in the UK was associated with an effect on mortality equivalent to nearly 29,000 deaths at typical ages in 2008, and that, depending on how this burden is spread in the whole population, this might correspond to impacts on life expectancy ranging from 6 months (if the air pollution effects was distributed across all deaths) to 11.5 years (if PM 2.5 were only implied in 29,000 deaths) [ 89 ].

The risk estimation relies on a more or less sophisticated combination of the exposure distribution under each counterfactual scenario with the ERF and with an estimate of the baseline disease frequency in the considered population, whether explicitly or hidden in the externally available disease burden. This estimation is repeated under each of counterfactual scenario, and the risk difference between the targeted and “baseline” scenario is computed. In practice, several ways to estimate the risk are used, which are not all strictly valid in theory. The main types of approaches correspond to:

• PAF-based formulas: An analytical (formula-based) estimation of a “population attributable fraction” associated with the exposure, multiplied either by the incidence of the disease or by an externally available estimate of the disease burden in the population (i.e., the impact of the disease in the considered population irrespective of all its possible causes, typically, expressed in DALYs);
• Person-years method: A simulation of the whole population in which new disease cases occur each year during the course of the study period under various counterfactual scenarios, from which attributable cases, differences in life expectancy, DALYs and other specific measures of risk can be estimated. This approach has the advantage of allowing to take into account the dynamics between death rates, population size and age structure [ 89 ].

Note that alternative approaches exist, such as compartmental models (generally, but not exclusively, used for infectious diseases) or approaches based on individual, as opposed to population-based modeling, such as microsimulations (see e.g. Mueller et al. [ 90 ] for a review). Compartmental models assume that subjects switch between mutually exclusive states (e.g., susceptible, infected, recovered, dead, for an infectious disease) and model the trajectory of each individual from the population across these states via deterministic or probabilistic approaches. They are particularly relevant to model the impact of interventions that may influence infectious diseases, and will not be detailed here (see [ 91 ] for an example). The lifetable (or person-years) approach mentioned below can in a way be seen as a particular example of compartmental models.

As already mentioned, one general issue relates to the consistency between the various metrics used; for example, the data on baseline disease frequency need to correspond to an estimate of disease incidence if the exposure–response function is derived from an etiologic study assessing the occurrence of new disease cases, such as a cohort or incident case-controls study.

The “PAF-based formula” approach can be illustrated taking the simple example in which a binary exposure level is changed from 1 to 0 (e.g., all smokers stop smoking) in the counterfactual situation and in which the overall burden associated with the disease, assumed to correspond to a dichotomous outcome (with/without disease) is available. The health impact is generally estimated in two steps, the first one corresponding to the estimation of the PAF, which, in a situation without confounding, is defined as:

Where Y corresponds to the disease, P(Y = 1) is the proportion of subjects developing the disease in the study period and X is the exposure of interest, with X = 0 corresponding to non-exposed subjects. Equivalently, the PAF can be defined as:

Were R e and R u are the risks of disease in the exposed and unexposed subgroups, respectively.

Note that the assumption about the lack of confounding can be conveniently ignored by relying on the structural causal modeling framework [ 92 ] and the do operator:

where do(X = 0) refers to a situation in which X is set to 0 through a minimally invasive intervention (in the terminology of Pearl [ 92 ]), all other variables possibly influencing X remaining constant; of course, the “reference” value X = 0 can be replaced by any other value or distribution, in the case of an exposure with more than two categories.

Coming back to the situation in which X is binary, the PAF is generally estimated relying on Levin’s formula, which can be derived from the previous ones:

Where P is the prevalence of exposure in the population, or P(X = 1), RR the relative risk associated with exposure (assumed here to be dichotomous). (Note that Rockhill et al. [ 6 ] explain that this formula is not valid in the context of confounding. This is true when one applies the formula ( 4 ) in the study population from which the RR is estimated but not, as in a risk assessment exercise, if one uses (4) to estimate the attributable fraction in a given population using a RR assumed to be unbiased estimated from another population.

The health impact is then estimated by combining the estimated PAF with the burden of the considered disease, BD, in the study population, generally available or approximated from external sources (or estimated via an ad-hoc study):

The unit of BD (e.g., deaths, DALYs, etc.) defines the unit of measure of the health impact.

In the case of a categorical assessment of exposure, the health impact is estimated as above in each exposure category, after which the overall impact is estimated by summing over all exposure levels. If exposure is continuous, formula ( 1 ) above is generalized by integrating the PAF over all exposure levels as shown here:

Where m is the maximal exposure level, P S1 is the observed distribution of exposure level (or baseline scenario), P S2 the distribution of exposure under the counterfactual hypothesis (which may correspond to a uniform distribution with all subjects at zero if zero is the targeted level) and RR the exposure–response function, with RR(x) providing the relative risk when exposure is x.

If the health parameter Y is continuous (e.g., blood pressure, birth weight…), then the impact of X on the average value of Y can be estimated as:

Where β corresponds to the exposure–response function describing the average value of the outcome Y as a function of the outcome. In the case of a binary exposure with prevalence P, the right-hand side of this formula simplifies to β x P. This value can be multiplied by the population size if one wants to express the impact in terms of units of Y (e.g., IQ points) due to the exposure in the population as a whole.

The person-years approach consists in simulating the cohorts corresponding to each of the considered counterfactual scenarios throughout the study period, with new disease cases appearing each year. It has several key advantages over the formula-based approach, including: 1) to make all assumptions more explicit; 2) to avoid issues related to the estimation of the expected number of cases [ 7 , 93 ], since the number of subjects still at risk in each cohort is explicitly estimated; 3) to be more flexible in coping with various exposures simultaneously, assuming various correlation structures between them, with scenarios implying gradual changes in exposure over time, considering sociodemographic changes in the study population, without having to work out an analytical solution. The cost of this approach is that it is generally much more complex to implement and compute.

The estimation needs to be repeated for the other health outcomes identified at the hazard identification step (Identification of exposures above) as being possibly influenced by the considered factor. It can also be repeated for other factors, as we now discuss.

Consideration of multiple risks factors

If several factors are considered (e.g., because one is interested in a prespecified set of exposures, or because the policy evaluated is expected to influence several physical, chemical, psychosocial factors), the estimation needs to be repeated for each of these factors, at least those for which the level of evidence regarding effects on one health outcome are above the selected level, if any (see Handling of the strength of evidence about the effect of environmental factors on health). A central issue here relates to the situation in which two or more of these factors can influence the same health outcome. Indeed, care is required to acknowledge the fact that the fraction of cases of a specific disease attributable to different risk factors can generally not be summed. This is a consequence of the multiplicative nature of risk and of the multifactorial nature of most diseases [ 94 ]; moreover, care is needed to consider possible relations, and in particular correlation, effect measure modifications or mediation between risk factors.

Again, the PAF-based formula and the person-years method can be used when considering several factors influencing the same health outcome, with the latter being more flexible. Regarding the former approach, if population attributable fractions have been estimated for each of the R risk factors influencing the considered outcome, and under some hypotheses (see below), then these can be aggregated with formula ( 6 ):

Where PAF r is the population attributable risk fraction associated with risk factor r estimated independently from the other risk factors. This approach makes strong assumption: that all risk factors act independently (in particular, that no effect of one factor is to some extent mediated by another factor, or modified by another factor) and are not correlated. Note that this formula is identical to that used in toxicology for the so-called case of independent action [ 95 ]. Under these assumptions, if two independent factors have attributable fraction of the cases of 40 and 50% respectively, then their joint action corresponds to an attributable fraction of 70% (40% plus 50% of the remaining 60%).

Some of these assumptions may not hold in real situations.

A first issue relates to the situation in which exposures to the considered factors (say, x 1 and x 2 ) are correlated. Formula ( 8 ) assumes that the fraction of cases attributable to B is the same in the whole population and in the population from which factor A has been removed (independent action), which does not hold if A and B are correlated because then the prevalence of exposure to B is not the same in the two populations. From this it appears that information on the relations between exposures to the considered factors (here, x 1 and x 2 ) in the study population is required to estimate the fraction of cases attributable to x 1 and x 2 . Specifically, their joint distribution P(x 1 , x 2 ) needs to be considered in the PAF estimation, as described in Ezzati et al. [ 96 ]. Formula ( 6 ) can be adapted by replacing each integral by:

this implies of course that information on the joint (and not only marginal) distribution of all relevant factors is available or collected in the population in which the risk assessment is conducted. Biomonitoring and exposome studies, provided they assessed multiple exposures simultaneously in the same participants, allow providing such a joint distribution [ 97 ]. Equation ( 9 ) assumes that the risk for a given combination (x 1 , x 2 ) of exposures X 1 and X 2 is the product of the relative risks associated with X 1 and X 2 , corresponding to a hypothesis of lack of effect measure modification of X 1 by X 2 .

If now there is evidence of effect measure modification (sometimes termed interaction) between risk factors then in principle RR 1 (x 1 ).RR 2 (x 2 ) in Eq. ( 9 ) should be replaced by RR(x 1 ,x 2 ), that is the relative risk function describing the joint effect of x 1 and x 2 , which can incorporate a different relative risk associated with x 1 at each given value of x 2 . To our knowledge, there are currently few examples of risk factors and outcomes for which this function is accurately characterized and available.

Another option to handle it is to consider different ERFs in different population strata; again, one needs information on the joint distribution of all relevant factors, as well as stratum-specific relative risks.

Another (non-exclusive) situation is that of mediation effects [ 98 ]. Consider the case of a disease D (say, lung cancer), influenced by several risk factors including active smoking (A) and green space exposure (B), the effect of which being partly mediated by changes in air pollution levels (C). Figure  7 provides a possible model of the assumed relations between A, B, C and D. Let us assume that one is interested in estimating the overall impact of factors A, B, C on D, that is, the number (or the fraction) of cases of disease D that would be avoided if A, B, and C all had “optimal” levels. The estimated impacts of B (improving green space exposure) and C (getting rid of air pollution) cannot be considered as being independent because a part of the effect of B is included in the effect of C (a mediation issue). Estimates of the share of the effect (in the sense of measure of association) of B on D that is not mediated by C (but that may be mediated by other factors not considered here, such as an increase in physical activity), termed the natural direct effect, can be provided by mediation analysis techniques [ 98 ]. This natural direct effect of B on the disease risk is by construction independent of the effect of C on disease risk, so that the corresponding attributable fractions can then be estimated and combined using formula ( 8 ) above. In the Global Burden Disease methodology, for each pair of risk factors that share an outcome, the fraction of the risk mediated through the other factor is estimated using mediation analysis techniques, if the relevant studies (in which B, C, D are altogether estimated) are available. A concern here (besides the usual assumptions required by mediation analysis [ 98 ]) relates to the transposability from one area to the other of such mediation analyses; for example, the change in air pollution level following a change in green space surface may be influenced by the local share of traffic-related air pollution among the total of the emissions of air pollutants from all sources, which may vary across areas.

Causal diagram summarizing the causal relations between hypothetical risk factors ( A , B and C ) and a disease D . Here, A and B are assumed to independently affect the probability of disease, while a part of the effect of B on D is mediated by C

In the case of a continuous health outcome, one option to estimate the joint impact of several factors, which assumes a lack of synergy (or of departure from additivity) is to sum the average changes in the outcome attributed to each exposure to obtain an estimate of the impact of the combined exposure.

Cessation lag

The cessation lag can be defined as the time lag between the implementation of the considered intervention and the consequent change in hazard rate. It is meant to take into account the fact that for most (chronic) clinical endpoints, the effect of changes in (external) risk factors does not manifest fully immediately. The COMEAP study of particulate matter impacts on mortality [ 89 ] provides an illustration of the impact of various cessation lags (see also Fig.  8 ). Such a cessation lag can be implemented in studies relying on person-year approaches. Whether a cessation lag needs to be considered depends on the availability of knowledge about when a change in exposure will start influencing the considered health outcomes, as well as on the question asked: if for example one is interested in knowing how health is likely to vary in the short and mid-term if one managed to implement a given intervention now (or in a near future), then considering a cessation lag is relevant; if the question of interest is the more theoretical one of quantifying how much better the population health would be today if a given exposure or set of exposure was absent (or if a specific intervention had been implemented a long time ago), then cessation lags might be ignored.

Illustration of possible cessation lags (in years) considered in the estimation of the impact of fine particulate matter exposure on mortality [ 89 ]. The first year of the intervention implementation is designated as year one

Socio-economic impact of attributable health effects

One may wish to go beyond the health impact estimates. Economic analysis can further inform choices between different project or policy alternatives considering dimensions beyond health. Such an analysis can be limited to quantifying the costs of implementation of the considered scenarios or policies (e.g., those required to induce a reduction of exposure to a specific factor), and relate these costs to the health benefits expressed in non-monetary terms, such as DALYs (corresponding to a cost–benefit analysis). Beyond this, cost–benefit analyses have the advantage of allowing to compare the costs and monetized health and non-health benefits. Such an analysis is more complete and may be more meaningful for decision-makers than one ignoring the costs of implementation of each of the considered alternatives.

The benefits include the monetization of health benefits, which takes into account both tangible and intangible costs. The tangible costs refer both to direct costs, in particular the costs to the health system (costs of specific treatments for each pathology: drugs, consultation, hospitalization, etc.) and indirect costs linked to absenteeism and the resulting loss of productivity. The intangible costs refer to the inclusion in the economic analysis of the loss of well-being due to anxiety, discomfort, sadness or the restriction of leisure or domestic activities due. These are therefore non-market costs for which it is necessary to reveal their value through survey methods or analysis of the behavior implicitly attributed to them. For example, Ready et al. [ 99 ] and Chilton et al. [ 100 ] valued the willingness to pay to avoid an air pollution-related morbidity episode. Among these intangible costs, the economic valuation of mortality is a delicate step from an ethical point of view in the absence of consensual values. It is generally based on the monetary valuation of a statistical life or a year of life lost. In recent years, the most popular approach in the economic literature to determining the statistical value of a human life is the willingness-to-pay approach. The statistical value of a human life is thus approximated by the amount of money a society is willing to pay to reduce the risk exposure of each of its members. This literature shows that the statistical value of a human life depends on characteristics such as age at death, time between exposure and death (i.e. latency), and nature of the underlying risk [ 101 , 102 ]. Empirical assessments have provided a range of values generally between €0.7 and €10 million per human life. The other approach used is that of revealed preferences, which is based on observed behavior: for example, the difference in wages between two branches of economic activity with different mortality risks.

The cost–benefit analysis, beyond the health benefits directly generated by the project or policy, can also integrate co-benefits not directly related to health. For example, Bouscasse et al. [ 29 ] considered the following co-benefits of measures to reduce fine particle pollution: the reduction of noise, the development of active mobility on health, which lead to health co-benefits, but also the reduction of greenhouse gas emissions, which includes benefits not related to health.

With regard to the evaluation of costs, it is necessary to define the scope: is it the cost of implementing the policy for the public authorities? Should the impact on household expenditure also be taken into account? This may be important, for example, in the case of measures aimed at reducing air pollution through actions on heating or transport, for which individuals carry a part of the cost. The assessment may also seek to quantify the impact on employment, for example, or on imports or exports of goods and equipment.

Finally, the time dimension is important in the implementation of a cost–benefit analysis. While costs usually occur at the beginning of the period (implementation of actions or policies, investments), benefits tend to occur later. Indeed, health benefits are generated over time depending i) on the speed of implementation of actions and of the progressiveness of the reduction of exposures, and ii) on the fact that health benefits may not be immediate following the reduction of this exposure (cessation lag).

The time lag between costs and benefits has another consequence in a cost–benefit analysis. We do not give the same value to €1 to be paid today or in several years, because of what economists call the preference for the present. This is introduced into cost–benefit analysis through a discount rate , which gives a present value to monetary flows in the future. The higher the discount rate used by the public authority or by the stakeholders, the lower the present value of benefits that occur later.

Sensitivity and uncertainty analyses

Sources of uncertainty in quantitative risk assessment studies.

Uncertainties exist at each step of risk assessment. For example, there may be uncertainties in the health outcomes influenced by the considered factor or policy, in the level of evidence relating the factor or policy with a given health outcome, in the corresponding dose–response function, in exposure distribution… We will here distinguish uncertainties due to the variability of the quantitative parameters considered in a risk assessment study (typically, in the dose–response function, but also possibly in parameters of higher dimension, such as the distribution of an exposure in the population) from more systemic uncertainties related to the model choice (sometimes termed systemic or epistemic uncertainties , e.g. related to the assumption that the risk factor(s) considered only influences a specific health outcome, possibly disregarding health effects not identified yet or for which a dose–response function is not available) [ 103 ]. A typology of sources of uncertainty in burden of diseases studies is presented in Knoll et al. [ 104 ]. We will focus here on uncertainty due to the variability in parameters, and touch upon systemic uncertainty in Sect. " Sensitivity and uncertainty analyses " below.

The consideration of the uncertainty related to variability typically implies to obtain an estimate of the uncertainties occurring at each step of the study, in order to combine these uncertainties (uncertainty propagation, or error propagation) and try providing an estimate of the resulting uncertainty on the overall study results.

Estimating the impact of uncertainties

In the simple case of a single source of uncertainty, the translation of this uncertainty on the overall results is in principle relatively straightforward; for example, if one only considers uncertainty on a dose response function expressed using a relative risk and (simplistically) assumes that this uncertainty is conveyed by the confidence interval of the relative risk, then the estimation of the health impact can be repeated using the limits of the confidence interval instead of the point estimate of the relative risk (of course, the confidence interval usually only conveys uncertainties related to the population sampling, more specifically to random error related to sampling.

However, there are multiple sources of uncertainty beyond those related to sampling error. Indeed, in the classical view of epidemiology (developed for measures of association), the uncertainty due to variability can be seen as having a random and a systemic component; only the former is easily estimated, while the estimation of bias requires quantitative bias assessment methods [ 105 ] that are seldom applied. In particular, sources of uncertainty related to exposure measurement error, to the assessment of disease frequency, possibly confounding bias or uncertainties in the shape of the exposure response function and the existence and shape of cessation lag, are not conveyed by the confidence interval of the relative risk and are worth considering – but rarely taken into account.

If one tries to simultaneously take into account several sources of uncertainty, then more complex approaches are required to propagate the uncertainty down to the impact estimate. Although analytical approaches (such as the delta-method) may be applicable in relatively simple situations, one more general approach corresponds to Monte-Carlo simulations. Monte-Carlo simulations rely on the principle of repeating a large number of times the health impact estimation, letting all underlying parameters (relative risks, exposure distribution, possibly the number of factors influencing the health outcome and the outcomes that they influence, if there are uncertainties at this level…) vary across realistic values [ 106 , 107 ]. They allow providing an estimate of the distribution of the possible values of the health impact. This requires knowledge or assumptions on the likely distribution of each parameter considered in the study. If such an approach is implemented, then authors will be able to report a distribution of the most likely value of the health impact or cost. The results will for example be conveyed in a way such as: “Taken all identified sources of uncertainty into account, there is a 90% probability that the number of deaths attributable to factor A in the area is above 10, a 50% chance that it is above 30 and a 10% chance that it is above 50 cases” (with of course specific explanations for a non-scientific audience). Alternative approaches to Monte-Carlo simulations also exist, in particular in the context of Bayesian modeling [ 108 ]. Provided relevant data are available, this framework can in principle accommodate both the uncertainty related to variability, but also systemic uncertainty [ 109 ].

In the absence of formal consideration of the systemic uncertainty in the uncertainty analysis, it remains essential for the investigators to state their model’s assumptions and limitations, including in particular the impacts related to specific risk factors or health outcomes that could not be taken into account in the quantitative assessment (see also 3.3 below).

Quantitative assessment studies are at the interplay between scientific, policy and legal issues; contrarily to what the deceptively simple epidemiological concept and formula of the “population attributable fraction” may let think [ 7 , 8 ], their implementation and interpretation is very challenging.

We have reviewed some of the possible approaches and issues at each step of risk assessment studies. We have made the choice not to discuss the steps of problem framing, study reporting, and issues related to population participation, which are presented elsewhere [ 19 , 21 ]. In the absence of broad methodological studies (e.g., via simulation approaches) in this field, we acknowledge that some of the choices we have done in presenting methods carry some amount of subjectivity and encourage the development of studies to quantitatively assess bias and trade-offs in this area to help investigators make more informed choices with regards to the methodological options. Such simulation studies could e.g., be used to select the most efficient approach to assess exposures in a given context.

To conclude, we will touch upon issues related to the terminology of risk assessment and HIA studies, the distinction between human-derived and animal-derived (toxicological) risk assessment studies, and research needs.

Issues related to terminology

Studies aiming at characterizing the health and societal impact of policies or environmental factors are riddled with many different terminologies and acronyms. This diversity of acronyms encompasses some real differences in aims or methodology, but is also due to the convergence of various research and application streams. Indeed, as already mentioned, these studies originate from the epidemiological research stream related to the concept of population attributable fraction, which dates back to the 1950s [ 7 ], from the development of legal requirements for environmental impact assessment before the development of new policies, plans or programs (which progressively also encompassed health issues), and from the applied stream of chemical risk assessment based on “regulatory toxicology” approaches and the risk assessment logic outlined in the USA National Research Council originally published in 1983 and also known as the “red book [ 1 , 2 ]. Three key expressions are used: 1) burden of disease; 2) risk assessment; 3) health impact assessment.

Burden of disease studies generally correspond to an assessment of the risk (e.g., in terms of attributable cases or DALYs) associated with a given disease in human populations, without referring to an exposure possibly causing the disease or to a policy aiming at limiting its impact. However, when used in relation with a factor or family of factors, then only the risk (or disease burden) associated with this factor is considered (e.g., in “environmental burden of disease”), so that there is no essential distinction anymore with what we have discussed here. In practice, health impact assessment is often used in relation with a policy or intervention likely to affect health, while environmental burden of disease is often used in the absence of explicit consideration of a policy or intervention (see below).

Regarding health impact assessment, a difficulty arises from the fact that much of the theory and examples of HIA studies has been published in the grey literature [ 10 ]. The term has most often been used to assess the potential impact of a public policy, plan, program or project not yet implemented [ 3 ]. HIA is defined by WHO as “a combination of procedures, methods and tools by which a policy, program or project may be judged as to its potential effects on the health of a population, and the distribution of those effects within the population”. In addition, the consideration of inequalities (i.e., considering the distribution of risk within a population rather than only its mean value) has been put forward as an essential part of HIAs, at least in principle [ 110 ]. Several distinctions exist within the field of HIAs, which refer to various notions and dimensions [ 10 , 110 , 111 ]. Some of these notions are at different levels and hence not mutually exclusive (for example some distinctions refer to the way health is conceptualized, other to the qualitative or quantitative nature of the study, others to the level of participation of the considered population), making it difficult to suggest a simple and unified terminology. A distinction between “broad focus” HIA studies, in which “a holistic model of health is used, democratic values and community participation are paramount and in which quantification of health impacts is rarely attempted” [ 10 ], and “tight focus” HIAs, based on epidemiology and toxicology and tending towards measurement and quantification, is sometimes done [ 10 ]. “Analytical HIA” is sometimes used synonymously to these “tight focus” HIAs. Harris-Roxas and Harris also quote distinctions such as between quantitative and qualitative HIAs, to those relying on “tight” or “broad” definitions of health, to HIAs of projects or policies [ 111 ]. What we have reviewed here is close (if not equivalent) to these “analytical”, “tight focus” or “quantitative” HIAs.

Such quantitative HIAs typically aim at answering a question about the future (“how is the planned policy expected to affect the health of the concerned population?”) while many risk assessment studies aim at answering a question about the present, and are sometimes presented as considering a single factor at a time (typically, “how many lung cancer cases would be avoided in this city if WHO air pollution guidelines were respected?”). We come back to these apparent differences in the Sect. "  Assessing the impact of policies versus the impact of exposures ".

As already stated, in practice, some HIA or “risk assessment” exercises fall short of providing a quantitative estimate of the risk, e.g., because of a lack of relevant dose response functions or support to collect missing information; they may for example stop at the step of hazard identification. It is, in a way, what happens with animal-based risk assessment studies.

Animal-based risk assessment studies

It is important to recall that in addition to the approach based on human-derived dose–response functions that we described above exists a whole stream of research and applied studies relying on animal-based dose–response functions and so-called toxicological reference values. The core approach of the benchmark dose (BMDL) consists in identifying an exposure level corresponding to an effect considered small in an animal model (say, a 5% decrease in organ weight or 5% increase in the frequency of a disease) [ 112 ]. The lower confidence bound of the benchmark dose is then divided by an uncertainty factor (typically, 100), to take into account between-species and within-species differences in sensitivity, and this value is used as a “daily tolerable dose”, or compared to the exposure distribution in a specific human population. Therefore, this approach aims at identifying a range of doses under which, under certain assumptions, there would be no “appreciable adverse health effects” of exposure. The comparison of the estimated daily tolerable dose with the exposure distribution in the considered population allows to identify if a substantial share of the population is above this daily tolerable dose, and thus finds itself at exposure levels that cannot be deemed safe (a qualitative rather than quantitative statement about risk). For this reason, these risk assessment approaches based on animal-derived reference values or dose–response functions rather correspond to safety assessment : they can allow to state that, given its exposure distribution, a given human population is “safe”, or unlikely to suffer appreciable adverse health effects, with respect to a specific exposure (or set of exposures if the approach is used in the context of mixtures of exposures) and do not strictly correspond to risk assessment as we defined it, i.e., the estimation of a number of disease cases (risks) attributable to an exposure in a population. To limit ambiguity, it might be relevant either to use the expression “safety assessment” when referring to the so-called risk assessment studies relying on animal derived toxicological reference values, or to distinguish “animal-based risk assessment” (which implies between-species extrapolation) from “human-based risk assessment” (which does not).

Assessing the impact of policies versus the impact of exposures

The factors influencing health considered in risk assessment studies range from single behaviors (e.g., smoking, physical activity), to physical and chemical factors such as particulate matter [ 35 , 113 ] or other environmental exposures including lead or families of factors such as endocrine disruptors [ 12 , 75 ]; these can be considered at various scales, from the neighborhood, region, country or at the planetary scale (e.g. as done by the Global Burden of Disease Studies, or by a study considering different ozone layer depletion scenarios [ 37 ]). In addition, as described above, the formalism of risk assessment studies used for a single exposure can be extended to the case of two or more exposures, at least under certain assumptions. Consequently, if one is interested in a project (e.g., the building of a road infrastructure, a factory) or a policy (regulating a behavior such as smoking, alcohol consumption, speed limit on highways, frequency of social contacts or exposure to a chemical or set of chemical factors, or consisting in taxes aiming at modifying exposures or behaviors), and if it is possible to provide a quantitative estimate of the expected changes in the factors affecting health impacted by this project or policy, then the methodology of risk assessment as described above can be used to provide an estimation of the impact of this project or policy. Symmetrically, evaluating the impact of a factor, such as an atmospheric pollutant, implies, as we have seen, to compare a given situation (usually, the current one or a future of this situation assuming a “business as usual” scenario) with a counterfactual situation (a hypothetical present in which a behavior or an exposure has been altered at the population level) that can be seen as resulting from a policy or an infrastructure. For example, assessing the risk associated with air pollution exposure implies to consider a counterfactual level (e.g., the WHO air pollution guidelines); the estimated risk is then identical to the health gain expected from a policy that would allow to lower air pollution from the current situation down to this guideline value (see Fig.  9 for an illustration). In other words, assessing ex-ante the impact of a hypothetical policy or infrastructure (what is sometimes termed health impact assessment) boils down to evaluating the health impact of its immediate consequences relevant for health (a set of factors); and assessing the impact of one or several behavioral, social or environmental factors (for which the expression risk assessment is usually reserved) is equivalent to considering the impact of a policy that would alter this or these factor(s). There may be some differences in implementation between the two specific questions (e.g., one may want to assume that it takes a few years for air pollution to reach the target value in the case of the evaluation of a policy, while an estimation of the current impact only requires to compare two “parallel worlds” with distinct air pollution levels), but these are not always considered and can be seen as minor technical differences. For these reasons, there are no essential differences in quantitatively assessing the effect of a single factor, of several factors, or of a policy or project. Recognizing this similarity in design of risk assessment and analytical/quantitative HIAs may allow to bring more clarity in the methodology and terminology; in particular, it may be relevant to adopt a unified terminology allowing to point to the differences that bear strong consequences, such as whether the study relies on human-based dose response functions (as illustrated here) or on dose–response functions derived from animal models.

Illustration of the similitude of the principles of risk assessment of an exposure ( A ) and of a policy or program ( B ). When considering an exposure ( A ), the fraction of disease cases attributable to a specific exposure (compared to a lower and theoretically achievable level) is estimated for time t (typically assumed to correspond to the current time). When considering a policy ( B ), the expected health benefit of the project or policy (consisting in changing the level of one or more environmental factors) is estimated, considering the population at the current time or at a later time t, comparing it to the situation without change. Both approaches can be seen as aiming to estimate the impact of a theoretical policy or intervention lowering (or, more generally, changing) the level of one or several environmental factors, compared to a reference situation considered at the same time period

The perils of quantification: leaving emerging hazards by the roadside

Risk assessment studies imply many steps requiring a large amount of data; this is all the more true in the case of studies considering simultaneously multiple exposures, exposures with effects on multiple health endpoints (such as tobacco smoke, particulate matter, lead, physical activity…), or policies likely to influence several exposures or behaviors (such as a “zero pollution action plan”, as envisioned by the European Commission, or a set of actions to limit greenhouse gas emissions in multiple sectors). In some (probably not infrequent) cases, only a fraction of the data relevant for the risk assessment will be available or possibly available within a limited time frame. Researchers are then facing several non-exclusive options:

• collect additional data (the scientifically rigorous approach); this may take a long time and be very expensive, since in many cases the missing data correspond to dose–response functions, which are typically generated by cohort studies. For example, in an ongoing exposome study conducted as part of ATHLETE project and considering 74 exposure-outcome pairs corresponding to effects with a level of evidence deemed likely or more than likely, a human dose–response function could be identified for only 70% of these possible effects (Rocabois et al., personal communication). Although working with high-quality data, even if this implies to delay the availability of the final results, is often the preferred option in science, such an option is problematic for health impact assessment studies, which often require to be conducted within a constrained time frame so that a decision about the planned policy or a possibly harmful exposure can be quickly taken, to bring potential health benefits to society or inform a legal process;
• perform the study with the limited data available in the constrained time frame (imperfect but timely approach); in this case, it is possible that only a fraction of the impact of the exposure(s) or policy will be assessed (because several dose response functions corresponding to the effects of the exposure or policy are available) and that the quantified fraction will be estimated with large uncertainties;
• perform a purely qualitative health impact assessment study (qualitative approach);
• not to perform the study (“analysis paralysis”).

In many cases, option 2), consisting in moving along with the limited data available, will be preferred. The consequence may be that a fraction, which may be large, of the impact, will be ignored. Thus, because of their relative complexity, health impact assessment studies, which aim to make health impacts visible, may paradoxically leave a large fraction of this impact on the roadside. Impacts left on the road side will often correspond to “emerging” (newly identified factors, newly identified effects) risks. Under this imperfect but timely approach, it is essential not only to try to provide a quantification of the uncertainty around the quantified part (see above, Sect. "  Sensitivity and uncertainty analyses "). It would also be relevant to attempt providing some estimate of the magnitude of what has obviously been left out (for example, the impact of a known exposure likely to affect an outcome for which no dose–response function is available), or at least to make the missing part (the “known unknown”) visible in some way.

Identified gaps

This review provided a general methodological framework for risk assessment studies and demonstrated their relevance to also consider the expected impact of policies and infrastructures, and therefore their closeness to health impact assessment studies; it illustrated recent development related to the diversity of approaches to assess factors at the individual levels (such as fine-scale environmental models and personal dosimeters), and the potentially strong impact of choices regarding exposure assessment tools, including the consideration of population density when environmental models are used. It also allowed to identify some gaps, challenges or pending issues in the methodology of risk assessment studies. These issues include 1) proposing a formal approach to the quantitative handling of the level of evidence regarding each exposure-health outcome pairs (see Handling of the strength of evidence about the effect of environmental factors on health); 2) more generally, develop more formal and if possible quantitative assessment of the health impacts not handled by a specific quantitative risk assessment study (the “know unknowns”); 3) confronting the approaches of risk assessment based on human dose–response function reviewed here with that relying on toxicological data; and 4) other technical issues related to the simultaneous consideration of several exposures (or of policies acting on health via changes in several environmental factors), in particular when some of these exposures are causally related.

Authors’ contributions

R.S. and M.R. wrote the main manuscript text, with contributions from S.M., J. Be. and J. Bu.All authors reviewed the manuscript.

The manuscript was conducted as part of HERA (Health Environment Research Agenda for Europe) project funded by Horizon Europe research and innovation programme from the DG Research of the European Commission (grant 825417).

Availability of data and materials

Declarations.

The authors declare no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

AI Risk Assessment: A Scenario-Based, Proportional Methodology for the AI Act

• Original Paper
• Open access
• Published: 07 March 2024
• Volume 3 , article number  13 , ( 2024 )

Cite this article

You have full access to this open access article

• Claudio Novelli   ORCID: orcid.org/0000-0003-4193-8604 1 ,
• Federico Casolari 1 ,
• Antonino Rotolo 1 ,
• Mariarosaria Taddeo 2 , 4 &
• Luciano Floridi 1 , 3  

769 Accesses

22 Altmetric

Explore all metrics

The EU Artificial Intelligence Act (AIA) defines four risk categories for AI systems: unacceptable, high, limited, and minimal. However, it lacks a clear methodology for the assessment of these risks in concrete situations. Risks are broadly categorized based on the application areas of AI systems and ambiguous risk factors. This paper suggests a methodology for assessing AI risk magnitudes, focusing on the construction of real-world risk scenarios. To this scope, we propose to integrate the AIA with a framework developed by the Intergovernmental Panel on Climate Change (IPCC) reports and related literature. This approach enables a nuanced analysis of AI risk by exploring the interplay between (a) risk determinants, (b) individual drivers of determinants, and (c) multiple risk types. We further refine the proposed methodology by applying a proportionality test to balance the competing values involved in AI risk assessment. Finally, we present three uses of this approach under the AIA: to implement the Regulation, to assess the significance of risks, and to develop internal risk management systems for AI deployers.

Similar content being viewed by others

Artificial Intelligence and Business Value: a Literature Review

Ida Merete Enholm, Emmanouil Papagiannidis, … John Krogstie

Cyber risk and cybersecurity: a systematic review of data availability

Frank Cremer, Barry Sheehan, … Stefan Materne

Ethical Decision-Making Theory: An Integrated Approach

Mark S. Schwartz

Avoid common mistakes on your manuscript.

1 Introduction: From Broad Scopes to Risk Scenarios

The European Artificial Intelligence Act (AIA) introduces a risk-based regulatory framework for AI systems (AIs), categorising them into four levels of risk: unacceptable, high, limited, and minimal. The legislator allocates regulatory burdens to AIs’ providers so that the greater the risk posed by AIs, the greater the legal safeguards to minimise it.

However, the AIA, in its capacity as a broad risk-based regulation, lacks a detailed risk assessment methodology for identifying risks, relying on a static view of AI risk. The four risk categories of AIs are mainly based on their technological features and the broad application areas. In short, AI is mostly seen as a product, akin to the EU product safety legislation. Footnote 1 , Footnote 2 In doing so, the AIA does not consider the interaction among hazard sources, vulnerability profiles, and exposed values, but treats them as stand-alone technical standards. This is coupled by the lack of a proportionality judgement between the risk mitigation measures and the principles and rights involved. As a result, the impact that AIs may have on European fundamental values and interests seems predetermined.

The AIA may misestimate the magnitude of AI risks—i.e., the likelihood of detriment and severity of consequences on values like health, safety, privacy, and others—and make the overall legal framework ineffective, that is, with rules that are either too stringent or too soft for the actual applications of specific AIs. The root of this problem is that the AIA has not yet progressed to the standardization phase, which is crucial for developing detailed guidelines. To address this challenge, the paper suggests a risk assessment methodology aimed at improving the accuracy and relevance of the AIA’s provisions. This methodology is not only applicable to enhancing the AIA but could also be beneficial for any other risk-based legislative framework governing AI.

We propose a risk assessment model that identifies and combines specific risk factors influencing real-world AI application scenarios. Footnote 3 While some legal arguments have been presented, suggesting a reading of the AIA’s risks approach considering tort law (Chamberlain, 2022 ), we draw from research and policy reports on climate change risk. In particular, we refer to the framework developed by the Intergovernmental Panel on Climate Change (IPCC) working groups and refined by the subsequent literature (Simpson et al., 2021 ). Accordingly, the risk of an event is assessed by the interplay between (1) determinants of risk (i.e., hazard, exposure, vulnerability, and responses), (2) individual drivers of determinants, and (3) other types of risk (i.e., extrinsic, and ancillary risks). This framework can provide a more accurate risk magnitude of AIs under a specific scenario. This is a measure defined based on hazard chains, the trade-off among impacted values, the aggregation of vulnerability profiles, and the contextualisation of AI risk with risks from other sectors.

This qualitative analysis is grounded on a quantitative assessment. In fact, the risk magnitude should be assessed by weighing the fundamental values (positively and negatively) affected by AIs against the intensity of the interference of AIA’s risk containment measures on the same values. This type of judgment for interference between constitutional principles is the object of the proportionality test by Robert Alexy (Alexy, 2002 ). The outcome of the test would indicate whether a risk category is appropriate for an AI under a specific risk scenario or whether it introduces grossly disproportionate limitations and trade-offs for competing values. Footnote 4

We consider three uses of this semi-quantitative risk framework under the AIA. First, for implementing the AIA with a different way of categorizing the risk. This means moving from a vertical approach, which categorizes AIs according to risk factors in isolation, to a horizontal approach, which categorizes AIs according to the interactions between various risk factors across different real-world scenarios. This nuanced assessment strategy is particularly suited for the delegated acts phase of the AIA, where implementation standards are established by the Commission with contributions from stakeholders and the scientific community. This phase offers an optimal setting for adopting the risk methodology recommended in this paper. Second, for enabling deployers of AIs initially classified as high-risk, based on the AIA’s predefined criteria, to dispute this categorisation. According to Recital 32, they must demonstrate that considering the severity, intensity, likelihood, duration, and potential targets, the overall risk is not significant. Footnote 5 Third, for setting up the internal risk management system for high-risk AI deployers as mandated by Article 9 of the AIA. This requirement involves identifying risks, considering AI uses and misuses, and evaluating new risks from post-market data. The second and third uses are seen as more feasible given the current legislative stage of the AIA.

The article is structured as follows. Section 2 presents the risk-based regulation of the AIA, bridging the risk model within the EU proposal and the ALARP principle. Section 3 discusses the strengths and weaknesses of the AIA risk-based regulation. Section 4 shows how to overcome the AIA’s model gaps by using the IPCC framework for climate change risk assessment updated by the relevant literature. Section 5 offers a quantitative support to the model through a proportionality test. Section 6 discusses the three potential uses of our semi-quantitative proposal under the AIA. Section 7 outlines the advantages of modifying the AIA’s strategy towards risk in its enforcement and regulation of General Purpose AI (GPAI). Section 8 concludes the article.

2 AIA’s Risk-Based Regulation

Generally, risk-based regulations consist of (at least) three phases: assessment, categorisation, and management (Millstone et al., 2004 ). In this article, we shall focus more on the first two phases and less on the AIA’s risk management system, that is, legal safeguards and requirements.

The AIA relies on the traditional conception that risk is the likelihood of converting a source of hazard into actual loss, injury, or damage. Footnote 6 Sources of danger are those uses of AI that are most likely to compromise safety, health, and other values. Footnote 7 Being the likelihood of damage, risk can be expressed through the ratio between hazard and safeguards so that, as the safeguards increase, the risk quotient decreases:

The risk may become untenable if safeguards do not offset severe hazards. The regulatory intervention should be proportionate to the hazards net of safeguards. Risk tolerance thresholds—in the AIA, the risk categories—indicate which risks are accepted without (strong) precautions and which instead require (further) mitigation practices.

In the AIA, the benchmark to calculate the risk of AIs is their potential adverse impact on health, safety, and EU fundamental rights. As a result, the AIA classifies AIs according to four risk categories: unacceptable, high, limited, and minimal (Kaplan & Garrick, 1981 ). Footnote 8 Stricter requirements are prescribed for suppliers and users of riskier AIs. This is explicitly stated in Recital 14 of the draft:

In order to introduce a proportionate and effective set of binding rules for AI systems, a clearly defined risk-based approach should be followed. That approach should tailor the type and content of such rules to the intensity and scope of the risks that AI systems can generate. Footnote 9

This is why the AIA modulates the legal requirements to make the risk of deploying AIs at least tolerable. The tolerance thresholds that constitute the AIA’s risk categorisation is compatible with the ALARP principle. ALARP is a general principle in UK law for risk management systems in safety-critical industries (Abrahamsen et al., 2018 ; Jones-Lee & Aven, 2011 ), and in the UK health system. Footnote 10 ALARP-type approaches involve a proportionality review of risk reduction measures so that they are not exorbitant to the improvement gained (Bai & Jin, 2016 ). Typically, ALARP provides the following risk tolerance ranges (Hurst et al., 2019 ):

This is a simplified version of the figure found in (Hurst et al., 2019 ). Note that the size of the three categories of the inverted pyramid is related to the severity and not to the numerousness of the relative risks.

Although the transposition into EU law of ALARP is limited Footnote 12 and controversial Footnote 13 , AIA’s risk categories overlap with the tolerance ranges shown in Fig. 1 . Footnote 14 These risk categories can be summarised as follows. Footnote 15

Unacceptable risk includes (AIA, Title II):

AIs that may cause significant harm through (a) subliminal manipulation of individuals’ consciousness that distorts their behaviour Footnote 16 or (b) exploitation of vulnerabilities—age, physical or mental disability—of a specific group of people that distorts the behaviour of its members.

AIs for social scoring that evaluate or classify natural persons or groups based on their social behaviour when social scoring leads to detrimental or unfavourable treatment (a) in social contexts that are unrelated to the contexts in which the data was originally generated or collected; (b) detrimental or unfavourable treatment are unjustified or disproportionate to the social behaviour of natural persons or groups.

AIs for biometric categorisation that categorise natural persons according to sensitive or protected attributes or characteristics (e.g., gender, ethnicity, political orientation, religion, disability) or based on the inference of those attributes or characteristics. Footnote 17

AIs for risk assessments of natural persons or groups to assess the risk for offending or reoffending or for predicting the occurrence or reoccurrence of (an actual or potential) criminal or administrative offence based on assessing personality traits and characteristics, such as the person’s location, past criminal behaviour of natural persons or groups of natural persons.

AIs for inferring emotions of a natural person in the areas of law enforcement, border management, in workplace and education institutions.

High-risk includes (AIA, Title III):

AIs used as safety components of products covered by the European New Legislative Framework (NLF) and other harmonised European regulations (Annex II, Sects. A and B). Regulated areas include, e.g., automotive, fossil fuels and medical devices.

AIs deployed in (a) biometric identification (when this is not forbidden) (b) management and operation of critical infrastructure, (c) education and vocational training, (d) employment, worker management and access to self-employment, (e) access to and enjoyment of essential private services and public services and benefits (e.g., healthcare), (f) law enforcement (g) migration, asylum and border control management, (h) administration of justice and democratic processes (Annex III).

Limited risk includes (AIA, Title IV):

AIs that interact with natural persons, e.g., chatbots, when this is not obvious from the circumstances and the context of use or is not permitted by law to detect, prevent and investigate criminal offences.

AIs that generate or manipulate images, audio, or video to simulate people, objects, places or other existing entities or events (i.e., deep fakes).

Minimal risk includes (AIA, Title IX):

Residual AIs, some examples are AIs for video games or spam filters.

AIs posing unacceptable risks fall into the ALARP ‘Intolerable’ risk range, i.e., situations whose risk cannot be justified except in extraordinary circumstances. Under the AIA, specific exempt circumstances, like terrorist attacks, allow the time-limited use of AIs for remote biometric identification in publicly accessible spaces for law enforcement (Article 5(d)).

AIs posing high and limited risks fall into the ‘Tolerable’ risk range. That is where the ALARP principle comes fully into play: risk is tolerated only if all reasonably practicable mitigation measures are implemented. However, what counts as ‘reasonably practicable’ might be tricky to determine. A predominant interpretation is that: “Efforts to reduce risk should be continued until the incremental sacrifice is grossly disproportionate to the value of the incremental risk reduction achieved. Incremental sacrifice is defined in terms of cost, time, effort, or other expenditures of resources” (Baybutt, 2014 ).

This judgement should therefore consider the expected utility of risk containment. In the AIA, reasonable efforts consist of the legal requirements and guarantee mechanisms that providers (and deployers) must comply with to place high-risk AIs on the single market (Article 6 et seq.). We shall analyse the ALARP principle, seeking to improve its enforcement in the AIA, in greater detail in Sect. 5 .

AIs posing minimal risks fall into the ALARP ‘Broadly Accepted’ risk range. In these cases, the risk is tolerable enough that no specific intervention is required, except to ensure compliance with good practices. This is also what the AIA prescribes by encouraging the adoption of voluntary codes of conduct either by individual providers of AIs or by their representative organisations (Article 69).

Much of the legal framework concerns high-risk AIs, prescribing conformity assessment procedures, technical documentation, and certification duties to place them on the market (e.g., Article 43). Sometimes these safeguards involve post-market monitoring (e.g., Article 61). The other three risk categories produce fewer and simpler regulatory burdens: AIs that pose unacceptable risks are prohibited (Article 5), those that pose limited risk trigger a general transparency obligation (Article 52), while for those that pose minimal risks the AIA fosters voluntary codes of conduct (Article 69). Footnote 18 An exception to these rules is provided in the AIA insofar as it requires the Member States to introduce regulatory sandboxes: controlled environments in which AIs can be developed and tested for a limited time, before putting them on the market, prioritising small providers and start-ups (Article 53 seq.).

3 Strengths and Weaknesses of the AIA’s Risk Regulation

The supranational legislator expects the regulation of AI to increase legal certainty in this field and to promote a well-functioning internal market: reliable for consumers, attractive for investment, and technologically innovative. Footnote 19 This might trigger the Brussels effect, ensuring a competitive advantage over other international policy-makers while shaping their regulatory standards (Bradford, 2020 ). Nevertheless, should the AIA prove to be unsustainable or ineffective, the EU may lose its attractiveness for the production and commercialisation of AI technologies. To prevent this, the AIA must introduce norms that promote safety while not disincentivising the production or deployment of AIs. Footnote 20 In this regard, the AIA’s risk-based approach has its strengths and weaknesses. Let us start with the strengths.

First, risk-based regulations rationalise governance interventions by setting their priorities and objectives. Well-delineated priorities and objectives facilitate accountability mechanisms towards the policy-maker (Black, 2010b ). In this respect, the AIA declares its priorities and objectives: the protection of the fundamental values and rights of the Union and the development of the AI market.

Second, risk-based regulations facilitate the fair distribution of resources (e.g., for supervision and certification) and costs. For example, costs are distributed according to the specific risks posed to a target community, and they are so transparently, as the criteria for distributing resources and costs are made evident in the regulation (Black, 2010a ). As the compliance cost is proportional to the risk, AIA introduces a kind of Pigouvian tax on the negative externalities of high-risk AIs (Baumol, 1972 ). To be acceptable, the AIA should allocate costs and resources efficiently among market players. However, the AIA does not consistently distribute resources in the best possible way, as we shall see when discussing its weaknesses.

Third, risk-based regulations cope with the uncertainty of phenomena—i.e., “when there is a lack of knowledge in qualitative or quantitative terms” Footnote 21 , Footnote 22 —for example, by qualifying predictions about the occurrence of specific hazards probabilistically (Rothstein et al., 2013 ). Moreover, risk-based regulations adapt to the political context or technological and market changes (Black & Baldwin, 2010 ). Footnote 23 In this regard, the AIA offers the possibility of updating its list of risky AIs at Articles 84–85. Unfortunately, the current version allows new AIs to be added only if they fall within the already established scopes. For this reason, some suggestions have been made to include reviewable risk categorisation criteria (Smuha et al., 2021 ).

By contrast, one of the main limitations of the AIA is the uncertainty about criteria for reviewing risk categorisation, which depends instead on the broad scopes of AIs. AI providers may be reluctant to invest in the EU’s AI market due to the perceived rigidness of the AIA guidelines and the absence of a mechanism for revising or adapting limitations and prohibitions as technological advancements occur. These advancements could potentially make certain AI systems that are currently considered risky less so. The AIA may preclude adapting risk categorisation to the interplay of hazard sources, vulnerability profiles of the exposed community, or values and interests at stake. No doubt, the model enshrined in the AIA heavily relies on a fundamental rights-based approach—as confirmed by the amendment introducing a fundamental rights impact assessment (AIA, Article 29a)—which characterizes the entire structure of the legislative proposal and, more broadly, the most recent pieces of legislation adopted at EU level in the digital context (Ufert, 2020 ). However, as legal compliance always comes at a cost (Khanna, 2021 ), if there is no possibility to ease regulatory burdens by a proportionality assessment, then the AIA might become unsustainable for AIs providers or deployers. This would be a severe loss for the EU AI strategy, disincentivising innovation and losing the benefits AI technologies can bring to those values the AIA aims to protect. The May 2023 amendment significantly advanced the regulation by allowing revisions to high-risk system classifications based on an assessment of the risk’s significance, i.e., its probability, severity, intensity, and potential population impact (AIA, Recital 32). However, this revision process is currently without a defined methodology or metrics. Our goal is to furnish support and clarify this mechanism (among others) by introducing a semi-quantitative risk assessment approach.

4 Addressing the Model Flaw: The IPCC Framework for Risk Assessment

The model flaw results from an insufficiently granular risk assessment model: the relevant factors of AI risk are not accurately identified and/or combined.

As argued in Sect. 2 , the AIA’s risk model is compatible with the ALARP principle and considers mainly two risk factors (a) the inherent risk of AI technology and (b) a value asset consisting of fundamental principles and rights of the Union. The EU legislator prescribes risk mitigation measures proportionate to the risk magnitude. As a result, risk management measures are allocated according to the four risk categories of the AIA.

Hence, the risk considered in the AIA is legal in nature, expressing the potential detriment that comes from the violation of a legal norm by an AIs (i.e., principles and rules) (Mahler, 2007 ). Footnote 24 However, the AIA’s risk assessment model does not fulfil the distinctive nature of the legal risk as it does not evaluate comparatively and proportionately the specific weight of legal norms. Quite the opposite, risk assessment in the AIA seems modelled as a neutral tool that treats legal norms as technical standards which are either met or not (Smuha et al., 2021 ). Consequently, the risk is categorised through a list of AI scopes potentially detrimental to fundamental principles and rights. But risk assessment is not a neutral tool: it reflects the risk appetite of a specific community, weighing the costs and benefits of risk mitigation (Krebs, 2011 ), balancing the interests and values of that community, and all this dynamically and diachronically; while promoting a legal value, it may be the case that the unexpected demotion occurs of other equally fundamental legal values. Accordingly, risk management measures should be modulated according to the outcome of such a balancing process. This adaptability is what the AIA needs to incorporate. In fact, despite claiming to be informed by the trade-off between economic development interest and the protection of fundamental rights, Footnote 25 the AIA seems to predetermine the proportionality judgment that settles the interference between values. Also significantly, not only the list of fundamental rights protected by the proposal is particularly rich, but it also includes interrelated rights, Footnote 26 making thus difficult a horizontal balance between competing fundamental rights.

The model flaw does not concern only the lack of granularity in the analysis of values and rights. The AIA also lacks an accurate representation of the hazards’ sources of AIs, of what makes people vulnerable to these hazards, and of whether hazards and vulnerabilities are mitigated by mechanisms, including legal ones, that already exist (i.e., the net risk) (Black & Baldwin, 2012 ).

Against this background, the May 2023 compromise text’s requirement for deployers of high-risk AI systems to conduct a fundamental rights impact assessment before market introduction is a progressive move. The methodology we propose in this section aims to enhance the accuracy of the proposed assessment outlined by the EU policymaker.

To improve the implementation of the AIA (Simpson et al., 2021 ), we propose a risk assessment methodology that includes multiple risk factors, and their interferences, and provides a proportionality judgement to review risk categories. This, however, without dismantling or multiplying the draft’s tolerance ranges. On the contrary, we suggest applying the four risk categories horizontally to each of the Ais listed in the AIA, so that under varying conditions—e.g., a specific interference among fundamental rights involved—the same system can be treated as unacceptable, high-risk, limited-risk or minimal-risk. This implies that risk categories would not depend by default on AI scopes, but on the real-world risk scenarios associated with the application of AI systems due to the incidence and combination of multiple risk factors.

To build risk scenarios, the Intergovernmental Panel on Climate Change (IPCC) provides a multifaceted risk assessment model, which has then been refined the subsequent literature (Simpson et al., 2021 ) and which we can use to assess risks of AIs. The risk magnitudes associated with both climate change and AI are influenced by a range of interacting factors, resulting in context-dependent outcomes. Recognizing this, we look to the IPCC model, which offers a detailed and widely recognized framework for assessing the trade-offs inherent in devising risk mitigation strategies.

The IPCC has often conceived the climate change risks—e.g., disaster risk—as the consequence of three determinants: hazard (H), exposure (E), and vulnerability (V). Footnote 27 Broadly speaking, hazard refers to the sources of potential adverse effects on exposed elements; exposure refers to the inventory of elements within the range of the hazard source; vulnerability refers to the set of attributes or circumstances that makes exposed elements susceptible to adverse effects when they impact the hazard source (Cardona et al., 2012 ; Liu et al., 2018 ). Footnote 28 The IPCC’s approach can be developed further, as in the framework for climate change risk assessment proposed by (Simpson et al., 2021 ), which evaluates risk at a lower level of abstraction by including the individual components of the risk determinants, i.e., the drivers. Simpson et al. expand the IPCC approach by incorporating a fourth risk determinant: the response (R), which refers to existing measures that counteract or mitigate risk. They also contextualise risk assessment by including multiple types of risk with their own determinants. Thus, according to their framework, the overall risk results from the interaction among (1) determinants, (2) drivers, and (3) risk types (Fig. 2 ). These three sets of relations occur at stages of increasing complexity. The AIA only considers the lowest complexity stage, where the relevant risk factors are the determinants taken statically, that is, overlooking interactions among their drivers (or with cross-sectorial risk types).

Three categories of increasingly complex climate change risk by (Simpson et al., 2021 )

The weight of each determinant is given by the drivers and their interactions, both within and across determinants. Interactions among drivers may be (i) aggregate, if drivers emerge independently of each other but jointly influence the overall risk assessment; (ii) compounding, if drivers produce a specific effect on risk assessment when combined, unidirectionally or bi-directionally; (iii) cascading, when drivers trigger others which themselves may produce further drivers in a cascading process. The same applies to interactions between multiple risk types (Simpson et al., 2021 ). Footnote 29 Figure 2 below shows the three sets of interactions.

In climate change, the drivers of the hazard (H) can be natural or human-induced events. In AI, these drivers may be either purely technological or caused by human-machine interactions: e.g., the opacity of the model, data biases, interaction with other devices, and mistakes in coding or supervision. The last three hazard drivers interact in an aggregate way. Interactions are compounded when, e.g., low data representativeness compounds with overfitted machine learning models or biased data. The interaction between drivers is cascading when, e.g., model opacity triggers cascading hazards of unpredictability, unmanageability, or threats to security and privacy. An accurate reconstruction of these interactions can provide evidence about the simplicity or complexity of the causal chain between hazard and harm, as well as its likelihood and distribution (Black & Baldwin, 2012 ).

Drivers of exposure (E) in climate change risk may be people, infrastructure, and other social or economic assets. For AI risk, exposure drivers may be tangible assets, like goods or environment, or intangible assets, like values and rights. As already stressed, the exposed asset of the AIA mainly consists of fundamental rights and values, such as health, safety, employment, asylum, education, justice, and equality. Interactions between drivers of exposure may be aggregated if, e.g., an AIs has adverse effects on the right to asylum and the privacy of asylum seekers. It is compounded when, e.g., an AI’s adverse effect on the environment compounds with those on health. The interaction between drivers of exposure is cascading when, e.g., an AI’s adverse effect threatens access to education, and thus equality and democratic legitimacy (and so on).

Vulnerability (V) drivers of climate change risk may concern the propensity to suffer adverse effects of communities—e.g., poverty—and infrastructure—e.g., lack of flood containment. Drivers of vulnerability in AI risks are multiple and overlapping, e.g., income, education, gender, ethnicity, health status, and age. The lack of appropriate control bodies, procedures, or policies should be included among the drivers of vulnerability for AI risk. Footnote 30 The AIA shows two conceptions of vulnerability: a generic one, whereby the mere entitlement to fundamental rights entails the propensity to suffer adverse effects of hazards; and a more specific one, whereby all those AIs that “[…] exploits any of the vulnerabilities of a specific group of persons due to their age, physical or mental disability”, (AIA, Article 5) should be banned. In the latter case, the list of vulnerability drivers is rather poor.

The interaction between vulnerability drivers is aggregated when, e.g., an AIs is deployed in a vulnerable environment, and there are few surveillance or feedback mechanisms. The compounding interaction is perhaps the most interesting one, as an intersectional reading of vulnerabilities can also be advocated in AI risk: ethnicity, gender, health, age, education, economic status, and other characteristics are profiles of vulnerability that have to be considered in the way they intersect and influence each other. The vulnerability stems from various interconnected social processes that lead to multiple dimensions of marginalisation (Kuran et al., 2020 ). In this sense, the intersectional approach to vulnerability is a risk management principle that enables policy-makers to identify the most appropriate measures to counter hazards to individuals and groups. These interactions make vulnerability a multi-layered condition (Luna, 2019 ). The interaction between vulnerability drivers is cascading when, e.g., the absence of AIs liability rules triggers several other vulnerabilities for those under the adverse effect of AIs use. Footnote 31

The analysis by Simpson et al. introduces a fourth determinant, i.e., response (R), which concerns existing measures that counteract or mitigate risk. The response indicates the environment’s resilience to a specific risk and includes governance mechanisms. Regarding AI risk, the response drivers can be institutional safeguards on the development, design, and deployment of AIs or data quality rules. Consequently, risk assessment and categorisation within the AIA should consider already existing legal measures to avoid the adverse effects of AI technologies, e.g., those contained in the GDPR. Footnote 32

Adaptation and mitigation responses may increase or decrease the risk level of specific AIs. As a result, the response determinant can be used to discriminate intrinsic from net risk, the latter adjusted to risk management measures:

[…] where the potential harm is higher than for the intrinsically lower risks, but the probability and/or impact is reduced by risk management and other control measures, or by systems of resilience – such as capital requirements in financial institutions, or engineered safety controls in power stations, or by the possibility of remediation (Black & Baldwin, 2012 , 5).

Simpson et al. also introduce a third stage of interaction, between climate change risk and other types of risk, which are extrinsic to it and have their own determinants. Risk types that interact with AI risk may be, e.g., market, liability, and infrastructure risks. Some of these risk types are created by the AI risk itself—i.e., cascading interactions—others are independent but may affect the overall assessment of AI risk—i.e., aggregate or compounded interactions. For instance, an aggregate interaction occurs between AI risk and policy risk, in the sense that adverse effects of ineffective policies or regulations—perhaps external to AI—cumulate with the adverse effects of AIs’ deployment. In a healthcare setting, for example, an AI system used for skin cancer diagnoses might inaccurately diagnose patients due to algorithmic biases (Gupta et al., 2016 ). Concurrently, outdated tort liability laws may not adequately address AI’s role in healthcare. This leads to compounded risks: the AI’s misdiagnoses are exacerbated by unclear liabilities, heightening overall risk magnitude in patient care. AI risk can then compound with the risk of the digital infrastructure in which an AIs operates.

Finally, AI risk can cascade into multiple other types of risk, the risk to innovation, to digital sovereignty, to economic sustainability, to power concentration, and so forth.

This third stage of interaction should be linked to that of ancillary risks, i.e., risks posed or increased by the risk regulation itself. For example, banning AIs should be justified also against the loss of opportunity benefit of their use, the potential barriers to technological innovation that the ban raises, and the risk posed by the systems replacing the banned ones (Sunstein, 2004 ). The AIA’s regulatory choices cannot be justified just by their positive impact on the intended scope—i.e. the protection of fundamental rights—but also by the (difference between) the marginal gains and harms they generate for other values at stake (Karliuk, 2022 ).

To sum up, the risk magnitude of each AIs listed in the AIA should be assessed in terms of the interactions among determinants, drivers, and other risk types. Although AIA considers some interactions among determinants—e.g., the scale and the likelihood of adverse effects on values—it does not account for the interaction among the individual drivers of those determinants, nor does it evaluate the risk of AIs in relation to other types of risk. Therefore, the AIA misestimates the AI risk magnitude and anchors risk categories to static, coarse-grained factors.

Once the determinants, drivers and external types of risk are identified, adaptation and mitigation become easier, i.e., to reduce the risk of AI by planning actions (including policies) that address the factors of hazards, exposure, and vulnerability (Simpson et al., 2021 ).

The granular risk assessment we propose has a higher degree of variability. The risk categories of the AIA become risk scenarios (Renn, 2011 ), which change depending on the interactions among risk factors. This leads to a more accurate representation of the risk magnitude—i.e., the likelihood of detriment and severity of consequences on values—with connections among risk factors being made explicit. Even if the EU legislator intends to keep the current framework—where risk categories are pre-determined based on the AI’s scopes—this model can aid in the proposed additional assessments that could revise the risk categorisation (i.e., risk significance). However, what we have presented in this section, is just a general framework. While risk magnitudes may correspond in the abstract to risk categories, as a preliminary evaluation, this assignment also must pass the proportionality test that we shall describe in the next section.

5 A Quantitative Basis for the Model: The Proportionality Test

Though not directly mentioned in the AIA, an issue shared with the ALARP principle is setting risk management measures, without defining what qualifies as a “grossly disproportionate” containment measure.

A way to offer quantitative support for ALARP-based legislative choices is through the traditional cost-benefit analysis (CBA) (French et al., 2005 ). While the ALARP allows the costs of risk mitigation to exceed the benefits as long as they are not exorbitant, the CBA specifies that intervention is justified only if costs are less than or equal to the benefits. CBA does not account for uncertain costs and benefits (Jones-Lee & Aven, 2011 ). Despite this drawback, CBA can support ALARP as a preliminary informational input: as far as possible, CBA quantifies known costs and benefits so that this information can be combined with a qualitative assessment of what is “reasonably practicable” (Ale et al., 2015 ). The risk assessment model presented in the previous section helps us to combine CBA, the ALARP principle, and the AIA to account for the likelihood and distribution of adverse effects, the causal chain between hazards and harms, the effects of AI risk regulation (i.e., ancillary risks), and alternative measures for risk mitigation. However, CBA remains an imperfect tool for the AIA, as the former expresses the value of things with a single numerical parameter, usually market prices, while the latter concerns a legal risk, whose exposed asset consists of fundamental rights and values, which, respectively, are intended to represent “principles of [EU] law of a constitutional nature” Footnote 33 and the “very identity” of the EU legal order. Footnote 34

However, we suggest an alternative (semi-)quantitative approach to ascertain when sacrifices to mitigate risk are “grossly disproportionate” (within the scope of the AIA). This quantitative assessment should be seen as complementary—a second step—to the risk assessment model of the previous section: to assign the appropriate risk category for a specific scenario, we need to compare the impact each risk category has on the assets served by the intended scope of the AIs ( P x )—e.g., law enforcement—against those of the exposed asset ( P y )—e.g., safety, health, and equality. Thus, if applying the high-risk category to AIs for law enforcement under a specific risk scenario has a sub-optimal impact on the joint realisation of principles and rights, it is desirable to opt for an alternative risk category. Whereas, if the marginal gains to law enforcement outweigh the marginal harms to other rights, then the risk category is justified.

Robert Alexy proposed a well-known method in legal theory to quantify this type of choices (Alexy, 2002 ). According to this approach, a legal norm that interferes with fundamental values Footnote 35 is legitimate when it meets a proportionality test characterised by the following optimisation principles:

Suitability , which “excludes the adoption of means obstructing the realisation of at least one principle without promoting any principle or goal for which they were adopted” (Alexy, 2003 ). In the AIA, the legislative choice of assigning a risk category R 1 to an AI that negatively impacts one principle P 2 is suitable if it impacts positively another principle P 1 .

Necessity , which “requires that of two means promoting P 1 that are, broadly speaking, equally suitable, the one that interferes less intensively in P 2 ought to be chosen” (Alexy, 2003 ). In other words, R 1 with a negative impact on P 2 is necessary if it has a positive impact on P 1 and there is no alternative, R 2 , having a higher positive impact on P 2 and non-inferior on P 1 (Sartor, 2018 ). In the AIA, as in many other cases, Pareto-optimality equilibria are rather unstable: multiple values are involved, and a principle P 3 that is negatively interfered with by R 1 can easily occur. These unavoidable costs call, according to Alexy, for a third principle.

Proportionality in the narrow sense , which states that “The greater the degree of non-satisfaction of, or detriment to, one principle, the greater the importance of satisfying the other” (Alexy, 2002 , 102). This principle provides a basis for determining whether or not the importance of satisfying P 1 with R 1 justifies the impairment or failure to satisfy P 2 . When multiple values are involved, as in the AIA, we will say that R 1 with a negative impact on P 2 is balanced if there is no alternative R 2 having a lower negative impact on P 2 and a higher overall utility on P 3 , P 4 … P n . (Sartor, 2018 ).

Such a proportionality test, which is (by and large) in line with the proportionality test the EU Court of Justice applies while balancing competing rights and values (Alexy, 2003 ; Tridimas, 2018 ), may support legislative choices and trade-offs within the AIA, i.e., the exposed asset of AI risk. Footnote 36 We suggest that it may serve to justify trade-offs between fundamental values/rights that (should) inform the risk categorisation of AIs. The outcome of the test may warrant the ascription of a risk category R 1 (e.g., high-risk) to specific AIs or shifting an AI to a new category R 2 (e.g., minimal risk). For this purpose, proportionality in the narrow sense should be broken down into three evaluations:

(1) the intensity of interference ( I x ), the degree of non-satisfaction or detriment to a principle P x to the benefit of a competing one P y (2) the concrete importance ( C y ) of satisfying P y (3) the concrete weight of P x ( W x , y ), namely the ratio between I x and C y, which determines whether the importance of satisfying P y justifies the non-satisfaction or detriment to P x (Alexy, 2003 ).

Finally, the abstract weights of P x ( W x ) and P y ( W y ) also play a role in the overall balance. Footnote 37

In some cases, P x will prevail over P y , e.g., when I x is severe, and C y is weak. In other cases, P y will prevail over P x . There may also be cases where there is no prevalence between P x and P y , I x = C y , which creates deadlocks, increasing discretion in balancing. The outcome of the ratio between the intensity of the interference on a specific principle and the concrete importance of the competing one is expressed by the following, simplified version, of the weight formula (Alexy, 2003 ):

Applying the weight formula to the AIA, I x would correspond to the degree of interference a risk category, with its containment measures, has on a (set of) value(s) served by the intended scope of AIs: e.g., the interference to public safety ( P x ) as served by biometric categorisation systems. C y would correspond to the concrete importance of satisfying a competing (set of) value(s) explicitly protected by the AIA, which is part of risk-exposed asset in biometric categorisation systems: e.g., the right to privacy ( P y ). The concrete importance expressed in C y depends on qualitative assessments in relation to the risk scenario, i.e., what are the hazard factors, vulnerability profiles and response mechanisms that determine the magnitude of risk in the concrete scenario (as described in the framework shown in the previous section). Therefore, whether the EU legislator is authorised to restrict the use of AIs for biometric categorisation will depend on whether the magnitude of the privacy risk posed by these systems ( C y ) justify the impairment of public safety caused by the measures of the relevant risk category ( I x ).

Although the weight formula relies on non-numerical premises—like judgments about the degree of interference of a risk category or the abstract weight of principles (Alexy, 2003 )—numerical values can still be assigned to I x and C y . This can be done using a geometric sequence, like 2 0 , 2 1 , 2 2 , 2 4 , to assign numerical ranges to the AIA’s four risk categories according to the degree of interference, or non-satisfaction, they cause to the intended scope of an AIs ( I x ): unacceptable risk = 16, high-risk = 4, limited risk = 2, minimal risk = 1. The same numerical ranges may be assigned to the importance of satisfying the competing principle—( C y ): major = 16, severe = 4, moderate = 2, light = 1 Footnote 38 —and to the abstract weights of principles ( W x and W y ). As shown below, where the asset served by the intended scope of an AIs prevails over the exposed asset, the concrete weight W x , y will be greater than 1. Conversely, W x , y will be less than 1.

I x · W x (16 · 4)/ C y · W y (8 · 2) = 4

I x · W x (4 · 4)/ C y · W y (8 · 16) = 1/8

This quotient describes the concrete weight of the asset served by the intended scope of an AIs given the interference of a risk category on it (I x ) and a competing asset protected for being partly exposed to the AIs (C y ). The inclusion of the vulnerability and response determinants’ values in the ratio can make the proportionality test fully aligned with the risk assessment model outlined in Sect. 4 .

To sum up, the quotient of the weight formula is a quantitative criterion to assess whether the risk control measures are “grossly disproportionate” in the AIA, given the balance of relevant values, and therefore whether a risk category is suitable for the risk scenario of an AIs or whether it should be changed. In particular, what is grossly disproportionate can be quantified over a range. In our example, according to the numerical parameters we employed, it is reasonable to argue that the quotient of the weight formula should not be less than 1 or greater than 4. If it falls outside this range, then the balancing between principles is disproportionate and it is advisable to alter the risk category. Indeed, out of the range, a specific risk category may be inadequate for the risk scenario, with measures too stringent or too soft to balance competing EU values, like privacy and technological innovation. In this way, the AIA fails to achieve one of its main objectives: a uniform protection of EU fundamental rights.

We are aware that compulsory numerical values of EU principles and fundamental rights cannot be pre-assigned. Also, attempts to establish a strict hierarchy among EU fundamental values and rights have so far failed. Footnote 39 While acknowledging the importance of these circumstances, we believe that a quantitative method for assessing risk containment measures could help relevant actors make policy decisions and avoid significant imbalances when implementing the AIA. Numerical values have been assigned to the coefficients in the proportionality test to enhance clarity, but these coefficients can also be compared through non-numerical preferences or magnitudes, such as the Paretian superiority illustrated in (Sartor, 2018 ).

On a different note, we cannot ignore the role that EU institutions—and, in particular, the role that the EU Court of Justice—shall play in preserving the constitutional framework of the Union and the untouchable core of the EU legal order, which include its fundamental values and rights. This is why in the next section, we shall discuss the allocation of competences and roles in scenario building and proportionality assessments.

6 Using the Semi-Quantitative Risk Approach: Three Applications Under the AIA

We illustrate three potential uses of this semi-quantitative risk framework under the AIA. These applications are not mutually exclusive; they can be implemented simultaneously. The first is intended for policymakers, while the second and third are aimed at deployers of high-risk AI systems.

(1) When implementing the AIA. This implies transitioning from a scope-oriented categorisation of risk to a scenario-based model that considers the interplay of multiple factors in specific situations (as in Sect. 4 ). The four risk categories should then be applied horizontally to AIs so that, under varying risk scenarios, the same system can be estimated as unacceptable, high-risk, limited-risk or minimal-risk.

However, this application would pose some practical issues. In fact, although the categorisation of risk in the AIA is coarse-grained, its strategy of connecting risk measures to broad scopes of AIs makes it easier to approve and monitor them. Indeed, as previously highlighted in the context of regulation by design in the GDPR (Almada et al., 2023 ; Michelakaki & Vale, 2023 ), a finely-grained approach may fall short in offering sufficient guidance to regulated actors. In contrast, a legal framework with risk scenarios built on interacting factors and tested by proportionality-based balancing, as the semiquantitative model we are presenting, might complicate the procedures laid down in the AIA.

This issue is still manageable: under the existing AIA framework, national supervisory authorities (AIA, Title VI) could undertake the task of constructing risk scenarios. However, this approach would alter the governance structure of the AIA, which currently operates predominantly at a supranational level for regulating high-risk AIs. For this reason, it would be crucial to determine the competences, functions and interactions of supranational institutions and national bodies in the risk assessment of AIs. Considering the shared nature of the competences exercised by the EU legislator to adopt the AIA, Footnote 40 it remains undisputed that the EU legislator should retain a primary role in shaping the risk-assessment model at stake. Footnote 41 Meanwhile the European Commission should keep its role of guardian of the AIA enforcement and the EU Court of Justice’s authority in judging whether the risk assessment is consistent with the essential core of EU fundamental values (Lenaerts, 2019 ). This is particularly important considering the systematic backsliding on fundamental values and rights taking place in some EU countries.

More to the point, under our semi-quantitative model, the implementation acts of the EU Commission might establish (a) the key drivers of the four risk determinants, (b) the extrinsic types of risk to account for and (c) the (abstract) weight of the principles involved in the proportionality test. This task may be done by the AI Office, new body set up by the AI Act within the European Commission. This entity is designed to ensure a harmonized application of the AI Act through standardisation, provide guidance, and coordinate joint cross-border investigations. Anyway, these factors could be linked precisely to the scopes already identified in the AIA through the broader risk categorisation (e.g., Annex III). Footnote 42 In this way, the broad scopes of AIs would still play a primary role in risk regulation—which means that the text of the AIA would not require substantial changes—and EU institutions would limit the discretion of Member States. National authorities would be responsible for assessing risk in particular cases—thereby enhancing their powers over what is in the AIA—through scenarios and proportionality tests. Footnote 43

Detractors could claim that the proposed solution may lead to a partially diversified enforcement of the AIA within the EU, conflicting with the EU’s uniform values and rights. However, this position overlooks the necessity of context-sensitive risk assessment. Moreover, our method does not necessarily weaken the effectiveness of the EU’s fundamental values and rights as it is based on the idea of introducing a robust rational procedure, under the strict supervision of the European Commission and the ultimate control exercised by the EU Court of Justice.

To conclude, while some would prefer centralized risk assessment, our proposal to recalibrate the regulation during implementation seems more feasible. This means adhering to the AIA’s existing risk categories but applying them with a focus on the interplay of risk factors and proportionality. At the same time, among the three uses of the scenario-model we are discussing, this one appears to be the least practicable. This is due to the characteristics of the normative environment surrounding the AIA—specifically, the product safety framework (Almada & Petit, 2023 )—and the legislative stage at which the AIA currently stands. The next two applications we shall explore are less constrained by path dependency and the AIA’s normative integrity. However, they also are finely-grained and this generates challenges that we shall address at the end of this section.

(2) When Evaluating the Significance of Risk (AIA, Recital 32a). The assessment of risk significance was initially introduced by the EP compromise text from 11 May 2023, and later confirmed by the consolidated version dated January 26, 2024, as per the COREPER text. According to the proposal, AI systems to be classified as high-risk must also pose what is called a ‘significant risk’, requiring evaluation of the risk’s severity, intensity, likelihood, duration, and potential targets, whether an individual, multiple people, or a specific group (e.g., AIA, art. 3 (1b)). Footnote 44 If a deployer can demonstrate that her system does not pose a significant risk, contrary to the initial categorisation based on broad AI scopes, she can then reclassify the risk level of their system. The compromise text mandates these evaluations to be conducted by deployers but with the obligation to inform national supervisory authorities, relevant stakeholders, and representative groups of individuals who may be impacted by the application of the (high-risk) AI system (e.g., Recital 32 and Article 29a). We believe this option could be extended to systems initially categorized as posing unacceptable risks.

However, the proposal does not offer a methodology for integrating these metrics. Our model addresses this, offering a way to calculate risk significance. This approach enables a flexible review procedure, allowing deployers to reduce regulatory burdens when they can provide evidence of lower risk.

(3) When Implementing Deployers’ Internal Risk Management (AIA, Art. 9). Deployers of high-risk AI systems, as mandated by Article 9 of the Artificial Intelligence Act (AIA), must establish an internal risk management system. This system is aimed at ensuring compliance with the AIA and the reliability of high-risk AI systems. Article 9 requires deployers to identify both known and foreseeable risks associated with their AI systems, considering their intended uses and potential misuses. This includes the assessment of additional risks that may emerge, informed by data from post-market monitoring. Deployers are required to implement effective risk management measures, such as risk elimination or reduction, through thoughtful design and development processes. The risk management systems must be thoroughly documented and regularly updated throughout the AI system’s lifecycle.

The AIA’s outline of this risk management system, while not highly detailed, aligns with international standards (such as ISO) Footnote 45 and the historical evolution of the concept (Dionne, 2013 ). However, the AIA does not provide a shared procedure for these in-house risk management systems. Likely, further details will be provided in the Commission’s delegated and implementing acts.

In this context, our scenario-based model offers a common framework, a detailed terminology, granular risk factors, and a dynamic methodology for integrating them in real-life situations. And the internal risk management system may benefit from it. The scenario-based model may help identifying “known risks” and assists in evaluating risks as more or less “foreseeable” (as per Article 9, letter (b)). This evaluation depends on factors like response mechanisms (e.g., prevention measures), hazard drivers and the way they combine (e.g., aggregating or cascading). Our model also includes extrinsic and ancillary risks and their impact on the main risk magnitude, as outlined in Sect. 4 , which may help assessing “other possibly arising risks based on the analysis of data gathered from the post-market monitoring system” (Article 9, letter c). This approach enhances the identification of “suitable risk management measures” as required by Article 9, letter d, of the AIA. It clarifies the effectiveness of current response mechanisms in mitigating vulnerability drivers and explains how hazard drivers impact these vulnerabilities. This insight is crucial for understanding the overall risk magnitude, allowing for a more targeted and effective refinement of risk management strategies within the AI system. Furthermore, the scenario analysis may also be probabilistically qualified through the risk matrix that we shall illustrate in Sect. 7 .

Finally, the proportional testing of rights and interests, discussed in Sect. 5 , plays a significant role in various aspects of the internal risk management system. This includes identifying mitigating measures and testing procedures (art. 9 point 2 and point 6) that are both appropriate and proportional to the community’s risk appetite (seen as the ratio of values and the interests involved).

Unrelated to risk management yet somewhat linked to the obligations of high-risk systems’ deployers, proportional testing could also be relevant for the fundamental rights impact assessment (FRIA) recently introduced in the draft. Footnote 46

There is a common aspect to all three applications that needs consideration. Granular risk categorisation based on scenarios which combines multiple risk factors, as we suggest, might be supererogatory and complicate the AIA procedures. To figure out the severity of this, we need to distinguishing short-term from long-term aspects.

In the short run, a scenario-based risk assessments may indeed deter AI deployment and investment. To mitigate this, different strategies might be recommended to make our proposal more sustainable. Firstly, European legislation might indicate, in the AIA’s implementing acts, the key risk drivers for each broad AI scopes already outlined in the regulation (e.g., in the Annex III). This would ease the task of deployers and minimizes arbitrariness in the AIA’s enforcement. We shall illustrate this in the next section. Second, automating risk identification and management can streamline processes. Finally, a phased, iterative approach starting with a granular risk assessment only for a few deployers—maybe with lower risky systems and then with lower compliance costs—might enable procedural refinement and prepares others for a smoother implementation.

In the long run, these short-term costs will be offset by the benefits of decreased compliance costs as contextually tailored risk assessments yield less over-inclusive risk categories and more effective risk prevention or mitigation measures.

7 An Illustration Case for Showing the Contributions of the Semi-Quantitative Approach

Our analysis offers two contributions to the enforcement of the AIA and the regulation of general-purpose AI (GPAI).

First, the risk assessment model shown in Sect. 4 , supported by the quantitative proportionality test shown in Sect. 5 , improves the enforcement of the AIA. This improvement applies whether it is used for the horizontal implementation of risk categories, evaluating risk significance, or within deployers’ internal risk management systems. It would provide risk management measures that are more appropriate to estimate and contain the dangers of AI, more specific for national regulators (and judges), more sustainable for AI providers, and ultimately more likely to achieve the AIA’s goals of protecting all fundamental EU values involved. Ideally, such granular risk management measures can help avert, or more effectively handle, issues related to the under-inclusiveness or over-inclusiveness of risk categories (Hacker, 2023 ).

To show how risk scenario building might work in the AIA, let us consider a case study, that of AIs used to assess the recidivism rate of natural persons in criminal trials. The semiquantitative approach consists of two stages: risk-scenario building and the proportionality test. The risk drivers here identified can be easily inferred from the AIA. Of course, applying our proposed assessment model during the AIA implementation stage would necessitate enhanced legislative transparency in setting the drivers and extrinsic risks.

Starting from the risk-scenario building, the four determinants of AI risk, the interaction among their drivers and with other risk types may be thus combined:

Hazards. These drivers of an AI for recidivism rate assessment may be the inner opacity of the system, its reliability, the poor quality or misuse of the training data (e.g., outdated or incomplete data sets), and its validity. Validity is critical to ensure that the instrument measures what it is intended to measure (Quattrocolo, 2020 ). When these hazard drivers compound, they can lead to the AIs perpetrating discrimination biases. For instance, the AI might consistently and incorrectly predict higher recidivism rates for certain demographic groups due to poor training data, compounded by its opacity and unreliability. This risk magnitude differs from a scenario where these drivers do not compound, such as when the decision-making process is transparent, allowing stakeholders to identify and rectify biased data issues more readily. The greater these hazard drivers are, and the more likely they combine to produce such wrongdoing, the “heavier” the hazard determinant will be in the specific risk scenario. What is more, the hazards must be connected to the vulnerability drivers of a specific environment in which AIs are deployed, not least because these will be inclined to replicate the social discriminations of the environment.

Exposure. These driving factors include the directly impacted parties, such as defendants, and, indirectly, the fundamental values potentially affected by the use of an AI to assess the recidivism rate. This would involve the interest to an efficient trial together with some substantive legal principles—e.g., the principle of criminal culpability and of equality—or procedural ones—e.g., the principle of transparency and the right of/to defence (Garrett & Monahan, 2020 ). These drivers also interact with each other, and where they interfere, it is necessary to balance them to assess the overall weight of the exposure determinant. This also requires balancing those values that the use of AIs is intended to enhance (consistent with the proportionality test in Sect. 5 ), such as the principle of predictability, legal certainty, safety, and efficiency. An example of interference occurs when a recidivism assessment AI system is designed for procedural fairness and equality by only using unbiased data. This approach neglects defendants’ personal narratives and circumstances, clashing with the interest for an individualized justice. This focus on depersonalized data can overlook important personal factors like rehabilitation efforts or life challenges, crucial for a fair assessment.

Vulnerability. These drivers may be the attributes that make individuals or groups susceptible to the adverse effects of automatic recidivism rate assessment, such as ethnicity, economic conditions, and education. So, for instance, the lack of data literacy among judges, lawyers, and defendants may compromise their ability to grasp the workings of AI systems, their limitations, and the implications of their use in legal settings. This lack of data literacy and awareness can lead to overreliance on or misinterpretation of AI-generated recidivism predictions. When these drivers interact with each other, perhaps compounding or cascading, vulnerability should be treated as a multi-layered condition (Luna, 2019 ): e.g., the compound of ethnicity and socio-economic conditions often leads to a heightened sensitivity to the biases of prediction systems. As mentioned above, drivers of vulnerability compound with hazard drivers: e.g., biases in the recidivism rate assessment will be greater where social discriminations are already in place.

Response. These are pre-existing strategies designed to mitigate the risks associated with automated recidivism rate assessments. They might be governance measures, like standards for data quality and data collection, transparency, bias examination, and human oversight. An example includes regulations mandating the exclusion of specific indicators that, while predicting some degree of social dangerousness, are directly or indirectly linked to ethnic or social background, e.g., the postal code (van Dijck, 2022 ).

Extrinsic risks. The risk of AIs for recidivism rate assessment would finally interact with extrinsic risk types. Some extrinsic risks, in this case, would be compliance risk, liability risk, and economic risk. Indeed, AI risk may be influenced by the lack of effective rules for the allocation of liabilities for adverse effects. If the system’s prediction leads to an unjustly prolonged detention, the developers or users of the AI could face lawsuits. This scenario could then lead to or exacerbate economic risks within the AI market, as potential legal repercussions might deter investment. The overall risk magnitude should also consider ancillary risks. For example, risks arising from inefficient regulation or poor risk management can impact innovation, lead to lost opportunities, and affect digital sovereignty. The introduction of regulatory burdens, or entry barriers, on AIs’ providers may weaken technological innovation and, in the case of a radical ban, resulting in the loss of opportunity for the general social interest.

The interactions among these risk factors determine the two input variables of the overall risk magnitude of the specific scenario: (1) the likelihood of the event depend on the interaction between hazard drivers and response drivers (e.g., preventive measures); (2) likewise, the severity of the detriment can be higher or lower depending on the hazard sources, exposed asset, and vulnerability profiles. Footnote 47 As a result, risk magnitude is associated with the four risk categories of the AIA—i.e., unacceptable (U), high (H), limited (L) and minimal (M) risk—as illustrated in the risk matrix below Footnote 48 (Table 1 ):

The five levels of severity are described qualitatively, and those of likelihood in percentages in a range between 0 and 1 (where 0.20 − 1 is remote risk, while 0.80 − 1 the risk is almost certain). Under this matrix, the intersection of the input variables correlates with one of the four risk categories of the AIA. Footnote 49

The second step is to evaluate the suitability of the resulting risk category in relation to the asset exposed to the use of an AIs, by means of the proportionality test. Let us assume that the risk magnitude for a specific recidivism rate assessment system matches its current categorisation in the AIA, i.e., unacceptable risk (U). One of the principles served by AIs for assessing recidivism rates is safety (P x ) and, according to the geometric sequence seen in Sect. 5 , its abstract weight can be quantified with a score of 4 (W x ). The degree of interference (I x ) of the AIA’s high-risk category on legal certainty is 16. In the denominator of the Weight Formula, the abstract weight of a competing principle, e.g., criminal culpability (P y ), might be 4 (W y ) as well as the concrete importance of satisfying it (C y ). Applying all these values to the ratio—W x,y = (I x ∙ W x )/(C y ∙ W y )—the outcome would be 4, which falls within the proportionality range we have assumed. As a result, we might conclude that the risk category is appropriate as it correctly balances the values involved. Of course, if the competing principle was deemed to be less significant, for instance, it held a light value such as 2, then the outcome of the equation might not fall within the range and the risk category should be revised.

The second contribution of the risk assessment model presented here concerns one of the regulatory issues that emerged from the debate on the AIA: the governance of general-purpose AI (GPAI), which is the label that includes the so-called Generative AI (e.g., large language models like ChatGPT). The issue was raised in an amendment proposing a definition of GPAI and classifying them as high-risk systems. Footnote 50 In the consolidated version, they have an autonomous three-tier risk classification. Anyway, GPAIs are systems that can be deployed in multiple fields and with different tasks, some of which were unintended by the developers). This definition would also include open-source AI models (e.g., open-source datasets).

Indeed, if the intended purposes are not foreseeable, due to their scalability and vast flexibility, neither are the fundamental values that AIs would affect and based on which their risk would be categorised. This implies that the application of the AIA would be even more static than for AIs with intended purposes. Therefore, the construction of risk scenarios based on determinants, drivers and types seem the only way to categorise and regulate GPAI in a granular manner and avoid treating them all the same. Given these AI technologies’ success on the market, undifferentiated regulatory treatment might negatively impact AI industry innovation.

The semi-quantitative model outlined in this article would facilitate risk assessment and categorisation for all those situations that the AIA leaves uncovered, for example, where it recognizes the discretion of the European Commission in updating or modifying the list (and to remove use-cases) of high-risk AIs provided that:

[…] (b) the AI systems pose a significant risk of harm to health and safety, or an adverse impact on fundamental rights, to the environment, or to democracy and the rule of law, and that risk is, in respect of its severity and probability of occurrence, equivalent to or greater than the risk of harm or of adverse impact posed by the high-risk AI systems already referred to in Annex III. (AIA, Article7, point 1 (b))

However, to determine whether the risk associated with new AI systems matches or exceeds those already classified as high-risk, thus justifying their addition to the high-risk list, a robust and transparent risk assessment methodology is necessary. This is something the AIA does not currently offer, but an attempt to provide such a methodology has been made in this paper.

8 Conclusions

In this paper, we have offered a semi-quantitative approach to AI risk assessment, articulated in two stages: (1) the construction of risk scenarios and (2) a proportionality-based quantitative assessment. For scenario construction, we have referred to the IPCC’s theoretical framework and the literature on climate change risk. Accordingly, risk results from the interaction among four determinants, among individual drivers of determinants, and among extrinsic types of risk. For the second stage, we have referred to the quantitative approach developed by Alexy for balancing legal principles. Such a quantitative assessment aims to check whether the risk category assigned following the scenario construction is proportionate to the values involved in employing AIs.

We have discussed three applications of a semi-quantitative risk framework under the AI Act (AIA). Firstly, the framework supports the horizontal implementation of the AIA by introducing a scenario-based risk model. It would categorize AI systems variably as unacceptable, high-risk, limited-risk, or minimal-risk, depending on specific situations. Secondly, it aids in the evaluation of risk significance, providing deployers with the opportunity to reassess the risk levels of their AI systems, which could lead to reduced regulatory burdens. Thirdly, it facilitates and harmonizes the implementation of the internal risk management system as outlined in Article 9 of the AIA, by offering a comprehensive framework to help deployers in identifying, evaluating, and managing risks. Our findings suggest that the latter two applications—evaluating risk significance and implementing internal risk management—are more practical and less challenging to realize, considering the AIA’s framework and legislative process.

Finally, we have pointed out that a semi-quantitative approach can improve the enforcement of the AIA and help address issues uncovered by the EU regulation, e.g., risk assessment for the GPAI, without undermining the protection of EU fundamental values and rights. Future research should investigate further governance issues, including identifying which institutional bodies are called upon to apply the semi-quantitative risk analysis, with what specific faculties and with how much discretion in evaluating risk factors.

Data Availability

Not applicable.

Cf. European Commission, Explanatory Memorandum to AIA, para 1.3.

Still, it is compatible with a risk management standard employed in safety-critical industries in other legislations. An example we shall consider is the United Kingdom’s ‘As Low As Reasonably Practicable’ (ALARP) principle, which we will use as a framework to introduce the AIA’s risk classification.

We shall use the expression ‘risk factors’ to refer in a general way to all variables potentially able to increase or decrease the risk of an event. We shall specify its meaning by referring to determinants and drivers.

Robert Alexy has been a pioneer in developing quantitative approaches to balancing principles. His work is widely respected for its clarity and rigor, making him a valuable reference for scholars and practitioners alike. Additionally, his Weight Formula offers a promising framework for implementing the fundamental rights impact assessment (RIA) recently introduced in the draft legislation.

The risk significance evaluation has been introduced by the European Parliament compromise text on the AIA from 11 May 2023.

This conception can easily be deduced from some sections of the draft, for instance, Recital 32 referring to high-risk systems: “[…] high-risk AI systems other than those that are safety components of products…it is appropriate to classify them as high-risk if, in the light of their intended purpose, they pose a high risk of harm to the health and safety or the fundamental rights of persons, taking into account both the severity of the possible harm and its probability of occurrence […]”.

The AI Act will with the OECD definition of AI: “An AI system is a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that [can] influence physical or virtual environments. Different AI systems vary in their levels of autonomy and adaptiveness after deployment”. Source: https://www.euractiv.com/section/artificial-intelligence/news/ai-act-eu-policymakers-nail-down-rules-on-ai-models-butt-heads-on-law-enforcement/ .

The text refers to three categories, but a fourth sub-category of high-risk systems can be derived from the presence of lighter obligations.

Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts, Com/2021/206 final, Recital 14.

UKHSE. Risk management: Expert guidance - ALARP at a glance. https://www.hse.gov.uk/managing/theory/alarpglance.htm . However, the principle is also recognised in other legal systems, like the US, sometimes under the formula “as low as reasonably achievable” (ALARA)

A specific reference may be found in the EU legislation on medical devices: cf. Annex I, Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, [2017] OJ L117/1.

In particular, the debate developed when the European Commission argued that the use of ALARP (in the homologous version of SFAIRP) in the UK’s Health & Safety at Work Act was not consistent with the European “Framework Directive” for occupational safety and health (Directive 89/391/EEC), asking thus the Court of Justice to declare that the Member State failed to fulfil its obligation to correctly transpose the Directive. However, the European Court of Justice, without taking a specific position on the compatibility of ALARP with the Directive’s provision, dismissed the action brought by the European Commission, maintaining that the EU institution did not clearly identify the legal standard enshrined in the Directive that the UK failed to implement (Case C-127/05 Commission v UK EU:C:2007:338, para 58).

Indeed, a textual reference to ALARP can be found in the section where the AIA describes the mandatory risk management system for high-risk systems: “In identifying the most appropriate risk management measures, the following shall be ensured: (a) elimination or reduction of risks as far as possible through adequate design and development; […]”.

This list is updated to the May 2023: Draft Compromise Amendments on the Draft Report (COM(2021)0206 - C9 0146/2021 - 2021/0106(COD)). However, the text is not yet conclusive.

Literature indicates that the notion of subliminal techniques remains ambiguous under the AI Act (Neuwirth, 2022 ).

When the exonerating circumstances provided for in Articles 5(1)(d) and 5(2)(4) are not met.

Codes of conduct can be created by individual providers or their representative organisations.

These are explicitly stated objectives of the AIA draft (p. 3).

Of course, other factors will determine the success of the European AI strategy, like taxation and administrative efficiency. However, in this paper, we will only address the regulatory framework, namely the risk-categorisation of the AIA.

van der Heijden J. Risk as an approach to regulatory governance: An evidence synthesis and research agenda. SAGE Open 2021;11(3):215.

Sometimes, the concepts of risk and uncertainty are kept separate, the former being considered calculable and the latter not. For this purpose, the distinction between epistemic and aleatory uncertainty may be relevant, with only the latter being effectively addressable through risk assessment. On this, see Renn O. Risk governance: coping with uncertainty in a complex world. London: Routledge 2011; 368.

At the same time, excessive uncertainty must be seen as a limitation of any risk model.

This is at least one of the meanings that the concept of legal risk can take, and it is the one associated with the Basel Committee on Banking Supervision’s definition: “Legal risk includes, but is not limited to, exposure to fines, penalties, or punitive damages resulting from supervisory actions, as well as private settlements”.

This is clearly stated in the Explanatory Memorandum of the Proposal: “To achieve those objectives, this proposal presents a balanced and proportionate horizontal regulatory approach to AI that is limited to the minimum necessary requirements to address the risks and problems linked to AI, without unduly constraining or hindering technological development or otherwise disproportionately increasing the cost of placing AI solutions on the market”.

Cf. European Commission, Explanatory Memorandum to AIA, para 3.5.

This conceptual approach is clearly set out in (Cardona et al., 2012 ). This approach also emerges in special IPCC reports, e.g., Special Report on Climate Change and Land—IPCC site 2019 https://www.ipcc.ch/srccl/ and Special Report on the Ocean and Cryosphere in a Changing Climate 2018 https://www.ipcc.ch/srocc/ .

That hazard, exposure, and vulnerability are relevant to risk assessment is also widely believed in the literature other than climate change, such as in (Renn, 2011 ). In studies on global environmental change and sustainability, the same four determinants were considered as parts of a risk sequence chain (Turner et al., 2003 ).

Simpson NP, et al. (n 6).

Further specifications can be made. For instance, it has been proposed to categorize exposure and vulnerability drivers into four types: ontological, passive, active, and intentional, as in (Liu et al., 2018 ).

In the proposed fundamental rights impact assessment, interest in vulnerability is emphasized: “This assessment should include […] (f) specific risks of harm likely to impact marginalised persons or vulnerable groups” (Article 29a, (f)).

Consider, for example, art. 35 on data protection impact assessment: “1. Where a type of processing in particular using new technologies, and taking into account the nature, scope, context and purposes of the processing, is likely to result in a high risk to the rights and freedoms of natural persons, the controller shall, prior to the processing, carry out an assessment of the impact of the envisaged processing operations on the protection of personal data.”

Joined cases C-402/05 P and C-415/05 P Kadi and Al Barakaat International Foundation EU:C:2008:461, para 276.

Case C-156/21 Hungary v European Parliament and Council of the European Union EU:C:2022:97, para 127

In Alexy’s theory, these fundamental values are typically constitutional principles.

Other formal models have been developed to balance the impact of actions and decisions on values in the legal domain. For instance, consider the model developed by (Maranhão et al., 2021 ).

Alexy also includes another variable in his weight formula, namely the epistemic reliability of the balancing premises. For simplicity of exposition, we will not consider them here.

This requires assuming that the abstract weights have the same impact on the concrete weight as the intensity of interference.

This incapacity can stem from non-commensurability of values involved. While our model assumes commensurability, Amartya Sen has demonstrated that even in the presence of non-commensurability, rational decisions can still be made using a systematic approach. This approach involves acknowledging the limitations of our value judgments and making reasoned choices based on available information and analysis. Sen proposes the ‘weighted average principle’ as a method for making decisions when direct comparisons are not possible, considering the relative differences between options (Sen, 2004 , 50).

As it is well-known the AIA proposal is based in the first place on Article 114 TFEU, providing a EU shared competence in adopting measures to ensure the establishment and proper functioning of the internal market. In addition, the proposal is based on Article 16 TFEU, due to its connection to the processing of personal data.

On the fundamental role the EU legislator should play in this respect, see (Fontanelli, 2016 ).

In the next section, we offer a case study that illustrates how key drivers of the four determinants and extrinsic risk types may be identified in connection to the scope of an AIs, i.e., justice (Sect. 7 ).

Or any other type of proportionality-based balancing.

A second update from the comprimse text mandates deployers of high-risk systems to conduct a fundamental rights impact assessment and develop a risk mitigation plan in coordination with the national supervisory authority and relevant stakeholders before market entry (e.g., AIA, Recital 58a and Article 29a).

ISO 31000:2018 Risk Management—Guidelines, Clause 3.2: “coordinated activities to direct and control an organization with regard to risk”

The FRIA was introduced in the European Parliament proposal through Article 29a.

These are the same input variables of the conception of risk magnitude embraced by the AIA (e.g., Title III, art. 7).

The risk matrix approach is widespread in semi-quantitative risk assessments, such as the one we are suggesting (Ni et al., 2010 ).

For example, someone else might think it more correct that a moderate detriment with a probability between 0.20 and 0.40 percent should correspond to a high-risk category.

GPAIs were excluded from the previous draft of the AIA. However, they are given more room in the compromise text, as implementations of foundation models, and are no longer equate with high-risk systems. They must still adhere to certain documentation and transparency rules. For instance, generative foundation models must always disclose that the content was AI-generated (AIA, Recital 60g).

Abrahamsen, E. B., Abrahamsen, H. B., Milazzo, M. F., & Selvik, J. T. (2018). Using the ALARP principle for safety management in the energy production sector of chemical industry. Reliability Engineering & System Safety , 169 (January), 160–165. https://doi.org/10.1016/j.ress.2017.08.014

Article   Google Scholar  

Ale, B. J. M., Hartford, D. N. D., & Slater, D. (2015). ALARP and CBA All in the Same Game. Safety Science , 76 (July), 90–100. https://doi.org/10.1016/j.ssci.2015.02.012

Alexy, R. (2002). A theory of constitutional rights . Oxford University Press.

Google Scholar  

Alexy, R. (2003). On balancing and subsumption. A structural comparison. Ratio Juris , 16 (4), 433–449. https://doi.org/10.1046/j.0952-1917.2003.00244.x

Almada, M., Maranhão, J., & Sartor, G. (2023). Art. 25. Data protection by design and by default . Nomos, Beck, and Hart Publishing. https://cadmus.eui.eu/handle/1814/75913

Almada, M., & Petit, N. (2023). The EU AI act: A medley of product safety and fundamental rights? Working Paper. European University Institute. https://cadmus.eui.eu/handle/1814/75982 .

Bai, Y., & Jin, W.-L. (2016). Chapter 38 - Risk assessment methodology. In Y. Bai & W.-L. Jin (Eds.), Marine Structural Design (2nd ed., pp. 709–723). Butterworth-Heinemann. https://doi.org/10.1016/B978-0-08-099997-5.00038-1 .

Chapter   Google Scholar  

Baumol, W. J. (1972). On taxation and the control of externalities. The American Economic Review , 62 (3), 307–322.

ADS   Google Scholar  

Baybutt, P. (2014). The ALARP principle in process safety. Process Safety Progress , 33 (1), 36–40. https://doi.org/10.1002/prs.11599

Article   CAS   Google Scholar  

Black, J. (2010a). Risk-based regulation: Choices, practices and lessons being learnt . OECD. https://doi.org/10.1787/9789264082939-11-en

Book   Google Scholar  

Black, J. (2010b). The role of risk in regulatory processes (R. Baldwin, M. Cave, & M. Lodge, Eds.) (pp. 302–348). New York, USA: Oxford University Press. http://ukcatalogue.oup.com/

Black, J., & Baldwin, R. (2010). Really responsive risk-based regulation. Law & Policy , 32 (2), 181–213. https://doi.org/10.1111/j.1467-9930.2010.00318.x

Black, J., & Baldwin, R. (2012). When risk-based regulation aims low: Approaches and challenges. Regulation & Governance , 6 (1), 2–22. https://doi.org/10.1111/j.1748-5991.2011.01124.x

Bradford, A. (2020, March). The brussels effect: How the European union rules the world . Faculty Books. https://scholarship.law.columbia.edu/books/232

Cardona, O. D., Van Aalst, M. K., Birkmann, J., Fordham, M., Mc Gregor, G., Rosa, P., Pulwarty, R. S., et al. (2012, January). Determinants of risk: Exposure and vulnerability. Managing the risks of extreme events and disasters to advance climate change adaptation: Special report of the intergovernmental panel on climate change , pp. 65–108. https://doi.org/10.1017/CBO9781139177245.005

Chamberlain, J. (2022, December). The risk-based approach of the European union’s proposed artificial intelligence regulation: Some comments from a tort law perspective. European Journal of Risk Regulation , 1–13. https://doi.org/10.1017/err.2022.38

Dijck, G. V. (2022). Predicting recidivism risk meets AI act. European Journal on Criminal Policy and Research , 28 (3), 407–423. https://doi.org/10.1007/s10610-022-09516-8

Dionne, G. (2013). Risk management: History, definition, and critique. Risk Management and Insurance Review , 16 (2), 147–166. https://doi.org/10.1111/rmir.12016

Fontanelli, F. (2016, January). The court of justice of the European union and the illusion of balancing in internet-related disputes. The internet and constitutional law: The protection of fundamental rights and constitutional adjudication in Europe , 94–118. https://doi.org/10.4324/9781315684048

French, S., Bedford, T., & Atherton, E. (2005). Supporting ALARP decision making by cost benefit analysis and multiattribute utility theory. Journal of Risk Research , 8 (3), 207–223. https://doi.org/10.1080/1366987042000192408

Garrett, B., & Monahan, J. (2020). Judging risk. California Law Review , 108 (2), 439–493.

Gupta, A. K., Bharadwaj, M., & Mehrotra, R. (2016). Skin cancer concerns in people of color: Risk factors and prevention. Asian Pacific Journal of Cancer Prevention: APJCP , 17 (12), 5257–5264. https://doi.org/10.22034/APJCP.2016.17.12.5257

Article   PubMed Central   PubMed   Google Scholar  

Hacker, P. (2023). The European AI liability directives—Critique of a half-hearted approach and lessons for the future. arXiv. https://doi.org/10.48550/arXiv.2211.13960

Hurst, J., McIntyre, J., Tamauchi, Y., Kinuhata, H., & Kodama, T. (2019). A summary of the ‘ALARP’ principle and associated thinking. Journal of Nuclear Science and Technology , 56 (2), 241–253. https://doi.org/10.1080/00223131.2018.1551814

Article   ADS   CAS   Google Scholar  

Jones-Lee, M., & Aven, T. (2011). ALARP—What does it really mean? Reliability Engineering & System Safety , 96 (8), 877–882. https://doi.org/10.1016/j.ress.2011.02.006

Kaplan, S., & Garrick, B. J. (1981). On the quantitative definition of risk. Risk Analysis , 1 (1), 11–27. https://doi.org/10.1111/j.1539-6924.1981.tb01350.x

Karliuk, M. (2022, October). Proportionality principle for the ethics of artificial intelligence. AI and Ethics . https://doi.org/10.1007/s43681-022-00220-1

Khanna, V. S. (2021). Compliance as costs and benefits. In B. van Rooij & D. D. Sokol (Eds.), The Cambridge handbook of compliance (pp. 13–26) Cambridge Law Handbooks. Cambridge University Press. https://doi.org/10.1017/9781108759458.002

Krebs, J. R. (2011). Risk, uncertainty and regulation. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 369 (1956), 4842–4852. https://doi.org/10.1098/rsta.2011.0174

Article   ADS   MathSciNet   Google Scholar  

Kuran, C. H. A., Morsut, C., Kruke, B. I., Krüger, M., Segnestam, L., Orru, K., Nævestad, T. O., et al. (2020). Vulnerability and vulnerable groups from an intersectionality perspective. International Journal of Disaster Risk Reduction , 50 (November), 101826. https://doi.org/10.1016/j.ijdrr.2020.101826

Lenaerts, K. (2019). Limits on limitations: The essence of fundamental rights in the EU. German Law Journal , 20 (6), 779–793. https://doi.org/10.1017/glj.2019.62

Liu, H.-Y., Lauta, K. C., & Maas, M. M. (2018). Governing boring apocalypses: A new typology of existential vulnerabilities and exposures for existential risk research. Futures, Futures of Research in Catastrophic and Existential Risk , 102 (September), 6–19. https://doi.org/10.1016/j.futures.2018.04.009

Luna, F. (2019). Identifying and evaluating layers of vulnerability—A way forward. Developing World Bioethics , 19 (2), 86–95. https://doi.org/10.1111/dewb.12206

Article   PubMed   Google Scholar  

Mahler, T. (2007). Defining legal risk. SSRN Scholarly Paper. Rochester, NY. https://papers.ssrn.com/abstract=1014364

Maranhão, J., de Souza, E. G., & Sartor, G. (2021). A dynamic model for balancing values. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL’21 , pp. 89–98. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3462757.3466143

Michelakaki, C., & Vale, S. B. (2023, May). Unlocking data protection by design & by default: Lessons from the enforcement of article 25 GDPR. https://policycommons.net/artifacts/3838751/fpf-article-25-gdpr-a4-final-digital/4644643/

Millstone, E., van Zwanenberg, P., Marris, C., Levidow, L., & Torgersen, H. (2004). Science in trade disputes related to potential risk: Comparative case studies. Other. Seville, Spain: European Commission. http://ftp.jrc.es/EURdoc/eur21301en.pdf

Neuwirth, R. J. (2022). The EU artificial intelligence act: Regulating subliminal AI systems. SSRN Scholarly Paper. Rochester, NY. https://doi.org/10.2139/ssrn.4135848

Ni, H., Chen, A., & Chen, N. (2010). Some extensions on risk matrix approach. Safety Science , 48 (10), 1269–1278. https://doi.org/10.1016/j.ssci.2010.04.005

Quattrocolo, S. 2020. Artificial intelligence, computational modelling and criminal proceedings: A framework for a European legal discussion. Vol. 4. Legal studies in international, European and comparative criminal law. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-52470-8 .

Renn, O. (2011). Risk governance: Coping with uncertainty in a complex world . Routledge. https://doi.org/10.4324/9781849772440

Rothstein, H., Borraz, O., & Huber, M. (2013). Risk and the limits of governance: Exploring varied patterns of risk-based governance across Europe. Regulation & Governance , 7 (2), 215–235. https://doi.org/10.1111/j.1748-5991.2012.01153.x

Sartor, G. (2018). A quantitative approach to proportionality. In C. Aitken, A. Amaya, K. D. Ashley, C. Bagnoli, G. Bongiovanni, B. Brożek, C. Castelfranchi, et al. (Eds.), Handbook of legal reasoning and argumentation (pp. 613–636). Springer Verlag.

Sen, A. (2004). Incompleteness and reasoned choice. Synthese , 140 (1/2), 43–59.

Article   MathSciNet   Google Scholar  

Simpson, N. P., Mach, K. J., Constable, A., Hess, J., Hogarth, R., Howden, M., Lawrence, J., et al. (2021). A framework for complex climate change risk assessment. One Earth , 4 (4), 489–501. https://doi.org/10.1016/j.oneear.2021.03.005

Article   ADS   Google Scholar  

Smuha, N., Ahmed-Rengers, E., Harkens, A., Wenlong, L., Maclaren, J., Piselli, R., & Yeung, K. (2021, August). How the EU can achieve legally trustworthy AI: A response to the European commission’s proposal for an artificial intelligence act. Artificial Intelligence - Law, Policy, & Ethics eJournal . https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3899991

Sunstein, C. R. (2004). Risk and reason . Cambridge Books, Cambridge University Press. https://ideas.repec.org/b/cup/cbooks/9780521016254.html

Tridimas, T. (2018). The Principle of Proportionality. In R. Schütze & T. Tridimas (Eds.), Oxford principles of European union law: The European union legal order: Volume I . Oxford University Press. https://doi.org/10.1093/oso/9780199533770.003.0010

Turner, B. L., Kasperson, R. E., Matson, P. A., McCarthy, J. J., Corell, R. W., Christensen, L., Eckley, N., et al. (2003). A framework for vulnerability analysis in sustainability science. Proceedings of the National Academy of Sciences , 100 (14), 8074–8079. https://doi.org/10.1073/pnas.1231335100

Ufert, F. (2020). AI regulation through the lens of fundamental rights: How well does the GDPR address the challenges posed by AI? European Papers - A Journal on Law and Integration , 5 (2), 1087–1097. https://doi.org/10.15166/2499-8249/394

Download references

Open access funding provided by Alma Mater Studiorum – Università di Bologna within the CRUI-CARE Agreement.

Author information

Authors and affiliations.

Department of Legal Studies, University of Bologna, Via Zamboni, 27/29, Bologna, 40126, Italy

Claudio Novelli, Federico Casolari, Antonino Rotolo & Luciano Floridi

Oxford Internet Institute, University of Oxford, 1 St Giles’, Oxford, OX1 3JS, UK

Mariarosaria Taddeo

Digital Ethics Center, Yale University, 85 Trumbull Street, New Haven, CT, 06511, USA

Luciano Floridi

Alan Turing Institute, British Library, 96 Euston Rd, London, NW1 2DB, UK

You can also search for this author in PubMed   Google Scholar

Contributions

Corresponding author.

Correspondence to Claudio Novelli .

Ethics declarations

Informed consent, competing interests.

Luciano Floridi is part of the Editorial Board of Digital Society.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Novelli, C., Casolari, F., Rotolo, A. et al. AI Risk Assessment: A Scenario-Based, Proportional Methodology for the AI Act. DISO 3 , 13 (2024). https://doi.org/10.1007/s44206-024-00095-1

Download citation

Received : 14 September 2023

Accepted : 15 February 2024

Published : 07 March 2024

DOI : https://doi.org/10.1007/s44206-024-00095-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Risk assessment
• Proportionality
• Artificial Intelligence
• Find a journal
• Publish with us
• Track your research