In response to the OECD's open public consultation on AI risk thresholds, I am pleased to have contributed my perspectives on the key questions raised. As AI systems evolve and become more deeply integrated into society, it is crucial to establish robust frameworks for managing potential risks. This consultation represents a valuable opportunity for stakeholders from diverse sectors to share insights that can inform the development of effective AI governance.

My responses aim to address the challenges and opportunities associated with establishing risk thresholds for advanced AI systems, taking into account both compute power and other relevant factors. I have also provided suggestions for how governments and companies might approach setting, identifying, and measuring these thresholds to ensure the responsible development and deployment of AI technologies. See my responses below.

1.     What publications and/or other resources have you found useful on the topic of AI risk thresholds?

Several publications and resources have been insightful in understanding AI risk thresholds, including:
i. NIST AI Risk Management Framework – This framework aims to provide a voluntary, rights-preserving, and flexible resource for organisations across all sectors to manage AI risks and promote the trustworthy and responsible development and use of AI systems. See (https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf)

ii. The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation by Brundage et al. – This report highlights the potential security threats posed by misuse of AI and outlines different risk scenarios, which can inform the development of AI risk thresholds. See (https://arxiv.org/pdf/1802.07228)

iii. EU Artificial Intelligence Act – The EU's AI Act categorises AI systems into risk levels, offering a legal framework that could serve as a basis for establishing thresholds.

iv. The International Scientific Report on the Safety of Advanced AI: Interim Report addresses the risks associated with general-purpose AI, categorizing them into categories such as malicious use risks, risks from malfunctions, systemic risks, and cross-cutting risk factors. Moreover, the report emphasizes the need for a shared, scientific understanding of these risks to ensure the safe development and deployment of AI technologies.

See (https://www.gov.uk/government/publications/international-scientific-report-on-the-safety-of-advanced-ai/international-scientific-report-on-the-safety-of-advanced-ai-interim-report#risks)

v. The paper titled On the Limitations of Compute Thresholds as a Governance Strategy critically examines the reliance on compute thresholds. Specifically, it argues that Governance that depends too heavily on computing overlooks the fact that the connection between computing power and risk is highly unpredictable and quickly evolving. The paper argues that, although compute thresholds are currently utilized by regulators in frameworks such as the EU AI Act and U.S. Executive Orders on AI Safety, their effectiveness is constrained by several significant factors. See (https://arxiv.org/html/2407.05694v1)

vi. The paper titled Risk thresholds for frontier AI discusses the risks associated with advanced AI systems and explores approaches to managing the risks of frontier ai systems namely, capability thresholds and risk thresholds. It argues for using risk thresholds to guide decision-making but acknowledges that capability thresholds are currently more practical for reliable evaluation. See (https://arxiv.org/pdf/2406.14713)

2. To what extent to you believe AI risk thresholds based on compute power are appropriate to mitigate risks from advanced AI systems?

AI risk thresholds based on compute power can serve as a useful tool for flagging potentially risky systems, particularly during the development of advanced models. However, as the paper On the Limitations of Compute Thresholds as a Governance Strategy suggests, relying solely on compute power is inadequate for a thorough risk evaluation. Moreover, the author Sara Hooker argues that the ever-evolving link between compute and AI performance, along with the uncertainty in deployment scenarios, demands a more flexible and comprehensive approach. Therefore, while thresholds based on compute are beneficial, they should be integrated into a broader, adaptive risk management framework that also takes into account the system’s intended use, the data it operates on, and the environment in which it functions. Focusing exclusively on compute power risks overlooking significant dangers in lower-compute systems that might be deployed in critical situations.

2.     To what extent do you believe that other types of AI risk thresholds (i.e., thresholds not explicitly tied to compute) would be valuable, and what are they?

According to Koessler et al., capability thresholds outline the conditions under which a high-risk activity can proceed, either by ensuring the model's capabilities remain below a certain threshold or by implementing sufficient safety measures if they exceed it. These thresholds serve as a specific form of risk acceptance criteria, distinct from traditional risk thresholds. In addition, capability thresholds are used by companies like Anthropic, OpenAI, and Google DeepMind to determine when additional safety measures are necessary, emphasizing that a model's capabilities, rather than just its compute power, are crucial for assessing and managing risk effectively. See (https://arxiv.org/pdf/2406.14713)

It is also important to consider the following risk thresholds.
i. Transparency and explainability – Setting thresholds for the level of transparency required in AI decision-making processes.

ii. Human oversight – Establishing thresholds for the extent of human involvement in AI operations to ensure accountability.

4. What strategies and approaches can governments or companies use to identify and set out specific thresholds and measure real-world systems against those thresholds?

To ensure responsible AI deployment, governments and organizations should involve diverse stakeholders, including technologists, ethicists, and the public, in setting acceptable AI risk levels. Pilot programs can test AI systems in controlled settings, while continuous monitoring can ensure adaptability to emerging risks. Multidisciplinary panels comprising experts from various fields should assess AI risks comprehensively. Scenario planning and stress testing can help to establish appropriate thresholds, which should remain adaptive to real-world data and societal values. Transparency in AI development and reporting can ensure that these thresholds are met and maintained.

5. What requirements should be imposed for systems that exceed any given threshold?

For AI systems exceeding risk thresholds, mandatory external audits can assess compliance with ethical and safety standards, while enhanced transparency mechanisms will require developers to provide detailed documentation on system design and decision-making processes. Organizations must conduct impact assessments and develop mitigation plans, with high-risk systems potentially requiring regulatory approval before deployment. Rigorous testing and validation of safety, reliability, and ethical implications are critical. Transparency reports and human oversight protocols, including accountability mechanisms, will ensure ongoing monitoring and proper governance in critical decision-making areas.

6. What else should the OECD and collaborating organisations keep in mind with regards to designing and/or implementing AI risk thresholds?

AI risk thresholds should be developed with international coordination to ensure consistency and prevent regulatory fragmentation across borders. While it is crucial to mitigate risks, these thresholds must be flexible enough to encourage innovation, allowing for experimentation while maintaining safety. Ethical considerations, such as fairness, transparancy and accountability, inclusivity, and human rights, should be at the core of their design. Also, thresholds need to be forward-looking, capable of adapting to rapid advancements in AI technology. Governments and organisations should also focus on educating stakeholders including the public about the importance and role of these AI risk thresholds in safeguarding society.

Kindly review the submission here and note there is an opportunity to provide your own insights until October 1, 2024.