Lessons Learned from Assessing Trustworthy AI in Practice.

lWe published the Lessons Learned from Assessing Trustworthy AI in Practice.

Dennis Vetter, Julia Amann, Frederick Bruneault, Megan Coffee, Boris Düdder, Alessio Gallucci, Thomas Krendl Gilbert, Thilo Hagendorff, Irmhild van Halem, Dr Eleanore Hickman, Elisabeth Hildt, Sune Holm, George Kararigas,Pedro Kringen, Vince Madai , Emilie Wiinblad Mathez, Jesmin Jahan Tithi, Ph.D , Magnus Westerlund, Renee Wurth, PhD, Roberto V. Zicari & Z-Inspection® initiative (2022)

Digital Society (DSO), 2, 35 (2023). Springer

Link: https://link.springer.com/article/10.1007/s44206-023-00063-1

Abstract

Building artificial intelligence (AI) systems that adhere to ethical standards is a complex problem. Even though a multitude of guidelines for the design and development of such trustworthy AI systems exist, these guidelines focus on high-level and abstract requirements for AI systems, and it is often very difficult to assess if a specific system fulfills these requirements.

The Z-Inspection® process provides a holistic and dynamic framework to evaluate the trustworthiness of specific AI systems at different stages of the AI lifecycle, including intended use, design, and development. It focuses, in particular, on the discussion and identification of ethical issues and tensions through the analysis of socio-technical scenarios and a requirement-based framework for ethical and trustworthy AI.

This article is a methodological reflection on the Z-Inspection® process. We illustrate how high-level guidelines for ethical and trustworthy AI can be applied in practice and provide insights for both AI researchers and AI practitioners. We share the lessons learned from conducting a series of independent assessments to evaluate the trustworthiness of real-world AI systems, as well as key recommendations and practical suggestions on how to ensure a rigorous trustworthiness assessment throughout the lifecycle of an AI system.

The results presented in this article are based on our assessments of AI systems in the healthcare sector and environmental monitoring, where we used the framework for trustworthy AI proposed in the Ethics Guidelines for Trustworthy AI by the European Commission’s High-Level Expert Group on AI. However, the assessment process and the lessons learned can be adapted to other domains and include additional frameworks.

An extended version of this article is available in Zicari et al. (2022)