AI and Ethics: Reports/Papers classified by topics

The Ethics of Artificial Intelligence (AI),  AI and Trust.

– Ethical and societal implications of algorithms, data, and artificial intelligence: a roadmap for research. Whittlestone, J. Nyrup, R. Alexandrova, A. Dihal, K. Cave, S. (2019), London. Nuffield Foundation.  Link to .PDF

– Ethics Guidelines for Trustworthy AI. Independent High-Level Expert Group on Artificial Intelligence. European commission, 8 April, 2019. Link to .PDF

– WHITE PAPER. On Artificial Intelligence – A European approach to excellence and trust.  European Commission, Brussels, 19.2.2020 COM(2020) 65 final. Link to .PDF

Ethics, Moral Values, Humankind, Technology, AI Examples.

– Perspectives on Issues in AI Governance, Lynette Webb, Charina Chou, Google White Paper, 2019. Link to .PDF

–  AI on the Case: Legal and Ethical Issues. Richard Austin, Deeth Williams Wall LLP , May 17, 2019. Link to .PDF

– Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System, Partnership on AI, 2019. Link

– Dirty Data, Bad Predictions: How Civil Rights Violations Impact Police Data, Predictive Policing Systems, and Justice (February 13, 2019). Richardson, Rashida and Schultz, Jason and Crawford, Kate, 94 N.Y.U. L. REV. ONLINE 192 (2019). Available at SSRN

Fairness, Bias and Discrimination in AI.  From Philosophy to Machine Learning.

– Improving Fairness in Machine Learning Systems: What Practitioners Need? K. Holstein et al. CHI 2019; May 4-0, 2019. Link to .PDF

– Ensuring, Fairness in Machine Learning to Advance Health, Alvin Rajkomar et al.  Equity, Annals of Internal Medicine (2018). DOI: 10.7326/M181990. Link to .PDF

– Putting Fairness Principles into Practice: Challenges, Metrics, and Improvements. Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Allison Woodruff, Christine Luu, Pierre Kreitmann, Jonathan Bischof, Ed H. Chi (Submitted on 14 Jan 2019). Link to .PDF

– Dissecting racial bias in an algorithm used to manage the health of populations. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Science. 2019 Oct 25;366(6464):447-453. doi: 10.1126/science.aax2342. Link to .PDF

-AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias, Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, Yunfeng Zhang, 2018, Paper linkOpen Source project linkPublished as IBM Journal of Research and Development 63(4/5), 2019

AI: Explainability, Transparency.

Experiences with Improving the Transparency of AI Models and Services. Michael Hind, Stephanie Houde, Jacquelyn Martino, Aleksandra Mojsilovic, David Piorkowski, John Richards, Kush R. Varshney (Submitted on 11 Nov 2019)arXiv:1911.08293v1Link to .PDF

– Petkovic D, Kobzik L, Re C. “Machine learning and deep analytics for biocomputing: call for better explainability”. Pacific  Symposium on Biocomputing Hawaii, January 2018;23:623-7, Link to .PDF

– Petkovic D, Kobzik L, Ganaghan R,“AI Ethics and Values in Biomedicine – Technical Challenges  and Solutions”, Pacific Symposium on Biocomputing, Hawaii January 3-7, 2020, Link to .PDF

Gunning D, Aha D.:”DARPA’s Exianable Artificial Intelligence Program”, AI magazine, Association for the Advancement of Artificial  Intelligence, Summer 2019, slides

– Ribeiro M, Singh S, Guestrin C. “Why Should I Trust You? Explaining the Predictions of Any Classifier”, KDD ’16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningAugust, 2016,  Link to .PDF or ACM PDF

– Ribeiro M, Singh S,  Guestrin C.: ”Nothing Else Matters: Model-Agnostic Explanations by Identifying Prediction Invariance”, 30th Conf. of Neural Information Processing Systems (NIPS 2016), Barcelona, Spain 2016, Link to .PDF

– Petkovic D, Altman R, Wong M, Vigil A.: “Improving the explainability of Random Forest classifier – user centered approach”. Pacific Symposium on  Biocomputing. 2018;23:204-15, Link to .PDF

– D. Petkovic, A. Alavi, D. Cai, J. Yang, S. Barlaskar:  “RFEX – Simple random Forest Model and Sample Explainer for non-ML experts”, Link to .PDF

– Barlaskar S, Petkovic D: “Applying Improved Random Forest Explainability (RFEX 2.0) on synthetic data”, SFSU TR 18.01, 11/27/20181; with related toolkit at

– One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques, Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilovic, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney, Dennis Wei, Yunfeng Zhang, 2019, Link to .PDF

– Explaining explainable AI, Michael Hind, XRDS: Crossroads, The ACM Magazine for Students 25(3), ACM, 2019, Link to .PDF

– Explaining decisions made with AI. – ICO and The Alan Turing Institute, May 2020, Link

Human in the loop, Security, and Accountability.

– Calvo, R.A., Peters, D. & Cave, S. Advancing impact assessment for intelligent systems. Nat Mach Intell, Vol 2, 89-91 (2020).  Link to. PDF

– Concrete Problems in AI Safety, Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. Link to PDF

– Anomalous Instance Detection in Deep Learning: A Survey – Saikiran Bulusu, Bhavya Kailkhura, Bo Li, Pramod K. Varshney, Dawn Song. Link to PDF

– Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. Miles Brundage et al. Link to PDF

Introduction to Z-inspection. A framework to assess Ethical AI

Mindful Use of AI.  Z-Inspection: A holistic and analytic process to assess Ethical AI – Roberto V. Zicari, Frankfurt Big Data Lab, July 2, 2020, Youtube video.  Copy of the slides: Zicari.Lecture.October15.2020

– Brusseau J. , What a Philosopher Learned at an AI Ethics Evaluation, AI Ethics Journal 2020, 1(1)-4,

Assessing AI use cases. Socio-Technical Scenarios.

– Ethical Framework for Designing Autonomous Intelligent Systems. J Leikas et al. J. of Open Innovation, 2019, 5, 1. Link

Assessing AI use cases. Ethical tensions, Trade offs.

– Algorithmic Impact Assessment: A Practical Framework for Public Agency Accountability, Reisman D., Schultz J, Crawford K, Whittake M, AI Now, April 2018. Link to .PDF

– FactSheets: Increasing trust in AI services through suppliers declarations of conformity. Arnold, M.; Bellamy, R. K. E.; Hind, M.; Houde, S.; Mehta, S.; Mojsilovic ́, A.; Nair, R.; Natesan- Ramamurthy, K.; Olteanu, A.; Piorkowski, D.; Reimer, D.; Richards, J.; Tsay, J.; and Varshney, K. R. 2019. IBM Journal of Research & Development 63(4/5). Link to .PDF

–  Datasheets for datasets. Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J. W.; Wallach, H.; Daume ́, III, H.; and Craw- ford, K. 2018.. In Proceedings of the Fairness, Accountability, and Transparency in Machine Learning WorkshopLink to .PDF

–  IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems, Ethically Aligned DesignA Vision for Prioritizing Human Well-being with Autonomous and Intelligent SystemsFirst EditionPp. 211 – 281.Link to .PDF

– COVID-19 Rapid Evidence Review: Exit through the App Store?, Nuffield Foundation. Link

EMPOWERING CITIZENS AGAINST COVID-19 WITH AN ML-BASED AND DECENTRALIZED RISK AWARENESS APPSpeaker : Professor Yoshua Bengio (Mila), Friday, May 8, 2020Recording: here

AI Ethics in Healthcare

–  Stealth research: Lack of peer‐reviewed evidence from healthcare unicorns, Ioana A. Cristea Eli M. Cahan John P. A. Ioannidis, European Journal of Clinical Investigation, 28 January 2019. Link

– Grote T, Berens P. On the ethics of algorithmic decision-making in healthcare, J Med Ethics 2019;0:1–7. doi:10.1136/medethics-2019-105586. Link to .PDF

– Schonberg D. Artificial intelligence in healthcare: a critical analysis of the legal and ethical implications.  International Journal of Law and Information Technology, 2019, Link

– Dorian Peters, et. al, Responsible AI- Two Frameworks for Ethical Design Practice. IEEE Transactions on Technology and Society, Vol. 1, No. 1, March 2020. Link to .PDF

Legal relevance of AI Ethics

– OECD, ‘Recommendation of the Council on Artificial Intelligence’ (22 May 2019), Link

– G20, ‘G20 Ministerial Statement on Trade and Digital Economy’ (9 June 2019), Link to .PDF

– Ethics Guidelines for Trustworthy AI. Independent High-Level Expert Group on Artificial Intelligence. European commission, 8 April, 2019, Link to .PDF

– F Möslein, ‘Robots in the boardroom: artificial intelligence and corporate law’ in  W Barfield and U Pagallo (eds), Research Handbook on the Law of Artificial Intelligence (Edward Elgar Publishing 2018), Link to .PDF

– F Möslein, ‘Regulating Robotic Conduct: On ESMA’s New Guidelines and Beyond’ in N Aggarwal and others (eds), Autonomous Systems and the Law (Beck, Nomos 2019) 45

– F Möslein ‘Leitlinien für den Einsatz künstlicher Intelligenz’ in D Linardatos (ed), Rechtshandbuch Robo-Advice (Beck, Vahlen 2020) 58

– Ethical Business Regulation:Understanding the Evidence, Christopher Hodges, Professor of Justice Systems, and Fellow of Wolfson College, University of Oxford February 2016. Link .PDF

– Ethical Theories, By Larry Chonko, Ph.D. The University of Texas at Arlington, SlidesNotes to slides

–  German Data Ethics Commision The Data Ethics Commission presented its Opinion to the Federal Government on 23 October 2019 at a closing ceremony at the Federal Ministry of Justice and Consumer Protection.