Resources

Offline Guide

An offline copy of this guide. Interactive components, such as the explorable, are left out. But it has been nicely formatted for you to print on sheets of dead tree matter.

Download the offline guide

Other Websites and Guides

Useful or interesting links related to algorithmic bias.

• Survival of the Best Fit - a game about algorithmic bias in hiring
• Google’s People + AI Guidebook and Inclusive ML Guide
• The Financial Modelers’ Manifesto written by Emanuel Derman and Paul Wilmott was written for quants and financial engineers amidst the fallout of the subprime mortgage crisis, but the lessons are very applicable to today’s AI engineers

Organizations and Conferences

• The AI Now Institute is working actively on AI ethics and has many great publications
• FAT ML and ACM FAT* are two of the main conferences in AI ethics - check out the conference websites for related publications

Datasets

Datasets for the more bias-aware.

• Gapminder’s Dollar Street images, which was used by DeVries et al. in Does Object Recognition Work for Everyone? and comprises over 16,000 images from 60 different countries across 138 categories - a downloadable set can be found via my GitHub repository
• Google’s Open Images Extended - Crowdsourced, - Google has also provided some notes on possible biases in this dataset
• Joy Buolamwini’s Gender Shades dataset can be requested here

Tools

Tools for diagnosing and mitigating algorithmic bias, complete with detailed tutorials.

• IBM’s AI Fairness 360 Open Source Toolkit
• Microsoft’s InterpretML
• Tensorboard’s What If

Readings

Academic publications related to algorithmic bias that I found useful.

• Do Artifacts have Politics? (Winner, 1980)
• Bias in Computer Systems (Friedman and Nissenbaum, 1996)
• Technologies of Humility (Jasanoff, 2007)
• Big Data’s Disparate Impact (Barocas and Selbst, 2016)
• Inherent Trade-offs in the Fair Determination of Risk Scores (Kleinberg et al., 2016)
• Interventions over Predictions: Reframing the Ethical Debate for Actuarial Risk Assessment (Barabas et al., 2017)
• Fairness Definitions Explained (Verma et al., 2018)
• Fairness and Abstraction in Sociotechnical Systems (Selbst et al., 2019)
• A Framework for Understanding Unintended Consequences of Machine Learning (Suresh and Guttag, 2019)

References