What is AI bias?

Reporters at Bloomberg asked a popular image tool to draw people in different jobs. Lawyers, judges, and politicians came back mostly as men with lighter skin, and women were rarely shown as doctors. Nobody told the tool to do that. It learned from huge numbers of pictures that people had posted and labeled over many years, and it repeated what it saw most. That was one tool, tested in 2023. But it shows what people mean by artificial intelligence (AI) bias.³

AI bias means a system's results are unfair to some groups of people, in a way that repeats. One wrong answer is a mistake. The same kind of wrong answer, landing on the same kind of person again and again, is bias. It usually starts in the data the system learned from, or in choices people made while building it.¹

A model learns the past it was shown. If that past was unequal, the patterns in it are too. In one well-known case, Amazon built a tool to sort job applications and trained it on resumes from past hires, who were mostly men. It learned to favor men's applications, and the company dropped it. The tool didn't invent that preference. It found it in the history it was given.¹

Bias can also come from who isn't in the data. If a system learns mostly from photos, voices, or medical records of one group, it gets less practice with everyone else, and its results for them can be worse. In medicine, some computer-aided diagnosis systems have been found to be less accurate for Black patients than for white patients. More varied examples help close that gap.¹

People also decide what a system should predict, and what counts as success. Those are human choices. They can carry bias even when the data looks fine. A 2019 study in the journal Science examined a widely used tool that helped health systems pick patients for extra care. It treated past health spending as a stand-in for need. Less had been spent on Black patients who were just as sick, so the tool rated them healthier than they were. The researchers showed that predicting health itself, instead of spending, largely removed the bias.²⁵

The National Institute of Standards and Technology (NIST), part of the United States government, sorts bias into three kinds.

• Systemic: unfairness already built into society and its institutions, which then shows up in the data.
• Statistical and computational: gaps and errors in the data and the math, like a sample that leaves people out.
• Human: the assumptions and shortcuts of the people who build and use the system.

Most real cases mix more than one. So fixing bias takes more than tidying a spreadsheet.²

Bias matters most where a result changes someone's life: a job interview, a loan, a diagnosis, how people are shown in pictures. And it can be measured. Teams test a system's results group by group and look for gaps. Then they change the data, the rules, or the goal, and test again. It's ongoing work, and it's work people know how to do. If a decision like that ever lands on you, you can ask how the system was tested.¹²⁴

When you hear that a tool is fair, or unfair, ask who was in the examples it learned from. Then ask who checked its results, group by group, and what they found. Think of one app you use that sorts or ranks people. Who do you suppose checked it?