Do AI detectors work?

A parent hears that a teacher ran an essay through a checker, and it scored the essay as 87 percent written by artificial intelligence (AI). The student says they wrote every word. Maybe you've heard a story like this, or lived one. A number like that sounds precise. It can't settle who's right.

An AI text detector doesn't know who wrote anything. It never sees the writer, the drafts, or the hours spent. It looks at the words and guesses how machine-like they seem. Many detectors lean on how predictable the text is. Chatbots tend to pick likely words, so writing full of likely words scores as more machine-like. The number on the screen is a guess about style.²

That's a problem for anyone who writes simply, because clear, plain sentences are predictable too. In a 2023 study from Stanford University, researchers ran essays from an English test, written by people whose first language isn't English, through seven popular detectors. On average, the detectors wrongly flagged 61 percent of those essays as AI-written. Essays by native English speakers were flagged far less often. The researchers' likely explanation: people writing in a second language often stick to common words and familiar sentence shapes, and that reads as predictable.¹²

It fails in the other direction too. A light rewrite can slip AI text past a detector. In one test that a teaching guide from the Massachusetts Institute of Technology (MIT) points to, a small change to the prompt cut detection of AI-written college essays from 100 percent to 13. So a detector can miss text a chatbot wrote, and flag text no chatbot touched.¹

In early 2023, OpenAI, the company behind ChatGPT, released its own tool for spotting AI-written text. In OpenAI's testing, it caught about a quarter of the AI-written samples, and it wrongly labeled human writing as AI 9 percent of the time. By July that year, OpenAI had taken it down, citing its low rate of accuracy.³

When someone needs to know how a piece of work was made, there's better evidence than a score. Drafts and notes show it taking shape. The version history in a shared document shows when the words appeared. And a short conversation about the work, what the writer meant and how they got there, tells you more than any number can. MIT's guide recommends that kind of evidence over detectors. None of it needs special software, and all of it treats the writer as someone to hear from.¹

Like any AI output, a detector's score can be confidently wrong, and topic 4, "Why does AI make things up?", explains how confident guesses miss. If a score ever lands on you or someone you know, ask what else supports it. A score can be one reason to look closer. It shouldn't be the only one. Which of your own drafts or notes could show how you made something?