What is a token?

Maybe you've seen it on a chatbot's pricing page. Or in a notice telling you you've reached your limit for the day. The limit isn't counted in words or pages. It's counted in tokens, and nobody stops to explain the word. So here it is. A token is the unit an artificial intelligence (AI) model reads and writes in, and once you know what it is, a few odd things about chatbots start to make sense.

A token is a small chunk of text. Often it's a whole common word, like "the" or "cat." A longer or rarer word may be split into two or three pieces. A comma usually gets a token of its own. So does a period. Before a model reads your message, a tool called a tokenizer cuts it into these chunks, and the model works with the chunks. When it replies, it writes chunks too, one after another, and the app turns them back into the words you see.¹²

For English text, OpenAI gives a rule of thumb: about 100 tokens make about 75 words. So a token is roughly three quarters of a word. Anthropic puts a Claude token at about three and a half English characters. Its newer models, from 2026, can split the same text into up to about a third more tokens, depending on the text, so a Claude token can cover closer to half a word. Those figures don't quite match, and that's the point. Each company cuts text its own way, and the count can change with the model and with the language you write in.¹²⁵

Why not just use whole words? Because people keep inventing them. New names, slang, typos, a product launched last week. A model that only knew a fixed list of whole words would be stuck on any word it had never seen. With a fixed set of pieces, it can build any word, the way a box of letter tiles can spell a name nobody has heard before. Common words get a single piece, which keeps things efficient. Rare ones are assembled from smaller parts.²

Tokens decide two things you'll run into.

• How much fits at once. A model can only work with so many tokens in one go, counting your message, any files, and its own reply. That limit is called the context window, and it has a topic of its own.
• What it costs. Companies that sell access to their models price it by the token, counting what goes in and what comes out.

That's why price lists and limit notices speak in tokens. They're counting the model's own unit of work.¹³

Tokens also explain a slip people love to point out. Ask a chatbot how many times a letter appears in a long word, and it may get the count wrong. That's because it never sees the letters one by one. The word may arrive as two or three chunks, and counting letters inside them is a little like counting the letters in a word you've only ever heard spoken.

If you're curious, OpenAI has a free tokenizer page where you can paste a sentence and watch it split into colored chunks. Try your own name. Then try a word you've just made up. Rare and invented words usually break into more pieces than everyday ones, so see how many yours takes.⁴