How does AI copy a voice?

Think of the audiobook you fell asleep to, or the voice that reads your directions aloud in the car. Some of those voices never read the lines you heard. A person may have recorded hours of speech once, and then software learned how they sound and produced the rest. That's called text to speech, and artificial intelligence (AI) has made it sound remarkably human.

The older reading voices sounded robotic for a reason. Many worked by stitching together tiny recorded pieces of speech, and you could hear the seams. Newer systems use neural networks, programs that learn patterns from examples, trained on many hours of real speech. They work out the rhythm, the stresses, and the timing of each sentence, then produce the sound itself. So the voice rises and falls the way a person's does.¹

Every voice has a pattern. How high or low it sits. How fast it moves, where it breathes, the texture of certain sounds. A model trained on lots of different speakers learns what voices have in common, and that lets it learn what sets one particular voice apart. With that, it can say brand-new sentences that sound like a specific person. People call that voice cloning.

Some models need very little to go on. In March 2024, OpenAI described a model called Voice Engine that could make natural-sounding speech resembling a person from a single fifteen-second recording. OpenAI chose not to release it widely, because of the risks of misuse. Other companies offer voice copying too, each with its own rules.²

The same ability does a lot of good.

• Audiobooks and videos can be read in a natural voice, or translated into another language while keeping the speaker's sound.
• People who find reading hard can have any text read to them in a voice that's easy to listen to.
• People who are losing their speech to an illness can record their voice early and keep using a version of it on a speaking device.

OpenAI's early partners for Voice Engine included a hospital team helping patients who had lost their speech.¹²³

Scammers use the same trick. The Federal Trade Commission (FTC) warns that a cloned voice can make a fake call from a family member far more convincing, and that a short clip posted online may be all it takes. The defense doesn't depend on spotting the fake. If a voice asks for money or secrets, hang up and call the person back on a number you already know is theirs.⁴⁵

Responsible voice tools draw the line at permission. OpenAI's rules for its Voice Engine partners required explicit, informed consent from the person whose voice was used, and clear disclosure that the voice was made by AI. That's a fair promise to look for in any voice tool. If you ever wanted your own voice copied, for an audiobook or to keep it safe, what would you want the tool to promise you?²