Text to speech on Apple Silicon with MLX
In this lesson I make this Mac speak a sentence with a text-to-speech model that runs on its own graphics processor, the GPU. Nothing is sent to a cloud service. At the end you will know how fast the model is and how much memory it needs. Before you start you need a Mac with Apple silicon and two tools from Homebrew UV which manages Python and FFmpeg which converts and measures audio. Here are the versions. The course has a starter repository called LocalVoiceMac. It holds a makefile, the pinned Python requirements and the five small scripts that the lessons use. MacEnv creates a Python virtual environment with UV and installs the pinned packages, then it prints the three that matter. MLX is Apple's machine learning framework for Apple silicon. It runs models on the GPU and because the GPU shares its memory with the processor, nothing is copied between them. MLX audio is a library of speech models ported to MLX. MLX Whisper is the Whisper speech recognition model ported to MLX2. MLX sees the GPU so the models will run there. The models come from Hugging Face, a hub for open models. Each one downloads the first time you use it, about 16 gigabytes in total, so I downloaded them before before recording. Make models list them with their size. There are three text-to-speech models, Chatterbox by Resemble AI, QAN3 TTS by Alibaba and Vox CPM2 by OpenBMB. The tokenizer is a part that Chatterbox needs, Whisper Turbo is the speech recognition model I use later to check what the voices say. Say.py speaks text with one of the models, VoxCPM2 can speak without any reference voice. So I start with it, one sentence, written to takes slash first.wave. The script loads the model into memory, then generates the audio on the GPU. The first run takes a little longer, so I speed up the video. Say dot PY prints two lines. The first says how long the model took to load, the second has the length of the audio, the time it took to generate it, and the real-time factor. The real-time factor, or RTF, is the time taken divided by the length of the audio. Below 1 the model speaks faster than real-time. Vox CPM2 at full precision is above 1 on this Mac, so it is slower than real time. Your numbers depend on the chip and on what else is running. The last number is the peak memory of the GPU that tells you how much of your Mac's memory the model needs. Now let me listen. AFPlay is the audio player built into macOS. Hello? This sentence was spoken by a model running on the GPU of this Mac. That voice is the model's own, Vox CPM2 chose it because I gave it no voice to copy. The other two models need a reference clip, a short recording of the voice to clone. That is a text to speech running locally. One command, one sentence, measured on your own GPU.