What to keep in the cloud and what to run locally
In this lesson I decide which work stays with a cloud model and which work a local model can do and I measure how fast two local models answer on this Mac and how much memory they take. Every token that a local model handles is a token you do not pay a cloud provider for. Before you start you need Ollama, the local model server with two models, on a Mac with homebrew that is brew install Ollama. Brew services start Ollama and Ollama pool for Gemma 4E4B and Qwento 827B. One command checks it. Version 0.40.16 and both models are on disk. Gemma 4E4B is the small fast model that does the offloaded work, Quento 827B is a much bigger model. In this course it stands in for the cloud model so that nothing leaves the Mac and no key is needed. I work in the starter repository of the course local AI offload. The rule comes first. A task goes to the local model when it is repetitive, the same prompt on many inputs, So code can tell a good answer from a bad one, low risk or private when the data must not leave the machine. A task stays with the cloud model when it is long and has many steps when it is ambiguous and needs judgment, when a mistake is expensive or when the best quality is exactly what you sell. Many tasks fit both columns. Then the local model does the work, a check catches its mistakes and only the failures, or a short summary goes to the cloud. Speed is the second question. Ulama run with the verbose flag prints, timing figures after the answer. Think false turns off the model's thinking step for a direct answer. First the small model. The EVAL RATE line is the output speed in tokens per second, a token is a piece of the world, the unit that models RAID, write and cloud providers, bill, the load duration is the time to load the model into memory, paid once. Now the big model with the same prompt has many more parameters. So it takes longer to load and every token costs more work. I speed up the weight. Compare the eval rates on this Mac, the small model writes 2 to 5 times faster than the big one depending on what else uses the graphics processor. For a short explanation like this one, both answers are usable. The third question is Memory Ollama. PS lists the loaded models and their size in memory. Both models run 100% on the GPU, the graphics processor. With short prompts the small model takes about 10GB and the big one about 18. Long prompts need more for the context. On Apple Silicon the GPU shares one pool of memory with the processor, so the models must fit in that pool next to everything else you run. This Mac has 48GB. So the small model alone works on a Mac with 16GB and both together are comfortable with 48GB. Models stay loaded for 5 minutes after the last request. Ola must stop, freeze the memory now. You now have the rule for what runs locally and what stays in the cloud and the speed the memory figures of both models on your own Mac.