Llama Stack on a local model
Llama Stack on a local model. In this video you'll put one API server in front of the model on your Mac, and send it a first request. Your client speaks the OpenAI API to Llama Stack. Llama Stack picks a provider for each API, here Ollama for inference, and keeps state in SQLite. All of it runs on your Mac. In this lesson you will configure Llama Stack, start it, and send it a first request. Before you start, you need Ollama with gemma4:e4b, and the starter repository's virtual environment. make check tests both. Ollama answers on this Mac, with gemma4, size e4b, a small chat model from Google. The repository's virtual environment has Llama Stack zero point seven point three, the MCP software development kit, and the OpenAI Python client. Llama Stack is configured by one file. Each API gets a provider. Inference goes to the remote Ollama provider, limited to one model. The Responses API keeps its state in SQLite, and the tool runtime can call MCP servers. The server itself listens on one two seven dot zero dot zero dot one only, port eight three two one. The server runs in the background, and its log goes to a file. First, which models does it offer? One model, named after its provider: ollama slash gemma4. Now a first request, in the OpenAI chat completions format. Llama Stack translated the request for Ollama, and gave back the answer in the same format. Reasoning effort none switches off the model's thinking, so the answer comes in seconds. The same from Python. hello dot py uses the OpenAI client, pointed at Llama Stack by its base URL, and calls the Responses API, the newer interface that agents use. The server also serves its own API reference, on the docs page. Every group of the Llama Stack API is here: conversations, inference, models, responses, and the tool connectors. I try the models endpoint from the page. The answer is the same JSON that curl printed: one model, served by the Ollama provider. One configuration file, one server on your Mac, and any OpenAI client can talk to the local model through it. On OpenShift AI, the same API server is the Llama Stack distribution, and the model behind it is usually served by vLLM on KServe. In this lesson you learned to configure Llama Stack with the Ollama provider, to start it on your Mac, and to call it from curl, from Python and from its API page.