Why distributed inference
Why distributed inference. In this video you'll see what a large language model server spends its time on, and the topology that routes requests by load and cache, running on kind without a GPU. A model server answers in two phases. Prefill reads the whole prompt once and fills the KV cache, the key-value cache of the attention layers: that sets the time to the first token. Decode then writes one token at a time, reading that cache. Each server runs only a few requests at a time and its cache is limited. So with many servers, where a request lands matters: on a busy server it waits, on a server that already holds its prefix it skips work. That is the router's job. Here, llm-d's inference simulator plays the model servers. It reproduces the timing, the queues and the cache metrics of a vLLM server, not the speed of a real model on a GPU. In this lesson you will look at the topology, stream one answer to see the two phases, and read a server's metrics. Before you start, you need the lab cluster: in the llm-d-mac repository, run make up, and check that kubectl get pods shows every pod running. This is the lab, on a kind cluster on the Mac. The gateway's proxy receives the requests. The chat e p p pod is the endpoint picker, which chooses a model server for each request. And the two sim pods are llm-d's inference simulator, playing two vLLM model servers. Prometheus scrapes their metrics, and Grafana shows them. stream dot py sends one chat request through the gateway on port eight thousand and eighty eight, and prints the answer as it arrives. The text is made up: what matters is the timing. The first token came after a few hundred milliseconds: that is prefill, the whole prompt read at once. Then twenty nine more tokens, about forty milliseconds each: that is decode, one token at a time. Now twelve clients send requests at the same time, in the background, for twenty five seconds. What does each server see? Four running requests on each pod, never more: each server has four slots. On a GPU, that limit comes from memory. And this is how full each server's KV cache is. The cache holds the prompts and the answers being written, so it fills up with the running requests. Each request took about two seconds alone, yet the slowest ones took longer: they had to wait for a free slot. Where a request waits, and on which server, is what distributed inference is about. Headlamp, a web user interface for Kubernetes, runs in this cluster too. I logged in before recording. The pods list shows the same topology. The gateway's proxy, the endpoint picker, and the two simulated model servers. Each one is a single small container: no GPU, no model weights. Its details show the arguments that set its behaviour: at most four requests at a time, three hundred milliseconds to the first token, forty milliseconds per token, and a KV cache of sixty four blocks. On OpenShift AI, the same topology is distributed inference with llm-d: vLLM pods on GPU nodes behind the same gateway and endpoint picker. The simulator stands in for those pods here. In this lesson you learned that prefill sets the time to the first token and decode the rest, that each server has a few slots and a limited KV cache, and that the simulator gives you real timing and metrics without a GPU.