What to measure for AI workloads
What to measure for AI workloads. In this video you'll measure one question end to end, load the service, and read the indicators that matter on a dashboard. A user asks the assistant a question. The gateway fetches documents, asks the model, calls a tool, and streams the answer back. The gateway measures what the user feels: time to first token, errors and cost. The model server measures its own work: the queue, the key-value cache, or KV cache, and tokens per second. In this lesson you will ask one question and read its indicators, read the model server's own metrics, and load the service while you watch the dashboard. Before you start, you need the lab cluster: make up. One question through the assistant. The answer is made-up text: the model server is a simulator. What matters is the measurements under it. The user waited about a second for the first token. That time includes the retriever, the tool call and the model's queue, not just the model. Then the tokens arrived at a steady rate, and the gateway priced the question from its tokens in and out. The model server measures its own work, in vLLM's metric names: requests running and waiting in its queue, and how full its KV cache is: the cache that holds the prompts' keys and values in accelerator memory. On a GPU, that is the memory you run out of first. Sixteen clients ask at once for thirty seconds. Two model servers take four requests each, so half of them wait. I speed up the wait. The median user now waits several times longer for the first token, and the slowest tenth even more. No error, no crash: just a queue. That is why queue depth sits next to time to first token on every AI dashboard. The same indicators in Grafana, while sixteen clients keep asking. Each tile shows the latest value and a small trend. Time to first token at the ninetieth percentile, as the user feels it, and as the model server reports it. That one stays low: the simulator starts its clock when a request leaves its queue, so the waiting shows in the queue depth and in the user's number. Output tokens per second, the throughput of the model servers. The queue depth, the first number to watch when users complain, and the KV cache use, which stands in for accelerator memory here. Errors and cost per question complete the picture: is it working, and what does it cost. On OpenShift AI, the model servers expose the same vLLM metrics, the platform shows them in its model serving dashboards, and GPU nodes add accelerator metrics through the NVIDIA DCGM exporter. In this lesson you learned that the user's time to first token includes retrieval and tools, that a growing queue explains most of the slowdown under load, and that cost per request comes from tokens times their price.