llm-d Distributed Inference Without GPUs
Use a Kubernetes simulation to observe how llm-d routes, queues, and scales inference requests, then compare the simulator with a GPU serving setup.
What you'll learn
- Inspect the inference gateway and KServe resources
- Observe routing, priority queues, and prefix caching
- Scale inference replicas from queue metrics and inspect the GPU configuration
Who this is for
- • DevOps engineers deepening their expertise
- • SREs building production-grade systems
- • Backend developers expanding into infrastructure
Prerequisites
- • A Mac that can run a local Kubernetes cluster, and familiarity with kubectl.
Course content
2 modules, 6 lessons, 22m of video
Module 1
3 lessons, 13 min
5 min
Prefill, decode and the KV cache, seen on llm-d's inference simulator behind a gateway.
5 min
Gateway API with the inference extension: an HTTPRoute to an InferencePool, an endpoint picker per request.
4 min
KServe's LLMInferenceService creates the model server, the pool, the picker and the route for you.
Module 2
3 lessons, 10 min
4 min
InferenceObjectives and the endpoint picker's flow control: premium traffic first, best effort waits.
4 min
KEDA scales the simulator on the gateway's queue and running requests, read from Prometheus.
3 min
Swap the simulator for vLLM on a GPU: the pods change, the routing, flow control and scaling stay.
Get started
Start Free LessonFirst lesson is always free — no signup required
This course includes
Want all courses?
See Pro plan →Your instructor

Founder and CEO, Open Empower. About the author
More in AI Homelab on a Mac mini
All courses in this trackTurn a Mac mini into a reproducible homelab server
You will turn an Apple Silicon Mac mini into an always-on home server. You start by planning hardware choices and initializing a Git repository to track your setup steps. This foundation ensures your configuration is documented and ready for automation. You then configure remote access using SSH and Tailscale for secure connectivity. You also adjust power settings to keep the machine running continuously without interruption. Next, you use Homebrew with a Brewfile to manage software packages consistently. You create a Makefile that journals every command and output for full reproducibility. Finally, you add a read-only health check to verify system status. You learn to rebuild the entire setup on a clean machine from your repository.
Beginner · 4 lessons · 15 min · Updated Nov 2026
Preview free lesson
Virtual machines on Apple Silicon
You will build a Linux virtual machine on your Apple Silicon Mac using Colima and the Virtualization framework. You will point Docker to this VM and manage it with simple commands. You will share files between your Mac and the VM using virtiofs mounts. You will run AMD64 container images efficiently using Rosetta translation instead of slow emulation. You will size your VM correctly for your specific Mac hardware to avoid resource issues. You will create a plain Ubuntu VM with Lima for lab environments. You will configure launchd to start your VM automatically when you log in. You will learn to stop and clean up these virtual machines when you are done.
Beginner · 6 lessons · 20 min · Updated Nov 2026
Preview free lesson
Kubernetes on your Mac
You will build a multi-node Kubernetes cluster on your Apple Silicon Mac using kind. You start by creating a single-node cluster and then expand it to three nodes using a configuration file. You install applications with Helm and reach them from your Mac using ingress or Gateway. You also manage persistent storage and resource limits for your pods. You finish by tearing down and rebuilding the entire environment in minutes. You use a Makefile to automate the setup and cleanup process. This ensures your cluster is reproducible and easy to manage. You gain hands-on experience with a real cluster that runs entirely on your laptop.
Beginner · 6 lessons · 23 min · Updated Nov 2026
Preview free lesson