Skip to main content
🎤 Luca Berton is speaking at Red Hat Summit & KubeCon EU 2026!Learn more →
intermediate

llm-d Distributed Inference Without GPUs

Use a Kubernetes simulation to observe how llm-d routes, queues, and scales inference requests, then compare the simulator with a GPU serving setup.

6 lessons22mUpdated November 2026

What you'll learn

  • Inspect the inference gateway and KServe resources
  • Observe routing, priority queues, and prefix caching
  • Scale inference replicas from queue metrics and inspect the GPU configuration

Who this is for

  • • DevOps engineers deepening their expertise
  • • SREs building production-grade systems
  • • Backend developers expanding into infrastructure

Prerequisites

  • • A Mac that can run a local Kubernetes cluster, and familiarity with kubectl.

Course content

2 modules, 6 lessons, 22m of video

Module 1

3 lessons, 13 min
1

5 min

Prefill, decode and the KV cache, seen on llm-d's inference simulator behind a gateway.

2
The inference gateway

5 min

Gateway API with the inference extension: an HTTPRoute to an InferencePool, an endpoint picker per request.

3
LLMInferenceService

4 min

KServe's LLMInferenceService creates the model server, the pool, the picker and the route for you.

Module 2

3 lessons, 10 min
4
Flow control

4 min

InferenceObjectives and the endpoint picker's flow control: premium traffic first, best effort waits.

5
Autoscaling on queue depth

4 min

KEDA scales the simulator on the gateway's queue and running requests, read from Prometheus.

6
From the simulator to GPUs

3 min

Swap the simulator for vLLM on a GPU: the pods change, the routing, flow control and scaling stay.

Get started

Start Free Lesson

First lesson is always free — no signup required

This course includes

6 hands-on lessons
Intermediate level
22m total
Final assessment

Want all courses?

See Pro plan →

Your instructor

Luca Berton

Founder and CEO, Open Empower. About the author

More in AI Homelab on a Mac mini

All courses in this track

Turn a Mac mini into a reproducible homelab server

You will turn an Apple Silicon Mac mini into an always-on home server. You start by planning hardware choices and initializing a Git repository to track your setup steps. This foundation ensures your configuration is documented and ready for automation. You then configure remote access using SSH and Tailscale for secure connectivity. You also adjust power settings to keep the machine running continuously without interruption. Next, you use Homebrew with a Brewfile to manage software packages consistently. You create a Makefile that journals every command and output for full reproducibility. Finally, you add a read-only health check to verify system status. You learn to rebuild the entire setup on a clean machine from your repository.

Beginner · 4 lessons · 15 min · Updated Nov 2026

Preview free lesson

Virtual machines on Apple Silicon

You will build a Linux virtual machine on your Apple Silicon Mac using Colima and the Virtualization framework. You will point Docker to this VM and manage it with simple commands. You will share files between your Mac and the VM using virtiofs mounts. You will run AMD64 container images efficiently using Rosetta translation instead of slow emulation. You will size your VM correctly for your specific Mac hardware to avoid resource issues. You will create a plain Ubuntu VM with Lima for lab environments. You will configure launchd to start your VM automatically when you log in. You will learn to stop and clean up these virtual machines when you are done.

Beginner · 6 lessons · 20 min · Updated Nov 2026

Preview free lesson

Kubernetes on your Mac

You will build a multi-node Kubernetes cluster on your Apple Silicon Mac using kind. You start by creating a single-node cluster and then expand it to three nodes using a configuration file. You install applications with Helm and reach them from your Mac using ingress or Gateway. You also manage persistent storage and resource limits for your pods. You finish by tearing down and rebuilding the entire environment in minutes. You use a Makefile to automate the setup and cleanup process. This ensures your cluster is reproducible and easy to manage. You gain hands-on experience with a real cluster that runs entirely on your laptop.

Beginner · 6 lessons · 23 min · Updated Nov 2026

Preview free lesson