The inference gateway
Lesson in llm-d Distributed Inference Without GPUs ยท 4 min
Gateway API with the inference extension: an HTTPRoute to an InferencePool, an endpoint picker per request.
Sign in to continue
Sign in to access this lesson
Lesson in llm-d Distributed Inference Without GPUs ยท 4 min
Gateway API with the inference extension: an HTTPRoute to an InferencePool, an endpoint picker per request.
Sign in to access this lesson