What Actually Happens When You Send an LLM Request to Kubernetes?

Duration: NaN:NaN:NaN

Speaker: Atharva Mhaske

Type: CLS - Lightning Talk (Open Tech Summit)

Expertise:

Event: Devcon

Date: Nov 2026

Platform teams ship LLM features fast. Then production asks a basic question: where did this request go? The answer is rarely one Pod. A chat completion crosses DNS or a Service, often an ingress or gateway, then a model proxy that routes by model name. The serving controller can scale a Pod from zero. The model server loads weights, runs inference, and streams tokens back. Each hop has its own failure mode. A 404 on the wrong API path. A hang while a cold model wakes. A gateway timeout that kills a long stream. Request-count rate limits that ignore token cost. Teams debug the symptom because they never mapped the path. This talk draws that map. We start from a client call to an OpenAI-compatible endpoint. We follow the request into Kubernetes networking, into the serving proxy, into the Model resource and the Pod that runs Ollama on CPU or vLLM on GPU. We use OpenEverest as a concrete case: an Instance CR declares intent, a provider creates a KubeAI Model, and status writes the phase and endpoint back. The point is the path, not the brand. The same hops appear with KServe or a hand-rolled stack.