Prefill, Decode, and Despair: Disaggregated LLM Serving Made Boring

Duration: NaN:NaN:NaN

Speaker: Sergey Pronin

Type: CLS - Talk (Open Tech Summit)

Expertise:

Event: Devcon

Date: Nov 2026

KServe can serve large language models on Kubernetes - but past "hello world," it gets sharp. You face disaggregated prefill/decode, four kinds of parallelism, autoscaling that behaves differently under KEDA versus HPA, a CPU profile that runs out of memory before your model loads, and gateway wiring where one wrong setting silently breaks routing. Miss a field and you don't get an error - you get a slow, wrong, or unreachable model. This talk shows how the OpenEverest's KServe provider hides all of that behind a single, technology-agnostic Instance resource. In ten lines of YAML, you get an OpenAI-compatible endpoint. We'll look at the real traps it saves you from: container args that silently drop your defaults, a gateway backend that bypasses request routing, and KV-cache settings that refuse to start.