In-House LLM Serving at Netflix

Netflix took a different approach to large language models — instead of calling hosted APIs like most organizations, the company built its own end-to-end serving infrastructure integrated into their existing production environment. Netflix TechBlog reports the system uses NVIDIA Triton and vLLM as the core inference engine, treating LLMs no differently from other ML models the company deploys through the same infrastructure and rollout pipelines. The technical team published their design decisions and the trade-offs they encountered — including problems that only surfaced under production load, like managing model version changes without dropping user requests.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton