Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

Netflix shared its engineering approach to building a real-time service dependency map across millions of servers. Per the Netflix TechBlog, the first version worked perfectly in development but hit production obstacles—Kafka consumers fell behind, instances ran out of memory, and garbage collection pauses consumed CPU. The team solved it by switching from batch processing to a streaming-first architecture with backpressure, allowing the system to handle millions of flow records per second while slowing gracefully under load instead of dropping data or crashing. The result: topology visibility within tens of minutes instead of hours or days old data.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton