Smaller, faster, safer: running Kimi and GLM at scale

Large language models are powerful but memory-hungry. Cloudflare is now serving two of the most capable open-source models, Moonshot's Kimi and Z.ai's GLM, more efficiently by stacking three optimizations. According to Cloudflare's blog, the first technique quantizes the key-value cache — the structure that stores attention data as the model generates text — from sixteen-bit down to eight-bit precision. On Kimi, that doubles the context length the model can hold in memory, and lets Cloudflare serve sixty-four concurrent requests instead of thirty-two, achieving forty-one percent higher throughput at roughly thirty percent lower cost per token. The second technique compresses GLM's model weights from eight-bit to four-bit integers, shrinking the checkpoint by forty percent, which speeds up token generation. Third, because hundreds of requests now share the same GPU cache, Cloudflare added integrity checking — a safety layer that costs less than one percent in overhead and prevents any request from reading another's cached data. Model accuracy stays identical across all three, so Cloudflare can support more customers at lower cost with no compromise.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton