How we could save petabytes of cache storage with Zstandard and Pingora

Cloudflare Blog reports that the company has prototyped a feature called Cache Transcoding, which compresses eligible text assets with the Zstandard algorithm inside its Pingora proxy before writing them to disk. In internal tests the compression reduced on‑disk size to roughly one third of the original, a savings of about two point eight times, while adding only a few percent extra CPU load. The system only targets uncompressed text larger than four kibibytes, leaving already compressed media such as images and video untouched. By keeping the compressed representation as the asset moves through Cloudflare’s Tiered Cache, the approach cuts both storage needs and the bandwidth used between data centers, effectively expanding petabytes of cache capacity. The blog notes that the modest CPU cost is paid once per cache fill, with decoding occurring on each request.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton