SQLite compressed text-history prototypes

Simon Willison's Weblog documents a novel approach to storing text revision histories in SQLite databases: bundling all prior versions into a JSON array and compressing the whole collection with zlib or zstandard. The results are striking—one thousand simulated document revisions totaling twenty point four megabytes of raw text compressed down to just eighty point three kilobytes. After discussing the concept with ChatGPT's voice mode, Willison prompted a large language model to build experimental Python prototypes, which worked well enough that he's now exploring refinements like splitting history across multiple rows to avoid recompressing the entire array on each edit.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton