Why Models Are AI’s Next Training Dataset with Damian Borth - #772

The exponential scaling strategy that's powered AI progress—ever-larger models trained on ever-larger datasets—is hitting a wall. As high-quality training data grows scarce and pretraining costs soar, researchers are looking for new approaches. According to TWIML AI Podcast, Damian Borth, a professor of AI and machine learning at the University of St. Gallen, argues we've been overlooking a crucial resource: the trained models we already have. His work on weight space learning treats neural networks themselves as data, learning from the distilled results of millions of GPU hours of optimization instead of starting from raw information each time. By reusing and combining existing models, this approach could dramatically reduce the cost of building specialized models and potentially reshape how foundation models are developed—moving away from consuming ever-growing datasets toward training on collections of pre-existing models.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton