Untie Squared ReLU variant

Researchers at LessWrong tested a modified approach to how transformer neural networks apply activation functions during training. Rather than a simple activation, they introduced two separate, independent activation branches whose results are multiplied together—inspired by recent efficiency research. Per the post, the variant achieved similar training quality compared to standard baselines but with costly trade-offs: about eighteen percent slower execution per training step and one point five times the computational operations. The experiment underscores a persistent lesson in optimization research: clever-sounding changes often carry hidden performance penalties that only become visible through careful measurement.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton