Canonical-basis realignment for Transformer LLMs: every hidden axis becomes independently measurable and controllable

A new open-source tool on GitHub offers a way to rotate a Transformer's internal coordinate system into a canonical basis, aligning it with the model's own weight matrices without changing outputs. Per Lobsters, applying this transform to models like Qwen or Pythia reveals hidden structures—such as bipolar oscillators where axes fire in opposition, and a homeostatic mechanism that erases perturbations within a couple of layers. The technique also suggests that a half-billion-parameter model's effective correlation rank might be as low as eleven independent patterns, giving researchers a standardized lens for studying how LLMs reason.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton