An Induction Head in Disguise: Chasing Grammar in a Character-Level Transformer

Per LessWrong, a researcher trained a character-level transformer on Nietzsche texts and analyzed a specific attention head to understand what it does. Using eigenvalue analysis, they first suspected it enforced quotation grammar—pairing quotes. That failed. They next hypothesized it handled parenthesis balancing. Also wrong. The actual finding: it's an induction head that copies tokens based on patterns seen earlier in the sequence. It can copy opening brackets, but its query-key circuit lacks sufficient routing to complete the full pairing mechanism.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton