The true test dataset for a generalised task

Stuart Armstrong, writing on LessWrong, argues that the standard machine learning test-set approach—using data from the same distribution as training—only works for narrow tasks. For a general task like 'understand how code runs', the test set should be deliberately out-of-distribution: train on Python and C, but test on something entirely different, like Java. If the model succeeds on a foreign language, it learned the actual task, not just 'how Python works'. This catches a subtler form of overfitting—fitting to the training distribution's patterns rather than the underlying task itself. Armstrong warns these out-of-distribution test sets should be single-use; reusing them defeats the purpose by turning them into validation sets.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton