Pragmatic FDT, and predictors as game theory

According to Stuart Armstrong writing on the AI Alignment Forum, functional decision theory faces persistent criticism from academic philosophers, yet continues attracting rationalist defenders. Armstrong proposes a pragmatic version that sidesteps theoretical problems by defining algorithm equivalence pragmatically: two algorithms are the same if the equivalence can be built, using ordinary tools to judge computational equivalence, not abstract metaphysics. His approach searches for isomorphisms between the agent's decision process and the world, evaluates the utility of adopting each, then chooses the highest-utility option or defaults to classical decision theory if none improve the outcome. Armstrong's second insight: whenever predictors make counterfactual guesses about your behavior, you're no longer solving a pure decision problem. You've entered game theory. That's why paying the blackmailer—seemingly irrational in isolation—can be rational when you account for the predictor's response.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton