Researchers introduce EAST to evaluate Theory of Mind in LLMs

The authors introduce the Epistemic Asymmetry Schelling Task (EAST), a two-player dialogue game designed to benchmark robust and generalizable Theory of Mind (ToM) abilities in Large Language Models. This approach addresses limitations in traditional text-based evaluations, such as the Sally-Anne task, which can be gamed due to pre-training exposure.

• EAST requires LLM-LLM dyads to independently converge on semantic Schelling points under varying states of epistemic transparency.
• The study reveals a significant capability gap in functional social reasoning, with only frontier models successfully navigating the tasks.
• Analysis shows coordination failures are primarily driven by epistemic tracking errors, such as conflating private knowledge with mutual knowledge.

The findings indicate that despite high performance on traditional static benchmarks, robust social reasoning and epistemic tracking remain critical bottlenecks for future LLM evaluation and development.