A study evaluates six frontier language models on a two-agent communication task where a questioner identifies a target from N Wikipedia paragraphs using exactly log2(N) yes/no questions. The experiment ran 408 games across document sets of 4 to 1024 paragraphs, revealing significant performance disparities among the tested models.
- Claude Opus won only 28 of 68 games, while GLM-5.3, GPT-5.6 Sol, Grok 4.6, Gemini 3.8 Flash, and Kimi K3 achieved win rates between 45% and 56%.
- The top five models showed a win rate decline with set size (r=-0.973), fitting the formula win=p^(log2 N) with p=0.928.
- Errors were split evenly between answer mistakes and discrimination failures, with Claude Opus making 32 of 34 unanimous errors by incorrectly answering "No".
- Information per question correlated positively with win rate (r=+0.88), and only two models extracted a full bit per question by partitioning on document titles.
- Reasoning token expenditure varied 4.5x across models but showed little relation to success, as trace growth did not match reliability gains.
The results indicate that while most leading models are marginally separable in this asymmetric information setting, Claude Opus significantly underperforms compared to its peers.