This paper investigates deliberative large language model (LLM) agents in cooperative joint decision-making tasks characterized by partial and asymmetric observations. The authors formalize this scenario as a problem requiring information exchange through deliberation to achieve a shared reward.
- Introduces a scalable benchmark instantiating the problem across multiple task settings and domains.
- Establishes a reference scaffold and evaluation protocol for deliberative agents.
- Conducts a systematic evaluation of representative LLMs on these tasks.
- Finds that complex deliberative collaboration continues to challenge state-of-the-art language models.
- Notes that while external mathematical tools help, models may still fail in alignment or reasoning.
- Reveals that deliberation allows for reflection and error correction, sometimes improving performance over centralized baselines.
The work establishes a foundation for evaluating and improving LLM agents in deliberative collaboration and provides insights into the strengths and limitations of current multi-agent systems.