GiulioDER has released AMB, an open and preregistered benchmark designed to evaluate the effectiveness of memory layers in coding agents. Unlike traditional benchmarks that focus on retrieval relevance, AMB tests whether retrieved memories actually help an agent complete real coding tasks or cause worse outcomes.
- The benchmark runs Claude Code inside a sandboxed repository where executable tests grade the resulting artifacts.
- It is adversarially constructed with absent, stale, superseded, contradictory, adjacent, and irrelevant memories.
- The public feed contains 195 pre-authored session transcripts.
- A separate hard corpus scales to 4,900 documents, including 196 real documents and 4,704 synthetic distractors, with roughly 143,000 chunks.
- The current suite mainly measures the read path and does not yet measure whether a memory layer learns from the agent’s own work across sessions.
The author is inviting small memory vendors to test their systems without a participation fee, allowing them to publish their configuration and results.