Researchers introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations rather than relying solely on final-answer accuracy. The framework represents each memory event with a structured trace specifying triggers, targets, and state transitions to disentangle heterogeneous causes of failure.
- MemOps embeds operations like remembering, forgetting, and updating into long, task-oriented conversations using a controllable generation pipeline.
- It produces gold operation traces alongside six categories of operation-level probes evaluated under adjacent-evidence and long-context settings.
- The benchmark reveals that session-level retrieval outperforms turn-level retrieval in current systems.
- Long-context models remain notably weak at reconstructing ordered memory-state trajectories.
This approach moves long-term memory evaluation from black-box scoring toward interpretable, operation-level diagnosis to better assess reliability.