AI agents are getting better at remembering past conversations and earlier tasks but remembering something doesn’t guarantee an agent can use it to do better work next time. Letta’s new FrontierMemory benchmark, released Wednesday, tests that gap by measuring whether agents can learn from experience rather than by just retrieving stored information. Early results show a widespread in performance across agent harnesses, the the infrastructure that manages how an agent operates.

Letta Dream came out on top with a 73.3% score, followed by Prime Agent at 60%. OpenAI Agents SDK scored 40%, while Claude Managed Agents and Hermes Agent trailed at 33.3% and 26.7%, respectively.

“Modern ‘memory’ benchmarks are saturated since they measure recall of facts,” Letta co-founder and CEO Charles Packer writes on X, saying that in the coding harness and bash loop era, recall is solved, while continual learning in token space, meaning learning through the context an agent keeps rather than through weight updates, remains unsolved.

Sarah Wooders, also a Letta co-founder, points to another problem on X. Most coding and knowledge-work benchmarks evaluate unrelated tasks with all the necessary context provided upfront. But that’s not how people actually work. Wooders compares it to starting a new job: A new hire needs plenty of direction, while someone who’s been there for years can often get the job done with little explanation.

Packer’s premise is that “if AI-generated memory is good, it should actually improve downstream task performance.”

FrontierMemory takes a different approach. Agents receive transcripts of earlier work, including successful and failed attempts, and then generate their own memories to help complete later tasks.

Source link