Agent Framework Outperforms Model in New Memory Learning Test

Whether artificial intelligence agents can genuinely learn from prior interactions—rather than simply retrieving stored facts—remains an open question in the field. Letta's FrontierMemory benchmark, unveiled this week, probes this distinction by evaluating whether agents improve their performance on new tasks by drawing on lessons from earlier work. The testing uncovered substantial performance variation across different agent frameworks, the software infrastructure governing how agents function.

Letta Dream emerged as the top performer with a 73.3% score, trailed by Prime Agent at 60%. OpenAI Agents SDK achieved 40%, while Claude Managed Agents and Hermes Agent scored 33.3% and 26.7% respectively.

Recall isn't learning

Existing memory benchmarks have become oversaturated because they measure only factual recall, according to Letta co-founder and CEO Charles Packer. In the current era of coding frameworks and scripted loops, recall has largely been solved, yet continual learning through the context window an agent maintains—rather than through updates to model weights—remains an unsolved challenge.

Co-founder Sarah Wooders identifies a related limitation: most coding and knowledge-work benchmarks present unrelated tasks with all necessary information available at the start. This diverges sharply from how work actually unfolds. She draws a parallel to onboarding at a company: newcomers require extensive guidance, whereas experienced employees often complete assignments with minimal instruction.

if AI-generated memory is good, it should actually improve downstream task performance.

Charles Packer

FrontierMemory operates differently. Agents receive transcripts from earlier work—including both successes and failures—and must synthesize their own memories to support completion of subsequent tasks. Developers have already begun implementing this principle, constructing agents that learn continuously by combining a language model with memory systems and feedback mechanisms.

Letta adapted existing coding and knowledge-work tests to require agents to apply lessons from previous tasks rather than receiving complete instructions upfront. Following refinement and correction of the original assessments, Letta narrowed FrontierMemory 1.0 to 15 tasks.

Harness beats model

The framework itself may prove equally or more important than the underlying language model. Letta evaluated five agent systems to assess their capacity to extract value from prior work and apply those insights to fresh challenges.

When both systems used GPT-6.1 Sol for memory generation and GPT-6 Sol for task execution, Letta Dream outperformed OpenAI Agents SDK 60% to 40%. The gap widened considerably when using Claude Opus 5.5: Dream achieved 73.3%, Prime Agent reached 60%, and Claude Managed Agents scored 33.3%.

According to Packer, the method by which agents conduct "dreaming"—the process of reviewing prior work and converting learned insights into reusable instructions or memories—produces significant differences in outcomes. Prime Agent's superiority over Hermes Agent further demonstrates the variability across these systems. One drawback emerges: without safeguards governing what gets retained, agents may internalize and perpetuate errors.

FrontierMemory also assessed seven frontier models on their ability to generate useful memories. Anthropic's Claude Opus 5.5 and Z.ai's GLM-5.3 tied at 73.3%, followed by Kimi K3 at 66.7%. OpenAI's GPT-6.1 Sol and Google's Gemini 4 Argon both scored 60%, Muse Wonder reached 46.7%, and Grok 4.7 finished at 33.3%.

Opus 5.5 and GLM-5.3 are significantly better at generating useful memories than GPT-6.1 Sol.

Charles Packer

Cheaper memory, lower scores

OpenAI maintained a substantial cost advantage. Processing each task with GPT-6.1 Sol cost approximately $0.77, compared with $2.80 for Opus 5.5 and $3.58 for GLM-5.3. Despite lower benchmark scores, OpenAI's approach proved significantly more economical.

Letta employed its Dream framework to generate memories using each model, then measured how effectively those memories performed on tasks executed in Letta Code. The findings illustrate a fundamental tension between superior memory quality, stronger task performance, and reduced expenses.

Memory that travels

Letta investigated why certain memories proved more effective than others. In a test centered on company pricing information, Opus 5.5 generated a single skill instructing the agent when to apply specific data. GPT-6.1 Sol created four separate skills, yet the agent failed to locate the necessary one.

Gemini 4 Argon made a distinct error by memorizing particular file sizes, which offered little utility when subsequent tasks involved different files. Opus 5.5 instead instructed the agent to check both the standard file location and email for information. This approach remained effective even as tasks shifted. Memories produced by Opus 5.5 successfully enabled GPT-6 Sol to complete tasks, suggesting developers might employ a costlier model for memory creation paired with a less expensive model for execution.

The benchmark has attracted attention from other developers. Following Wooders' post about FrontierMemory on X, the OpenClaw account expressed interest in integrating the test into its platform. The account noted that its multiplayer setting could serve as a valuable testing ground for the benchmark, raising questions about memory behavior when multiple agents operate within the same framework.

Yes definitely – would love to add OpenClaw to the harness comparison! Extending to multiplayer memory is also a very interesting future direction.

Sarah Wooders

What the benchmark misses

FrontierMemory remains in early stages. With only 15 tasks and a single attempt per task, any individual result can shift scores by roughly 6.7 percentage points. The current benchmark emphasizes agents learning from completed work rather than updating memories during ongoing execution. Packer indicates the forthcoming release will incorporate longer-duration tasks and real-time memory updates.