Ping An's EEAgent Teaches Robots to Self-Improve Without Retraining, Hits 92.2%
A new framework gives robotic agents human-style long and short-term memory, letting them refine prompts from past successes and failures without retraining.
Takeaways
- EEAgent framework uses VLMs plus a dual-memory reflection system to self-improve on robotic manipulation tasks without training.
- LSTRO splits memory into long-term principles and short-term task hints, updated via reflection on successes and failures.
- Error localization uses image-description and action-to-instruction consistency checks to pinpoint faults before retrying.
- Achieves 92.2% average across six VIMA-Bench tasks, beating Instruct2Act, CoT-Prompt, and learning-based baselines.
- Environment interpreter uses SAM plus GPT-4o function calling; policy planner outputs JSON action sequences.
- No code release; performance depends heavily on VLM scale and SAM size (Huge recommended).
A team from Ping An Technology has proposed a framework that lets a robotic manipulation agent improve itself between tasks without touching a single model weight. Their Evolvable Embodied Agent (EEAgent) pairs a vision-language model with a memory system inspired by how humans consolidate lessons, and it posts the top average score on six VIMA-Bench manipulation tasks.
The core idea splits the agent's memory into a long-term store for principles that generalize across tasks and a short-term store for hints tied to the current instruction. When a task fails, the agent reflects on the failure, localizes the error, and pushes the resulting insights back into those two memory banks as natural-language prompts.
Why prompt-only self-evolution is hard
Two mainstream options for building adaptable robots both have pain points. Fine-tuning large models for general-purpose robotics is prohibitively expensive given the volume of weights and the heterogeneity of downstream tasks, and it also raises timeliness and interpretability issues. Retrieval-augmented generation avoids retraining but leans on an external memory bank and can miss on retrieval accuracy.
Reflection-style methods like Reflexion looked promising because they generate language critiques of failures and feed them back as prompts. But the ultimate success of that paradigm hinges on the quality and broad applicability of the derived reflections, and there are significant challenges in getting VLMs to reflect accurately and produce truly generalizable insights. In practice, reflections tend to be task-specific anecdotes rather than transferable rules, and there is no principled way to consolidate them.