RPent (Recursive Physical Agent) is an open framework for building embodied agents that continuously evolve through recursive interaction with the physical world. Rather than prescribing a single foundation model, RPent provides a recursive agent framework that harnesses heterogeneous intelligence, including perception, reasoning, memory, execution, and self-evolution, into a unified physical agent. Through continuous interaction, reflection, and adaptation, RPent enables physical agents to acquire new capabilities and evolve beyond their initial design. We build RPent upon a foundation of service-oriented, standardized, and composable design principles, ensuring the framework remains highly extensible.
RPent is built for four kinds of users:
- Embodied intelligence researchers targeting high success rates on embodied tasks and benchmarks — especially long-horizon manipulation. RPent's memory-guided, agentic composition consistently lifts task success beyond what a frozen VLA delivers alone.
- Online-learning and reinforcement-learning researchers studying self-evolving embodied agents. RPent's recursive interaction, reflection, and memory-distillation loops provide a ready substrate for continual and reinforcement learning in the physical world.
- Robotics application developers deploying embodied solutions on real robot hardware. RPent's service-oriented, standardized architecture and customizable agentic control logic improve real-world success rates and shorten the path from prototype to production.
- End users of deployed embodied agents — the customers of the developers above. Install RPent together with the relevant real-robot extensions and run the predefined tasks out of the box, with no ML expertise required.
- [2026/08] 🔥 RPent supports the non-reasoning mode, which reduces average execution time by ~40%.
- [2026/08] 🔥 RPent supports exploration mode for LIBERO. Doc: LIBERO exploration mode.
- [2026/08] 🔥 RPent supports RoboTwin with LingBot-VLA for dual-arm manipulation tasks. Doc: RoboTwin.
- [2026/08] 🔥 RPent supports RoboCasa with RLDX-1 as manipulation model. Doc: RoboCasa.
- [2026/07] 🔥 Our first RPent publication, Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents, is released.
| Agentic Planner | Action Primitive | Simulator | Real World |
|---|---|---|---|
|
|
1. Install RPent with a single pip install.
git clone https://github.com/RLinf/RPent rpent &&cd rpent
pip install -e ".[full]".[full] is the default end-to-end stack (openpi Pi0.5 VLA + LIBERO-PRO and RoboCasa365 simulators + SAM 3.0 on the RLinf runtime).
If you don't need the whole stack, see the installation docs for narrower extras.
2. Download the LIBERO-PRO simulator assets.
liberopro-download-assets --skip-existing💡 Slow connection to Hugging Face? Download through the mirror:
HF_ENDPOINT=https://hf-mirror.com liberopro-download-assets --skip-existing.
See the installation docs for other simulators.
3. Configure keys and checkpoints, then run.
# Anthropic key; no need to export the base url if you use the official endpoint.export ANTHROPIC_BASE_URL=https://xxx
export ANTHROPIC_API_KEY=sk-xxx
# VLA checkpoint — download from# https://huggingface.co/RLinf/RLinf-Pi05-LIBERO-130-fullshot-SFT
hf download RLinf/RLinf-Pi05-LIBERO-130-fullshot-SFT \
--exclude optimizer.pt \
--local-dir ./checkpoints/RLinf-Pi05-LIBERO-130-fullshot-SFT
export PI05_CHECKPOINT_PATH=$PWD/checkpoints/RLinf-Pi05-LIBERO-130-fullshot-SFT
# SAM 3.0 checkpoint — download from# https://modelscope.cn/models/facebook/sam3
pip install -U modelscope
modelscope download facebook/sam3 \
--local-dir ./checkpoints/sam3
export SAM3_CHECKPOINT_PATH=$PWD/checkpoints/sam3/sam3.pt
export LIBERO_TYPE=pro
# Run one task: libero_object_swap, task 2, seed 0, using Claude Code# with Claude Opus 4.8.
rpent --robot libero --suite libero_object_swap --task 2 --seed 0 \
--cuda-device 0 --planner claude_code --model claude-opus-4-8See the planner docs to configure other planners (api, codex) and model providers.
For the exploration workflow and local-memory evaluation, see LIBERO exploration mode.
Add --interactive (-i) to steer the agent live from your terminal. At the you> prompt, the built-in task is pre-filled — press Enter to use it or replace it with your own — then type any message while it runs to steer the agent at the next turn (/help lists commands; /quit or Ctrl-D ends). Requires an interactive terminal (TTY).
rpent --robot libero --suite libero_object_swap --task 2 --seed 0 \
--planner claude_code --model claude-opus-4-8 --interactiveAdd --dashboard to start a local Dashboard and print its URL in the terminal. Open the URL and confirm the configuration; once the services are ready, start a task with /rpent-task <suite> <task> <seed>. The page streams agent reasoning, camera views, and the action timeline, and you can submit another task after the current one finishes. Use --dashboard-language zh-cn for the Chinese UI.
rpent --robot libero --dashboard --dashboard-language zh-cn \
--planner claude_code --model claude-opus-4-8For a complete list of CLI options, see the Key CLI options table in the Quick Start docs. RoboCasa and RoboTwin use their own entrypoints and CLI — see the RoboCasa and RoboTwin docs.
For more detailed documentation, see the RPent documentation.
If you find RPent or Harness VLA helpful, please cite the paper:
@article{zhang2026harnessvla,
title={Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents},
author={Zhang, Yixian and Zhang, Huanming and Gao, Feng and Li, Xiao and Liu, Zhihao and Zhu, Chunyang and Qiu, Jiaxing and Yan, Yuchen and Liu, Jiyuan and Tang, Wenhao and Fang, Zhengru and Nie, Yi and Wei, Changxu and Wang, Yu and Ding, Wenbo and Yu, Chao},
journal={arXiv preprint arXiv:2607.08448},
year={2026},
url={https://arxiv.org/abs/2607.08448}
}RPent builds on the simulators, VLA models, and training infrastructure of RLinf, and on the agent SDKs of the broader open-source community — pydantic-ai, the Claude Agent SDK, and the OpenAI Codex SDK. Thanks to the teams behind LIBERO, RoboCasa, robosuite, MuJoCo, and openpi.