Practical reinforcement learning for adaptive robotic manipulation: Sample efficiency, sim-to-real transfer, and context inference
Date
2025
Authors
Enayati, Amir Mehdi Soufi
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Modern robotic systems seek the ability to adapt to novel tasks and environments in a sample-efficient and robust manner. A framework is proposed to enable such adaptability through three interrelated and complementary contributions in the field of reinforcement learning (RL) for robot manipulation. The central challenges compromising the practical use of RL are addressed, including data efficiency, sim-to-real transfer, and context-aware generalization to unseen tasks.
The first contribution addresses the sample-efficiency challenge. Demonstration Exploitation by Abstract Symmetry of Environments (Demo-EASE), introduces a sample-efficient RL framework with limited demonstrations that can be augmented by exploiting the symmetry. By identifying and leveraging symmetry in manipulation environments, abstract demonstrations are reused across multiple sub-regions of the task space. The implemented masked behavior cloning allows online adaptive balance between pure RL and imitation learning. Demo-EASE shows effective knowledge transfer, improved learning efficiency, and fewer interactions required while generalizing on the workspace of the expert policy.
The second contribution focuses on improving the reliability of sim-to-real transfer. A novel concept, Real-Time Intrinsic Stochasticity (RT-IS), is introduced, demonstrating that inherent noise of real-time simulations can be beneficial when approximating real-world uncertainty. Experimental validation on simulated and physical robot tasks confirms that RT-IS improves deployability, requiring less explicit tuning than domain randomization and relaxing the threshold on modeling precision.
The third contribution addresses the challenge of inferring task representation in a meta-RL setting. A transformer-based belief model, Context Representation via Action-Free Transformer encoder-decoder (CRAFT), is developed to infer variational latent task belief from sequences of states and rewards, without access to the agent's actions. This action-agnostic approach improves adaptability in partially observable environments and supports effective zero-shot learning. Tested on the MetaWorld benchmark, CRAFT outperforms existing baselines in generalization with meaningful task inference quality.
Together, these contributions complete the roadmap to create a framework for adaptive reinforcement learning in robotics. The results demonstrate how structured data, intentional noise, and agent-agnostic long-horizon attention can create efficient, robust, and adaptable learning systems. This work lays a theoretical and practical foundation for future developments in adaptive robotics, enabling robots to operate across a broad spectrum of environments and objectives while cutting down on retraining.
Description
Keywords
Reinforcement learning, Adaptive robotics, Robot manipulation, Task inference, Sim-to-real transfer, Behavior cloning, Bayes-adaptive MDP, General abstract symmetry