Value Sim-to-Real: Simulation Values
for Efficient Real-World RL

Real-world reinforcement learning (RL) is a promising approach for advancing robotic manipulation, but how to obtain informative and robust reward signals remains a central challenge. Recent works have explored reward models trained on real robotic data to enable efficient real-world RL. However, this formulation can be expensive and infeasible due to the high-dimensional state space and its dependence on real-world expert data. In this paper, we address this limitation with Value Sim-to-Real (VSR), a novel framework that leverages simulation-trained value functions to define informative, high-quality reward functions for efficient real-world RL. We evaluate our method with both sim-to-sim and sim-to-real experiments on several vision-based contact-rich tasks. We show that VSR learns a smooth value function that strongly correlates with task progress and improves the performance and sample efficiency of downstream RL with its induced reward function.