Humanoid Teleop with Full Body Tracking Using the Meta Quest 3 and IsaacSim (simintel.co)

A team has developed a cost-effective wireless VR teleoperation system using the Meta Quest 3 and NVIDIA's IsaacSim to enhance the collection of synthetic datasets for training humanoid robots. This innovative approach leverages the Meta Quest 3’s advanced body tracking capabilities, streaming data through ALVR and SteamVR. By offering a low-cost alternative—significantly less expensive than traditional motion capture setups—it opens the door for students, independent researchers, and startups to collect high-quality teleoperation data, which is crucial for developing human-like robotic behaviors. The system captures motion from nine key body joints, providing full 6DOF tracking and real-time visual feedback within a simulation environment. Technical advancements include an OSC-based receiver architecture, which allows for seamless integration of tracking data into IsaacSim, and dynamic retargeting to humanoid robot models. While challenges such as leg tracking accuracy and occlusion issues remain, this project establishes a strong foundation for future developments in humanoid robot learning, including a comprehensive dataset for loco-manipulation tasks. The ongoing work aims to create robust controllers and models that will drive further improvements in embodied AI systems.