Human-Robot Interaction: Real-Time Control of Robotic Systems via ROS2 and Unity

Abstract

This paper explores the integration of Virtual Reality (VR), Unity, and Robot Operating System 2 (ROS2) to create an immersive and scalable control interface for robotic systems. As the complexity of robots increases, the basic simulation and control environments such as Gazebo often fail to deliver intuitive or complete representations of real-world challenges. In response, we propose an architecture that allows real-time, bi-directional communication between the user and a robotic arm—either physical or simulated—through a VR interface. Unity is employed as a visual and interactive platform, offering flexibility through community-developed plugins for both VR and ROS2. ROS2 provides robust, real-time communication capabilities with support for distributed systems, enhanced security, and industrial scalability. Together, they form a powerful ecosystem for developing human-robot interaction systems. Use cases in education and industrial robotics are presented to demonstrate the effectiveness of the proposed system. In educational settings, VR reduces costs and risks while enhancing engagement and skill acquisition. In industrial contexts, remote operation through VR improves safety, situational awareness, and control accuracy. This type of architecture confirms that integrating VR with ROS2 and Unity is not only feasible but highly beneficial for modern robotic applications, particularly within the scope of Industry 4.0. Future work includes adding AI-driven control, gesture recognition, and haptic feedback.

Authors

* External Author