AgentHands: Generating interactive hand gestures for spatially grounded agent conversations in XR

AgentHands is a research prototype that gives AI conversational agents the ability to make expressive hand gestures synchronized with their speech in extended reality (XR) environments. The system helps bridge the gap between verbal instructions and physical understanding by having virtual hands point to, outline, and demonstrate actions in a user's actual surroundings rather than relying on flat visual overlays. A study comparing this approach to speech-only guidance found that participants significantly improved at locating objects and following complex procedural tasks when the agent used synchronized gestures. This matters because as AI assistants move from text interfaces toward immersive spatial platforms, embodied communication through hand gestures can make assistance feel more natural and effective for physical, real-world tasks.