Google Research published a description of AgentHands — a research prototype for augmented reality (XR/Android XR) in which a conversational AI agent accompanies its speech with synchronized animated hands: points to real objects in the room, shows movements, and provides visual warnings. The work is presented at the CHI 2026 conference.
What happened
The AgentHands pipeline consists of four steps. First, the system registers objects by the user's gaze and reconstructs the scene with 3D frames. Then, a library of gesture events in three semantic categories is applied: deictic, iconic, and expressive. After that, the LLM generates a response with embedded GestureEvents tied to specific words. At the final step, execution occurs on the headset: TTS audio is synchronized with hand animation by word-level timestamps. In demonstration scenarios, the agent outlines the aerial roots of an orchid when watering, shows the sequence of button presses on a 3D printer panel, and warns of the risk of burns with a red glow near a hot nozzle. In an experiment involving 12 people, in which interaction with the agent and a purely voice-based baseline were compared, gestures significantly facilitated object search and understanding of instructions (p < 0.05) on tasks of orchid care and 3D printer control.
Context
Today, voice assistants in augmented reality — Project Astra and Gemini Live — provide hints as "flat" rectangular blocks on a phone screen. AgentHands takes a step toward spatially grounded agents working in AR glasses: hand gestures become part of the LLM output layer on par with text and speech. Importantly, the spatial component itself relies on already known methods of scene reconstruction and gaze tracking, not on new results in the field of 3D perception. The value of the work is in the pipeline as a whole: the connection of 3D scene reconstruction, a semantic taxonomy of gestures, and the binding of LLM output to audio timestamps. For the ML community, this is an application-level publication: there are no new models, SOTA results, or benchmarks in it.
Why this is important for the industry
For the industry, AgentHands is a specific, transferable design pattern: the LLM acts as a source of structured GestureEvents tied to words, and a local parser on the device synchronizes them with TTS audio by word-level timestamps. The output layer of a conversational agent is thereby expanded from text and speech to gestures, and the publication provides a ready-made reference design: a taxonomy of gestures, the format of GestureEvents, and a synchronization scheme. For teams already building AR agents in the Android XR ecosystem, this is confirmation that the pattern has passed the "does it work at all" stage and is a guide for design. In the longer term, "structured events tied to speech timestamps" may become a standard output channel for agents on wearable devices alongside text and speech.
Why this is important for users
The practical effect for users is currently limited: there is no public demo, API, SDK, or repository, and the system cannot be reproduced. The value for the reader is in understanding where voice assistants in AR are heading. A familiar problem, where an agent says "over there" but shows nothing, is solved literally: the agent shows with its hand where the object is, how to use it, and what to be careful of. The Google Research post details the taxonomy of gestures and the system's design, so it serves as an introduction to the UX of spatial agents. If the pattern takes root in Android XR, "showing, not just talking" may become a basic expectation for AR assistants in glasses.
What is still unknown / limitations
The publication does not disclose which specific LLM is used and how GestureEvents are generated: the blog does not describe a specific model, fine-tuning, or prompt engineering. There is no data on latency and cost, no public code or demo. The experiment is small: 12 participants, narrow task domains (orchid care, 3D printer control), and the absence of objective metrics in the announcement, so conclusions about the quality of interaction should be taken with caution. Full details of the experiment will likely appear in the CHI paper.
Sources
Author
Look at AI, editorial team