TECH

AI helps digital humans move more naturally as they speak

People communicate with more than words. A raised hand, a change in posture or a facial expression can help convey meaning and emotion. Now, researchers have developed an artificial intelligence (AI) system that can translate speech into coordinated movements of the body, hands and face, potentially making digital characters and virtual presenters more expressive and lifelike.

Developed by researchers at Tohoku University and the University of Hong Kong, DynaConTalk generates three-dimensional (3D) human movements that correspond to spoken content. Unlike approaches that generate only individual gestures, the system coordinates different parts of the body while maintaining smooth movement over longer speeches. Users can also guide the generated motion by specifying poses or providing reference examples.

Generating realistic full-body movement from speech presents several challenges. Large changes in body posture can overshadow smaller movements, such as finger gestures and facial expressions, making them difficult for AI models to learn. This can result in repetitive gestures or unnatural shaking. Another problem occurs when separately generated movement segments are joined together, sometimes creating noticeable changes in speed or rhythm.

To address these issues, DynaConTalk separates movement into slower and faster components while keeping them synchronized. This is similar to separating sound into low- and high-frequency components. By learning these different types of movement together, the system can capture both broad changes in posture and subtle, brief gestures. It also dynamically adjusts how much it relies on different features of speech, including rhythm, voice characteristics and spoken words, as it generates movement.

The system also uses information from previously generated movements to guide the next segment. This allows it to maintain continuity across longer sequences, reducing abrupt transitions and helping produce more natural movement during extended speeches.

The researchers evaluated DynaConTalk using two publicly available datasets of speech and human movement, BEAT2 and SHOW. They also conducted a study in which 30 participants compared videos generated by three different systems across 15 speech examples. DynaConTalk received the highest average ratings in all four evaluation categories: naturalness, correspondence with spoken content, synchronization and movement diversity. Automated evaluations further showed that its generated movements more closely resembled the distribution of recorded human movements.

The technology could have applications in character animation for video games and films, virtual presenters, online education and remote communication. For example, creators could specify a character's initial pose or provide a reference movement, then let the AI generate the surrounding gestures and expressions. This offers a more flexible way to create animated characters whose movements complement what they are saying.

A sequence of poses generated by DynaConTalk from speech. The changing posture, hand gestures and facial expressions illustrate how the system coordinates movement across the body over time. ©Yifei Zhu et al.

The team hopes to extend the approach to a wider range of whole-body movements, including walking. However, existing datasets contain relatively limited examples of large-scale movement, while speech alone does not provide enough information to determine how a person should move through space. The researchers plan to explore larger motion datasets and additional guidance about movement paths to address these limitations.

Researchers at Tohoku University and the University of Hong Kong have developed an AI system called DynaConTalk that translates human speech into lifelike, coordinated 3D movements across a digital character's body, hands, and face.

AI helps digital humans (avatars) move more naturally while speaking by translating speech, text, and audio cues into coordinated whole-body, hand, and facial movements.

Key AI techniques and methods:

• Diffusion Models and Wavelet Constraints: Newer frameworks—such as research from Tohoku University—utilize specialized diffusion models (like wavelet-constrained diffusion) to generate long-form, controllable 3D co-speech motions for the entire body, hands, and face instead of isolated gestures.

• Multimodal Speech-to-Motion Mapping: AI analyzes the rhythm, pitch, pauses, and phonemes of spoken audio alongside language semantics. This allows the avatar to naturally time its nods, hand shrugs, weight shifts, and eye contact to match the emotional flow of the conversation.

• Lip-Sync and Facial Alignment: Advanced decoders analyze audio or speech transcripts to map out mouth shapes—prioritizing clear lip-sync milestones first and filling in contextual transitions—so facial expressions align tightly with the spoken words.

• Real-Time LLM Orchestration: Platforms like UneeQ use real-time large language model (LLM) orchestration to coordinate language generation, voice synthesis, and physical movement simultaneously within milliseconds, enabling fluid, interactive live conversations.

• Reference-Based and Pose Guidance: Creators can specify an initial pose or reference movement, letting the AI procedurally generate surrounding gestures, expressions, and posture shifts that complement the dialogue

How DynaConTalk works...Generating realistic full-body motion from audio is difficult because large posture shifts often drown out subtle finger gestures or facial expressions, leading to robotic shaking or repetitive loops. DynaConTalk solves this through a few key innovations:

• Frequency Separation. The system splits motion into slower and faster components—much like separating audio into low and high frequencies. Learning them together allows the AI to capture broad posture changes alongside rapid, subtle hand and facial movements.

• Dynamic Speech Balancing. As the AI generates animation, it continuously adjusts how heavily it relies on different speech cues, including vocal rhythm, voice characteristics, and the actual spoken words.

• Long-Form Continuity. To prevent awkward jumps or speed changes during long speeches, the model uses previously generated motion segments to guide upcoming frames.

• Creator Control. Animators can input a specific starting pose or reference movement, and the AI generates the surrounding gestures and expressions to match.

Key results & applications:

• Human & Automated Testing. Evaluated on the BEAT2 and SHOW motion datasets and tested with 30 human participants, DynaConTalk outperformed competing systems in naturalness, speech synchronization, content relevance, and gesture diversity.

• Practical Uses. The system is designed to streamline character animation for video games, films, virtual presenters, online education, and remote communication.

• Publication & Future Work. Published in ACM Transactions on Graphics in November 2026 and scheduled for presentation at SIGGRAPH Asia 2026, the team next plans to expand the model to handle full-body spatial movements like walking. You can explore technical demos and code details on the official DynaConTalk Project Page

mundophone