Generating Holistic 3D Human Motion from Speech

CVPR2023

Hongwei Yi^1*, Hualin Liang^2*, Yifei Liu^2*, Qiong Cao^3†, Yandong Wen¹, Timo Bolkart¹, Dacheng Tao³⁴, Michael J. Black^1†

¹Max Planck Institute for Intelligent Systems, Tübingen, Germany

²South China University of Technology   ³JD Explore Academy   ³The University of Sydney

* denotes equal contribution, † denotes joint corresponding authors

Abstract

This work addresses the problem of generating 3D holistic body motions from human speech. Given a speech recording, we synthesize sequences of 3D body poses, hand gestures, and facial expressions that are realistic and diverse. To achieve this, we first build a high-quality dataset of 3D holistic body meshes with synchronous speech. We then define a novel speech-to-motion generation framework in which the face, body, and hands are modeled separately. The separated modeling stems from the fact that face articulation strongly correlates with human speech, while body poses and hand gestures are less correlated. Specifically, we employ an autoencoder for face motions, and a compositional vector-quantized variational autoencoder (VQ-VAE) for the body and hand motions. The compositional VQ-VAE is key to generating diverse results. Additionally, we propose a cross-conditional autoregressive model that generates body poses and hand gestures, leading to coherent and realistic motions. Extensive experiments and user studies demonstrate that our proposed approach achieves state-of-the-art performance both qualitatively and quantitatively. Our novel dataset and code will be released for research purposes at https://talkshow.is.tue.mpg.de.

Given an input audio, TalkSHOW can generate realistic and diverse holistic human motions.

.

More examples.

Previous

Next

• 1
• 2
• 3

TalkSHOW can generate various style holistic motions for different characters.

TalkSHOW can generalize on the audio from unseen characters.

TalkSHOW can generalize on foreign languages, e.g. French, w/o fine-tuning.

SHOW: Synchronous Holistic Optimization in the Wild.

To train TalkSHOW, we propose SHOW to reconstruct holistic body motion from in-the-wild Youtube videos, to collect the synchronous audio and holistic body motion dataset.

Compared with SMPLify-X, PIXIE and PyMAF-X

Compared with Habibie et al. and its' Inverse Kinematic Results

Comprehensive Explanatory Video 

Poster

Paper

Code and Data

Acknowledgement. We thank Wojciech Zielonka, Justus Thies for helping us incorporate MICA into our reconstruction
method, Yao Feng, Zhen Liu, Weiyang Liu, Changxing Ding and Huaiguang Jiang for the insightful discussions, and Benjamin Pellkoferfor IT support. 
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B. 
Disclosure. MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon.
MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH.

@inproceedings{yi2023generating,
title={Generating Holistic 3D Human Motion from Speech},
  author={Yi, Hongwei and Liang, Hualin and Liu, Yifei and Cao, Qiong and Wen, Yandong 
and Bolkart, Timo and Tao, Dacheng and Black, Michael J},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 
pages={469-480},
month={June}, 
year={2023} 
}