Logo

MikuDance: Animating Character Art with Mixed Motion Dynamics

Jiaxu Zhang1,2, Xianfang Zeng2†, Xin Chen3, Wei Zuo2, Gang Yu2*, Zhigang Tu1*,
1Wuhan University, 2StepFun, 3ByteDance
†Project leader *Corresponding author
arXiv Code (Coming soon) Video Results

We propose MikuDance, a diffusion-based pipeline incorporating mixed motion dynamics to animate stylized character art.

Abstract

We propose MikuDance, a diffusion-based pipeline incorporating mixed motion dynamics to animate stylized character art. MikuDance consists of two key techniques: Mixed Motion Modeling and Mixed-Control Diffusion, to address the challenges of high-dynamic motion and reference-guidance misalignment in character art animation. Specifically, a Scene Motion Tracking strategy is presented to explicitly model the dynamic camera in pixel-wise space, enabling unified character-scene motion modeling. Building on this, the Mixed-Control Diffusion implicitly aligns the scale and body shape of diverse characters with motion guidance, allowing flexible control of local character motion. Subsequently, a Motion-Adaptive Normalization module is incorporated to effectively inject global scene motion, paving the way for comprehensive character art animation. Through extensive experiments, we demonstrate the effectiveness and generalizability of MikuDance across various character art and motion guidance, consistently producing high-quality animations with remarkable motion dynamics.

Method Overview

Given a reference character art and a driving video, the pixel-wise scene motion is predicted using the Scene Motion Tracking (SMT) strategy, which is combined with the character poses to form the multi-motion guidance. The Mixed-Control Diffusion model subsequently generates the animation in a latent space, guided by the character poses and the scene motion injected through the Motion-Adaptive Normalization (MAN) module.

Results Gallery



Technical Paper


Jiaxu Zhang, Xianfang Zeng, Xin Chen, Wei Zuo, Gang Yu, Zhigang Tu
MikuDance: Animating Character Art with Mixed Motion Dynamics
arXiv:2411.08656, 2024.

BibTeX

@misc{zhang2024mikudance,
      title={MikuDance: Animating Character Art with Mixed Motion Dynamics}, 
      author={Jiaxu Zhang and Xianfang Zeng and Xin Chen and Wei Zuo and Gang Yu and Zhigang Tu},
      year={2024},
      eprint={2411.08656},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}