arXiv:2606.02000: motion control in video diffusion via 3D mesh tokens

A new study published on arXiv under the designation arXiv:2606.02000, titled “Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization”, proposes an approach to human motion control in video generation that does not rely on classic rendering. The authors (Liang et al.) bypass 2D rendered guides and condition the model directly on compressed 3D human mesh tokens.

What is mesh tokenization?

A mesh is a 3D geometric model of the human body made of a network of polygons. Instead of first rendering that model into a 2D image that guides generation, the proposed method converts the geometry into tokens, discrete units that a transformer can process. According to the authors, such a representation “preserves full 3D geometric information” and enables a unified pipeline in which video tokens are processed together with motion tokens.

How does the architecture work?

The system uses a DiT (Diffusion Transformer) architecture in which the model “jointly reasons about appearance, 3D structure and camera angle” during video generation. Motion tokens and video tokens are processed simultaneously within the same transformer, which requires integrated reasoning across multiple modalities at once.

What are the results?

The method shows strong performance on benchmarks for human motion control with practical improvements: it reduces artifacts caused by a viewpoint-dependent 2D guide as well as mismatches between pose and trajectory during editing. The authors conclude that video diffusion models, equipped with mesh tokenization, better capture the complex 3D structures of the human body and their interaction with the environment.