The team led by Matthias Nessler from TU Munich released Video Diffusion Avatars — a method that turns the WAN2.2-Animate-14B video diffusion model into a controllable 3D avatar from a single photograph. The expensive part of the work, understanding the real world, is already built into the pre-trained diffusion model, so adaptation reduced to relatively cheap fine-tuning, and the avatar gained separate control channels for camera, expression, and body pose. Code and weights have been published, though only under the non-commercial license CC BY-NC-SA 4.0.

image
image

What happened

Matthias Nessler's group at TU Munich published the Video Diffusion Avatars project, accompanied by a paper on arXiv with number 2610.23772, code on GitHub, and model weights. The method fine-tunes the WAN2.2-Animate-14B video diffusion model into a controllable 3D avatar built from a single photo; an adapter of about 6.5 GB is trained on top of the base model, which took 34 hours of machine time on a single A100. The avatar has three independent control channels: the camera trajectory is set by any orbit around the head up to a full fly-around, expressions (including tongue and mouth) are transmitted via view-disentangled X-NeMo codes, and the upper body pose is controlled separately. Results were measured on the BecomingLit benchmark: compared to the best single-image 3D avatars, the method reduces face LPIPS by 29% and increases JOD by 66%. A distilled 4-step fast version streams video of arbitrary length on a KV-cache, outputting approximately 3.5 frames per second at 288x512 resolution on an A100 and maintaining more than 3000 frames without degradation.

Context

Controllable 3D avatars have traditionally been built on explicit representations: pipelines based on 3DMM and NeRF require separately modeling light, hair, and other physics. Video diffusion models are structured differently: they are pre-trained on massive volumes of real video and generate shadows, reflections, refractions, and hair dynamics themselves — effects unavailable to explicit 3D renderers. VDA demonstrates a paradigm shift: instead of building an explicit 3D representation, ready-made video diffusion is 'tamed,' forced to listen to external control signals while appearance and physics remain the responsibility of the diffusion model itself. The technical reason why simple direct fine-tuning would not have worked: learned expression codes are heavily dependent on the viewing angle, and without compensation, the avatar would 'slide' along with the camera. Hence the project's main training idea, camera-triplet: the same moment in time is shown to the model from three cameras, teaching it to think about the head in 3D and align expressions across views.

Why this matters for the industry

The economics of customization are changing: the expensive global visual prior is paid for by the pre-training of WAN2.2-Animate-14B, and specialization for a character costs tens of GPU hours and one adapter, if these figures are reproducible. Then creating an avatar transforms from a project to build a separate 3D pipeline into inexpensive fine-tuning on top of ready-made video diffusion. The camera-triplet trick itself is not architectural and is reusable: multi-view alignment of a single moment can be transferred to other controllable fine-tunings of video diffusion, for example for bodies, scenes, and multi-character scenes. For teams building virtual speakers, VFX, or virtual avatars, this is a path to consistent multi-video characters with three 'knobs' instead of a classic 3D rig. The ability to stream long videos without quality degradation is an argument that transformer-based video diffusion models can be made streaming without explicit 3D representation, and a likely wave of derivative works — hosted-API, distillation for more accessible GPUs, and integration of control codes with voice pipelines.

Why this matters for users

Practical value for the reader: a character is assembled from a single photograph, and this works even with drawn heroes, and the camera, face, and body can be controlled independently, like a director, not a 3D modeler. It can be tried now: code and weights are open, the environment deploys in about 30 minutes, but it will require about 65 GB on disk, a GPU with 48–80 GB of video memory of the A100 or H100 class, and a free FLAME account. With such a threshold, a team can assemble a non-commercial demo in a few days: a previz roll, avatar check from a photo, or their own pipeline from photo to controlled video session. The non-commercial license CC BY-NC-SA 4.0 means that commercial use is currently prohibited, so for the general public this is a research project, not a ready-made service.

What is still unknown / limitations

The gap on BecomingLit should be read with caution: face LPIPS and JOD are perceptual metrics, and there is only one benchmark, so part of the advantage may be explained by the favorability of diffusion textures for these metrics, rather than the geometric consistency of avatars, and verification on other data is needed. There is no direct comparison with classic 3DMM, NeRF, and 3DGS pipelines in their strong scenarios, stability of geometry and identity across views, in the material, so the thesis that VDA already devalues explicit avatar graphics is premature. There are no product guarantees either: research code has no API, pricing, or SLA, and the real-time question is not solved. Open questions remain: whether the adaptation figures are reproducible on other base VDMs, whether the speed will catch up with live scenarios, and whether such avatar adapters will remain a separate class of models or be absorbed by base diffusion models.

Sources

Author

Look at AI, editorial team