Latent-to-4D: Generating 4D Scenes Directly from Video Model Latents

Zhejiang University introduced the Latent-to-4D method, which generates dynamic 4D scenes directly from the latents of video diffusion models without intermediate RGB decoding.

Compiled by Sergey KostenchukPublished 2026-08-13Updated 2026-09-18

2026-08-13 Research HuggingFace

🤖 Latent-to-4D: 4D scenes directly from latents

Zhejiang University introduced a method that generates 4D scenes without intermediate RGB. A single checkpoint on ~1000 clips works with multiple video DiTs. It outperformed Wan+4RC by +2.88–5.81 DINO-F1, with 66.8–72.1% human preference.

🌍 Bypassing RGB eliminates error accumulation in cascaded pipelines — a shift from rigid architectures to a modular approach.

👤 Code and weights are in preparation. A demo of all modes is on the project page.

Source 1: https://arxiv.org/abs/2608.10744 Source 2: https://hayd-zju.github.io/Beyond-Pixels/

Sources