Researchers at Zhejiang University published the paper Beyond Pixels (arXiv 2608.10744), describing the Latent-to-4D method, which generates dynamic 4D scenes directly from the latent space of video diffusion models, bypassing the classic cascading chain through RGB pixels. The method outperformed the best cascading approaches on two benchmarks and received the majority of votes in human evaluation, however the code and model weights have not yet been released.
What happened
On August 11, 2026, the ReLER/CCAI group at Zhejiang University published the Latent-to-4D method on arXiv, which aligns the final latents of video-DiT on a token grid of a pre-trained 4D decoder and refines them using frame-wise and global spatiotemporal attention. The model was trained on approximately 1000 clips and works with several compatible video-DiTs from the same VAE family without retraining. On the Text4D-200 and I4D-200 benchmarks, the method outperformed the Wan+4RC cascade by 2.88–5.81 DINO-F1 points and received 66.8–72.1% preference in human evaluation for geometry quality and temporal stability. Supported modes include Text-to-4D, Image-to-4D, Audio-driven 4D, Video-as-prompt 4D, Pose-controlled, and Motion-controlled.
Context
Before the emergence of Latent-to-4D, 4D scene generation was built on a cascading principle: first, a video generator created a sequence of frames in RGB, then a separate module reconstructed 3D geometry with a temporal component from them. Such a pipeline accumulates errors at each stage and suffers from distribution mismatch between the video generator and the 3D reconstructor. The latent space of video diffusion models contains structured information about geometry and motion, and the idea of Latent-to-4D is that this structure can be used directly, without the losses that arise during intermediate decoding to pixels.
Why this is important for the industry
Bypassing RGB decoding eliminates error accumulation and distribution mismatch in cascading 4D generation pipelines. A unified latent interface allows transferring one trained model to multiple video generators without retraining — this is a fundamental shift from hard binding to a specific architecture to a modular approach. For teams developing cascading pipelines, there is a risk of current architectures becoming obsolete. It is expected that upon publication of the code and weights, the method will quickly become a baseline for 4D generation, and latent alignment as a technique will spread to adjacent tasks of mapping between generative models and structural representations.
Why this is important for users
The paper has just been published, and the weights and code have not yet been released. The hayd-zju/Beyond-Pixels repository on GitHub has collected 75 stars, and the work takes first place in HuggingFace Daily Papers. The project page contains interactive demos of all 81 frames of generation — the quality of 4D scenes can be evaluated right now, without downloading models. For researchers and developers, this is a signal to start designing integration prototypes and API contracts for future integration of the latent approach into their pipelines.
What is still unknown / limitations
The code and model weights have not been published, making it impossible to reproduce the results and evaluate latency. The method has only been tested with video-DiTs from one VAE family — compatibility with other architectures has not been verified. Metrics of computational cost, memory consumption, and inference time are absent in the work. The Text4D-200 and I4D-200 benchmarks are limited in scale — results on larger datasets may differ.
Sources
- Beyond Pixels: From Video Priors to 4D Worlds — arXiv 2608.10744
- Beyond Pixels: From Video Priors to 4D Worlds — Project Page
- hayd-zju/Beyond-Pixels — GitHub
Author
Look at AI, editorial team