A new framework, BRDFusion, has been introduced that combines Physically Based Rendering (PBR) with generative models for high-quality inverse rendering of urban locations based on video.
What Happened
The BRDFusion framework has been developed, utilizing 3D Gaussian Splatting (3DGS) and diffusion models to reconstruct geometry, materials, and HDR lighting from video recordings. The system enables relighting, night-time simulation, and the insertion of dynamic objects into the scene.
Context
Traditional methods often face a gap between the physical accuracy of rendering and the visual realism characteristic of generative models. BRDFusion addresses this issue using a hybrid PBR + Generative approach, where 3DGS provides efficient geometry reconstruction at urban scales, while diffusion models eliminate artifacts and enhance image quality.
Why It Matters for the Industry
For the AI and autonomous vehicle industries, this technology opens up possibilities for creating high-quality synthetic simulations for training agents and testing sensors. This could lead to the emergence of specialized libraries and plugins for automating data preparation in robotics and the creation of digital twins for cities.
Why It Matters for Users
Content creators and researchers gain a tool for editing real-world urban video scenes: one can change the time of day or add objects so that they appear physically correct and natural within the ambient lighting.
What Is Not Yet Known / Limitations
At the current stage, the project is a demonstration prototype. The observed image blurring indicates difficulties in preserving high spatial frequencies when using diffusion models, which limits the method's application in high-precision production.
Sources
Author
Look at AI, Editorial Staff