The ByteDance Seed team, in collaboration with researchers from Peking University and Zhejiang University, has presented Lucida — a research system that reconstructs a real room from video or a set of frames as a collection of individual editable 3D objects, rather than a single static model. The arXiv preprint 2608.30821 was submitted on August 31, 2026. Based on the number of authors, the system significantly outperforms known baselines on the R2S-Scene and CA-1M benchmarks, and the results can already be viewed in a browser-based demo on PlayCanvas.
What Happened
The ByteDance Seed research group, together with scientists from Peking University and Zhejiang University, has published the arXiv preprint 2608.30821 describing the Lucida system, which builds a scene from individual editable 3D objects from video or a set of frames of a room. The pipeline consists of three stages. Parse builds a scene graph from key frames with masks, 3D boxes, partial point clouds, and text prompts. Generate synthesizes a complete object, filling in parts occluded by other items. Place arranges the objects using the GizmoAct module: a VLM in a closed render → edit → re-render loop moves the object in a 3D editor until it matches the original frames, and it decides for itself when to stop. On the R2S-Scene test, Lucida raised the scene F-Score from 0.794 for SAM 3D to 0.924, and the object pose accuracy ADD-SB@0.05 on CA-1M increased from 57.8% for RecGen to 83.4% when using up to 4 viewpoints. The preprint is accompanied by an official project page and an interactive demo on PlayCanvas featuring an apartment reconstructed from real footage.
Context
Lucida operates in the real-to-sim domain: transferring a real environment into a simulation, which is needed for robotics, digital twins, and virtual production. Typical video reconstruction produces a single static model of a room, where furniture is part of the baked geometry, so this form is of little use for simulators that require manipulable individual assets. The compositional approach changes the target form of the result: at the output, a chair, wardrobe, or lamp remain independent objects. The authors' key methodological claim is that accuracy is achieved at the end of the pipeline, at the Place stage, and is not required at the input: the cyclic verification with the original frames explains the claimed robustness to noisy real footage, and if this is confirmed on independent data, the requirements for input video quality will be significantly reduced. Methodologically, the most significant contribution appears to be the GizmoAct technique: object placement is treated not as direct pose regression, but as a multi-turn GUI interaction of a VLM with a terminal stopping condition, and this template is transferable to other positioning and parameter tuning tasks.
Why This Matters for the Industry
For the industry, Lucida is a signal that real-to-sim transfer from ordinary video is moving from static room reconstruction to sim-ready individual objects: the pipeline output is individual mesh assets suitable for robotics simulators and digital twins, where each item must remain controllable. The claimed improvements over named baselines are non-trivial, although all metrics are currently author-reported. The GizmoAct technique is valuable in its own right: treating object placement as a multi-turn GUI interaction of an agent with a stopping condition is reusable in other positioning and parameter tuning tasks, and the work can already be cited in research on real-to-sim and agentic positioning. For founders and teams, this means that the production of editable 3D assets from video is becoming an active race, and 3D platforms will start adding scene segmentation and similar features. Engineering-wise, however, this is a pure demonstration: there is no open code, API, or data on latency and cost, so the system cannot be taken into production today.
Why This Matters for Users
Right now, you can open the interactive demo on PlayCanvas and rotate the apartment reconstructed from real footage in 3D, manually assessing the quality of the reconstruction and how well each item remains an independent controllable object. The arXiv preprint 2608.30821 is open for reading, and from it you can understand the structure of the Parse → Generate → Place pipeline. If the authors release the code, it will be possible to obtain an editable scene for robot simulation, virtual production, or location preparation from ordinary room video, where each item remains a controllable object rather than part of monolithic geometry. Until the implementation is open, the pipeline cannot be reproduced on your own videos, and the practical benefit is limited to the demo and studying the materials.
What Is Still Unknown / Limitations
All metrics are author-reported measurements: there are no independent reproductions on R2S-Scene and CA-1M yet. The key claim that accuracy is achieved at the end of the pipeline and not required at the input is currently confirmed only by the team's own experiments. There is no open code or API, and no data on latency and cost has been published, so the pipeline cannot be reproduced and the robustness to noisy footage cannot be tested outside the authors' conditions. The release of the implementation is expected within a few months after publication, but is not guaranteed; if the code is not released, Lucida will remain a demonstration work, and speculation about production application is premature.
Sources
- Lucida: Composable Real-to-Sim Scene Modeling — ByteDance Seed official project page
- Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling — arXiv preprint 2608.30821
- Lucida — interactive 3D demo of apartment reconstruction on PlayCanvas
Author
Look at AI, editorial team