🤖 ProxyPose: A New Method for 6-DoF Video Tracking
The ProxyPose framework has been developed, which uses LoRA adaptation of the Wan2.1-T2V-14B model to create "proxy videos." This allows for the extraction of precise 3D motion trajectories of an object using classical geometry, even if the original video contains complex textures or glares.
🌍 This combines generative AI and computer vision, allowing it to bypass the limitations of traditional methods like COLMAP when dealing with reflections and transparent surfaces. This is crucial for the development of AR/VR and robotics.
👤 Users can obtain an accurate 3D motion trajectory simply by specifying a point on a standard video, eliminating the need for expensive sensors.
Source 1: https://ruihangzhang97.github.io/proxypose/ Source 2: https://github.com/ruihangzhang97/proxypose