Non Rigid Registration for Topological Changes
Damien Heese, David Barac, Jonas Müller, Bolkar Eren, Dilara Kus
Imagine trying to match a digital 3D model of an object, say a scanned sculpture or a design from a computer, with photos or sensor scans of the same object in the real world. This process, known as registration, is widely used in fields like medicine, robotics, and film production. It allows computers to understand how objects move, change shape, or interact with their surroundings. But what happens when the object doesn’t just bend or stretch, but actually breaks or tears apart? Most existing methods for registration assume that objects only deform smoothly, like clay being reshaped. However, many real-world situations involve more dramatic changes, like a fractured bone, a torn piece of cloth, or a broken tool. In this project, we set out to address this much harder problem: how to match digital 3D models to real-world observations when the object has not only deformed, but also changed its structure. To tackle this, we took inspiration from the way objects are often captured in real-world imaging scenarios, using projective spaces, an intriguing mathematical setting that can elegantly handle observations from diverse imaging devices ranging from commercial cameras to X-ray devices. We developed two different approaches. The first uses mathematical optimization, where we carefully reformulate the problem so that it can be solved reliably through a process that always finds the best possible match under certain assumptions. The second method follows a more gradual, flexible approach, where the shape of the object is adjusted iteratively, adapting continuously to match the observed data, even as its structure changes. Our work is not just theoretical, we tested these methods on real-world data from cameras, and we will present visual examples showing how our techniques can successfully align objects that have undergone complex changes like tearing or breaking. This project opens up new possibilities for applications where objects are expected to undergo such changes, from medical simulations to robotics and digital effects. While we have kept the methods accessible, we believe some of the underlying mathematical ideas, particularly around projective geometry and optimization, will also spark interest among technical experts in the audience.