George Drettakis: Combining AI and physics to create lifelike 3D

The first, which I've been studying for years, starts with photos of an object or scene captured from different angles, but sharing pixels. These photos are used to create a 3D point cloud, which is then “densified”, to represent the scene in a allowing fast display, with high visual quality. But this method doesn't allow for physics-based calculations.  

The second approach is used for video games and special effects, but also architecture. It relies on physics to simulate lighting, and requires tedious modeling, typically done by artists to define geometry, materials and lighting. This is followed by expensive calculations to simulate light transport. On the other hand, these methods are difficult to adapt to the creation of 3D scenes from photos. 

The third, and more recent, technique is based on generative AI, for example Chat GPT. A scene is described by text in a system like Midjourney or Dall-E, and a computer-generated image is created rapidly and automatically. The problem is that this type of solution is neither very precise nor realistic, and it has no notion of the underlying physics. As a result, you have no control over the properties of what is displayed. 

It's very ambitious. And for some aspects, I still have only vague notions of how we'll go solving the problems! But ERCs are designed to explore such ambitious research directions and aim for breakthrough innovations.

It's hard to give you a precise answer. When a 3D scene creation tool performs well, it can have applications far beyond what you might have imagined. Less than a year after its publication, our 3D Gaussian Splatting approach is already being used in e-commerce, video games and cinema special effects, as well as in real estate, the inspection of engineering structures and damaged industrial sites...  

 With the solutions that will be proposed by NERPHYS, we can imagine much more advanced industrial uses in the same fields, and others entirely new, thanks to the physics-based control offered by our new algorithms. 

We're well equipped in terms of photo-based rendering techniques, but we'll need to draw on some outside expertise. For deep learning and physics-based synthesis, we'll be relying on other universities and research institutes in France, Europe and beyond, such as our long-time collaborator Professor Frédo Durand at MIT (USA).