Grant: Discovery Grants Program - Individual
Université de Montréal
$148,000.00
Extending the Frontiers of Deep Generative Modelling.
EF 2022-23
Résumé
Modern neural network-based models have made dramatic improvement in the quality of synthesize images. However the ability of these generative models to synthesize video or 3 dimensional (3D) scenes has received much less attention from researchers. In this proposal, I describe a series three of projects with the goal of addressing this gap: learning generative models of 3D scenes; learning generative models of video; and learning transformations between different representations, specifically between 2D images and 3D scenes.In developing good representations of 3D scenes, the 3D representation is key. Here we propose to use a representation known as splats. Splats represent 3D scenes as a set of independent surface points. Our approach will be to build a generative model that will implicitly represent the full 3D scene and output only the splats that correspond to the current viewpoint of the observer. This approach should allow us to scale to complex scenes with cluttered objects in a way that has not yet been demonstrated.Our approach to video will focus on two tracks of innovations. We will explore novel model architectures (i.e. the wiring pattern between different modules) that better capture the nature of video. Specifically, we will explore models that allow an easy pathway for pixel from one frame to propagate to the next frame relatively unchanged. This captures the observation that from frame to frame, videos change slowly. The other track of innovation includes exploring alternative objective for the learning algorithm. We will experiment with a variant of a new training strategy that frames the learning of a generative model as an adversarial game between the generator and a discriminator that is trying to distinguish between true data examples and the synthesized examples from the generator.Finally, we will explore methods to learn transformations between 2D images and 3D scenes. To do so, we will use the same adversarial training strategy as described above. One important challenge that we face in this project is that the transformation from images to 3D scenes is not fully specified, for instance the surfaces of occluded objects are not represented in the image, yet they would have to be inferred in the 3D representation. Dealing with this issue of missing information is the focus of this part of the proposal.With this proposal, we set out to lower the barrier between the dynamic 3D world in which we live and the virtual world of video gams and simulators. Providing the ability to enriching simulation this way has the potential to greatly increase the rate of progress in fields such as robotics and autonomous design.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2018-05920
Name nom
Courville, Aaron
Department département
Informatique et recherche opérationnelle - Informatique et recherche opérationnelle
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2018
Award amount
148000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Installment
inst:5;6
Partie
2
Nb partie
6
Application title
Extending the Frontiers of Deep Generative Modelling.
Application summary
Modern neural network-based models have made dramatic improvement in the quality of synthesize images. However the ability of these generative models to synthesize video or 3 dimensional (3D) scenes has received much less attention from researchers. In this proposal, I describe a series three of projects with the goal of addressing this gap: learning generative models of 3D scenes; learning generative models of video; and learning transformations between different representations, specifically between 2D images and 3D scenes.In developing good representations of 3D scenes, the 3D representation is key. Here we propose to use a representation known as splats. Splats represent 3D scenes as a set of independent surface points. Our approach will be to build a generative model that will implicitly represent the full 3D scene and output only the splats that correspond to the current viewpoint of the observer. This approach should allow us to scale to complex scenes with cluttered objects in a way that has not yet been demonstrated.Our approach to video will focus on two tracks of innovations. We will explore novel model architectures (i.e. the wiring pattern between different modules) that better capture the nature of video. Specifically, we will explore models that allow an easy pathway for pixel from one frame to propagate to the next frame relatively unchanged. This captures the observation that from frame to frame, videos change slowly. The other track of innovation includes exploring alternative objective for the learning algorithm. We will experiment with a variant of a new training strategy that frames the learning of a generative model as an adversarial game between the generator and a discriminator that is trying to distinguish between true data examples and the synthesized examples from the generator.Finally, we will explore methods to learn transformations between 2D images and 3D scenes. To do so, we will use the same adversarial training strategy as described above. One important challenge that we face in this project is that the transformation from images to 3D scenes is not fully specified, for instance the surfaces of occluded objects are not represented in the image, yet they would have to be inferred in the 3D representation. Dealing with this issue of missing information is the focus of this part of the proposal.With this proposal, we set out to lower the barrier between the dynamic 3D world in which we live and the virtual world of video gams and simulators. Providing the ability to enriching simulation this way has the potential to greatly increase the rate of progress in fields such as robotics and autonomous design.