Grant: Discovery Grants Program - Individual

AI and deep learning in particular are making significant forays into the graphics world. Most graphics applications address synthesis of visual objects and models. In general, deep learning solutions depend on large volumes of training data. During the course of developing learning based solutions for the graphics and games industries, major technical challenges that arise include the need for generalization of the solution, learning disentangled knowledge representations to enable user control over the final results, and quality results acceptable to the end user.  However in general, only a low amount of training data is available in real world industry applications, exacerbating these problems further. While the literature abounds with many solutions for these situations, they cannot as yet address the specificity needed in a synthesis application. Clearly, graphics applications have high quality and accuracy requirements. Further, support for user control over the final synthesized product is a must. This research proposal plans to address the above challenges. The primary methodology is to develop learning based applications and continuously analyze the development to abstract basic principles that can guide the use of deep learning techniques in graphics application development. The work is carried out in 4 parallel activities. Three activities are real world application development: (i) Structure-aware 3D reconstruction of monuments, with data captured by flying a mini drone over a heritage monument, (ii) Realistic garment simulation for game characters, for which training data is obtained from the game industry partner, and (iii) Realistic face reenactment for visual dubbing, for which training data is provided by the dubbing studio partner. In all the above, the data available for training is low, but since the application parameters and requirements are different, each will require their own innovative solutions. The goal is to gather abstracted knowledge on deep learning techniques for graphics applications. The fourth activity will continuously probe into the development of these applications to customize/devise new deep learning techniques for graphics applications. Most deep learning models are trained for hierarchic learning, and deep hierarchies require large training data. Non-hierarchic learning has to be investigated more. We will formulate a light weight attention mechanism suitable to graphics applications.  Investigations in this activity will both derive insights from the above applications, and feed back methodologies into the applications to address the challenges. All the above research derives its problems from collaboration with local industry and arts scholars. The solutions/methodology evolved will be fundamental and practical. The research and outcomes are of great interest both to academia and to industry in Canada and world-wide. It will enable high quality training of students at the PhD, MSc and BSc levels.