Grant: Alliance Grants
University of Toronto
$90,000.00
Generative video models for controllable digital human avatars
EF 2024-25
Résumé
This research proposal aims to advance the development of digital human avatars, which are crucial for creating immersive virtual environments in gaming, film, virtual reality, education, remote communication, and beyond. While recent breakthroughs in large-scale generative models have enabled photorealistic image and video generation, significant challenges remain in accurately replicating the complexities of human appearance and expression. For example, current methods, struggle with photorealism and issues like temporal consistency, shifts in appearance, and limited control over fine-grained details such as facial expressions and head movements.
To address these challenges, we propose a partnership with LG Electronics focused on three key objectives: first, we will leverage LG's advanced 3D face tracker to create a large-scale dataset of tracked human videos, which will augment existing human video datasets with dense 3D reconstructions of head poses and facial animations; second, we will develop a novel generative model for controllable digital human avatar generation, and we will train it on this large-scale dataset; and third, we will improve the efficiency of our approach to facilitate integration into LG's digital AI assistant pipelines. Overall, the collaboration aims to push the boundaries of digital human avatar technology, enabling more realistic and versatile avatars for a wide range of applications including entertainment, virtual technical support, and digital marketing.
Programme
Alliance Grants
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
600005-2024
Name nom
Lindell, David Brian DB
Department département
Computer Science (St. George Campus)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
90000
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Generative video models for controllable digital human avatars
Keywords
3d reconstruction; computer graphics; computer vision; diffusion models; digital human; facial animation; generative models; machine learning; neural rendering; video generative models
Application summary
This research proposal aims to advance the development of digital human avatars, which are crucial for creating immersive virtual environments in gaming, film, virtual reality, education, remote communication, and beyond. While recent breakthroughs in large-scale generative models have enabled photorealistic image and video generation, significant challenges remain in accurately replicating the complexities of human appearance and expression. For example, current methods, struggle with photorealism and issues like temporal consistency, shifts in appearance, and limited control over fine-grained details such as facial expressions and head movements.
To address these challenges, we propose a partnership with LG Electronics focused on three key objectives: first, we will leverage LG's advanced 3D face tracker to create a large-scale dataset of tracked human videos, which will augment existing human video datasets with dense 3D reconstructions of head poses and facial animations; second, we will develop a novel generative model for controllable digital human avatar generation, and we will train it on this large-scale dataset; and third, we will improve the efficiency of our approach to facilitate integration into LG's digital AI assistant pipelines. Overall, the collaboration aims to push the boundaries of digital human avatar technology, enabling more realistic and versatile avatars for a wide range of applications including entertainment, virtual technical support, and digital marketing.