Grant: Alliance Grants
University of Toronto
$165,000.00
Implicit Representations for 4D Digital Humans
FY 2024-25
Summary
In media such as cinema, television, and video games, characters are increasingly being portrayed using animated digital humans whose likeness and expressions are programmed to match those of an actor. This approach offers many advantages-digital humans can be placed into any simulated environment, and their actions, expressions, and appearance can be adjusted to any specification without the cost and burden of live capture. Digital humans play an especially important role in video games, where controllable characters are embedded in a completely virtual environment. However, current pipelines for creating digital humans involve a laborious process of 3D scanning and touch-ups by digital artists to fine-tune appearance and animation controls. Beyond being labor intensive, existing pipelines fail to realistically and automatically capture details of human appearance, including pores, wrinkles, and difficult-to-model elements such as hair. Improving the video game experience could have significant impact in Canada, where roughly 53 percent of people report gaming regularly, and where the video games industry contributed roughly 5.5 billion CAD to the gross domestic product in 2021. We propose to overcome these limitations by partnering with Ubisoft, a world-leading video game developer, to create new methods for capturing and rendering 4D digital humans using an emerging technique known as implicit representations. These representations model appearance and geometry using volumes or surfaces that are typically parameterized by a neural network; they have recently enabled photorealistic modeling of scenes from captured images. Leveraging Ubisoft's expertise in video game development as well as their unique access to motion capture and light stage capture equipment, this proposal seeks to (1) significantly improve the photorealism of digital humans, making video games more compelling and immersive; (2) reduce the burden of tedious 3D reconstruction and animation workloads using implicit representations; and (3) provide benefits to many Canadian sectors, including the video game industry and related areas such as film and visual effects.
Program
Alliance Grants
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
592551-2023
Name nom
Lindell, David Brian DB
Department département
Computer Science (St. George Campus)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
165000
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Implicit Representations for 4D Digital Humans
Keywords
3d reconstruction; animation; character; digital human; implicit representation; light stage; machine learning; motion capture; neural rendering; video games
Application summary
In media such as cinema, television, and video games, characters are increasingly being portrayed using animated digital humans whose likeness and expressions are programmed to match those of an actor. This approach offers many advantages-digital humans can be placed into any simulated environment, and their actions, expressions, and appearance can be adjusted to any specification without the cost and burden of live capture. Digital humans play an especially important role in video games, where controllable characters are embedded in a completely virtual environment. However, current pipelines for creating digital humans involve a laborious process of 3D scanning and touch-ups by digital artists to fine-tune appearance and animation controls. Beyond being labor intensive, existing pipelines fail to realistically and automatically capture details of human appearance, including pores, wrinkles, and difficult-to-model elements such as hair. Improving the video game experience could have significant impact in Canada, where roughly 53 percent of people report gaming regularly, and where the video games industry contributed roughly 5.5 billion CAD to the gross domestic product in 2021. We propose to overcome these limitations by partnering with Ubisoft, a world-leading video game developer, to create new methods for capturing and rendering 4D digital humans using an emerging technique known as implicit representations. These representations model appearance and geometry using volumes or surfaces that are typically parameterized by a neural network; they have recently enabled photorealistic modeling of scenes from captured images. Leveraging Ubisoft's expertise in video game development as well as their unique access to motion capture and light stage capture equipment, this proposal seeks to (1) significantly improve the photorealism of digital humans, making video games more compelling and immersive; (2) reduce the burden of tedious 3D reconstruction and animation workloads using implicit representations; and (3) provide benefits to many Canadian sectors, including the video game industry and related areas such as film and visual effects.