Grant: Discovery Grants Program - Individual
University of Toronto
$55,000.00
Representations and Interfaces for storytelling with 3D animated characters
FY 2024-25
Summary
Representations and Interfaces for storytelling with animated characters
Across culture and time, storytelling with anthropomorphic characters lies at the very core of the human spirit. Humans have evolved with stories and storytelling in turn has evolved with humans. The creation and sharing of stories, through text, audio, illustration, animation, image, or video, is all increasingly digital. Such creative digital content has a pivotal role in shaping and connecting humanity. The digital platform has unprecedentedly democratized self-expression, publishing, recording, and sharing of stories through blogs, images, audio and video. Animation as a medium for storytelling however, remains beyond the reach of most creative people. This proposal will research digital representations, tools and interfaces for the creation of stories involving animated characters.
Aspects of animation are getting easier: assets are readily available online (eg. Sketchfab, Metahuman); character motion can be similarly accessed (eg. Mixamo); facial animation can be driven by speech or video (eg. JALI, faceware); and these scenes can be assembled in animation software and game engines (eg. Maya, Adobe Char. animator, Unreal). Credible animation however, requires a space-time integrity not realized by naively assembling disparate assets. Even today, storytellers and directors at animation studios execute creative visions frame-to-frame, through teams of skilled animators.
The non-linear temporal pacing and causality of events in story-time however, is not inherently captured or preserved as a linear sequence of frames. While data-driven techniques can learn space-time co-relations in animation, their latent spaces do not provide users' with meaningful manipulation handles. This research will thus focus on performance driven storytelling, structured into three themes.
Face/body animation representations:
Faces are typically represented as a linear combination of FACS-based shapes, and bodies as joint skeletons. The onus of maintaining the complex co-relations between various face/body parameters, in static pose and animation, is difficult. Representations, such as staggered pose manifolds, will be explored, to ease user control of animated characters.
Space-time animation frameworks:
Stories can have a complex, branching non-linear space-time structure. This theme will conduct research into animation techniques that eschew the traditional linear time slider, in favour of stacked storyboards, layered animation behaviour, and audience-driven story abstraction and re-timing.
Character animation interfaces:
This theme will research a variety of novel interfaces, ranging from facial doodles, latent space animation layouts, animation curve handles, transcript and speech-driven storyboards, and direct 3D motion control in XR.
Animated character storytelling as addressed by this proposal is critically important, as digital humans become extensions of physical lives.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2022-04328
Name nom
Singh, Karan
Department département
Computer Science (St. George Campus) - Computer Science (St. George Campus)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2022
Award amount
55000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2707
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Graphics
Research subject fr
Infographie
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Representations and Interfaces for storytelling with 3D animated characters
Keywords
character animation; facial animation; sketch interfaces; autonomous characters; directorial interface; lip-synchronization; emotion modeling; time-warping; storyboarding; motion editing
Application summary
Representations and Interfaces for storytelling with animated characters
Across culture and time, storytelling with anthropomorphic characters lies at the very core of the human spirit. Humans have evolved with stories and storytelling in turn has evolved with humans. The creation and sharing of stories, through text, audio, illustration, animation, image, or video, is all increasingly digital. Such creative digital content has a pivotal role in shaping and connecting humanity. The digital platform has unprecedentedly democratized self-expression, publishing, recording, and sharing of stories through blogs, images, audio and video. Animation as a medium for storytelling however, remains beyond the reach of most creative people. This proposal will research digital representations, tools and interfaces for the creation of stories involving animated characters.
Aspects of animation are getting easier: assets are readily available online (eg. Sketchfab, Metahuman); character motion can be similarly accessed (eg. Mixamo); facial animation can be driven by speech or video (eg. JALI, faceware); and these scenes can be assembled in animation software and game engines (eg. Maya, Adobe Char. animator, Unreal). Credible animation however, requires a space-time integrity not realized by naively assembling disparate assets. Even today, storytellers and directors at animation studios execute creative visions frame-to-frame, through teams of skilled animators.
The non-linear temporal pacing and causality of events in story-time however, is not inherently captured or preserved as a linear sequence of frames. While data-driven techniques can learn space-time co-relations in animation, their latent spaces do not provide users' with meaningful manipulation handles. This research will thus focus on performance driven storytelling, structured into three themes.
Face/body animation representations:
Faces are typically represented as a linear combination of FACS-based shapes, and bodies as joint skeletons. The onus of maintaining the complex co-relations between various face/body parameters, in static pose and animation, is difficult. Representations, such as staggered pose manifolds, will be explored, to ease user control of animated characters.
Space-time animation frameworks:
Stories can have a complex, branching non-linear space-time structure. This theme will conduct research into animation techniques that eschew the traditional linear time slider, in favour of stacked storyboards, layered animation behaviour, and audience-driven story abstraction and re-timing.
Character animation interfaces:
This theme will research a variety of novel interfaces, ranging from facial doodles, latent space animation layouts, animation curve handles, transcript and speech-driven storyboards, and direct 3D motion control in XR.
Animated character storytelling as addressed by this proposal is critically important, as digital humans become extensions of physical lives.