Grant: Discovery Grants Program - Individual
Simon Fraser University
$29,000.00
Image Analysis for Realistic Scene Manipulation
FY 2022-23
Summary
There are many photographs or scenes in real life that can induce strong cognitive responses in humans such as emotions like awe or fear, or specific moods such as calmness or chaos. This intricate connection between visual stimuli and cognitive processes makes photography and movie making important venues for artistic expression. This connection also makes image analysis and synthesis that aims at understanding induced emotions and artistic expression a promising research direction to start conceptualizing such high-level cognitive processes in the context of artificial intelligence. The capabilities of current computer vision and computer graphics systems are not at a point that allows us to start looking at which aspects of a scene make it emotionally significant. The main goal of this 5-year research plan is to develop image analysis and manipulation methods that will enable exploring the cognitive effects of artistic choices in film production and photography. Film production and photography are selected as the main application targets because of the decades of traditions developed to effectively convey ideas and emotions visually. This established visual language provides a convenient starting point for bridging the cognitive effects of artistic choices with computational analysis of imagery. The proposed research are planned as three thrusts. The first two thrusts aim to generate scene representations that isolate specific aspects that contribute to the appearance of imagery: (i) Low-level, physical cues including illumination and material appearance, and (ii) High-level, structural cues including semantics and depth. Both research directions feature innovative uses of machine learning techniques to extend their applicability in computational photography, aiming realistic and quick manipulation of scenes through generic image representations as well as interactive tools. The third research thrust builds on the first two to apply scene manipulation methods to movie post-production to analyze the aspects of a scene that contribute to the significance of artistic choices. Effortless image manipulation that stems from physically realistic image representations is highly valuable in many fields. The proposed techniques will enable new application scenarios of deep learning in computational photography as well as the ability to extend existing computer vision datasets for more effective data-driven methods. Our interactive editing tools will be made publicly available to allow independent artists and low-budget films to achieve their artistic vision more effectively. Making advanced scene editing that is currently only possible with large production crews and budgets available to the large film production community in Canada has the potential to kick off impactful innovations in the art form of movie making and storytelling.
Program
Discovery Grants Program - Individual
Recipient
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2020-05375
Name nom
Aksoy, Yagiz
Department département
Computing Science, School of - Computing Science, School of
Organization id
5
Institution établissement
Simon Fraser University
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2020
Award amount
29000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2700
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information technology
Research subject fr
Technologie de l'information
Installment
inst:3
Partie
1
Nb partie
5
Application title
Image Analysis for Realistic Scene Manipulation
Application summary
There are many photographs or scenes in real life that can induce strong cognitive responses in humans such as emotions like awe or fear, or specific moods such as calmness or chaos. This intricate connection between visual stimuli and cognitive processes makes photography and movie making important venues for artistic expression. This connection also makes image analysis and synthesis that aims at understanding induced emotions and artistic expression a promising research direction to start conceptualizing such high-level cognitive processes in the context of artificial intelligence. The capabilities of current computer vision and computer graphics systems are not at a point that allows us to start looking at which aspects of a scene make it emotionally significant. The main goal of this 5-year research plan is to develop image analysis and manipulation methods that will enable exploring the cognitive effects of artistic choices in film production and photography. Film production and photography are selected as the main application targets because of the decades of traditions developed to effectively convey ideas and emotions visually. This established visual language provides a convenient starting point for bridging the cognitive effects of artistic choices with computational analysis of imagery. The proposed research are planned as three thrusts. The first two thrusts aim to generate scene representations that isolate specific aspects that contribute to the appearance of imagery: (i) Low-level, physical cues including illumination and material appearance, and (ii) High-level, structural cues including semantics and depth. Both research directions feature innovative uses of machine learning techniques to extend their applicability in computational photography, aiming realistic and quick manipulation of scenes through generic image representations as well as interactive tools. The third research thrust builds on the first two to apply scene manipulation methods to movie post-production to analyze the aspects of a scene that contribute to the significance of artistic choices. Effortless image manipulation that stems from physically realistic image representations is highly valuable in many fields. The proposed techniques will enable new application scenarios of deep learning in computational photography as well as the ability to extend existing computer vision datasets for more effective data-driven methods. Our interactive editing tools will be made publicly available to allow independent artists and low-budget films to achieve their artistic vision more effectively. Making advanced scene editing that is currently only possible with large production crews and budgets available to the large film production community in Canada has the potential to kick off impactful innovations in the art form of movie making and storytelling.