Grant: Discovery Grants Program - Individual
The University of British Columbia
$55,000.00
Scalable and Efficient Multimodal Visual Understanding and Generation
FY 2024-25
Summary
The main goal of the proposed research program is to push the boundaries of visual cognition and generative modeling. Computer vision and AI, as the broader fields encompassing these sub-topics, have experienced significant growth and unprecedented advances in recent years. However, despite successes, both academic and industrial (e.g., ChatGPT, DALL-E), a number of core challenges remain. Specifically, most effective forms and structure of predictive tasks for training large-scale models is still open for debate. Further, most existing models lack ability to synthesize knowledge from data effectively, being trained with many orders of human lifetimes of experience, while still exhibiting sub-human levels of intelligence. Finally, their ability to integrate and leverage data from various modalities (lingual, auditory, visual) is limited, particularly when dealing with long-form content (e.g., videos), which requires abstract memory mechanisms.
Proposed program is designed to focus on these fundamental technical limitations as well as questions surround trust in visually-capable AI agents and systems. Specifically, the program focuses on developing novel modularized multi-modal architectures with rich learned representations and capabilities. As part of this effort, we plan to (1) develop more effective model (pre-)training strategies that take into account uncertainty in the predictive distributions, as well as lead to more explicitly compositional and hierarchical representations; (2) develop mechanism for multi-modal 3D grounded inferences, commonsense reasoning and working memory -- capable of processing long-form video content in a streaming fashion; (3) develop new generative (diffusion) models capable of inferring complex structured representations (e.g., object boxes / segments and scene graphs) from images; and (4) develop mechanism that allow large-scale foundational models, to be adopted effectively using variable granularity annotations and limited data. Finally, we will also focus on (5) quantifying biases and accidental correlations in existing multi-modal architectures and develop tools and datasets to improve interpretability, and reduce biases of such models.
The program focuses on addressing fundamental limitations and advancing understanding of current visual AI models and systems. As such, it has a potential to have a significant and broad impact on this exceptionally fast growing field, that is poised to be transformational for entire sectors of economy (enabling new AI-enhanced creative & curation tools, autonomous driving and even precision agriculture capabilities).
This program will train 15 HQP (highly qualified personnel), including 6 PhD students, 4 MSc/PhD Track students, and 5 undergraduates, in computer vision and machine learning. The skills developed are transferable and include fundamental AI and machine learning skills, as well as design of visual automation systems and agents.
Program
Discovery Grants Program - Individual
Recipient
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2024-06085
Name nom
Sigal, Leonid
Department département
Computer Science
Organization id
2
Institution établissement
The University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
55000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Scalable and Efficient Multimodal Visual Understanding and Generation
Keywords
computer vision; machine learning; neural networks; representation and deep learning; multi-modal (visual+language) learning; generative modeling; image and video understanding; recognition, detection and segmentation; weakly- and semi-supervised learning; structured learning
Application summary
The main goal of the proposed research program is to push the boundaries of visual cognition and generative modeling. Computer vision and AI, as the broader fields encompassing these sub-topics, have experienced significant growth and unprecedented advances in recent years. However, despite successes, both academic and industrial (e.g., ChatGPT, DALL-E), a number of core challenges remain. Specifically, most effective forms and structure of predictive tasks for training large-scale models is still open for debate. Further, most existing models lack ability to synthesize knowledge from data effectively, being trained with many orders of human lifetimes of experience, while still exhibiting sub-human levels of intelligence. Finally, their ability to integrate and leverage data from various modalities (lingual, auditory, visual) is limited, particularly when dealing with long-form content (e.g., videos), which requires abstract memory mechanisms.
Proposed program is designed to focus on these fundamental technical limitations as well as questions surround trust in visually-capable AI agents and systems. Specifically, the program focuses on developing novel modularized multi-modal architectures with rich learned representations and capabilities. As part of this effort, we plan to (1) develop more effective model (pre-)training strategies that take into account uncertainty in the predictive distributions, as well as lead to more explicitly compositional and hierarchical representations; (2) develop mechanism for multi-modal 3D grounded inferences, commonsense reasoning and working memory -- capable of processing long-form video content in a streaming fashion; (3) develop new generative (diffusion) models capable of inferring complex structured representations (e.g., object boxes / segments and scene graphs) from images; and (4) develop mechanism that allow large-scale foundational models, to be adopted effectively using variable granularity annotations and limited data. Finally, we will also focus on (5) quantifying biases and accidental correlations in existing multi-modal architectures and develop tools and datasets to improve interpretability, and reduce biases of such models.
The program focuses on addressing fundamental limitations and advancing understanding of current visual AI models and systems. As such, it has a potential to have a significant and broad impact on this exceptionally fast growing field, that is poised to be transformational for entire sectors of economy (enabling new AI-enhanced creative & curation tools, autonomous driving and even precision agriculture capabilities).
This program will train 15 HQP (highly qualified personnel), including 6 PhD students, 4 MSc/PhD Track students, and 5 undergraduates, in computer vision and machine learning. The skills developed are transferable and include fundamental AI and machine learning skills, as well as design of visual automation systems and agents.