Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Launch Supplement

Université de Montréal

$12,500.00

Advancing Multimodal Vision--Language Learning

FY 2024-25

Summary
The long-term objective of my research program is to develop artificial intelligence (AI) systems that can `see' (i.e. understand the contents of an image: who, what, where, doing what?) and `talk' (i.e. communicate the understanding to humans in free-form natural language). Such systems, commonly known as vision-language (VL) systems, have many useful societal applications - aiding the visually impaired in understanding their surroundings, teaching children through interactive demos, and interacting with in-home physical robots. Over the last decade, VL research has seen impressive progress. We can now automatically describe images in natural language, answer natural language questions about them, and even generate images guided by text prompts. Despite such tremendous progress, current VL research faces several challenges that limit the applicability of state-of-art (SOTA) VL systems. Even very large scale systems such as GPT-4V struggle with counting objects in images, identifying fine-grained differences between similar images, and often hallucinate visual facts. In this research program, I aim to focus on the following outstanding challenges over the next 5 years: Compositional reasoning: Recent works have shown that SOTA VL systems struggle with visio-linguistic compositional reasoning - the ability to tackle complex visio-linguistic queries by reasoning about its constituent parts and how they are composed together. In this objective, we develop novel methods to advance compositional reasoning capabilities of VL systems. Robust automatic evaluation: The current automatic metrics for evaluating the correctness of responses generated by VL systems are not robust enough. Moreover, as the VL systems are getting stronger, it is getting harder to understand their failure modes, which in turn is important to help guide future research towards improving these systems. In this objective, we will develop novel robust automatic metrics for correctness evaluation, and novel methods for automatic discovery of failure modes of VL systems. Parameter and data efficient learning: SOTA VL systems are based on the general idea of adapting strong pretrained unimodal vision-only and language-only models for multimodal vision-language tasks. Such adaptation requires training billions of parameters using billions of paired image-text data, making such research limited to a handful of large industrial research labs. In this objective, I aim to build novel methods for parameter and data efficient adaptation of unimodal models for multimodal tasks. Our proposed research program has the potential to transform the current VL systems from research prototypes to AI tools suitable for real-world applications! The proposed program will also train several HQP in computer vision, natural language processing, deep learning and multimodal AI in general. Such a diverse skill set is currently in high demand in both academia and industry.
Program
Discovery Launch Supplement
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
DGECR-2024-00012
Name nom
Agrawal, Aishwarya
Department département
Department of Computer Science and Operations Research
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
12500
Program id
DGECR
Program name en
Discovery Launch Supplement
Program name fr
Supplément Tremplin vers la découverte
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Advancing Multimodal Vision--Language Learning
Keywords
vision and language; multimodal ai; deep learning / representation learning; visual question answering (vqa); large language models (llms); vision encoder; image captioning; visio-linguistic compositional reasoning; evaluation metrics; parameter and data efficient learning
Application summary
The long-term objective of my research program is to develop artificial intelligence (AI) systems that can `see' (i.e. understand the contents of an image: who, what, where, doing what?) and `talk' (i.e. communicate the understanding to humans in free-form natural language). Such systems, commonly known as vision-language (VL) systems, have many useful societal applications - aiding the visually impaired in understanding their surroundings, teaching children through interactive demos, and interacting with in-home physical robots. Over the last decade, VL research has seen impressive progress. We can now automatically describe images in natural language, answer natural language questions about them, and even generate images guided by text prompts. Despite such tremendous progress, current VL research faces several challenges that limit the applicability of state-of-art (SOTA) VL systems. Even very large scale systems such as GPT-4V struggle with counting objects in images, identifying fine-grained differences between similar images, and often hallucinate visual facts. In this research program, I aim to focus on the following outstanding challenges over the next 5 years: Compositional reasoning: Recent works have shown that SOTA VL systems struggle with visio-linguistic compositional reasoning - the ability to tackle complex visio-linguistic queries by reasoning about its constituent parts and how they are composed together. In this objective, we develop novel methods to advance compositional reasoning capabilities of VL systems. Robust automatic evaluation: The current automatic metrics for evaluating the correctness of responses generated by VL systems are not robust enough. Moreover, as the VL systems are getting stronger, it is getting harder to understand their failure modes, which in turn is important to help guide future research towards improving these systems. In this objective, we will develop novel robust automatic metrics for correctness evaluation, and novel methods for automatic discovery of failure modes of VL systems. Parameter and data efficient learning: SOTA VL systems are based on the general idea of adapting strong pretrained unimodal vision-only and language-only models for multimodal vision-language tasks. Such adaptation requires training billions of parameters using billions of paired image-text data, making such research limited to a handful of large industrial research labs. In this objective, I aim to build novel methods for parameter and data efficient adaptation of unimodal models for multimodal tasks. Our proposed research program has the potential to transform the current VL systems from research prototypes to AI tools suitable for real-world applications! The proposed program will also train several HQP in computer vision, natural language processing, deep learning and multimodal AI in general. Such a diverse skill set is currently in high demand in both academia and industry.