Grant: Discovery Grants Program - Individual
Université de Montréal
$71,000.00
Promoting systematic generalization in vision-and-language foundation models
FY 2024-25
Summary
The field of AI is undergoing a revolution underpinned by the tremendous progress of foundation models: large models trained on vast datasets. Foundation models can be used to solve a wide range of tasks at state-of-the-art level performance. However, there is a pattern emerging in the relative performance of these foundational models. Consider a recent study conducted by members of my group. We looked at the simple task of arithmetic: eg. adding one 3 digit number to another 3 digit number which the model can do with ease However the model's performance diminishes as the number of digits in the sum increases, despite the same underlying algorithm of addition being applicable. We found that if we provide the model with the algorithm in the form of step-by-step worked examples, then the performance on largeer sums increases dramatically. With the step-by-step examples, the models show a capacity to learning the algorithm.
Fodor and Pylyshyn (1988) introduced the term "systematic generalization" as defining this form of generalization. It is the sort of generalization that humans do routinely and that neural networks have typically failed to demonstrate. Yet, as described above, by decomposing the task into steps and conveying those steps to the LLM, we were able to demonstrate systematic generalization. This finding points to a general strategy for achieving systematic generalization: decompose the task into a series of simpler tasks and compose the result.
While our study of arithmetic addition points toward a path to systematic generalization, it also required that a solution strategy be given in the form of worked-out examples. This is a general limitation of existing task decomposition strategies. They are, for the most part, limited to tasks decompositions specified by a human operator. In the general case, if this is to a practical path toward systematic generalization, we will require that the model itself is able to provide the task decomposition without task specific supervision. In this proposal, I describe a research agenda focussed on developing learning strategies that use foundation models to support systematic generalization via task decomposition. In particular my goal is to enable agents to discover and develop their own novel task decompositions. Our focus will be on multimodal vision-language foundation models. that take images and/or text as input and generate text as output and thus are able to be applied to tasks including answering questions relating to an image.
Program
Discovery Grants Program - Individual
Recipient
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2024-06158
Name nom
Courville, Aaron
Department département
Department of Computer Science and Operations Research
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
71000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Promoting systematic generalization in vision-and-language foundation models
Keywords
machine learning; deep learning; neural networks; large language models (llms); foundational models; vision language models; reinforcement learning; systematic generalization; robustness and safety; planning and inference
Application summary
The field of AI is undergoing a revolution underpinned by the tremendous progress of foundation models: large models trained on vast datasets. Foundation models can be used to solve a wide range of tasks at state-of-the-art level performance. However, there is a pattern emerging in the relative performance of these foundational models. Consider a recent study conducted by members of my group. We looked at the simple task of arithmetic: eg. adding one 3 digit number to another 3 digit number which the model can do with ease However the model's performance diminishes as the number of digits in the sum increases, despite the same underlying algorithm of addition being applicable. We found that if we provide the model with the algorithm in the form of step-by-step worked examples, then the performance on largeer sums increases dramatically. With the step-by-step examples, the models show a capacity to learning the algorithm.
Fodor and Pylyshyn (1988) introduced the term "systematic generalization" as defining this form of generalization. It is the sort of generalization that humans do routinely and that neural networks have typically failed to demonstrate. Yet, as described above, by decomposing the task into steps and conveying those steps to the LLM, we were able to demonstrate systematic generalization. This finding points to a general strategy for achieving systematic generalization: decompose the task into a series of simpler tasks and compose the result.
While our study of arithmetic addition points toward a path to systematic generalization, it also required that a solution strategy be given in the form of worked-out examples. This is a general limitation of existing task decomposition strategies. They are, for the most part, limited to tasks decompositions specified by a human operator. In the general case, if this is to a practical path toward systematic generalization, we will require that the model itself is able to provide the task decomposition without task specific supervision. In this proposal, I describe a research agenda focussed on developing learning strategies that use foundation models to support systematic generalization via task decomposition. In particular my goal is to enable agents to discover and develop their own novel task decompositions. Our focus will be on multimodal vision-language foundation models. that take images and/or text as input and generate text as output and thus are able to be applied to tasks including answering questions relating to an image.