Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Alberta

$81,000.00

Foundation Models and Decision Making

EF 2024-25

Résumé
This research program will leverage concepts from decision making to address shortcomings in foundation model training and use. In particular, I will investigate how decision making and reinforcement learning techniques can be used to improve each of the core phases of language model development: (1) pre-training, where the outputs that naturally follow a particular sequence are captured, (2) fine-tuning, where the outputs that should be produced in response to an input are captured, and (3) prompting, which exploits the higher-order ability of language models to improve response quality via chained calls and tool use. For (1) I will leverage recent progress on "algorithm distillation" in reinforcement learning to exploit sequences that demonstrate implicit improvement processes (such as dialogues that successfully resolve a user problem) to accelerate the ability of a large model to reach a common understanding with end users. I will also explore how naturally-occurring reasoning chains can be used to integrate latent chain-of-thought prediction in the next-token prediction problem. For (2) I will exploit recent advances in off-line reinforcement learning to improve language model fine-tuning. The advent of parameter efficient fine-tuning techniques, such as low-rank adaptation, has made this research accessible within available resources. I will also investigate alternative matrix approximations that produce superior results than the standard low-rank approach. Finally, for (3) I will investigate the ability of large language models to acquire algorithmic behavior under different scenarios with increasing computational demand. In particular, I will consider what can be achieved by a single call to a language model, followed by multiple chained calls, then by leveraging external memory, and finally by exploiting search. To support this investigation, I will develop theory to explain how the higher-order prediction ability of large models is achieved, by exploiting properties of internal representations that provably enable few-shot learning. The search scenario poses a harder problem than traditional exploration in decision making, since the search space is self-generated and prone to hallucination, but evidence suggests that accuracy can still be improved by exploring alternative output paths. The proposed research program seeks to improve language models to better serve user intent. That is, rather than an autonomous agent, a language model can be viewed as a new type of central processor that operates with human understandable instructions. If successful, this research will enable a new generation of language models that can more efficiently align to user intent and reason more accurately, accelerating the development of prompt-based information products and services.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2024-05011
Name nom
Schuurmans, Dale
Department département
Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
81000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Foundation Models and Decision Making
Keywords
reinforcement learning; large language models; sequential decision making; contextual bandits; planning; reasoning; prompting
Application summary
This research program will leverage concepts from decision making to address shortcomings in foundation model training and use. In particular, I will investigate how decision making and reinforcement learning techniques can be used to improve each of the core phases of language model development: (1) pre-training, where the outputs that naturally follow a particular sequence are captured, (2) fine-tuning, where the outputs that should be produced in response to an input are captured, and (3) prompting, which exploits the higher-order ability of language models to improve response quality via chained calls and tool use. For (1) I will leverage recent progress on "algorithm distillation" in reinforcement learning to exploit sequences that demonstrate implicit improvement processes (such as dialogues that successfully resolve a user problem) to accelerate the ability of a large model to reach a common understanding with end users. I will also explore how naturally-occurring reasoning chains can be used to integrate latent chain-of-thought prediction in the next-token prediction problem. For (2) I will exploit recent advances in off-line reinforcement learning to improve language model fine-tuning. The advent of parameter efficient fine-tuning techniques, such as low-rank adaptation, has made this research accessible within available resources. I will also investigate alternative matrix approximations that produce superior results than the standard low-rank approach. Finally, for (3) I will investigate the ability of large language models to acquire algorithmic behavior under different scenarios with increasing computational demand. In particular, I will consider what can be achieved by a single call to a language model, followed by multiple chained calls, then by leveraging external memory, and finally by exploiting search. To support this investigation, I will develop theory to explain how the higher-order prediction ability of large models is achieved, by exploiting properties of internal representations that provably enable few-shot learning. The search scenario poses a harder problem than traditional exploration in decision making, since the search space is self-generated and prone to hallucination, but evidence suggests that accuracy can still be improved by exploring alternative output paths. The proposed research program seeks to improve language models to better serve user intent. That is, rather than an autonomous agent, a language model can be viewed as a new type of central processor that operates with human understandable instructions. If successful, this research will enable a new generation of language models that can more efficiently align to user intent and reason more accurately, accelerating the development of prompt-based information products and services.