Grant: Discovery Grants Program - Individual
Simon Fraser University
$48,000.00
Thinking Fast and Slow with Large Language Models
EF 2024-25
Résumé
Transformer neural networks were initially proposed in natural language processing (NLP) as a model for machine translation. It has since expanded its impact to all aspects of NLP and other fields in AI. The success of Transformers can be viewed in different ways: as an effective way to model long contextual information along with a GPU friendly computation model. It is instructive for future research to reflect upon the genesis of the core ideas of attention and self-attention that led to the ubiquitous Transformer model. The main idea behind attention was to decompose and compose the various parts of the source sentence that contribute to the language model decoder producing the target language tokens.
As we consider new advances beyond Transformers, the focus of this research will be on specific sets of tasks that are currently out of reach of large language models (LLMs) and what kinds of thoughts about language are needed to identify the next generation of models that can improve on current Transformer based large language models. Such tasks include complex ones like decipherment or simultaneous translation (aka interpreter-like translation) or long story generation to simple tasks like string reversal or Caesar cipher decipherment (both of which are challenging for LLMs like GPT4). Both fine-tuning methods and in-context learning methods will be explored.
Further research into LLMs that can handle longer contexts and the retrieval of information from prior data or prompts can benefit natural language processing (NLP) but also many other areas of machine learning research. Focusing on tasks that require long contexts, memory retrieval and decipherment methods can also make the search for new models grounded in improvements for these tasks. Improvements on multiple tasks simultaneously with the same model is an important goal and it is a goal taken seriously in this proposal.
We situate this research program within the framework of large causal language models tuned or prompted to provide high accuracy in a variety of NLP tasks. Our goal is to exploit the constraints we observe in natural language to enable computationally efficient and expressive machine learning models. We motivate these models using some specific NLP tasks. In particular, this research proposal focuses on Compositional Adapters, Outer-Product based Associative Memory, k-Nearest Neighbour memory networks, a novel concept of Entity-focused and Centering-based Transformers and Multilingual large language models.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
British Columbia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2024-06248
Name nom
Sarkar, Anoop
Department département
Computing Science
Organization id
5
Institution établissement
Simon Fraser University
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
48000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2801
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Natural language and speech understanding
Research subject fr
Langage naturel et reconnaissance de la parole
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Thinking Fast and Slow with Large Language Models
Keywords
natural language processing; deep learning; transformer models; recurrent neural networks; k-nearest neighbour models; holographic memory neural networks; machine translation; computational decipherment; attention networks
Application summary
Transformer neural networks were initially proposed in natural language processing (NLP) as a model for machine translation. It has since expanded its impact to all aspects of NLP and other fields in AI. The success of Transformers can be viewed in different ways: as an effective way to model long contextual information along with a GPU friendly computation model. It is instructive for future research to reflect upon the genesis of the core ideas of attention and self-attention that led to the ubiquitous Transformer model. The main idea behind attention was to decompose and compose the various parts of the source sentence that contribute to the language model decoder producing the target language tokens.
As we consider new advances beyond Transformers, the focus of this research will be on specific sets of tasks that are currently out of reach of large language models (LLMs) and what kinds of thoughts about language are needed to identify the next generation of models that can improve on current Transformer based large language models. Such tasks include complex ones like decipherment or simultaneous translation (aka interpreter-like translation) or long story generation to simple tasks like string reversal or Caesar cipher decipherment (both of which are challenging for LLMs like GPT4). Both fine-tuning methods and in-context learning methods will be explored.
Further research into LLMs that can handle longer contexts and the retrieval of information from prior data or prompts can benefit natural language processing (NLP) but also many other areas of machine learning research. Focusing on tasks that require long contexts, memory retrieval and decipherment methods can also make the search for new models grounded in improvements for these tasks. Improvements on multiple tasks simultaneously with the same model is an important goal and it is a goal taken seriously in this proposal.
We situate this research program within the framework of large causal language models tuned or prompted to provide high accuracy in a variety of NLP tasks. Our goal is to exploit the constraints we observe in natural language to enable computationally efficient and expressive machine learning models. We motivate these models using some specific NLP tasks. In particular, this research proposal focuses on Compositional Adapters, Outer-Product based Associative Memory, k-Nearest Neighbour memory networks, a novel concept of Entity-focused and Centering-based Transformers and Multilingual large language models.