Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Alberta

$41,000.00

Improving language models, inspired by the brain

EF 2022-23

Résumé
Language models (LMs) have markedly improved over the past 10 years, due in large part to the deep learning neural networks models they now use. These learned models can be fed a text snippet and generate words to meaningfully finish the thought. Current models can generate text that is so human-like that it can be difficult to differentiate from text written by people. These models have written segments of articles for the New Yorker and can answer some types of reading comprehension questions with accuracy at or above the accuracy of the average person. In fact, these models are so accurate, they are sometimes able to generate novel text that is better than what a human could generate. Remarkably, one such model generated the previous sentence. However, as convincing as current LMs are, they remain imperfect and sub-optimal in certain scenarios. For example, LMs perform poorly when tasks require inference, generalization, or commonsense knowledge. They also require huge amounts of data to train. Contrast this to young children who require only a fraction of the data our models do, and who are master generalizers and inference-makers. Where is that disconnect? Why are children so masterful with language in ways that our current models simply cannot replicate? Through this proposed research plan, I will study patterns in language development and usage, and use them as inspiration for improving computational LMs. I will study language acquisition and the onset of semantic composition in infants, the prioritization of different kinds of information in children, and the meta-cognitive processes governing language usage in fluent adults. Each of these points in the lifespan bring into focus different aspects of human language learning and usage, and novel ways of understanding and improving computational language learning. The impact of this work is multi-faceted. Firstly, these findings will allow me to create more accurate but also more efficient language models. More efficient LMs will train faster, with less data. This will improve multiple downstream usages of language models like dialogue generation and sentiment classification. Because more efficient models of language also require fewer resources to train, my work will impact the carbon footprint of the current natural language processing community. Secondly, my proposed research will contribute to our understanding of how the brain acquires and develops language, feed back into the basic neuroscience of learning, and help inform literacy education research. The impacts of my research will be as interdisciplinary as the research program itself.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2022-03580
Name nom
Fyshe, Alona
Department département
Computing Science - Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2022
Award amount
41000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Installment
inst:1
Partie
1
Nb partie
5
Application title
Improving language models, inspired by the brain
Application summary
Language models (LMs) have markedly improved over the past 10 years, due in large part to the deep learning neural networks models they now use. These learned models can be fed a text snippet and generate words to meaningfully finish the thought. Current models can generate text that is so human-like that it can be difficult to differentiate from text written by people. These models have written segments of articles for the New Yorker and can answer some types of reading comprehension questions with accuracy at or above the accuracy of the average person. In fact, these models are so accurate, they are sometimes able to generate novel text that is better than what a human could generate. Remarkably, one such model generated the previous sentence. However, as convincing as current LMs are, they remain imperfect and sub-optimal in certain scenarios. For example, LMs perform poorly when tasks require inference, generalization, or commonsense knowledge. They also require huge amounts of data to train. Contrast this to young children who require only a fraction of the data our models do, and who are master generalizers and inference-makers. Where is that disconnect? Why are children so masterful with language in ways that our current models simply cannot replicate? Through this proposed research plan, I will study patterns in language development and usage, and use them as inspiration for improving computational LMs. I will study language acquisition and the onset of semantic composition in infants, the prioritization of different kinds of information in children, and the meta-cognitive processes governing language usage in fluent adults. Each of these points in the lifespan bring into focus different aspects of human language learning and usage, and novel ways of understanding and improving computational language learning. The impact of this work is multi-faceted. Firstly, these findings will allow me to create more accurate but also more efficient language models. More efficient LMs will train faster, with less data. This will improve multiple downstream usages of language models like dialogue generation and sentiment classification. Because more efficient models of language also require fewer resources to train, my work will impact the carbon footprint of the current natural language processing community. Secondly, my proposed research will contribute to our understanding of how the brain acquires and develops language, feed back into the basic neuroscience of learning, and help inform literacy education research. The impacts of my research will be as interdisciplinary as the research program itself.