Grant: Discovery Grants Program - Individual
University of British Columbia
$55,000.00
Hyper-fast hyper-parameter tuning for the next generation of machine learning
EF 2022-23
Résumé
Machine learning (ML) uses large amounts data to set the parameters of a model, and is having success in a growing number of applications from speech recognition to computer vision to language translation. But ML has a problem with "hyper-parameters", the variables affecting the learning algorithm and the structure of the model. We typically still need to spend enormous amounts of time tuning hyper-parameters in order to obtain good performance. It was recently highlighted that the cost of tuning current ML language models has a comparable carbon output to five cars throughout their lifetime. This situation will only get worse as the next generation of models will have far more hyper-parameters. We will not be able to solve many important problems until we address this issue. The long-term goal of this research program is to develop algorithms that can train ML models on enormous datasets in a short amount of time. During the last 6 years we have focused on developing algorithms that converge faster, but we now need to turn to the pressing issue of dealing with the hyper-parameters. The short-term goal of this project is to focus on addressing the issues associated with the two typical sources of hyper-parameters: (A) The algorithm used to do the learning typically has hyper-parameters, such as the learning rate. (B) The model that is being used typically has hyper-parameters, such as the depth of a deep learning model. We have already made progress on (A). In 2019 we gave the first method that automatically tunes one of the most important hyper-parameters, the learning rate, during training. This method is guaranteed to perform at least as well as the best fixed learning rate for modern "over-parameterized" models. We plan to develop algorithms that are insensitive to other learning hyper-parameters and that do not require the over-parameterized assumption. My lab is uniquely positioned to address problem (B). Current strategies for tuning the parameters of network architectures tend to use discrete parameterizations of the hyper-parameters. In work 10 years ago I showed a variety of ways to use continuous parameterizations to yield high-quality approximate solutions to learning problems that involve searching over graph and hyper-graph structures. These continuous relaxations gave enormous speedups over previous approaches based on discrete parameterizations, and we will develop new methods like these to address tuning model hyper-parameters. There is a huge potential impact for the proposed research, with potential applications ranging from medicine to scientific discovery to self-driving cars. In ML we want to build algorithms and models that work across many applications (we want to "build a better hammer" that can be used for many tasks). Thus, breakthroughs on the algorithms underlying ML models immediately impact many applications that use ML (or will use it in the future).
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
British Columbia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2022-03669
Name nom
Schmidt, Mark
Department département
Computer Science - Computer Science
Organization id
2
Institution établissement
University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2022
Award amount
55000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Installment
inst:1
Partie
1
Nb partie
5
Application title
Hyper-fast hyper-parameter tuning for the next generation of machine learning
Application summary
Machine learning (ML) uses large amounts data to set the parameters of a model, and is having success in a growing number of applications from speech recognition to computer vision to language translation. But ML has a problem with "hyper-parameters", the variables affecting the learning algorithm and the structure of the model. We typically still need to spend enormous amounts of time tuning hyper-parameters in order to obtain good performance. It was recently highlighted that the cost of tuning current ML language models has a comparable carbon output to five cars throughout their lifetime. This situation will only get worse as the next generation of models will have far more hyper-parameters. We will not be able to solve many important problems until we address this issue. The long-term goal of this research program is to develop algorithms that can train ML models on enormous datasets in a short amount of time. During the last 6 years we have focused on developing algorithms that converge faster, but we now need to turn to the pressing issue of dealing with the hyper-parameters. The short-term goal of this project is to focus on addressing the issues associated with the two typical sources of hyper-parameters: (A) The algorithm used to do the learning typically has hyper-parameters, such as the learning rate. (B) The model that is being used typically has hyper-parameters, such as the depth of a deep learning model. We have already made progress on (A). In 2019 we gave the first method that automatically tunes one of the most important hyper-parameters, the learning rate, during training. This method is guaranteed to perform at least as well as the best fixed learning rate for modern "over-parameterized" models. We plan to develop algorithms that are insensitive to other learning hyper-parameters and that do not require the over-parameterized assumption. My lab is uniquely positioned to address problem (B). Current strategies for tuning the parameters of network architectures tend to use discrete parameterizations of the hyper-parameters. In work 10 years ago I showed a variety of ways to use continuous parameterizations to yield high-quality approximate solutions to learning problems that involve searching over graph and hyper-graph structures. These continuous relaxations gave enormous speedups over previous approaches based on discrete parameterizations, and we will develop new methods like these to address tuning model hyper-parameters. There is a huge potential impact for the proposed research, with potential applications ranging from medicine to scientific discovery to self-driving cars. In ML we want to build algorithms and models that work across many applications (we want to "build a better hammer" that can be used for many tasks). Thus, breakthroughs on the algorithms underlying ML models immediately impact many applications that use ML (or will use it in the future).