Grant: Discovery Grants Program - Individual
University of Toronto
$122,000.00
Unsupervised learning for perception
FY 2009-10
Summary
The backpropagation learning procedure that is often used to train artificial neural networks does not
work well for deep networks that have many layers of non-linear neurons between the input and the
output. My group has recently introduced a new approach in which deep networks are "pretrained" one
layer at a time using an efficient unsupervised algorithm that tries to model regularities in the
input data. After pretraining, the intermediate layers of a deep network already contain many useful
feature detectors and backpropagation is much better at fine-tuning a pretrained deep network than it
is at learning a deep network from scratch. In a recent Science paper we showed that this combination
of unsupervised pretraining followed by supervised fine-tuning outperforms all other learning methods
at a well-studied discrimination task and also for non-linear dimensionality reduction.
Using this new approach, we plan to apply deep neural networks to a variety of important discrimintion
tasks including object and face recognition. We will also use deep networks to model human motion,
speech, and video sequences. Using deep networks to extract low-dimensional codes from large, high
dimensional datasets, we will improve web-based recommendation systems. We will also develop deep
networks that can convert documents or images into binary addresses in such a way that semantically
similar documents have similar binary addresses. This allows extremely fast search for similar
documents or images. Finally, we will investigate ways in which these new learning algorithms might be
implemented in the cortex.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
9185-2007
Name nom
Hinton, Geoffrey
Department département
Computer Science
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2009
Competition year année de concours
2007
Award amount
122000
Program id
RGPIN
Program nane en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
331
Committee name en
Computing and Information Sciences - B
Committee name fr
Informatique - B
Area of application code
9999
Area of application group en
Not available
Area of application group fr
Non disponible
Area of application en
Not available
Area of application fr
Non disponible
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Partie
3
Nb partie
5
Application title
Unsupervised learning for perception
Keyword
NA
Application summary
The backpropagation learning procedure that is often used to train artificial neural networks does not
work well for deep networks that have many layers of non-linear neurons between the input and the
output. My group has recently introduced a new approach in which deep networks are "pretrained" one
layer at a time using an efficient unsupervised algorithm that tries to model regularities in the
input data. After pretraining, the intermediate layers of a deep network already contain many useful
feature detectors and backpropagation is much better at fine-tuning a pretrained deep network than it
is at learning a deep network from scratch. In a recent Science paper we showed that this combination
of unsupervised pretraining followed by supervised fine-tuning outperforms all other learning methods
at a well-studied discrimination task and also for non-linear dimensionality reduction.
Using this new approach, we plan to apply deep neural networks to a variety of important discrimintion
tasks including object and face recognition. We will also use deep networks to model human motion,
speech, and video sequences. Using deep networks to extract low-dimensional codes from large, high
dimensional datasets, we will improve web-based recommendation systems. We will also develop deep
networks that can convert documents or images into binary addresses in such a way that semantically
similar documents have similar binary addresses. This allows extremely fast search for similar
documents or images. Finally, we will investigate ways in which these new learning algorithms might be
implemented in the cortex.