Grant: Discovery Grants Program - Individual
Université de Montréal
$76,000.00
Deep Learning of Representations
FY 2015-16
Summary
Artificial intelligence requires computers to have knowledge of the world around us, knowledge they can use to answer questions and take decisions. Knowledge can be either hard-wired or learned from data and usually both sources of information are used. Machine learning algorithms endow computers with the ability to acquire knowledge from examples. In the age of big data, there is a great potential in the ability to tap this immense source of information. Representation learning algorithms are machine learning algorithms which also involve the learning of representations. The data (such as an image or a document) seen by a computer must be represented, and better representations make it easier to capture the statistical dependencies present and to learn to answer questions. Whereas representations are traditionally crafted by hand and engineered, recent years have shown that they can be learned and this can have drastic impact on the predictive ability of the learners. Deep learning, or learning of deep representations, is the subject of this proposal. It involves learning multiple levels of representation, corresponding to different levels of abstractions. In the past five years there has been a tremendous impact of research on deep learning, both at an academic level and with industrial breakthroughs. This proposal asks what are the current qualitative deficiencies of the current deep learning algorithms and proposes ideas for how to move beyond these limitations and towards AI. The main limitations that are discussed here regard the ways to scale up, because current models still have the size and capabilities of insect brains and do not even reach the perceptual abilities of rodents. Although faster and larger computers will matter, it is hypothesized that such advances in hardware will not suffice. For example, parallelization of the current learning procedures is not trivial and yet this is where computing power continues to grow. There also seem to be numerical optimization challenges that cannot be solved by simply pouring ten times more compute power, such as real or apparent local minima arising in the learning dynamics, for which it is not sufficient to restart thousands of times the optimization procedures: the result remains poor unless a different initialization procedure is used. There are also fundamental challenges regarding unsupervised learning: whereas most empirical progress of recent years has been with supervised learning (where humans have labeled data and told the computer what to answer to many questions), the greatest promise for future breakthroughs may come from unsupervised learning procedures (coupled with supervised learning). The main challenge there arises out of the intractability of the normalization constant involved in probabilistic models with many random variables and many high-probability modes separated by vast regions of low probability, a situation that is the most common in artificial intelligence applications. Although Monte-Carlo Markov chain methods are the most general solutions to this problem, new and maybe radically different ways of addressing the problem may be required and will be explored. Finally, this proposal considers the very basic question of what is a good representation, and proposes to view the different strategies we have to obtain good representations as priors that can help the learner in the very important task of disentangling the underlying factors of variation, i.e., of figuring out and separating from each other the factors that explain the data. As in past research of the applicant, these algorithmic explorations will be evaluated on challenging applications involving real data, from images and video to natural language text, or combining multiple modalities.
Program
Discovery Grants Program - Individual
Recipient
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2014-05917
Name nom
Bengio, Yoshua
Department département
Informatique et recherche opérationnelle
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2015
Competition year année de concours
2014
Award amount
76000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2709
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Other computing methods
Research subject fr
Autres méthodes informatiques
Partie
2
Nb partie
5
Application title
Deep Learning of Representations
Keyword
NSERC
Application summary
Artificial intelligence requires computers to have knowledge of the world around us, knowledge they can use to answer questions and take decisions. Knowledge can be either hard-wired or learned from data and usually both sources of information are used. Machine learning algorithms endow computers with the ability to acquire knowledge from examples. In the age of big data, there is a great potential in the ability to tap this immense source of information. Representation learning algorithms are machine learning algorithms which also involve the learning of representations. The data (such as an image or a document) seen by a computer must be represented, and better representations make it easier to capture the statistical dependencies present and to learn to answer questions. Whereas representations are traditionally crafted by hand and engineered, recent years have shown that they can be learned and this can have drastic impact on the predictive ability of the learners. Deep learning, or learning of deep representations, is the subject of this proposal. It involves learning multiple levels of representation, corresponding to different levels of abstractions. In the past five years there has been a tremendous impact of research on deep learning, both at an academic level and with industrial breakthroughs. This proposal asks what are the current qualitative deficiencies of the current deep learning algorithms and proposes ideas for how to move beyond these limitations and towards AI. The main limitations that are discussed here regard the ways to scale up, because current models still have the size and capabilities of insect brains and do not even reach the perceptual abilities of rodents. Although faster and larger computers will matter, it is hypothesized that such advances in hardware will not suffice. For example, parallelization of the current learning procedures is not trivial and yet this is where computing power continues to grow. There also seem to be numerical optimization challenges that cannot be solved by simply pouring ten times more compute power, such as real or apparent local minima arising in the learning dynamics, for which it is not sufficient to restart thousands of times the optimization procedures: the result remains poor unless a different initialization procedure is used. There are also fundamental challenges regarding unsupervised learning: whereas most empirical progress of recent years has been with supervised learning (where humans have labeled data and told the computer what to answer to many questions), the greatest promise for future breakthroughs may come from unsupervised learning procedures (coupled with supervised learning). The main challenge there arises out of the intractability of the normalization constant involved in probabilistic models with many random variables and many high-probability modes separated by vast regions of low probability, a situation that is the most common in artificial intelligence applications. Although Monte-Carlo Markov chain methods are the most general solutions to this problem, new and maybe radically different ways of addressing the problem may be required and will be explored. Finally, this proposal considers the very basic question of what is a good representation, and proposes to view the different strategies we have to obtain good representations as priors that can help the learner in the very important task of disentangling the underlying factors of variation, i.e., of figuring out and separating from each other the factors that explain the data. As in past research of the applicant, these algorithmic explorations will be evaluated on challenging applications involving real data, from images and video to natural language text, or combining multiple modalities.