Grant: Discovery Grants Program - Individual
University of Guelph
$26,000.00
Random Matrix Limit Theorems for Deep Neural Networks
FY 2021-22
Summary
Recent advances in deep neural networks (DNNs) have had a tremendous impact on the modern world. However, the theoretical understanding of these systems is still in its infancy. As a matter of course, the research in this area has been empirically driven and computationally focused rather than emphasizing mathematically rigorous results. There are many open theoretical questions which have been uncovered by empirical work that are now ripe for mathematical analysis. I propose a research program that will develop and apply tools from theoretical probability, specifically random matrix theory, to gain a better understanding of the theory of DNNs and other machine learning systems. I will focus on developing new limit theorems which describe behavior when the number of parameters and/or data becomes very large. These results will help us understand how DNNs work and help us design more effective systems in the future. My objectives of the research program include: 1. The neural tangent kernel: A random matrix that explains the behavior of large networks The neural tangent kernel (NTK) is a recently discovered non-random asymptotic object that explains the behavior of DNNs of fixed depth in the infinite width limit, when the number of neurons in each hidden layer tends to infinity. When applied to random data, the NTK gives a random matrix whose dimensions are the number of given data points. Analysis of this random matrix can explain how DNNs behave during training and can be used to understand the generalization error in deep neural networks. I propose to study this model using random matrix theory. 2. Applied free probability: Advanced tools for random matrix analysis The theory of free probability was originally developed in connection to pure problems in the field of operator algebras. More recently however, methods from free probability and its extensions have emerged as powerful tools for computing asymptotic features of complicated random matrix models. One application is to use free probability to compute the limiting spectrum of large random matrix models connected to DNNs. I also plan to investigate the use of operator valued free probability, a powerful extension of free probability, to study block random matrices related to DNNs. 3. Kardar-Parisi-Zhang (KPZ) universality: Fluctuations of random matrix eigenvalues The KPZ universality class is a collection of stochastic systems, including examples from stochastic PDEs and interacting particle systems, which all share the same type of universal asymptotic random behavior. An important application is the behaviour of the largest eigenvalues in many random matrix models. (As opposed to the bulk behavior of the spectrum captured by other random matrix tools). I plan to apply ideas from KPZ to random matrix problems coming from DNNs and other statistical learning models to analyze the evolution of the largest eigenvalues in these problems.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2021-02533
Name nom
Nica, Mihai
Department département
Mathematics and Statistics
Organization id
24
Institution établissement
University of Guelph
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2021
Competition year année de concours
2021
Award amount
26000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1508
Committee name en
Mathematics and Statistics
Committee name fr
Mathématiques et statistique
Area of application code
1205
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Mathematical sciences
Area of application fr
Sciences mathématiques
Research subject code
3006
Research subject group en
Statistics and probability
Research subject group fr
Statistique et probabilité
Research subject en
Probability theory
Research subject fr
Théorie de la probabilité
Installment
inst:1
Partie
1
Nb partie
5
Application title
Random Matrix Limit Theorems for Deep Neural Networks
Keyword
NSERC
Application summary
Recent advances in deep neural networks (DNNs) have had a tremendous impact on the modern world. However, the theoretical understanding of these systems is still in its infancy. As a matter of course, the research in this area has been empirically driven and computationally focused rather than emphasizing mathematically rigorous results. There are many open theoretical questions which have been uncovered by empirical work that are now ripe for mathematical analysis. I propose a research program that will develop and apply tools from theoretical probability, specifically random matrix theory, to gain a better understanding of the theory of DNNs and other machine learning systems. I will focus on developing new limit theorems which describe behavior when the number of parameters and/or data becomes very large. These results will help us understand how DNNs work and help us design more effective systems in the future. My objectives of the research program include: 1. The neural tangent kernel: A random matrix that explains the behavior of large networks The neural tangent kernel (NTK) is a recently discovered non-random asymptotic object that explains the behavior of DNNs of fixed depth in the infinite width limit, when the number of neurons in each hidden layer tends to infinity. When applied to random data, the NTK gives a random matrix whose dimensions are the number of given data points. Analysis of this random matrix can explain how DNNs behave during training and can be used to understand the generalization error in deep neural networks. I propose to study this model using random matrix theory. 2. Applied free probability: Advanced tools for random matrix analysis The theory of free probability was originally developed in connection to pure problems in the field of operator algebras. More recently however, methods from free probability and its extensions have emerged as powerful tools for computing asymptotic features of complicated random matrix models. One application is to use free probability to compute the limiting spectrum of large random matrix models connected to DNNs. I also plan to investigate the use of operator valued free probability, a powerful extension of free probability, to study block random matrices related to DNNs. 3. Kardar-Parisi-Zhang (KPZ) universality: Fluctuations of random matrix eigenvalues The KPZ universality class is a collection of stochastic systems, including examples from stochastic PDEs and interacting particle systems, which all share the same type of universal asymptotic random behavior. An important application is the behaviour of the largest eigenvalues in many random matrix models. (As opposed to the bulk behavior of the spectrum captured by other random matrix tools). I plan to apply ideas from KPZ to random matrix problems coming from DNNs and other statistical learning models to analyze the evolution of the largest eigenvalues in these problems.