Grant: Discovery Grants Program - Individual
The University of British Columbia
$74,000.00
Physics Based Architectures for Deep Neural Networks
EF 2023-24
Résumé
Deep Neural Networks (DNNs) have been the engine behind the recent deep learning revolution. Such networks have demonstrated remarkable capabilities in solving image classification problems, image segmentation problems and other computer vision tasks. Such tasks are common in applications such as face recognition, autonomous cars and medical image understanding. In recent years, modifications to these networks have been proposed, and in particular, residual networks have shown to improve over existing techniques, and yield even better results. More specifically, residual networks are easier to train, have been used with hundreds and even thousands of layers and can be made memory efficient. Encouraged from these results, a few researchers have attempted to explain the success of such networks by analyzing them as discrete dynamical systems, using theory from ordinary differential equations and optimal control. While initial results are interesting, there is still a large gap between the current technology of training and understanding DNNs, to other fields that use similar methodologies, such as time dependent optimal control, inverse problems and partial differential equations based optimization. Furthermore, DNNs do not have an over-arching continuous analog which makes them somewhat add-hoc and problem/data dependent. Understanding how to move networks between scales, changing the network parameters and understanding its properties are all difficult and in many cases, impossible. The goal of this work is to suggest and develop new ideas for the understanding of DNNs. In particular, we exploit the interpretation of neural networks as dynamical systems in order to propose physics based, or physically motivated networks. These networks represent physical processes such as nonlinear diffusion and nonlinear wave propagation similar to the partial differential equations that have been used for compressed sensing. Such networks can have favorable properties in term of their overall character and may be more amendable to fast optimization algorithms that utilize this structure such as multigrid methods. I will be training 3 Ph.D. students as well as 3-5 undergraduate research assistants in the fundamentals of machine learning and its application to problems in geoscience imaging.Overall, this research program will increase scientific knowledge in the computing of machine learning and will contribute to the growing number of companies working on AI in Canada. We expect students trained via the proposed research will pursue promising professional careers in either academia or industry.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
British Columbia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2019-04052
Name nom
Haber, Eldad
Department département
EOAS
Organization id
2
Institution établissement
The University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2019
Award amount
74000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1508
Committee name en
Mathematics and Statistics
Committee name fr
Mathématiques et statistique
Area of application code
1205
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Mathematical sciences
Area of application fr
Sciences mathématiques
Research subject code
2955
Research subject group en
Applied mathematics
Research subject group fr
Numerical analysis
Research subject en
Mathématiques appliquées
Research subject fr
Analyse numérique
Application title
Physics Based Architectures for Deep Neural Networks
Keyword
Control; Deep Neural Networks; Dynamical Systems; Inverse Problems; Partial Differential Equations
Application summary
Deep Neural Networks (DNNs) have been the engine behind the recent deep learning revolution. Such networks have demonstrated remarkable capabilities in solving image classification problems, image segmentation problems and other computer vision tasks. Such tasks are common in applications such as face recognition, autonomous cars and medical image understanding. In recent years, modifications to these networks have been proposed, and in particular, residual networks have shown to improve over existing techniques, and yield even better results. More specifically, residual networks are easier to train, have been used with hundreds and even thousands of layers and can be made memory efficient. Encouraged from these results, a few researchers have attempted to explain the success of such networks by analyzing them as discrete dynamical systems, using theory from ordinary differential equations and optimal control. While initial results are interesting, there is still a large gap between the current technology of training and understanding DNNs, to other fields that use similar methodologies, such as time dependent optimal control, inverse problems and partial differential equations based optimization. Furthermore, DNNs do not have an over-arching continuous analog which makes them somewhat add-hoc and problem/data dependent. Understanding how to move networks between scales, changing the network parameters and understanding its properties are all difficult and in many cases, impossible. The goal of this work is to suggest and develop new ideas for the understanding of DNNs. In particular, we exploit the interpretation of neural networks as dynamical systems in order to propose physics based, or physically motivated networks. These networks represent physical processes such as nonlinear diffusion and nonlinear wave propagation similar to the partial differential equations that have been used for compressed sensing. Such networks can have favorable properties in term of their overall character and may be more amendable to fast optimization algorithms that utilize this structure such as multigrid methods. I will be training 3 Ph.D. students as well as 3-5 undergraduate research assistants in the fundamentals of machine learning and its application to problems in geoscience imaging.Overall, this research program will increase scientific knowledge in the computing of machine learning and will contribute to the growing number of companies working on AI in Canada. We expect students trained via the proposed research will pursue promising professional careers in either academia or industry.