Grant: Discovery Grants Program - Individual
University of Alberta
$82,000.00
Distributed Optimization for Machine Learning on Decentralized Data and Features
EF 2023-24
Résumé
The need to scale up machine learning, in the presence of a rapid growth in both data volumes and model complexity, has sparked broad interests in developing distributed machine learning systems as well as parallel optimization algorithms. Most existing studies focus on partitioning computation among a tightly coupled cluster of machines either in a data-parallel fashion, to deal with large amounts of training samples, or a model-parallel fashion, to deal with large models, such as deep neural networks. In contrast, the goal of this research program is to glean insights and build models when the dataset (features, samples, labels or a combination of them) used for machine learning is inherently decentralized and owned by multiple participants/domains. Our long-term vision is to design reliable distributed algorithms and systems that can build models from decentralized data, without letting participants share original data with each other or to a central site. By effectively leveraging data from other domains, each participant is expected to enhance its predicting power over a model built only based on its local data, while the joint model built in a decentralized way is expected to approach and approximate the global model if all data were collected centrally. In the meantime, the sharing of model parameters among parties should also be minimized to preserve privacy and reduce communication overhead. Toward these objectives, we will introduce generic composite model structures that can jointly reap insights from data in different decentralization scenarios, including decentralization by features, by samples, by labels or decentralization by a combination of them. We will design theoretically inspired distributed optimization algorithms to solve these problems, develop effective communication compression techniques to reduce overhead, and also study implementation issues for specific applications. Specifically, our algorithms will be inspired by the recent advancements in the convergence of ADMM, stochastic gradient descent (SGD) and proximal SGD in an asynchronous and blockwise setting. Our communication compression techniques will be inspired by the opportunity to suppress model parameter transfers in flat regions of the optimization objective function, whereas existing literature mainly considers significance filters for gradients. Finally, we will use the proposed model architectures and algorithms to solve various real-world applications in cross-domain recommender systems, multitasked natural language understanding and collaborative mobile edge computing. We will design specific model composition and decomposition structures as well as distributed algorithms based on the data decentralization pattern inherent in each problem.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2019-04998
Name nom
Niu, Di
Department département
Electrical and Computer Engineering
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2019
Award amount
82000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2715
Research subject group en
Information technology
Research subject group fr
Optimization
Research subject en
Technologie de l'information
Research subject fr
Optimisation
Application title
Distributed Optimization for Machine Learning on Decentralized Data and Features
Keyword
ADMM; Communication Compression; Data Management; Decentralization; Distributed Optimization; Machine Learning; Parallel and Distributed Algorithms; Parameter Servers; Stochastic Optimization
Application summary
The need to scale up machine learning, in the presence of a rapid growth in both data volumes and model complexity, has sparked broad interests in developing distributed machine learning systems as well as parallel optimization algorithms. Most existing studies focus on partitioning computation among a tightly coupled cluster of machines either in a data-parallel fashion, to deal with large amounts of training samples, or a model-parallel fashion, to deal with large models, such as deep neural networks. In contrast, the goal of this research program is to glean insights and build models when the dataset (features, samples, labels or a combination of them) used for machine learning is inherently decentralized and owned by multiple participants/domains. Our long-term vision is to design reliable distributed algorithms and systems that can build models from decentralized data, without letting participants share original data with each other or to a central site. By effectively leveraging data from other domains, each participant is expected to enhance its predicting power over a model built only based on its local data, while the joint model built in a decentralized way is expected to approach and approximate the global model if all data were collected centrally. In the meantime, the sharing of model parameters among parties should also be minimized to preserve privacy and reduce communication overhead. Toward these objectives, we will introduce generic composite model structures that can jointly reap insights from data in different decentralization scenarios, including decentralization by features, by samples, by labels or decentralization by a combination of them. We will design theoretically inspired distributed optimization algorithms to solve these problems, develop effective communication compression techniques to reduce overhead, and also study implementation issues for specific applications. Specifically, our algorithms will be inspired by the recent advancements in the convergence of ADMM, stochastic gradient descent (SGD) and proximal SGD in an asynchronous and blockwise setting. Our communication compression techniques will be inspired by the opportunity to suppress model parameter transfers in flat regions of the optimization objective function, whereas existing literature mainly considers significance filters for gradients. Finally, we will use the proposed model architectures and algorithms to solve various real-world applications in cross-domain recommender systems, multitasked natural language understanding and collaborative mobile edge computing. We will design specific model composition and decomposition structures as well as distributed algorithms based on the data decentralization pattern inherent in each problem.