Grant: Discovery Grants Program - Individual
University of Waterloo
$36,000.00
Designing Serving Systems for Large Language Models with Low Latency, High Utilization, and Good Scalability
FY 2024-25
Summary
We are currently witnessing an explosion of large language models (LLMs). The latest models such as GPT-4 have achieved unprecedented performance in various tasks such as code generation, text classification, and problem reasoning. Such advancements have triggered a new generation of AI applications. These LLM-based applications demonstrate great potential to bring revolutionary changes to a wide variety of areas including commerce, science, and education.
However, serving LLM-based applications, i.e., deploying trained LLMs on a compute cluster and conducting model inference for incoming user requests, presents challenges in system design. First, LLMs introduce high execution latency due to the huge model size. GPT-4, for example, can take tens of seconds to generate responses, causing suboptimal user experiences. Second, serving LLMs requires expensive hardware accelerators (e.g., GPUs) and results in high resource consumption. ChatGPT is estimated to cost over $700,000 per day to operate. Moreover, scaling LLM-based applications to meet the demand of large user bases is challenging. For instance, deploying ChatGPT to power every Google search requires a cluster of over four million A100 GPUs. Taken together, serving systems for LLM-based applications highlight three critical demands: low execution latency to optimize user experience, high resource utilization to save costs, and good scalability to serve a massive number of requests.
The long-term goal of this program is to design LLM serving systems with high utilization, low latency, and good scalability. Meeting this goal requires fundamental redesign of existing model serving systems, as LLM-based applications introduce several unique new challenges. To this end, we break the long-term goal into three objectives, each aiming to address one challenge by proposing innovative solutions tailored for LLM serving: 1.Establish a framework for LLM serving systems that perform customized system optimizations based on input prompts; 2.Design an online scheduler that makes holistic and scalable scheduling decisions for LLM serving systems; and 3.Explore system optimizations that efficiently manage complex interactions between LLMs and external modules. The objectives explore new frontiers in AI systems, encompassing a cross-disciplinary approach across distributed systems, machine learning, and software engineering.
The program will generate significant research impact in facilitating the development and adoption of LLM-based applications. The outcomes contribute to Canada's international competitiveness in this AI revolution. It benefits Canada's economy by enabling tech companies and startups to develop and host LLM-based services at lower costs and with reduced latency. The high resource utilization achieved will positively impact the environment by lowering carbon footprints. The program will train students in machine learning systems, an expertise of high demand in both academia and industry.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2024-04095
Name nom
ZHANG, HONG
Department département
School of Computer Science
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
36000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2704
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Distributed and parallel processing
Research subject fr
Traitement réparti et simultané
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Designing Serving Systems for Large Language Models with Low Latency, High Utilization, and Good Scalability
Keywords
distributed systems; machine learning systems; model serving; large language models
Application summary
We are currently witnessing an explosion of large language models (LLMs). The latest models such as GPT-4 have achieved unprecedented performance in various tasks such as code generation, text classification, and problem reasoning. Such advancements have triggered a new generation of AI applications. These LLM-based applications demonstrate great potential to bring revolutionary changes to a wide variety of areas including commerce, science, and education.
However, serving LLM-based applications, i.e., deploying trained LLMs on a compute cluster and conducting model inference for incoming user requests, presents challenges in system design. First, LLMs introduce high execution latency due to the huge model size. GPT-4, for example, can take tens of seconds to generate responses, causing suboptimal user experiences. Second, serving LLMs requires expensive hardware accelerators (e.g., GPUs) and results in high resource consumption. ChatGPT is estimated to cost over $700,000 per day to operate. Moreover, scaling LLM-based applications to meet the demand of large user bases is challenging. For instance, deploying ChatGPT to power every Google search requires a cluster of over four million A100 GPUs. Taken together, serving systems for LLM-based applications highlight three critical demands: low execution latency to optimize user experience, high resource utilization to save costs, and good scalability to serve a massive number of requests.
The long-term goal of this program is to design LLM serving systems with high utilization, low latency, and good scalability. Meeting this goal requires fundamental redesign of existing model serving systems, as LLM-based applications introduce several unique new challenges. To this end, we break the long-term goal into three objectives, each aiming to address one challenge by proposing innovative solutions tailored for LLM serving: 1.Establish a framework for LLM serving systems that perform customized system optimizations based on input prompts; 2.Design an online scheduler that makes holistic and scalable scheduling decisions for LLM serving systems; and 3.Explore system optimizations that efficiently manage complex interactions between LLMs and external modules. The objectives explore new frontiers in AI systems, encompassing a cross-disciplinary approach across distributed systems, machine learning, and software engineering.
The program will generate significant research impact in facilitating the development and adoption of LLM-based applications. The outcomes contribute to Canada's international competitiveness in this AI revolution. It benefits Canada's economy by enabling tech companies and startups to develop and host LLM-based services at lower costs and with reduced latency. The high resource utilization achieved will positively impact the environment by lowering carbon footprints. The program will train students in machine learning systems, an expertise of high demand in both academia and industry.