Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

McGill University

$48,000.00

System Support for High Performance Inference of Retrieval-based Language Models

EF 2024-25

Résumé
Large language models (LLMs), with their inherent ability to comprehend and generate human-level texts, have revolutionized diverse fields such as chatbots, healthcare, education, finance, and more. As these models continue to evolve, they are becoming increasingly larger and more complex. The predominant focus starts to shift from training these models to efficiently making inferences with them. This observation is confirmed by leading companies including NVIDIA, as clearly announced by the CEO Jensen Huang in his 2023 GTC (GPU Technology Conference) keynote. Despite their superior performance, LLMs face certain limitations: it struggles to memorize the massive amount of factual knowledge; the memorized knowledge can be outdated; and the computation demand for inference using these LLMs is high. The emerging retrieval-based language models can help address these limitations. These models, composed of a retrieval mechanism and a language model, equip the system with a vast external memory, enabling it to dynamically access and leverage specialized knowledge. As a result, they can help address the above-mentioned limitations of LLMs. However, a retrieval-based language model (RBLM) requires interactions with large-scale external memory stores. The interaction time, along with the model computation time, directly impacts the overall inference time. Hence, current state-of-the-art retrievers (e.g., FAISS from Meta) are not efficient enough for many high-performance applications. In this proposed Discovery program, we aim to design systematic solutions to enable high-performance inference of retrieval-based language models. In recent years, we carried out research in closely-related areas including AI and machine learning, information retrieval, and storage systems etc. Results and insights from these research lay a solid foundation for this proposed program. Our long-term objective is to provide systematic solutions to shorten the inference time while maintaining the accurate response from RBLMs. The inference performance of an RBLM is determined by the time of handling two major cases: 1) The static data case, which refers to when the external storage is not being updated. 2) The dynamic data case, which refers to when the external storage is being updated. We divide the program into two main objectives accordingly and the corresponding short-term objectives, and they will be thoroughly investigated in this research program. The result will be a holistic solution that utilizes the software-hardware co-design approach to enable high-performance inference for modern RBLMs. Through this research program, we are committed to thoroughly training highly qualified personnel (HQPs) for their upcoming academic and industrial careers. The success of this program will make RBLMs more practical, hence significantly expand the adoption of LLMs and Generative AI, and have a positive impact on the World's and Canadian economy.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2024-04615
Name nom
Liu, Xue
Department département
Computer Science
Organization id
61
Institution établissement
McGill University
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
48000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2710
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information systems design
Research subject fr
Conception de systèmes d'information
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
System Support for High Performance Inference of Retrieval-based Language Models
Keywords
in-storage computing; retrieval-based language models; learned index; external memory; mixture-of-experts (moe); i/o management; nearest neighbor search
Application summary
Large language models (LLMs), with their inherent ability to comprehend and generate human-level texts, have revolutionized diverse fields such as chatbots, healthcare, education, finance, and more. As these models continue to evolve, they are becoming increasingly larger and more complex. The predominant focus starts to shift from training these models to efficiently making inferences with them. This observation is confirmed by leading companies including NVIDIA, as clearly announced by the CEO Jensen Huang in his 2023 GTC (GPU Technology Conference) keynote. Despite their superior performance, LLMs face certain limitations: it struggles to memorize the massive amount of factual knowledge; the memorized knowledge can be outdated; and the computation demand for inference using these LLMs is high. The emerging retrieval-based language models can help address these limitations. These models, composed of a retrieval mechanism and a language model, equip the system with a vast external memory, enabling it to dynamically access and leverage specialized knowledge. As a result, they can help address the above-mentioned limitations of LLMs. However, a retrieval-based language model (RBLM) requires interactions with large-scale external memory stores. The interaction time, along with the model computation time, directly impacts the overall inference time. Hence, current state-of-the-art retrievers (e.g., FAISS from Meta) are not efficient enough for many high-performance applications. In this proposed Discovery program, we aim to design systematic solutions to enable high-performance inference of retrieval-based language models. In recent years, we carried out research in closely-related areas including AI and machine learning, information retrieval, and storage systems etc. Results and insights from these research lay a solid foundation for this proposed program. Our long-term objective is to provide systematic solutions to shorten the inference time while maintaining the accurate response from RBLMs. The inference performance of an RBLM is determined by the time of handling two major cases: 1) The static data case, which refers to when the external storage is not being updated. 2) The dynamic data case, which refers to when the external storage is being updated. We divide the program into two main objectives accordingly and the corresponding short-term objectives, and they will be thoroughly investigated in this research program. The result will be a holistic solution that utilizes the software-hardware co-design approach to enable high-performance inference for modern RBLMs. Through this research program, we are committed to thoroughly training highly qualified personnel (HQPs) for their upcoming academic and industrial careers. The success of this program will make RBLMs more practical, hence significantly expand the adoption of LLMs and Generative AI, and have a positive impact on the World's and Canadian economy.