Grant: Alliance Grants
University of Toronto
$218,000.00
Security and privacy of foundation models: Towards assurance and governance in representation learning
FY 2024-25
Summary
Machine learning (ML) profoundly altered the way computer systems are designed, and its applications are remaking society. Yet, because ML algorithms do not provide worst-case guarantees of performance, models they output are known to be vulnerable to attacks. These attacks not only hinder our ability to trust ML in critical applications but also expose the data used to train models. This is unacceptable: Canadians require trust and accountability to ensure that applications of ML are responsible and beneficial to society.
In collaboration with ServiceNow Research (formerly Element AI) and the Defence Research and Development Canada (DRDC), itself an entity of the Department of National Defence (DND), we propose a new research partnership to develop the needed theory and practical methods for trustworthy ML. We focus our efforts on the study of representation learning. This new paradigm has led to several recent advances such as foundation models of which large language models, like GPT and LLaMA, are a prime example. Our research efforts will advance the science of trustworthy foundation models to protect the privacy and confidentiality of (cyber) data they analyze, develop assurance for the applicability of foundation models to downstream tasks, and systematize governance mechanisms for foundation models.
The propose partnership will not only improve our scientific understanding of the capabilities and limitations of foundation models, it will also enable Canadian industry and government to deploy ML approaches in a way that is both responsible and ensures beneficial impact to society thanks to the robustness and privacy guarantees afforded. In addition, through this research collaboration and the associated knowledge transfer, the proposed effort is expected to contribute to technology standardization and help the Canadian government define public policies and regulatory guidelines in strategic areas of the national economy.
Program
Alliance Grants
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
588484-2023
Name nom
Papernot, Nicolas NGR
Department département
Electrical & Computer Engineering, Edward S. Rogers Sr. Department of
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
218000
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Security and privacy of foundation models: Towards assurance and governance in representation learning
Keywords
computer security; data privacy; differential privacy; forensics in generative ai; foundation models; intellectual property protection; robust machine learning; trustworthy machine learning
Application summary
Machine learning (ML) profoundly altered the way computer systems are designed, and its applications are remaking society. Yet, because ML algorithms do not provide worst-case guarantees of performance, models they output are known to be vulnerable to attacks. These attacks not only hinder our ability to trust ML in critical applications but also expose the data used to train models. This is unacceptable: Canadians require trust and accountability to ensure that applications of ML are responsible and beneficial to society.
In collaboration with ServiceNow Research (formerly Element AI) and the Defence Research and Development Canada (DRDC), itself an entity of the Department of National Defence (DND), we propose a new research partnership to develop the needed theory and practical methods for trustworthy ML. We focus our efforts on the study of representation learning. This new paradigm has led to several recent advances such as foundation models of which large language models, like GPT and LLaMA, are a prime example. Our research efforts will advance the science of trustworthy foundation models to protect the privacy and confidentiality of (cyber) data they analyze, develop assurance for the applicability of foundation models to downstream tasks, and systematize governance mechanisms for foundation models.
The propose partnership will not only improve our scientific understanding of the capabilities and limitations of foundation models, it will also enable Canadian industry and government to deploy ML approaches in a way that is both responsible and ensures beneficial impact to society thanks to the robustness and privacy guarantees afforded. In addition, through this research collaboration and the associated knowledge transfer, the proposed effort is expected to contribute to technology standardization and help the Canadian government define public policies and regulatory guidelines in strategic areas of the national economy.