Grant: Discovery Grants Program - Individual
Université Laval
$31,000.00
Closing the Reality Gap in Machine Learning for Speech and Audio
EF 2024-25
Résumé
With the proliferation of deep learning, machine learning models have realized very substantial performance improvements on bench-marking datasets. Examples for this kind of success include domains such as computer vision, speech recognition, natural language processing, speech enhancement-separation, reinforcement learning, and so on.
However, the impressive performance obtained on the bench-marking datasets does not necessarily translate to real-life performance. For instance, machine learning models trained under a fixed set-of-conditions would not necessarily perform well under previously unseen conditions. For example a personal assistant might not perform well in reverberant rooms if it is trained under non-reverberant rooms, or speech recognizer in a smart phone might not work well in various types of environmental noise. With the support of this Discovery Grant, I would like to therefore establish a 5 year research program that would addresses the aforementioned shortcomings to increase and accurately measure the generalization performance of neural networks in real-life deployment for speech and audio applications.
To progress towards this goal, my first aim is to collect datasets that are representative to the challenges of real-life. As we have shown in our recent work, `REAL-M:Towards Speech Separation on Real Mixtures', it is possible to collect data in a scalable way through crowdsourcing, which promotes diversity in acoustic conditions as well. We will aim to collect data that can be used for tasks such as speech enhancement, speech separation, automatic speech recognition, multi-speaker speech recognition, speech diarization, sound and speech generation.
After completing this first step of data collection, we will investigate ways to improve performance under real-life conditions by improving upon the techniques from the established machine learning fields such as domain adaptation and domain generalization. In the later phases of the project we will also investigate the case where we would like to train models under evolving conditions, in order to aim to develop techniques that would effectively alleviate forgetting and improve domain generalization under the continual learning setup.
The potential outcome of this research project aims to significantly improve the performance of machine learning based speech and audio technologies under real-life conditions, for which there currently is a large commercial interest from big technology companies, as showcased by the prevalence of voice assistants. This program presents possibility of training 4 Highly Qualified Personnels (2 PhD students, 2 MSc students). These HQPs will develop an expertise in machine learning for speech/audio, which are currently in-demand competencies in the job market of Canada.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2023-05759
Name nom
Subakan, Yusuf Cem
Department département
Sciences et génie, faculté des - Sciences et génie, faculté des
Organization id
51
Institution établissement
Université Laval
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
31000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2801
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Natural language and speech understanding
Research subject fr
Langage naturel et reconnaissance de la parole
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Closing the Reality Gap in Machine Learning for Speech and Audio
Keywords
speech processing; machine learning for speech and audio; machine learning for signal processing; domain generalization; deep learning; conversational ai; out-of-domain generalization; continual learning; domain adaptation; speech enhancement and separation
Application summary
With the proliferation of deep learning, machine learning models have realized very substantial performance improvements on bench-marking datasets. Examples for this kind of success include domains such as computer vision, speech recognition, natural language processing, speech enhancement-separation, reinforcement learning, and so on.
However, the impressive performance obtained on the bench-marking datasets does not necessarily translate to real-life performance. For instance, machine learning models trained under a fixed set-of-conditions would not necessarily perform well under previously unseen conditions. For example a personal assistant might not perform well in reverberant rooms if it is trained under non-reverberant rooms, or speech recognizer in a smart phone might not work well in various types of environmental noise. With the support of this Discovery Grant, I would like to therefore establish a 5 year research program that would addresses the aforementioned shortcomings to increase and accurately measure the generalization performance of neural networks in real-life deployment for speech and audio applications.
To progress towards this goal, my first aim is to collect datasets that are representative to the challenges of real-life. As we have shown in our recent work, `REAL-M:Towards Speech Separation on Real Mixtures', it is possible to collect data in a scalable way through crowdsourcing, which promotes diversity in acoustic conditions as well. We will aim to collect data that can be used for tasks such as speech enhancement, speech separation, automatic speech recognition, multi-speaker speech recognition, speech diarization, sound and speech generation.
After completing this first step of data collection, we will investigate ways to improve performance under real-life conditions by improving upon the techniques from the established machine learning fields such as domain adaptation and domain generalization. In the later phases of the project we will also investigate the case where we would like to train models under evolving conditions, in order to aim to develop techniques that would effectively alleviate forgetting and improve domain generalization under the continual learning setup.
The potential outcome of this research project aims to significantly improve the performance of machine learning based speech and audio technologies under real-life conditions, for which there currently is a large commercial interest from big technology companies, as showcased by the prevalence of voice assistants. This program presents possibility of training 4 Highly Qualified Personnels (2 PhD students, 2 MSc students). These HQPs will develop an expertise in machine learning for speech/audio, which are currently in-demand competencies in the job market of Canada.