Grant: Research Tools and Instruments

In US, 127.1 million users are using voice control in their vehicles. Surveys conducted by Voicebot.ai show that smart speaker use frequency over time in the UK and Germany is quite high indicating a clear adoption of the technology. Although deep learning has greatly advanced the state-of-the-art in natural language understanding, translation and generation, more work is needed to reach human level communication in terms of adapting to the listeners cognitive abilities, responding with emotion and compassion, understanding context, and deliver the best response from both unstructured and structured knowledge sources in real time. We propose to build a voice and video communication framework to provide cloud services and address the following research challenges: a) develop multi-stream voice and video data fusion to allow improved machine cognition, b) generate voice responses adapting to users' voice, tone, vocabulary, and speed of speech for effective communication and better adoption of the technology by users of all age group and diversity, and c) enable real time rule based and information retrieval based question answering on the cloud using a hierarchical knowledge base combining in-memory and stored structured and unstructured data. We did extensive work on the IBM Watson voice and video communication system, a dialogue and rule-based question answering (QA) system, an information retrieval based three-phase real time QA system, and a hierarchical multilayer knowledge management system using in-memory graph clustering techniques for distributed knowledge management. However, none of these work addressed voice and video communication in real time for cloud service provisioning based on a hierarchical hybrid knowledge base, which we intend to address in the proposed research. Real time service provisioning on the cloud must address security aspects of voice and video data and the knowledge base, execute 3rd party voice and video processing tools and applications putting the network at risk based on the users' request, and most importantly require robust deep learning models trained on domain specific voice, video, and text data to enhance language understanding and response generation. A stand-alone cloud server must be used to test the real time usability of the system and to enable public access without risking the academic network with a custom distributed knowledge management system. We request a compute server with high memory GPUs, firewall, and storage for robust training of deep learning models, creating a publicly accessible cloud server to address security aspects, and validating the system for student support, health data analytics and knowledge services with the help of real people and research collaborators. Voice and video-based service provisioning on the cloud became a viable alternative during the pandemic and our research will extend the technology to enable its applications to home care and support services in Canada.