Grant: Discovery Grants Program - Individual

The human kind has arrived at an "era of experience", where understanding, enhancing, and enriching the human perceptual experience has become ever more important. This coincides with the rapidly growing interest in novel applications in virtual environments such as "metaverse", where seamlessly reproducing the perceptual multimedia quality-of-experience (QoE) in the real world becomes an essential enabling technology. Meanwhile, the past few years have witnessed some dramatic advancement in generative artificial intelligence (AI) models, which have resulted in some remarkable achievements in challenging tasks such as natural language processing, represented by successful models like ChatGPT and other large language models. However, state-of-the-art mainstream QoE research still focuses on predicting single individual quality scores of multimedia content and using such predictions for perceptual optimization purposes. Successful stories of generative AI methodologies in multimedia QoE research remain limited. This Discovery project aims to develop a systematic framework for the understanding, enhancement and enrichment of perceptual multimedia QoE by exploiting generative AI methodologies. The objective is to revolutionize the traditional ways of interpreting multimedia QoE and create large novel space to explore for the enhancement and enrichment of multimedia QoE in various application environment. First, we will develop a novel framework to interpret multimedia QoE. Unlike traditional approaches where multimedia QoE is often assessed in the form of an individual score obtained either by subjective testing or objective model prediction, our novel framework interprets multimedia QoE as a graph that represents the relationship between scene objects, image attributes, and quality characteristics, from which various linguistic descriptions may be generated. Second, we will develop generative language models in the domain of the novel multimedia QoE interpretation by leveraging large language models and by learning from training datasets composed of multimedia content of diverse perceptual distortion types and levels. Third, we will develop generative AI models to enhance the perceptual QoE and enrich the content of multimedia assets. The target content types include 2D images and videos, as well as 3D or more sophisticated content for virtual reality and augmented reality environment such as stereoscopic images/videos, 360-degree omnidirectional images/videos, 3D point clouds, and textured meshes. The project, if successful, is expected to help strengthen Canada's reputation and global leadership in the academic field of multimedia signal processing and AI. By transferring the technology to Canadian industrial partners, Canadian companies will improve their competitiveness in the global market. In addition, it will also help produce a significant number of highly qualified personnel and create new jobs in the information technology sector.