Grant: Discovery Grants Program - Individual

Volumetric media is one emerging media format that provides a highly immersive and interactive user experience for many applications. Different from 2D and 360-degree videos, a volumetric video consists of a 3D representation of objects, enabling users to watch a scene from various directions with six-degrees-of-freedom (6DoF). While volumetric media has great potential for new applications (e.g., remote medicine, digital twin, real estate, tourism, historical spaces preservation, entertainment), it introduces tremendous new challenges at all stages of media production, distribution, and consumption. While volumetric media provides a user with unheard-of fidelity to explore a scene freely from various directions, such media also requires vast amounts of bandwidth for their network distribution. We propose to investigate methods to distribute bandwidth-hungry volumetric media over new high-speed mobile networks such as 5G networks. While these networks can provide high throughput, this is achieved by utilizing very high radio frequencies (mmWave at 26.5-27.5GHz band in Canada). Unfortunately, such radio waves have low range and poor propagation qualities, i.e., they are easily blocked by walls, buildings, or even human bodies. This means that 5G cells need to be densified and a mobile user will constantly switch between cells, and even worse, sometimes alternate between 4G (or 5G/6GHz) and 5G/28GHz. Often when a user enters a building her mobile device will need to urgently switch to WiFi, or 5G microcells/femtocells need to be deployed close by or indoors. So while we can expect some very high peak bandwidth there will also be significant volatility and a large difference between the minimum and maximum bandwidth that a user may experience. In the proposed research, we will address four key challenges holistically together in the distribution of volumetric media, given the new network realities: how to T1) optimally package 6DoF media to allow T2) flexible and robust quality adaptations under extremely T3) varying bandwidth conditions and with T4) very low latency. The research will be built upon a foundation of novel packaging standards like the HTTP adaptive streaming and AI-driven adaptation algorithms. We will build prototype systems to verify the viability of the designed methods and publish the proposed research team's findings in top-tier conferences and journals. This research program will train 10 Highly Qualified Personnel (HQP) (3 PhDs, 3 Master students, and 4 undergrad students). The HQP will develop expertise in networked multimedia systems and applied AI, which are competencies highly demanded in the job markets. Training HQP in this field is of paramount importance for the ambitious AI and immersive technology ecosystem development plan of Canada. This research program will significantly contribute to Canada's innovation ecosystem competitive advantage in the years to come.