Grant: Discovery Grants Program - Individual

Head worn display technologies such as virtual/mixed/augmented reality (XR) have recently brought new levels of realistic immersive experience to users on a large scale. While impressive, these technologies are still limited in terms of the need to wear a headset or glasses. In contrast, Field of Light Displays (FoLDs) are also being developed to provide compelling immersive experiences, without the need to wear head or body--mounted devices. FoLDs provide the potential to be configured as a paneled whole--room display experience, creating a Holodeck that provides an unprecedented immersive experience. The ideal FoLD would be capable of reproducing patterns of light (known as a "light field"") as it is encountered in viewing natural imagery. In practice, this field is approximated, but still provides additional perceptual cues over conventional displays such as stereoscopy, motion parallax and focal cues. This has been shown to result in improved visualization task performance, decreased cognitive load and greater visual comfort. Overall, this results in a more realistic and useful visual experience, showing great promise in improving situational awareness, training and enhancing entertainment experiences. In order to provide content for these displays, field of light (FoL) imaging must be used in tandem in order to enable the full benefits of this display paradigm. In theory, field of light images provide more scene information and thus enable the potential for more insightful imaging in monitoring and situational awareness use cases, as well as for more realistic images with enhanced post-capture processing potential in cinematographic contexts. FoLDs pose enormous bandwidth requirements in terms of raw input image pixel count. Some have estimated that usable full--parallax FoLDs will require raw video bandwidth of at least 10 Tbits/s, driving 10s of billions of individually--addressable light rays. Conventional 8K 2D displays have ~33 million pixels. FoLDs require orders of magnitude more bandwidth than is currently provided by both existing interactive rendering systems and streaming video transmission codecs. FoL imagery intuitively contains more redundancy than conventional imagery. The question is, how to exploit this to improve rendering, storage and transmission performance for immersive display systems? In this research program, we will develop new representations and systems to enable (1) interactive rendering of immersive content and (2) codecs for storage and transmission of pre-recorded and live immersive content.