Grant: Discovery Grants Program - Individual

Volumetric Video-often referred to as a 4D video-is a computational fusion of digital video and depth data, resulting in a spatialized, and potentially navigable `hologram'. Volumetric video is becoming increasingly prevalent and sophisticated, driven by interest in augmented (AR) and virtual reality (VR) as VV affords a level of realism not commonly present in traditional 3D modeling and animation approaches. Staying true to its name, volumetric video is essentially a `video' (where each frame is a 3D model) in a 3D space retaining all of the advantages and limitations of `video' as a medium. Volumetric video aims to be a true recording of events occurring in the performative space and as such can be highly realistic making it a valuable medium for free-viewpoint playback with no user interaction other than standard video controls (e.g. play/pause). This proposal investigates the effectiveness of volumetric video for interactive simulation and training scenarios, and improves the state-of-the-art by developing new algorithms that fuse modern computer animation techniques with volumetric video to add interactivity to a highly dynamic yet immutable medium. Evolving volumetric video with modern animation techniques (originally developed for use with motion-capture) has the potential to impact the simulation and training industries enabling higher levels of realism, and providing interactive simulation designers the capability to quickly and effectively prototype new ideas. Although volumetric video has the potential to increase visual realism of real-time training scenarios, demonstrating whether volumetric video actually improves the user experience leading to improved training outcomes beyond traditional methods is yet to be determined. Does the increased visual fidelity of VV fall into the `uncanny valley' due to small imperfections in the result leading to an inferior training experience? How interactive should volumetric video assets be in order to retain immersion and improve training outcomes? This leads to the following driving research questions: is volumetric video an effective medium for simulation and training in VR and can we improve the interactivity of volumetric video? This research program will explore these questions in two phases: (1) exploring the `effectiveness' of volumetric video for simulation-based learning through a user testing methodology, (2) evolving animation techniques originally designed for motion capture to work with the new volumetric medium.