Grant: Discovery Grants Program - Individual

The key trends report from the Canada Media Fund (CMF) identifies Transmedia (Augmented and Virtual Reality (VR)) as “poised for wide consumer release”. While “VR is still 5 to 10 years from mainstream adoption, content businesses must prepare now if they hope to compete” [excerpts from the CMF report]. The proposed research comprises two relevant problems pertaining to transmedia applications, 3D content (virtual worlds) creation and human interaction with virtual objects for a fully immersive VR experience. ******Virtual worlds, i.e., polygonal meshes of cities at scale, are increasingly used in VR based applications, e.g., Google Earth VR. In 2018, an area of 50 sq. kms of Helsinki, Finland, was digitized by Umbra technologies targeting at AR, VR gaming int real city settings [www.forbes.com]. We position our research to support the ongoing efforts of the graphics and gaming industry to utilize digitized cities in transmedia applications. The current practices in large scale city digitization include acquiring raw measurements of the physical world using LiDAR sensors and abstracting coarse models from the LiDAR data. However, such coarse building models lack geometric details and hence realism, and do not represent a good choice for 3D navigation, virtual reality and video game creation. So, we investigate into a fully automatic computational framework that creates highly detailed and semantically rich urban models at large scale. The work is a follow up to my research in LiDAR based urban modeling which began in 2015.******A core enabler of interaction in VR spaces is real time full body tracking, e.g., a fully-immersive VR game brings human interaction to virtual objects in real time and such interactions call for robust and highly accurate full body tracking systems which are seamlessly deployable. We target highly robust and low-latency full-body motion capture in the presence of interfering geometry (e.g., joystick) using a multi camera system. A multi-camera 3D data acquisition system will be designed and set up, which would open up further avenues of research in futuristic applications such as virtual cloth fitting in online stores, and bio-mechanical studies in sports and fitness. We plan to collect annotated human motion data and make it publicly available for the geometric deep learning community.******The result of this research will be valuable to industry practitioners as well as the academic community in developing better practices and tools for virtual world creation and AR/VR applications. Helping the Canadian VR industry to become a key player in the global VR market will result in a positive impact on the local economy, with capital inflow and the associated requirement for high tech workers in the booming VR/AR industry. The proposed LiDAR based reconstruction framework is expected to play a key role in digitizing existing cities around the globe which may be used as 3D content for VR based games or tours and 3D maps for navigation.