Grant: Collaborative Research and Development Grants

Augmented reality (AR) and mixed reality (MR) are two very recent applications that are drawing much market interest, being the core technology behind flagship features of high-tech products, for example the iPhoneX and Microsoft Hololens. Under the hood of these new technology is the ability of machines to model and track what it is observing. Unfortunately, despite the phenomenal advancements from deep learning, these modeling and tracking methods still rely heavily on decade-old algorithms and human- designed ideas. The recent success of deep learning, exemplified by AlphaGo that surpassed humans, suggests that we can do much better.In this proposal we will teach machines to see in 4D (space and time), by developing learning-based approaches for real-time 4D visual tracking. We will let machines provide much more accurate modeling and tracking results, ultimately resulting in seamless integration of the reality and virtual world, enhanced human machine interaction based on more accurate and detailed gesture recognitions, and allow machines to intelligently interact with the world. All in real-time.The success of this project will push Canada forward and allow Canada to lead the newly emerging industry of AR/MR products and autonomous robotics.