Grant: Discovery Grants Program - Individual

Visual Geometry is the process of understanding the environment and estimating the position of the observer from images. It is at the core of Autonomous Vehicles (AV) such as self-driving cars and delivery drones, and determines how well a machine can automatically navigate and interact with the world. For Augmented Reality (AR) and Mixed Reality (MR), Visual Geometry is one of the deciding factors on how realistic and stable one can mix the virtual world with the real world. Thus, advancing Visual Geometry is essential in bringing these new technologies into our daily lives. However, despite its importance, existing methods for Visual Geometry is not accurate enough to be deployed in the wild. The main reason for the limited performance is that we still rely on traditional methods when it comes to Visual Geometry, in contrast to other areas in Computer Vision, where machines now outperform humans in many tasks through the help of Deep Learning. Moreover, the current state- of-the-art Deep Learning-based methods for Visual Geometry are not performing as well as in other applications, for example recognizing images. I argue that the main reason for this contrast is that typically, for the applications where Deep Learning brought incredible success, the problem is mathematically over-determined, whereas Visual Geometry is under-determined due to the loss of depth information when projecting 3D real-world data into 2D images. To overcome this problem, I aim towards building the Deep Visual Geometry Machine, a collection of Deep Networks dedicated to each task of the Visual Geometry pipeline: extracting key information from images, finding correspondences between images, and recovering the state of the observer through correspondences. By limiting each network to a specific sub task, we can formulate the Visual Geometry problem as a collection of over-determined sub problems, which Deep Learning has shown to be good at solving. By doing so, I expect a significant improvement in the accuracy of the estimated position of the observer, as well as the mapping of the environment, possibly surpassing the humans. The proposed research will have immediate broad impact on many applications including, but not limited to, autonomous driving, delivery drones, domestic robots, augmented reality, and mixed reality. Furthermore, the proposed technology would enable machines to learn to plan and act, thus allowing even more complicated applications. This could include, for example, allowing autonomous vehicles and drones to operate in remote regions in the north of Canada. The current proposal also is expected to contribute to the training of HQP in the area of Machine Learning and Computer Vision. Currently, both areas are in high demand of HQP in Canada and world-wide.