Grant: Discovery Grants Program - Individual

Environment perception and modeling from image sequences is a challenging task of intelligent vehicles in dynamic environments. As a critical component of visual perception, how to efficiently model and represent the world has been an active research area during the past decades. Although considerable progress has been made both in theory and in practice, the problem of environment perception remains far from being solved, owing primarily to three critical challenges: (i) how to recover the 3D structure of dynamic natural scenes coupled with rigid, nonrigid, articulated, and moving objects; (ii) how to efficiently and robustly represent the working environment in practical navigation scenarios; and (iii) how to increase the generalization ability of a machine learning model. These issues are very complex and have thus seldom been well-addressed in the literature. The main objective of this project is to solve these challenges and promote the 3D perception ability of intelligent systems by developing an efficient and robust representation of dynamic natural environments. Specifically, we will investigate three critical research problems in this project. The first one is on scene depth estimation and semantic segmentation. We propose a unified learning model for depth estimation and semantic segmentation using multi-task learning. By integrating the geometric constraint and the temporal information of the sequence, the proposed model can recover the depth and semantic information simultaneously with high accuracy. The second problem is semantic object detection. We propose a new learning model for object detection with the ability to make reasoning and deduction from training examples by mimicking human brains. A new learning paradigm is developed to increase the model generalizability via active and reinforcement learning. For the third problem, we propose a novel unified framework to represent dynamic environments by exploring all information from the scene to form a hybrid representation. The study envisions a novel perspective for environment perception by integrating all information from the scene. In particular, the project emphasizes new algorithmic aspects for problems of depth estimation, semantic segmentation, object detection, and 3D mapping. The proposed technique makes the environment mapping more efficient and effective for autonomous systems to navigate in unknown dynamic environments. This research is not only academically significant but also urgently needed by automotive and robotic industries. This research is a fundamental step towards the investigator's long-term research endeavor: building a highly intelligent system through visual cues. We will also integrate the proposed framework and research results into our education and outreach activities to enhance existing course curriculums and attract more students, especially those from underrepresented minorities, to work in computer science and engineering.