Grant: Discovery Grants Program - Individual

The development of intelligent vehicles with a high level of autonomy requires advanced perception capabilities. These vehicles are generally equipped with three main sensor types: cameras, lidar and radar. All these sensors are intrinsically very complementary and the strength of one can compensate for the weakness of the other. Sensor fusion aims at combining the information coming from different sensors and thus improve the perceptual ability of an autonomous vehicle. Three basic strategies can be used: early fusion, late fusion and mid-level fusion. Early fusion consists in simply merging the input representations built from the different sensors and used this hybrid data as input to a detection and classification network. Late fusion proceeds by first processing independently each sensor and combine the different outputs into one decision. Mid-level fusion processes each sensor independently to produce high--level information and then integrates the different branches into a detection and classification network. The general objective of the proposed research program is to study the impact of sensor fusion approaches on advanced driver assistance systems and autonomous driving applications. The research will focus on the development of new methods and algorithms to perform optimal fusion of data captured by different sensors. We also want to investigate under which conditions and scenarios sensor fusion is beneficial in perception systems. Finally, we will study how each type of sensors can benefit from the complementary information extracted by another sensor modality and which sensor combination benefit the most from data fusion. An important objective of this research program will consist in the capture and annotation of driving scenes recorded under adverse weather conditions, in particular winter conditions. These datasets will be used to assess the individual performance of each sensor and better understand their failure mode and how they deteriorate as the vehicle evolve into the adverse weather. Based on this analysis, new sensor fusion network architectures will be proposed and studied. Our focus will be on mid-level fusion architectures. This fusion strategy has not been well explored in the literature because of its complexity even if it has the potential of surpassing other simpler fusion schemes. The strength of mid-level fusion resides in its ability to consider and exploit the intrinsic properties of each sensor while, at the same time, combining and integrating higher level features in order to perform reliable detection and classification. Data representation has also an important impact on network performances; 2D and 3D schemes will be investigated.