Grant: Discovery Grants Program - Individual

Multimodal information fusion analyzes signals extracted from several types of sensory data or modalities and normally leads to better and more reliable performance than that obtained from a single modality. The need to coherently understand and process information from different modalities is becoming more and more demanding in many applications, particularly in biometrics and in multimedia information processing. Without a satisfactory resolution to multimodal information fusion, human mastery of the information era will remain an elusive dream.*There are two major advantages for multimodal fusion that make multimodal systems more reliable: 1) the information embedded in different modalities is usually complementary and multimodal processing is more informative than single modality, e.g. the human biological computing machine; and 2) When one modality becomes corrupted by noise or the environment, we can rely on other modalities to partially make up for the missing information.*A literature review reveals that the bulk of research to date lacks analytical foundations and has largely been motivated by special needs. Such approaches are ad hoc, unsystematic, and often, difficult to generalize. Without a proper approach, the field of information fusion will continue to be empirical at best. There is, therefore, an urgent call for a systematic resolution to multimodal information fusion. In this program, we propose a new framework based on the estimation of information entropy. Entropy is a measure of disorder or unpredictability. Instead of assuming the popular second order statistics, entropy based methods allow arbitrary distributions and higher order statistics. Entropy is closely linked to confidence level in information fusion. In general, high entropy indicates a low confidence in the corresponding modality, and vice versa. The evidence is combined with the estimated reliability through posterior entropy to provide a means to predict if the integration of multimodal information will lead to better performance, selecting the most appropriate fusion strategy for the application on hand and making a fusion system tolerant to quality changes of individual modalities, and even to the temporary corruption of a subset of the modalities. *The expected outcome of the proposed research is to provide knowledge and tools for developing state-of-art multimedia information processing systems. The gained knowledge will generate significant societal benefits to Canada and provide enabling technology for a broad range of applications, such as new media production and delivery, preservation of cultural heritage, infrastructure planning, training of skilled professionals, marketing and advertising, e-health, smart homes, security/surveillance, gaming and e-entertainment, and creating life-like experience with immersive multimedia technology. *