Grant: Discovery Grants Program - Individual

The rapid development of digital technologies continues to profoundly change the way we access information and communicate. As the digitization and encoding of different types of digital data become more affordable, computers and web-based database systems are required to store and manipulate voluminous digital data in multimodality. Although we have witnessed significant progress in technologies for analyzing, processing and managing unimodal data, such as image, video, audio, graphics, studies in multimodal data in large archives have not kept pace. This program will address a fundamental issue, multimodal information fusion - the coherent interaction and integration of different types of sensory and/or media data, in order to achieve a breakthrough in multimedia information processing. Specifically, we will extend the famed coupled hidden Markov model to the fusion of multimodalities, and further propose a novel fusion model, the fused Max-Margin Markov Networks (F-M3N) for multimodal information integration. We will use a unique statistical tool to analytically and experimentally demonstrate the effectiveness and robustness of the proposed methods in multimodal information fusion. We will also employ knowledge discovery techniques, particularly a dynamic self-organizing model: The Self-Organizing Tree Map (SOTM), to acquire indepth understanding of the characteristics of the multimodal data. In the endeavor of this program, broad and indepth knowledge and expertise in information processing and technology such as image and video processing, pattern recognition, human-computer interaction, understanding and processing of large scale databases are required. The developed technologies will be applied to, among others, a) multimedia information processing such as multimedia retrieval, automatic learning of audio-visual concepts, and new media applications; and b) multimodal biometrics for human intention/activity detection for security, e-health, and digital homes.