Grant: Discovery Grants Program - Individual

The ubiquity of imaging devices and internet connectivity has resulted in the production and sharing of enormous amounts of visual data. Recently, there has been progress in recovering information and making decisions based on these visual data, yet we are still limited in such developed methods. Algorithms that allow holistic and detailed understanding of visual data to the pixel-level, especially in videos, are beneficial in a broad range of applications. They are useful in the automotive industry (e.g., obstacle detection), social media videos analysis (e.g., retrieval and tagging) and medical video analysis (e.g., detection of abnormalities). My research targets the underpinning theoretical and engineering principles of pixel-level video understanding, which can partition the video to identify objects and their boundaries, estimate their motion, depth and actions. A corner stone in developing such algorithms is "learning", which allows computers to learn patterns from labelled datasets. This proposal focuses on Deep Learning, that trains deep architectures which can encode hierarchical representations from large-scale data. Part of deep learning, there is an emergence in foundation models that are pre-trained on broad datasets with tasks that enable generalization even when presented with few demonstrations. Although they are quite pervasive in textual data, such as ChatGPT series, video understanding foundation models are under-explored. Moreover, there is limited attention to their explainability which can impact complex and sensitive applications. My research program focuses on three main questions: i) How can we teach machines to understand videos by showing them only a few examples? (ii) What kind of tasks and data should we use to create powerful foundation models for video understanding? (iii) How can we understand and explain the decisions made by these models and the data used to train them? Key novelty comes about via investigating the aforementioned questions in pixel-level tasks which is currently under-explored. My research contributes to Canada in three important ways. First, it advances the discipline of computer vision allowing Canada to compete successfully in such a critical area. Second, it enables me to guide and mentor students, through hands-on experience resulting in skilled and successful personnel pursuing industry and/or academia. Third, in the recent trends of growing fear from artificial intelligence, it is important to tackle interpretability to assess the trustworthiness of the developed methods. Moreover, it is equally important to tackle data-efficiency to establish fairness for populations with access to limited labelled data towards decolonizing computer vision. Furthermore, part of my research program is concerned with social media videos as an application area that can protect Canadians from being manipulated by fake news and falsified information.