Grant: Discovery Grants Program - Individual

In this research program, we aim to explore new neural computing models to address some deficiencies of the current deep learning approaches for natural language understanding. First of all, we will study new methods to perform effective unsupervised or semi-supervised learning of neural networks for language understanding. The current successes of deep learning approaches largely rely on the popular error back-propagation based supervised learning algorithm, which requires a large amount of labeled data as prerequisites for any effective learning. Unlike speech and vision, it is much more challenging to collect human-labeled data for any language understanding tasks. On the other hand, we may easily have access to tons of unlabeled text from many sources on the Web. A successful neural computing model for natural language understanding needs to use a more effective unsupervised learning method to bootstrap itself from these unlabeled data, and then fine-tune towards a specific understanding task using only a small amount of labeled data. This sort of semi-supervised learning is a promising and viable approach to build natural language understanding systems in the near future. Secondly, the current neural networks are quite weak on long-term memory mechanisms, which are crucial in understating natural language. A same sentence may have very different meanings when appearing in various contexts, and even worse, the exact understanding of plain text normally stems from the background knowledge, which is not part of given text. In many cases, the given text simply serves as a trigger to retrieve some background knowledge or long-term memory to output as understanding. We will explore a novel approach to combine the neural networks based connectionist models with traditional symbolic approaches to address this issue since the symbolic approaches have shown tremendous advantages in reasoning over knowledge. The neural networks will be used as a flexible and powerful model to link surface text to background knowledge sources for effective reasoning. At last, we propose to investigate an open-domain question answer task as the main test bed for our research on both how to perform effective unsupervised learning and to combine connectionist models with traditional symbolic approaches.