Grant: Discovery Grants Program - Individual

Representation learning, enabled by the modeling and computational benefits of neural networks, has revolutionized the field of natural language processing (NLP). A general model that is applicable to many different sub-areas of NLP is the encoder-decoder neural network. An informal survey of all the papers published in the ACL 2017 conference (the most prestigious international conference in NLP) showed that one in four papers used encoder-decoder networks. This is an unprecedented amount of conformity for NLP researchers even though this model was used for many different NLP tasks. As a result, the focus of this proposal is likely to benefit a large number of NLP researchers and industry practitioners. Encoder-decoder networks with attention have been applied to several tasks: neural machine translation (NMT), language models, summarization, captioning of images, neural Turing machines (modeling the semantics of simple programming languages), visual and non-visual question answering systems, dialog systems, and many other applications. Further research into encoder-decoder networks is likely to have a significant impact broadly in NLP research. The addition of attention models to encoder-decoder networks has been an essential improvement, resulting in radically improved accuracies in many different tasks. Currently attention models use relatively simple weighted average models. There is considerable room for improvements to attention models in order to model various complex dependencies in natural language and in order to expand their utility to new NLP tasks and settings. In this proposal we explore various improvements to the encoder-decoder network models that have broad applicability to natural language processing and machine learning.