Grant: Discovery Grants Program - Individual

This proposal extends the current deep learning paradigm in NLP from narrow implementations of monolithic systems that are specially crafted for a specific task and dataset towards systems that encompass achievements of previous systems, that can capitalize on existing ontological resources developed mainly for human users, and that can combine different neural architectures for deeper and more complex analysis. To this end, we have developed the DAM architecture, which allows easy incremental additions of modules representing these ontological, grammatical, or legacy system resources. The potentially large number of modules is managed by a recurrent controller, that selects the k modules likely to contribute most (where k is a system parameter). Extensive experiments have shown that many disparate modules can in fact interoperate successfully even when they overlap or contradict each other. System analysis has shown that the sparsity enforced by a suitably small value of k effectively contains the parameter space below the size of other, comparable systems. These advantages will be further explored, the entire architecture will undergo increasingly challenging stress tests. The long-term objective of this research is to deepen our understanding of current neural architectures and how to expand them to accumulate knowledge from outside, static resources or from previous machine learning systems. Successive system prototypes will be made available as open source projects. Various student systems will contribute further modules and application extensions. The entire project aims at transfer and continuous learning behaviour.