Grant: Discovery Grants Program - Individual

It has been argued that deep learning is the only viable approach to building artificial intelligence systems that can operate in complicated real-world environments. Deep learning is a powerful and robust framework which represents the real-world as a nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract representations computed in terms of less abstract ones. Sum-product networks (SPNs) are a deep learning model with tractable probabilistic inference. This is an attractive feature when compared to probabilistic graphical models in general, including Bayesian networks (BNs), where inference is intractable. Furthermore, a required step in learning is probabilistic inference. Darwiche, a leading expert in the field, proposed arithmetic circuits (ACs) as a deep learning model that can perform inference in linear time, but viewed ACs as being compiled from BNs. He showed that various reasoning tasks could be answered with an upward pass followed by a downward pass in an AC. ACs were extended as SPNs and viewed as a model on their own right that could be learned from data. An SPN is a rooted directed acyclic graph in which the leaf nodes are univariate distributions and all other nodes represent either summation or product operations. The representational differences between SPNs and ACs are that ACs use indicator nodes and parameter nodes as leaves, while SPNs attach all parameters to the outgoing edges of summation nodes. With this in mind, we use the terms ACs and SPNs interchangeably. The overarching themes of this proposal are to improve our understanding of semantics in SPN structure and inference and to utilize SPN properties in practical settings. Although SPNs have shown great promise, there is disagreement regarding the semantics of an SPN. Furthermore, there remain gaps in our understanding of the semantics of SPN inference, specifically in the upward pass of SPN inference. In practice, SPNs are commonly learned using expectation maximization. However, the Wasserstein distance is a measure used to learn another popular deep learning model, called generative adversarial networks. The Wasserstein distance has several theoretical advantages over other measures including expectation maximization and it learns more accurate generative adversarial networks in practice. We will show how to exploit the SPN completeness and decomposability properties to learn SPNs with Wasserstein. Finally, we will introduce an SPN architecture ideally suited as a natural ``one-stop shop'' to tackle critical practical issues in prediction, including managing missing data, model uncertainty, and data generation.