Grant: Discovery Grants Program - Individual
University of Alberta
$29,000.00
Unsupervised Neural Text Generation by Stochastic Searching
EF 2021-22
Résumé
Natural language generation (NLG) is an important field of artificial intelligence. NLG aims to synthesize natural language text (e.g., sentences) in a variety of tasks, including text summarization, paraphrase generation, and dialogue systems. State-of-the-art NLG systems are based on deep neural networks, which typically synthesize a sentence by predicting one word at a time in an autoregressive fashion. Such approaches significantly outperform traditional rule/template-based NLG in terms of expressiveness and naturalness. However, existing neural NLG has two major drawbacks: 1) Neural networks are usually data-hungry. For example, millions of pairs of parallel sentences are required to train a neural translation system. 2) These methods suffer from the "error accumulation" problem, i.e., the quality of text could drop drastically as the generation proceeds, due to the autoregressive nature (e.g., left-to-right generation). The long-term goal of this proposed research is to investigate unsupervised approaches to text generation. In our previous studies, we tackled this problem by sampling from the probabilistic continuous latent space of a variational autoencoder. More recently, we proposed a novel Metropolis-Hastings (MH) sampler that directly samples a sentence from the discrete word space. In this way, text generation would be more data-efficient, and could be easily adapted to various real-world applications. Based on our previous work, this proposed research program would systematically explore unsupervised text generation by stochastic search, with the following short-term goals: 1) Development of stochastic searching algorithms. Despite our MH sampler, I plan to explore stochastic search algorithms, such as simulated annealing and genetic algorithms, because most NLG tasks are better formulated as a discrete optimization problem than sampling. Here, we would also design searching operations (e.g., word/phrase editing) suitable for text generation. They perform edits in a distributed way over the entire sentence, so our approach does not suffer from the "error accumulation" problem. 2) Applications of unsupervised text generation. Our searching framework provides a flexible way of text generation, because we can easily manipulate the searching objective function and also because such approach does not require parallel data for training. I plan to address a few important generation tasks in NLP, including text summarization, sentence simplification, and style-transfer text generation. 3) Combining searching and learning for text generation. I would like to integrate our search algorithms into a learnable model. On the one hand, a parametric learning model could not only smooth the manually defined searching objective, but also improve inference efficiency for sentence generation. On the other hand, the search procedure could also help train a learning machine, especially in the reinforcement learning setting.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2020-04465
Name nom
Mou, Lili
Department département
Department of Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2021
Competition year année de concours
2020
Award amount
29000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2801
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Natural language and speech understanding
Research subject fr
Langage naturel et reconnaissance de la parole
Installment
inst:2
Partie
1
Nb partie
5
Application title
Unsupervised Neural Text Generation by Stochastic Searching
Keyword
NSERC
Application summary
Natural language generation (NLG) is an important field of artificial intelligence. NLG aims to synthesize natural language text (e.g., sentences) in a variety of tasks, including text summarization, paraphrase generation, and dialogue systems. State-of-the-art NLG systems are based on deep neural networks, which typically synthesize a sentence by predicting one word at a time in an autoregressive fashion. Such approaches significantly outperform traditional rule/template-based NLG in terms of expressiveness and naturalness. However, existing neural NLG has two major drawbacks: 1) Neural networks are usually data-hungry. For example, millions of pairs of parallel sentences are required to train a neural translation system. 2) These methods suffer from the "error accumulation" problem, i.e., the quality of text could drop drastically as the generation proceeds, due to the autoregressive nature (e.g., left-to-right generation). The long-term goal of this proposed research is to investigate unsupervised approaches to text generation. In our previous studies, we tackled this problem by sampling from the probabilistic continuous latent space of a variational autoencoder. More recently, we proposed a novel Metropolis-Hastings (MH) sampler that directly samples a sentence from the discrete word space. In this way, text generation would be more data-efficient, and could be easily adapted to various real-world applications. Based on our previous work, this proposed research program would systematically explore unsupervised text generation by stochastic search, with the following short-term goals: 1) Development of stochastic searching algorithms. Despite our MH sampler, I plan to explore stochastic search algorithms, such as simulated annealing and genetic algorithms, because most NLG tasks are better formulated as a discrete optimization problem than sampling. Here, we would also design searching operations (e.g., word/phrase editing) suitable for text generation. They perform edits in a distributed way over the entire sentence, so our approach does not suffer from the "error accumulation" problem. 2) Applications of unsupervised text generation. Our searching framework provides a flexible way of text generation, because we can easily manipulate the searching objective function and also because such approach does not require parallel data for training. I plan to address a few important generation tasks in NLP, including text summarization, sentence simplification, and style-transfer text generation. 3) Combining searching and learning for text generation. I would like to integrate our search algorithms into a learnable model. On the one hand, a parametric learning model could not only smooth the manually defined searching objective, but also improve inference efficiency for sentence generation. On the other hand, the search procedure could also help train a learning machine, especially in the reinforcement learning setting.