Grant: Discovery Grants Program - Individual
University of Waterloo
$31,000.00
Approaching 100 Percent Recall for Requirements and Software Engineering Tools
FY 2018-19
Summary
A hairy requirements or software engineering task involving natural language (NL) documents is one that is not inherently difficult for NL understanding humans on a small scale but becomes unmanageable in the large scale. Examples include identification of abstractions, ambiguities, synonyms, and trace links. A hairy task demands tool assistance. Because humans need far more help in carrying out a hairy task completely than they do in making the local yes-or-no decisions, a tool for a hairy task should have as close to 100% recall (that the tool finds all desired information) as possible, even at the expense of high imprecision (that not all the information that the tool finds is desired). A tool that falls much short of 100% recall may even be useless, because to find the missing desired information, a human has to do the entire task manually anyway.*Any such tool based on NL processing (NLP) techniques inherently fails to achieve 100% recall, because even the best parsers are no more than 91% correct. Therefore, for a tool to achieve 100% recall for a hairy task, it needs to be based on something other than traditional NLP.*The reality is that a tool's achieving exactly 100% recall, which may be impossible anyway, may not be necessary. It suffices for a human working with the tool on a task to achieve better recall than a human working on the task entirely manually.*The proposed research is to discover and test a variety of non-traditional approaches to building tools for hairy tasks to see which, if any, allows a human working with the tool to achieve better recall than a human working entirely manually.*If the research succeeds, we will be able to build tools for hairy tasks that demonstrably out-perform humans on these tasks. Therefore, when a software or requirements engineer is faced with one of these hairy tasks, he or she will trust the completeness of the output of the tool and will not feel compelled to do the same task manually.*
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2016-04029
Name nom
Berry, Daniel
Department département
Computer Science, David R. Cheriton School of
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2018
Competition year année de concours
2016
Award amount
31000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2720
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Computer systems software
Research subject fr
Logiciel de systèmes informatiques
Partie
3
Nb partie
5
Application title
Approaching 100 Percent Recall for Requirements and Software Engineering Tools
Keyword
NSERC
Application summary
A hairy requirements or software engineering task involving natural language (NL) documents is one that is not inherently difficult for NL understanding humans on a small scale but becomes unmanageable in the large scale. Examples include identification of abstractions, ambiguities, synonyms, and trace links. A hairy task demands tool assistance. Because humans need far more help in carrying out a hairy task completely than they do in making the local yes-or-no decisions, a tool for a hairy task should have as close to 100% recall (that the tool finds all desired information) as possible, even at the expense of high imprecision (that not all the information that the tool finds is desired). A tool that falls much short of 100% recall may even be useless, because to find the missing desired information, a human has to do the entire task manually anyway.*Any such tool based on NL processing (NLP) techniques inherently fails to achieve 100% recall, because even the best parsers are no more than 91% correct. Therefore, for a tool to achieve 100% recall for a hairy task, it needs to be based on something other than traditional NLP.*The reality is that a tool's achieving exactly 100% recall, which may be impossible anyway, may not be necessary. It suffices for a human working with the tool on a task to achieve better recall than a human working on the task entirely manually.*The proposed research is to discover and test a variety of non-traditional approaches to building tools for hairy tasks to see which, if any, allows a human working with the tool to achieve better recall than a human working entirely manually.*If the research succeeds, we will be able to build tools for hairy tasks that demonstrably out-perform humans on these tasks. Therefore, when a software or requirements engineer is faced with one of these hairy tasks, he or she will trust the completeness of the output of the tool and will not feel compelled to do the same task manually.*