Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

University of Waterloo

$26,000.00

Machine Understanding of Software Use Through Text Corpora

FY 2014-15

Summary
The web is home to vast repositories of instructional content for software use. This content can be found in manuals, tutorials, forums, and videos. Embedded within this content is a wide range of knowledge: procedural knowledge (the steps necessary to achieve an outcome), alternative workflows (where a workflow represents one specific strategy and sequence of steps for solving a problem), recommendations for parameter settings, and advice on best practices. While instructional materials can be located via search engines, existing software has few capabilities for understanding, manipulating, or directly applying the content of instructional resources. Instead, users must select, interpret, and transform these materials to their particular needs. This manual process is error-prone and time consuming. The long-term aim of this research is to develop techniques that enable machine understanding of instructional content to make possible intelligent user interfaces that can understand and act on natural language instructions. Specifically, I will create methods for transforming instructional content (documents, images, and videos) into structured knowledge bases that enable systems to directly utilize this information within mixed-initiative and emerging ""natural user interfaces"". These systems will apprentice themselves by ""reading"" the web, then produce knowledge bases that map natural language goals to specific sequences of operations and problem solving strategies optimized for each task. The research will initially focus on text-based instructions, then consider the problem of instructional videos, which introduce unique additional challenges. As an example of the utility of the outputs of this research, consider the following scenario. Imagine a user wishes to use Photoshop or GIMP (an open source alternative to Photoshop) to ""selectively color"" an image (i.e., make some parts of the image greyscale, and other parts color). In systems envisioned by this research, a user first communicates this goal to the software using natural language (e.g., by typing or speaking their goal). Our knowledge bases will then be able to translate that goal to a set of learned sequences of operations, tools, and best practices tuned to solving that specific problem. With this information, the application can customize the workspace for this task, preprocess the data for the user, and present the user with a set of alternative solution paths from which to choose. For example, the software could automatically produce a set of alternative greyscale transformations of the source image for the user to choose from, set up layer masks to support selective exposure of portions of the original color image, and remove all tools and commands from the interface except the ones needed to complete the task. These operations would not only save the user the time and tedium of preparing the work environment and data, they also set up the solution space using best practices, namely, by employing a series of non-destructive operations that provide the user the greatest flexibility for long-term, iterative problem solving. Thus, the user can also learn optimal problem solving strategies by example. Critically, all of these operations and best practices can be learned through existing online instructional materials?what is missing are methods for computers to learn this information (the focus of this proposal).
Program
Discovery Grants Program - Individual
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2014-06432
Name nom
Terry, Michael
Department département
School of Computer Science, Cheriton
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2014
Competition year année de concours
2014
Award amount
26000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2710
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information systems design
Research subject fr
Conception de systèmes d'information
Partie
1
Nb partie
5
Application title
Machine Understanding of Software Use Through Text Corpora
Keyword
NSERC
Application summary
The web is home to vast repositories of instructional content for software use. This content can be found in manuals, tutorials, forums, and videos. Embedded within this content is a wide range of knowledge: procedural knowledge (the steps necessary to achieve an outcome), alternative workflows (where a workflow represents one specific strategy and sequence of steps for solving a problem), recommendations for parameter settings, and advice on best practices. While instructional materials can be located via search engines, existing software has few capabilities for understanding, manipulating, or directly applying the content of instructional resources. Instead, users must select, interpret, and transform these materials to their particular needs. This manual process is error-prone and time consuming. The long-term aim of this research is to develop techniques that enable machine understanding of instructional content to make possible intelligent user interfaces that can understand and act on natural language instructions. Specifically, I will create methods for transforming instructional content (documents, images, and videos) into structured knowledge bases that enable systems to directly utilize this information within mixed-initiative and emerging ""natural user interfaces"". These systems will apprentice themselves by ""reading"" the web, then produce knowledge bases that map natural language goals to specific sequences of operations and problem solving strategies optimized for each task. The research will initially focus on text-based instructions, then consider the problem of instructional videos, which introduce unique additional challenges. As an example of the utility of the outputs of this research, consider the following scenario. Imagine a user wishes to use Photoshop or GIMP (an open source alternative to Photoshop) to ""selectively color"" an image (i.e., make some parts of the image greyscale, and other parts color). In systems envisioned by this research, a user first communicates this goal to the software using natural language (e.g., by typing or speaking their goal). Our knowledge bases will then be able to translate that goal to a set of learned sequences of operations, tools, and best practices tuned to solving that specific problem. With this information, the application can customize the workspace for this task, preprocess the data for the user, and present the user with a set of alternative solution paths from which to choose. For example, the software could automatically produce a set of alternative greyscale transformations of the source image for the user to choose from, set up layer masks to support selective exposure of portions of the original color image, and remove all tools and commands from the interface except the ones needed to complete the task. These operations would not only save the user the time and tedium of preparing the work environment and data, they also set up the solution space using best practices, namely, by employing a series of non-destructive operations that provide the user the greatest flexibility for long-term, iterative problem solving. Thus, the user can also learn optimal problem solving strategies by example. Critically, all of these operations and best practices can be learned through existing online instructional materials?what is missing are methods for computers to learn this information (the focus of this proposal).