Grant: Discovery Grants Program - Individual
Simon Fraser University
$40,000.00
Building joint models of language and the 3D world
FY 2023-24
Summary
My research aims to connect natural language with 3D representations of the world. People live in a physical, three-dimensional world. We use natural language to talk about the world and our interactions with the world. Visual representations of the physical world and natural language are two critical, interconnected modalities for human communication. To create AI that can understand and assist us in our daily lives, we need to give machines the ability to talk about and understand the physical world. My aim is to develop algorithms and models that can allow an AI assistant (or robot) to resolve references such as "black mug on the dining table" to the 3D representation of the object. Moreover, the AI should understand the state change in the physical world associated with commands such as "take the mug from the table to the nightstand on the left of the bed". How do we develop AI that can talk with people about the real world? There are several research problems to address: understanding what people are referring to in a 3D environment, how to interact with the world and what language people use to describe interactions, and parsing complex dialogue in the world involving multiple agents. My research addresses these problems and explores fundamental questions that underpin a human-level understanding of the connection between language and the visual world. Progress in this challenging problem area can have immense impact. By tackling this problem, we can enable robotic assistants that can communicate with us using natural language about the world we live in. We can also make 2D and 3D content creation easier through language-driven UIs. Another application is in educational settings where we can use the learned mapping between language and visual concepts to teach foreign language speakers in virtual environments about how these concepts are carved out in a new language. Imagine, for example, a virtual reality game where you can look through your living room and see all the objects described in Spanish. Beyond these concrete applications, progress towards jointly understanding language and the physical world will form a foundation for general artificial intelligence, and is one ingredient for helping to combine machine perception and symbolic reasoning, a long-standing grand challenge problem in AI. Progress in this area will reinforce the strength and global leadership of Canadian academic institutions on topics related to artificial intelligence and machine learning. This proposal aims to build a joint model of language and the world by making the following contributions: 1) Build an online interactive simulation platform in which multiplayer dialogue can happen alongside manipulation of a 3D environment. 2) Investigate representations that jointly model the world and language. 3) Leverage these representations to design algorithms that can link language to the physical world.
Program
Discovery Grants Program - Individual
Recipient
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2020-07196
Name nom
Chang, Angel
Department département
School of Computing Science
Organization id
5
Institution établissement
Simon Fraser University
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2020
Award amount
40000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
9999
Research subject group en
Not available
Research subject group fr
Not available
Research subject en
Non disponible
Research subject fr
Non disponible
Application title
Building joint models of language and the 3D world
Keyword
3D representations; multimodal reasoning; natural language grounding; natural language understanding
Application summary
My research aims to connect natural language with 3D representations of the world. People live in a physical, three-dimensional world. We use natural language to talk about the world and our interactions with the world. Visual representations of the physical world and natural language are two critical, interconnected modalities for human communication. To create AI that can understand and assist us in our daily lives, we need to give machines the ability to talk about and understand the physical world. My aim is to develop algorithms and models that can allow an AI assistant (or robot) to resolve references such as "black mug on the dining table" to the 3D representation of the object. Moreover, the AI should understand the state change in the physical world associated with commands such as "take the mug from the table to the nightstand on the left of the bed". How do we develop AI that can talk with people about the real world? There are several research problems to address: understanding what people are referring to in a 3D environment, how to interact with the world and what language people use to describe interactions, and parsing complex dialogue in the world involving multiple agents. My research addresses these problems and explores fundamental questions that underpin a human-level understanding of the connection between language and the visual world. Progress in this challenging problem area can have immense impact. By tackling this problem, we can enable robotic assistants that can communicate with us using natural language about the world we live in. We can also make 2D and 3D content creation easier through language-driven UIs. Another application is in educational settings where we can use the learned mapping between language and visual concepts to teach foreign language speakers in virtual environments about how these concepts are carved out in a new language. Imagine, for example, a virtual reality game where you can look through your living room and see all the objects described in Spanish. Beyond these concrete applications, progress towards jointly understanding language and the physical world will form a foundation for general artificial intelligence, and is one ingredient for helping to combine machine perception and symbolic reasoning, a long-standing grand challenge problem in AI. Progress in this area will reinforce the strength and global leadership of Canadian academic institutions on topics related to artificial intelligence and machine learning. This proposal aims to build a joint model of language and the world by making the following contributions: 1) Build an online interactive simulation platform in which multiplayer dialogue can happen alongside manipulation of a 3D environment. 2) Investigate representations that jointly model the world and language. 3) Leverage these representations to design algorithms that can link language to the physical world.