Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Alberta

$48,000.00

Natural Language Interfaces to Databases

EF 2024-25

Résumé
Natural language interfaces to databases strive to make it easier for end users to access data in a relational database. For example, given the utterance "find employees who make more than their managers" and the schema of tables for employees and manages, one may want to generate a query in SQL that retrieves those employees from a database. This proposal aims to study the potentials and limitations of Large Language Models (LLMs) in various data management tasks including query generation, table understanding, and data integration and annotation. The particular challenges to be studied are: (1) expressing and decomposing different data management tasks within the limitations of LLMs, (2) Investigating the uncertainty of generated answers, and (3) evaluation of the models and verification of the results. These challenges and the relevant research questions are briefly discussed next. (1) LLMs have shown great potential in various data management tasks, such as query generation, table-based question answering and fact verification under zero-shot and few-shot settings. However, their performance depends on multiple factors, such as how complex the task is, the number and types of examples that are provided, how well the task is broken down and the relationships between different components and steps. The size of context for many LLMs is also limited, whereas the size of data in databases can be huge. We will study these challenges to gain a deeper understanding on how the performance of the model is affected by different factors, what principles govern the behaviour of the models on different data processing tasks, and how to overcome the limitations of the models in terms of the size of task description and the cost. (2) There is some uncertainty associated with the performance of LLMs, and this limits their deployment. In particular, the answers generated by LLM may or may not be correct, and neither the user nor the language model may know this. This raises a number of issues. One issue is understanding the variables that affect the uncertainty for different data processing tasks and if this uncertainty can be controlled or bounded. Another issue is quantifying the uncertainty. Even if the uncertainty cannot be controlled, understanding when the answers are uncertain and how this uncertainty is manifested in answers can be quite useful. (3) Evaluation is an important and critical part of measuring progress, and benchmarks have been developed for some tasks (e.g., text-to-SQL and question answering) but not for many new domains and applications. Also, the existing benchmarks may not be effective as the performance of the models get closer to the level of humans. We will study these issues that affect the evaluation and systematic approaches for addressing them. We will also study some of the principles behind evaluation such as the difficulty of verification, the space of possible answers, and the exhaustiveness of testing.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2024-04868
Name nom
Rafiei, Davood
Department département
Computer Science Department
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
48000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2711
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Database management
Research subject fr
Gestion de bases de données
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Natural Language Interfaces to Databases
Keywords
database systems; natural language interfaces to databases; novel database interfaces; database queries; text to data; data to text
Application summary
Natural language interfaces to databases strive to make it easier for end users to access data in a relational database. For example, given the utterance "find employees who make more than their managers" and the schema of tables for employees and manages, one may want to generate a query in SQL that retrieves those employees from a database. This proposal aims to study the potentials and limitations of Large Language Models (LLMs) in various data management tasks including query generation, table understanding, and data integration and annotation. The particular challenges to be studied are: (1) expressing and decomposing different data management tasks within the limitations of LLMs, (2) Investigating the uncertainty of generated answers, and (3) evaluation of the models and verification of the results. These challenges and the relevant research questions are briefly discussed next. (1) LLMs have shown great potential in various data management tasks, such as query generation, table-based question answering and fact verification under zero-shot and few-shot settings. However, their performance depends on multiple factors, such as how complex the task is, the number and types of examples that are provided, how well the task is broken down and the relationships between different components and steps. The size of context for many LLMs is also limited, whereas the size of data in databases can be huge. We will study these challenges to gain a deeper understanding on how the performance of the model is affected by different factors, what principles govern the behaviour of the models on different data processing tasks, and how to overcome the limitations of the models in terms of the size of task description and the cost. (2) There is some uncertainty associated with the performance of LLMs, and this limits their deployment. In particular, the answers generated by LLM may or may not be correct, and neither the user nor the language model may know this. This raises a number of issues. One issue is understanding the variables that affect the uncertainty for different data processing tasks and if this uncertainty can be controlled or bounded. Another issue is quantifying the uncertainty. Even if the uncertainty cannot be controlled, understanding when the answers are uncertain and how this uncertainty is manifested in answers can be quite useful. (3) Evaluation is an important and critical part of measuring progress, and benchmarks have been developed for some tasks (e.g., text-to-SQL and question answering) but not for many new domains and applications. Also, the existing benchmarks may not be effective as the performance of the models get closer to the level of humans. We will study these issues that affect the evaluation and systematic approaches for addressing them. We will also study some of the principles behind evaluation such as the difficulty of verification, the space of possible answers, and the exhaustiveness of testing.