Grant: Discovery Launch Supplement
Université de Montréal
$12,500.00
Programming with AI Image Recognizers
EF 2024-25
Résumé
Programming computers is hard, requiring years of specialized training. To make programming easier, computer scientists have, over many decades, devised an array of tools and infrastructure -from debuggers, to integrated development environments, to social platforms like StackOverflow. Today, advances in artificial intelligence (AI) are transforming programming even further. With models such as ChatGPT, users can ask an AI about a programming problem by typing in natural language, and receive a knowledgable (but sometimes incorrect) response.
However, textual input is not always the optimal means of communication. Many disciplines employ visual notations to convey ideas and encode information -from quantum computer scientists, to data visualization experts, to game designers. Yet when they wish to convey their intent to a machine, often they are required to lay out their thoughts in sequences of text. What might the recent advances in multi-modal AI and machine learning mean for programming? Might AI technologies hold more radical potential to transform programming practice beyond the textual, typewritten status quo? As prior work has shown, the earliest programming languages were in fact handwritten, not typed, and showed a diversity of styles that broke with clear distinctions between the 'textual' and the 'visual.' With advances in computer vision and widespread hardware such as the Apple Pencil, we are due to question how programming practice might evolve accordingly.
This research will explore programming environments which incorporate visual notations and handwritten means of input, driven by new developments in machine learning (ML) technology. How might visual notations interact fluidly with natural language or textual code (for instance, sketching a website page while chatting with an AI)? How can we develop debugging techniques and pipelines to support and streamline such interfaces? Finally, what are the trade-offs between visual, non-typewritten means of communicating programmatic intent to a machine, and typing in a traditional, textual programming language?
To explore these questions, we will explore how to integrate AI image recognizers into programming workflows. To accomplish this, we will develop or apply computer vision-based AI recognizers that convert images (e.g., drawings, screenshots, diagrams, sketches, etc.) to internal textual representations that can then be manipulated either graphically or textually. Research will target two related input formats: images, and pen input ("notational programming"), such that users can insert diagrams or handwrite notation which inter-operates with a surrounding programming environment. Outcomes of this work will shape how future systems for programming with multi-modal foundation models are designed (e.g., interaction paradigms), enhance the efficiency of programming practice, and impact real people by deploying systems in the wild that hold up under real-world scenarios.
Programme
Discovery Launch Supplement
Bénéficiaire
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
DGECR-2024-00368
Name nom
Arawjo, Ian
Department département
Department of Computer Science and Operations Research
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
12500
Program id
DGECR
Program name en
Discovery Launch Supplement
Program name fr
Supplément Tremplin vers la découverte
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2705
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Software and development
Research subject fr
Logiciels et développement
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Programming with AI Image Recognizers
Keywords
human-computer interaction; programming environments; artificial intelligence; image recognition; pen-based computing; computer vision; programming languages; visual notation; quantum computer programming
Application summary
Programming computers is hard, requiring years of specialized training. To make programming easier, computer scientists have, over many decades, devised an array of tools and infrastructure -from debuggers, to integrated development environments, to social platforms like StackOverflow. Today, advances in artificial intelligence (AI) are transforming programming even further. With models such as ChatGPT, users can ask an AI about a programming problem by typing in natural language, and receive a knowledgable (but sometimes incorrect) response.
However, textual input is not always the optimal means of communication. Many disciplines employ visual notations to convey ideas and encode information -from quantum computer scientists, to data visualization experts, to game designers. Yet when they wish to convey their intent to a machine, often they are required to lay out their thoughts in sequences of text. What might the recent advances in multi-modal AI and machine learning mean for programming? Might AI technologies hold more radical potential to transform programming practice beyond the textual, typewritten status quo? As prior work has shown, the earliest programming languages were in fact handwritten, not typed, and showed a diversity of styles that broke with clear distinctions between the 'textual' and the 'visual.' With advances in computer vision and widespread hardware such as the Apple Pencil, we are due to question how programming practice might evolve accordingly.
This research will explore programming environments which incorporate visual notations and handwritten means of input, driven by new developments in machine learning (ML) technology. How might visual notations interact fluidly with natural language or textual code (for instance, sketching a website page while chatting with an AI)? How can we develop debugging techniques and pipelines to support and streamline such interfaces? Finally, what are the trade-offs between visual, non-typewritten means of communicating programmatic intent to a machine, and typing in a traditional, textual programming language?
To explore these questions, we will explore how to integrate AI image recognizers into programming workflows. To accomplish this, we will develop or apply computer vision-based AI recognizers that convert images (e.g., drawings, screenshots, diagrams, sketches, etc.) to internal textual representations that can then be manipulated either graphically or textually. Research will target two related input formats: images, and pen input ("notational programming"), such that users can insert diagrams or handwrite notation which inter-operates with a surrounding programming environment. Outcomes of this work will shape how future systems for programming with multi-modal foundation models are designed (e.g., interaction paradigms), enhance the efficiency of programming practice, and impact real people by deploying systems in the wild that hold up under real-world scenarios.