Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Toronto

$29,000.00

Declarative Query Processing Over Real Time Video Streams

EF 2023-24

Résumé
In the last few years, Deep Learning (DL) has become a dominant artificial intelligence (AI) technology in industry and academia. It has managed to revolutionize certain important practical applications. Video data abound; as of this writing 500 hours of video are uploaded on Youtube every minute. Numerous applications benefit from advanced techniques to process and understand video content ranging from video surveillance and video monitoring applications, to news production and autonomous driving. State of the art DL algorithms assess the presence of specific objects in an image, assess their properties (e.g. color, texture), their location relative to the frame coordinates as well as track an object from frame to frame delivering impressive accuracy. In order to deliver solid results however, state of the art object detection techniques are far from real time and require significant hardware resources. Current technology, enables us to extract a schema from a video applying video classification/detection and tracking algorithms at the frame level. This presents an opportunity for data management. Our research explores declarative query processing on streaming video sources utilizing the schema extracted from video. Deep learning primitives (e.g., object detection) can be realized as User Defined Functions (UDF's) in a declarative framework. Our research explores the efficient identification of objects relevant to a query on the video streams, utilizing query specific networks (filters) that are built on demand, as opposed to large and more expensive deep learning models. Processing frames with cheaper query specific models, can quickly remove frames not relevant to the query and can dramatically improve frame processing rate. We explore options for applying such filters and explore optimization trade offs over speed and accuracy. Spatio-temporal query processing on detected objects is of vast importance. Queries can be expressed using spatial (e.g., human in-front-of car, etc) and/or temporal (e.g., car next-to stop-sign for 10 minutes) constraints in monitoring applications. At the same time interactions between objects (e.g., human breaking glass) are prevalent. This research studies how such predicates (spatial, temporal, actions, interactions) can be expressed and evaluated in a streaming video scenario when objects detected in frames are involved. We plan to support general conjunctive normal form queries on such predicates involving video objects. Last but not least, we extend the framework to support multiple video streams, as opposed to a single video stream. This raises numerous interesting research directions in correlating and matching objects across streams under spatial and temporal constraints and associated optimizations. This research is  timely from a training perspective. It merges data management and deep learning primitives to build a real time video query processing system.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2020-07238
Name nom
Koudas, Nikolaos
Department département
Computer and Mathematical Sciences (Scarborough) - Computer and Mathematical Sciences (Scarborough)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2020
Award amount
29000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
9999
Research subject group en
Not available
Research subject group fr
Not available
Research subject en
Non disponible
Research subject fr
Non disponible
Application title
Declarative Query Processing Over Real Time Video Streams
Keyword
data management; data streams; deep learning; machine learning; operator scheduling; performance evaluation; query optimization; query processing; video monitoring; video query processing
Application summary
In the last few years, Deep Learning (DL) has become a dominant artificial intelligence (AI) technology in industry and academia. It has managed to revolutionize certain important practical applications. Video data abound; as of this writing 500 hours of video are uploaded on Youtube every minute. Numerous applications benefit from advanced techniques to process and understand video content ranging from video surveillance and video monitoring applications, to news production and autonomous driving. State of the art DL algorithms assess the presence of specific objects in an image, assess their properties (e.g. color, texture), their location relative to the frame coordinates as well as track an object from frame to frame delivering impressive accuracy. In order to deliver solid results however, state of the art object detection techniques are far from real time and require significant hardware resources. Current technology, enables us to extract a schema from a video applying video classification/detection and tracking algorithms at the frame level. This presents an opportunity for data management. Our research explores declarative query processing on streaming video sources utilizing the schema extracted from video. Deep learning primitives (e.g., object detection) can be realized as User Defined Functions (UDF's) in a declarative framework. Our research explores the efficient identification of objects relevant to a query on the video streams, utilizing query specific networks (filters) that are built on demand, as opposed to large and more expensive deep learning models. Processing frames with cheaper query specific models, can quickly remove frames not relevant to the query and can dramatically improve frame processing rate. We explore options for applying such filters and explore optimization trade offs over speed and accuracy. Spatio-temporal query processing on detected objects is of vast importance. Queries can be expressed using spatial (e.g., human in-front-of car, etc) and/or temporal (e.g., car next-to stop-sign for 10 minutes) constraints in monitoring applications. At the same time interactions between objects (e.g., human breaking glass) are prevalent. This research studies how such predicates (spatial, temporal, actions, interactions) can be expressed and evaluated in a streaming video scenario when objects detected in frames are involved. We plan to support general conjunctive normal form queries on such predicates involving video objects. Last but not least, we extend the framework to support multiple video streams, as opposed to a single video stream. This raises numerous interesting research directions in correlating and matching objects across streams under spatial and temporal constraints and associated optimizations. This research is  timely from a training perspective. It merges data management and deep learning primitives to build a real time video query processing system.