Grant: Discovery Grants Program - Individual
University of Saskatchewan
$24,000.00
Efficient and Reproducible Image Annotation for Supervised Deep Learning with Small Data
EF 2023-24
Résumé
Machine learning is currently one of the hottest and most quickly evolving technologies. Everyone wants to use "deep learning" to develop systems for everything: face recognition, identity tracking, quality control in manufacturing, tracking objects in videos, diagnosing disease, precision agriculture, etc. Deep learning works best when there are very large numbers of images where the "correct answer" is known from which the machine learning algorithm can learn. It is relatively easy for people to tag an image of a car, a boat, or a plane, or even different types of cars with an appropriate label. It is much more challenging to create large datasets to train systems for diagnosing cancer, or counting the number of flowers on a plant. Labeling medical images requires highly experienced experts. When marking all of the flowers on a plant, it is easy to miss one, or for a flower to be partially occluded by a leaf or another flower, or to be imprecise in specifying a flower's exact location. Moreover, in order to obtain enough annotated images to learn from, typically multiple annotators label a dataset but each image is annotated by only one annotator. In such circumstances we have no way of determining whether different annotators are annotating consistently, that is, we can't quantify the inter-annotator agreement. Annotator disagreement arises from biases in their annotations, and degrades the quality of the training dataset. Inter-rater agreement is affected by the difficulty of the annotation task and the nature of the instructions given to annotators. If we can find ways of obtaining more consistent annotations across multiple annotators, training dataset quality will improve, and hence the performance of the learned system also improves. The proposed research program will study how to obtain better quality annotations from annotators with higher inter- and intra-annotator agreement. We will create augmented annotation tools that provide problem-specific semi-automation to assist annotators and quantify the resulting benefits to annotator agreement and trained system performance. We will quantify the relationship between annotator agreement and model performance. We will explore the degree to which contextual factors such as annotation type, instructions given, pressure, and distractions can influence annotator agreement and develop best practices for mitigating their effects. By studying factors that influence annotator agreement and the performance of the systems that are learned from annotated datasets, we will be able to develop new standardized methodologies for training "deep learning" models with limited data. This will allow better prediction of the optimal amount of resources to invest in annotation, reduce the reliance on trial-and-error methods to obtain the best trained system performance, and make successful machine learning less reliant on deep technical expertise.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Saskatchewan, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2021-02428
Name nom
Eramian, Mark
Department département
Computer Science
Organization id
16
Institution établissement
University of Saskatchewan
Province en
Saskatchewan
Province fr
Saskatchewan
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2021
Award amount
24000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2708
Research subject group en
Information technology
Research subject group fr
Image processing
Research subject en
Technologie de l'information
Research subject fr
Traitement des images
Application title
Efficient and Reproducible Image Annotation for Supervised Deep Learning with Small Data
Keyword
annotation; annotator agreement; applications; computer vision; deep learning; small data
Application summary
Machine learning is currently one of the hottest and most quickly evolving technologies. Everyone wants to use "deep learning" to develop systems for everything: face recognition, identity tracking, quality control in manufacturing, tracking objects in videos, diagnosing disease, precision agriculture, etc. Deep learning works best when there are very large numbers of images where the "correct answer" is known from which the machine learning algorithm can learn. It is relatively easy for people to tag an image of a car, a boat, or a plane, or even different types of cars with an appropriate label. It is much more challenging to create large datasets to train systems for diagnosing cancer, or counting the number of flowers on a plant. Labeling medical images requires highly experienced experts. When marking all of the flowers on a plant, it is easy to miss one, or for a flower to be partially occluded by a leaf or another flower, or to be imprecise in specifying a flower's exact location. Moreover, in order to obtain enough annotated images to learn from, typically multiple annotators label a dataset but each image is annotated by only one annotator. In such circumstances we have no way of determining whether different annotators are annotating consistently, that is, we can't quantify the inter-annotator agreement. Annotator disagreement arises from biases in their annotations, and degrades the quality of the training dataset. Inter-rater agreement is affected by the difficulty of the annotation task and the nature of the instructions given to annotators. If we can find ways of obtaining more consistent annotations across multiple annotators, training dataset quality will improve, and hence the performance of the learned system also improves. The proposed research program will study how to obtain better quality annotations from annotators with higher inter- and intra-annotator agreement. We will create augmented annotation tools that provide problem-specific semi-automation to assist annotators and quantify the resulting benefits to annotator agreement and trained system performance. We will quantify the relationship between annotator agreement and model performance. We will explore the degree to which contextual factors such as annotation type, instructions given, pressure, and distractions can influence annotator agreement and develop best practices for mitigating their effects. By studying factors that influence annotator agreement and the performance of the systems that are learned from annotated datasets, we will be able to develop new standardized methodologies for training "deep learning" models with limited data. This will allow better prediction of the optimal amount of resources to invest in annotation, reduce the reliance on trial-and-error methods to obtain the best trained system performance, and make successful machine learning less reliant on deep technical expertise.