Grant: Discovery Grants Program - Individual

Training artificial intelligence (AI) models requires collecting and annotating (i.e., labeling) large data sets, but enterprises implementing AI often face data challenges such as not collecting enough data, realizing poor annotations, and mismanaging annotation workers. This is most pressing for computer vision companies with massive image data sets processed by thousands of workers. Machine learning (ML) research has studied what data can improve a model's downstream accuracy, but there has been little work on how to manage the operations of the data-centric pipeline to reduce costs and workflow delays. As such, engineering managers often make ad hoc decisions that can waste resources or in the worst case, prevent deployment. Our long-term goal is an operations research (OR) framework of data-centric AI, which can optimize management-level metrics and bring much-needed guidance to managers. We previously optimized how much data to collect and what to annotate with limited resources. This current proposal optimizes data quality, labour costs, and delays from annotation. First, image data sets are often annotated by teams of workers that sequentially perform tasks such as annotating a specific part of each data point or error-checking prior work. Here, the assignment problem of determining which tasks to assign to which worker in which sequence faces combinatorial growth. Moreover to handle multiple data sources, companies must staff workers and schedule them to different priority groups. Finally, emerging applications in medicine or manufacturing feature small data sets and worker pools, wherein optimizing what to annotate and who to assign can be susceptible to erroneous estimates of data value and worker reliability. Our research program combines ML methods (e.g., belief propagation, active learning) that estimate the value of data points with OR models (e.g., optimization, queuing) to generate decisions. We can address the task assignment problem by repeatedly estimating an annotator's reliability and an image's annotation as functions of the task, annotator, and prior version of the image, and using integer optimization to assign workers to tasks. To solve the staffing and scheduling problems, we can extend active learning to prioritize incoming data and model the flow of the data streams as queues before determining how many workers to staff and how to schedule them. Finally, we can develop robust optimization models for active learning and task assignment to account for the uncertainty present in the small-data regime. Ultimately, our research program can further cement Canada's unique position as a worldwide research leader in ML and OR: (i) by bridging the research gap between data-centric AI and management science; (ii) by introducing new applications of OR models to AI deployment; and (iii) by producing operational guidelines for AI developers.