Grant: Discovery Grants Program - Individual

This proposal addresses the challenge of understanding group activity in videos, within the constraints of sports videos. The video can come from one or more static or moving cameras, and can be accompanied by commentary, annotation, or simply audio such as the play-by-play. The results of our system will aid coaches by helping in player evaluation and by identifying how teams operate in a variety of situations. More importantly, fans demand more game information online and in statistical summaries. To support these needs we will explore customizable interaction and visualization. Our goal is generate information to fill all these needs from the video.Activity understanding in video is of broad interest, ranging from interpreting videos of unconstrained events to very focused videos. Much information can be extracted from the video: the position of objects or persons, the pose of persons, as well as motion of objects or people. Coordinated motions suggest purposive actions and cause interactions with objects and people. Collections of actions form activities.We study sports because each sport has a limited action set with specific roles under the rules of the game and group activity has explicit purposes. We use videos from broadcast cameras so that amateurs can take advantage of our advances. We will refine our abilities to track and identify players, describe their pose and their actions. Many of these advances will rely on deep learning and recurrent neural networks to capture the broad range of actions and activities. The challenge is how to represent the structure of the game and the necessary information such as player appearance, styles, and roles, physical constraints and game semantics.We will integrate event descriptions from several viewpoints and will continue our work in camera planning so that several cameras can follow the movements of the players from different viewpoints and ranges of view (closeups). We need richer representations of game semantics - the human camera operator is telling a story requiring attention to salient game events, hence closeup shots depend on game evolution. We intend also to explore textual input, even game commentary, and naturally will explore text generation - reports for coaches and fans, as computer vision is rapidly expanding its ability to annotate images and videos with narratives. The methods we devise will generalize beyond sports to unrestricted videos.