Grant: Idea to Innovation

Deaf and hearing impaired rely on caption to be informed and enjoy TV and videos. Skilled people (captioners) edit caption to establish accuracy, clarity and proper reading rate. Caption can be produced either on-line during the broadcast or off-line. For off-line captioning, captioners have to place caption based on their assessment of the value of the visual information. Typically, they place the caption not too far from the source of speech (i.e. the speakers) and try not masking any visual element that may be relevant to the understanding of the content. This task can be labor-intensive and could require up to 18 hours to caption one hour of content. This project aims at developing an intelligent tool to assist off-line captioners to reduce the production time by automatically suggesting a position for the captions on a number of consecutive frames using a set of production rules based on automatic visual content detection (faces, motion activity area, text, banners) and speech/non-speech segmentation.