Grant: Discovery Grants Program - Individual

Due to the development of electronic technologies, high definition TV and broadcasting have become common in most countries. All the needed facilities and hardware are there ready for 3D and high-resolution videos, but we lack 3D and high-resolution video content. Most existing video content is standard definition video. It would be extremely important, practical, and cost-effective if we could convert existing standard resolution video to high and ultra-high resolution video. Super-resolution (SR) is the technique to generate a high-resolution (HR) image/video with sharp and clear details from its original low-resolution (LR) image/video. In this proposed program, we will research on advanced algorithms on 2D/3D video super-resolution. The chroma-keying is the most widely used compositing technique in photography processing, film making, and television production. Seemingly easy, it is actually extremely difficult to generate satisfactory keying results in different types of studios and for different application setups, due to the uneven background, shading, lighting, transparency, reflection, etc. To date, there are still many difficult issues that need to be solved for 2D image/video keying, and very little has been done to 3D video keying in both academia and industry. Based on the knowledge and experience we have obtained in 2D keying, in this program we will research and develop advanced 3D video keying algorithms. In the proposed research, depth-map generation from 2D and 3D video will be studied, which will enable 2D/3D video conversion and color-depth 3D representation, and provide vital information to chroma-keying and super-resolution, The quality evaluation of super-resolved HR image/video, the chroma-keying, depth-maps, and 3D video will also be studied. Chroma-keying and depth-map generation are mutually dependent. Super-resolution (SR) improves both chroma-keying and depth-map generation; in turn chroma-keying and depth-map can guide super-resolution. Quality evaluation of super-resolution, chroma-keying, depth-map and 3D video is critically important. To respect their own importance and wide applications, we clearly enumerate the four enabling components of 3D video processing in the title: super-resolution, chroma-keying, depth-map generation, and quality evaluation. Just in recent several years, the rapid development of neural network techniques has provided new possibilities for many research fields such as super-resolution, alpha-matting, depth-map generation, and video quality evaluation. In this research program, in addition to conventional techniques, we will also research and develop deep learning based techniques. The proposed research will definitely benefit the academia and industry in autonomous systems & robotics, video broadcasting, film making, image segmentation, object extraction, HDTV, remote sensing, medical imaging, 3D video rendering & coding, etc.