Grant: Discovery Grants Program - Individual

While significant progress has been achieved in 3D modeling of objects for video games, numerous other applications such as extended reality-based interactive environments for training, virtual prototyping for precision-sensitive industrial computer-aided-design-based processes, and complex interactive robot applications such as dexterous manipulation in robotic assembly, tele-robotics, medical and assistive robotics, still mainly rely on simplistic models of 3D objects; when in fact numerous real-world objects are mostly unsymmetrical, compliant, and exhibit alterable shapes. To fully reach their potential, such applications must evolve toward realistic and accurate representations of objects based on real measurements instead of hard-coded, approximate computer-generated models. Rising up to this challenge, the long-term goal of this research program is to develop enhanced sensing and modeling techniques for applications that require higher realism. This will be achieved by means of innovative AI-powered plausible 3D object representations built upon real multisensory measurements collected on physical deformable objects, along with their integration into practical applications. More specifically, inspiration from human visual and touch sensing, optimal sampling strategies using various sensors, computer vision algorithms and AI techniques will be adapted and refined for use on real measurements. They will be strategically combined for the formulation of novel object representations based on multisensory visual, shape, tactile and audio data. Original AI methods will be proposed to efficiently extract abstract knowledge from various sources of information to overcome the bottleneck in the measurement of kinesthetic and tactile object properties. The methods will be designed to support seamless adaptation and merge of multisensory models in interactive environments. The models will be integrated and validated in the context of practical applications, including for virtual-reality based cognitive rehabilitation and for safe data-centric guidance of deformable objects manipulation with robots. The proposed program will contribute the theoretical and practical knowledge required to build the next generation of immersive data-centric virtual and extended reality applications that will rely on plausible representations of 3D real-world objects. Advancements in this area of research will have a significant impact on applications where enhanced human-machine interaction is desired and that require high accuracy and improved realism. This research work will support the training of highly qualified specialists with cutting edge approaches and technologies, who are in demand for the strong multimedia, gaming, robotics, and virtual reality industry in Canada.