Grant: Research Tools and Instruments

A fundamental challenge in the field of robotics is dexterous manipulation, which refers to the skillful manipulation of objects with a robotic hand with a level of dexterity akin to that of a human hand. Successfully solving this challenge would allow robots to interact with a broad variety of objects in real world unstructured environments. While it is relatively straightforward to develop specialist robotic hand controllers for manipulating particular objects, it is challenging to develop generalist controllers that work across a wide array of objects of different shapes and sizes, including those are previously unseen. Such controllers must therefore adapt their behaviour based on the 3D geometry of objects. Among the sensors that are typically onboard a robotic system, cameras are the most efficient at capturing rich information about the object and the environment. 3D geometry is not directly perceived by cameras, however, since cameras only capture 2D images. It had been difficult to infer explicit 3D geometry from 2D images, so in existing work, the controller either does not adapt to 3D geometry of objects at all, or assumes the true 3D geometry to be known. The former cannot handle objects with complex geometry that require skillful manipulation, and the latter does not function well outside of simulation. We recently developed a new technique known as Proximity Attention Point Rendering (PAPR) for inferring accurate 3D point clouds from real world 2D images from multiple viewing angles, which was presented at NeurIPS 2023 as a spotlight, and an extension to dynamic scenes, which was presented at CVPR 2024 as a highlight. In this proposal, we propose leveraging PAPR to develop controllers that can adapt to 3D geometry of objects in the real world and generalize across many objects, including those with complex geometry that are previously unseen. We propose several lines of research that tackle different aspects of this problem, namely generalization to unseen real world objects, selection of viewing angles and handling of deformable objects. Essential to all lines of research is the purchase of a robotic hand, for which we request support in this proposal. The proposed research would significantly advance the state-of-the-art in dexterous manipulation and open the way to versatile generalist home robots that can assist humans with a variety of daily tasks.