Grant: Discovery Grants Program - Individual

This proposal focuses on visual attention and active observation asking first how do we as humans solve complex three-dimensional spatial tasks and second, can we create artificial agents to do the same? The kind of task we are considering, for example, is how does one determine if two objects are the same (3D same-different)? Another might be how to find a particular object in a large cluttered space (3D visual search). Naturally, sometimes colour, or shape reveals the answer immediately. But this is not always the case.  Sometimes your viewpoint makes all the difference (such as in well-known illusions or sculptures). This ability is used daily and is a core competency for real-world robotic assistants. Suppose you are asked to find a specific part or tool during an assembly task, found in a nearby bin (e.g., assembly of furniture). Children's building toys require one to perform this many times while constructing a block configuration, either copying from a plan, mimicking an existing one or building from one's imagination. Solving the easy cases are not so interesting; humans can do the hard ones too. This problem is not well studied. There are many experiments that test our visual ability when looking at something on a screen. However, humans did not evolve to categorize still cut-outs of their visual world on a screen, seated in a chair; they are active observers of their world. Making progress on this, both experimentally and computationally, is the key objective for this proposal. It is easy to conclude that attention must be involved; but how? Thus, my overall program has 3 themes: A) visual attention, B) its role in functional vision, and 3) its embodiment in intelligent agents. Regardless of biological or artificial realization, an intelligent agent must know the why, how, when, where, and what of its perceptually-guided behaviour. Deciding why you look (so you know what to do with what you see, i.e., task guidance), what to look at (to chose targets for your task, i.e., task-relevant candidates), when to look (at what time should I look at a task relevant target, i.e., sequences of attentive actions), where to look from (to get the right viewpoint, i.e., optimizing 3D imaging geometry), and how to look (to adapt sensory parameters, i.e., optimizing imaging sensitivity), is at least as important as the analysis of the seen image. We are working towards understanding these various capabilities, developing a theory to explain how they work in detail, and using that theory to develop intelligent agents that can be active assistants in the natural 3D world. There are both scientific and practical impacts of this work. On the science side we hope to develop algorithms that might codify how complex visual tasks might be solved that would generalize to more than the 1 or 2 tasks we will work on. On the practical side, the science will enable us to develop useful robotic assistants as industrial assembly or home caregiver assistants.