Grant: Discovery Grants Program - Individual
University of Waterloo
$32,000.00
Predictive Modeling of User Performance for Validated Effectiveness Measures of Search Engine Quality
EF 2014-15
Résumé
Effectiveness measures play a critical role in the development of new search engine algorithms. The effectiveness measures in wide use today have little or no explicit model of user behavior, no model of the user interface, and no estimate of the time it takes a user to find relevant information --- all user actions are falsely assumed to take the same amount of time and all users are falsely assumed to behave the same. As a result, these measures only produce scores that are correlated with user performance. In a recent breakthrough, we have produced an effectiveness measure that explicitly models the user, the user interface, and is calibrated with the time user actions take. This new effectiveness measure, time-biased gain, directly produces validated estimates of user performance and can even estimate the distribution of performance by modeling the different abilities and probabilistic actions of users. My research program studies user behavior with search engines and then tries to model this behavior for use in new effectiveness measures, as well as, to help produce new ways of interacting with the user to improve people's ability to satisfy their information needs. The next 5 years of my research program will focus on the further development of effectiveness measures with a long term objective of making the evaluation of search engine quality synonymous with the estimation of user performance. Previous to the results of my research program, user performance was primarily estimated through expensive and rarely conducted laboratory user studies with human participants; the developers of new retrieval algorithms were required to optimize their methods against the proxy values produced by existing measures. Specific objectives of my research program include i) accurately modeling multiple-query sessions, ii) creating better models of the time spent searching, and iii) making time-biased gain and other validated effectiveness measures a complete solution for researchers and engineers. The impact of this research will be widely felt and far reaching. Search engines have become an integral part of Canada's knowledge-based economy. We use search engines to help us in nearly endless ways. Engineers can find answers to their detailed questions in minutes rather than days, doctors as well as patients use search engines to research medical treatments for life-threatening illnesses, and lawyers depend on search engines to find past cases with legal e-discovery becoming an industry in itself. Better effectiveness measures, where better means more accurate predictions of user performance, will allow researchers and engineers to speed the rate at which they improve retrieval algorithms. In building better effectiveness measures, we will produce fundamental knowledge about human information processing as well the decision making processes of humans when faced with uncertainty, and this knowledge should have application in the fields of psychology, management sciences, and of course, the improvement of other knowledge-based software applications. The proposed research program will provide ample opportunities for students to gain experience in high value areas such as modeling and prediction of human behavior as well as large scale data processing.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2014-03642
Name nom
Smucker, Mark
Department département
Management Sciences
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2014
Competition year année de concours
2014
Award amount
32000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2700
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information technology
Research subject fr
Technologie de l'information
Partie
1
Nb partie
5
Application title
Predictive Modeling of User Performance for Validated Effectiveness Measures of Search Engine Quality
Keyword
NSERC
Application summary
Effectiveness measures play a critical role in the development of new search engine algorithms. The effectiveness measures in wide use today have little or no explicit model of user behavior, no model of the user interface, and no estimate of the time it takes a user to find relevant information --- all user actions are falsely assumed to take the same amount of time and all users are falsely assumed to behave the same. As a result, these measures only produce scores that are correlated with user performance. In a recent breakthrough, we have produced an effectiveness measure that explicitly models the user, the user interface, and is calibrated with the time user actions take. This new effectiveness measure, time-biased gain, directly produces validated estimates of user performance and can even estimate the distribution of performance by modeling the different abilities and probabilistic actions of users. My research program studies user behavior with search engines and then tries to model this behavior for use in new effectiveness measures, as well as, to help produce new ways of interacting with the user to improve people's ability to satisfy their information needs. The next 5 years of my research program will focus on the further development of effectiveness measures with a long term objective of making the evaluation of search engine quality synonymous with the estimation of user performance. Previous to the results of my research program, user performance was primarily estimated through expensive and rarely conducted laboratory user studies with human participants; the developers of new retrieval algorithms were required to optimize their methods against the proxy values produced by existing measures. Specific objectives of my research program include i) accurately modeling multiple-query sessions, ii) creating better models of the time spent searching, and iii) making time-biased gain and other validated effectiveness measures a complete solution for researchers and engineers. The impact of this research will be widely felt and far reaching. Search engines have become an integral part of Canada's knowledge-based economy. We use search engines to help us in nearly endless ways. Engineers can find answers to their detailed questions in minutes rather than days, doctors as well as patients use search engines to research medical treatments for life-threatening illnesses, and lawyers depend on search engines to find past cases with legal e-discovery becoming an industry in itself. Better effectiveness measures, where better means more accurate predictions of user performance, will allow researchers and engineers to speed the rate at which they improve retrieval algorithms. In building better effectiveness measures, we will produce fundamental knowledge about human information processing as well the decision making processes of humans when faced with uncertainty, and this knowledge should have application in the fields of psychology, management sciences, and of course, the improvement of other knowledge-based software applications. The proposed research program will provide ample opportunities for students to gain experience in high value areas such as modeling and prediction of human behavior as well as large scale data processing.