Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Collaborative Research and Development Grants

University of Waterloo

$64,504.00

User behavior models for information access evaluation

FY 2017-18

Summary
To improve search engines, and other information access systems, their performance must be measured in human terms. A key measure is the time taken to find required information and to complete the task at hand. Whether a user is searching for a single fact, or is researching a complex problem, the user wants to satisfy their information as fast as possible. Both researchers and engineers need a means to quickly and affordably estimate the rate at which users can satisfy their information needs with a given search engine. To produce accurate estimates of user performance, effectiveness measures must be designed that can adjust to different user tasks and computing devices. For example, users searching for a simple fact are expected to behave differently than users researching a complex problem. Likewise, the device used to access information can have a significant impact on how a user behaves. For mobile search, device constraints such as a small screen and a virtual keyboard need to be taken into consideration. Traditional effectiveness measures are ill-suited to estimate user effectiveness across different user tasks and access devices given their lack of user behavior models, which prevents adapting them to the search context. This work proposes to further develop the time-biased gain effectiveness measure, which can be adapted to the search context.The research undertaken by this project builds on existing work by Professors Smucker and Clarke. This existing work proposes and develops time-biased gain (TBG) effectiveness measures for evaluating search engine performance. Unlike traditional measures of search engine performance, TBG makes time a key part of the measure. Computations of TBG are calibrated to times and decision processes extracted from usage logs and user studies. As changes are made to a search engine's ranking algorithms, interface, and other components, TBG may be used to predict the impact of these changes on user performance. In order to advance this work to more complex scenarios and domains, the research will incorporate insights uncovered by Professor Cormack into relevance prediction. NSERC funding under this project serves to extend an existing Google Focus Award on context-aware mobile social networking. While developing technologies under that proposal, a need for improved evaluation methodologies became apparent. In addition to supporting the needs of this Google Focus Award, this project advances our general understanding of information access evaluation.
Program
Collaborative Research and Development Grants
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
468812-2014
Name nom
Smucker, Mark
Department département
Management Sciences
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2017
Competition year année de concours
2014
Award amount
64504
Program id
CRDPJ
Program name en
Collaborative Research and Development Grants
Program name fr
Subventions de recherche et développement coopérative
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2700
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information technology
Research subject fr
Technologie de l'information
Partie
3
Nb partie
3
Application title
User behavior models for information access evaluation
Keyword
NSERC
Application summary
To improve search engines, and other information access systems, their performance must be measured in human terms. A key measure is the time taken to find required information and to complete the task at hand. Whether a user is searching for a single fact, or is researching a complex problem, the user wants to satisfy their information as fast as possible. Both researchers and engineers need a means to quickly and affordably estimate the rate at which users can satisfy their information needs with a given search engine. To produce accurate estimates of user performance, effectiveness measures must be designed that can adjust to different user tasks and computing devices. For example, users searching for a simple fact are expected to behave differently than users researching a complex problem. Likewise, the device used to access information can have a significant impact on how a user behaves. For mobile search, device constraints such as a small screen and a virtual keyboard need to be taken into consideration. Traditional effectiveness measures are ill-suited to estimate user effectiveness across different user tasks and access devices given their lack of user behavior models, which prevents adapting them to the search context. This work proposes to further develop the time-biased gain effectiveness measure, which can be adapted to the search context.The research undertaken by this project builds on existing work by Professors Smucker and Clarke. This existing work proposes and develops time-biased gain (TBG) effectiveness measures for evaluating search engine performance. Unlike traditional measures of search engine performance, TBG makes time a key part of the measure. Computations of TBG are calibrated to times and decision processes extracted from usage logs and user studies. As changes are made to a search engine's ranking algorithms, interface, and other components, TBG may be used to predict the impact of these changes on user performance. In order to advance this work to more complex scenarios and domains, the research will incorporate insights uncovered by Professor Cormack into relevance prediction. NSERC funding under this project serves to extend an existing Google Focus Award on context-aware mobile social networking. While developing technologies under that proposal, a need for improved evaluation methodologies became apparent. In addition to supporting the needs of this Google Focus Award, this project advances our general understanding of information access evaluation.
University of Waterloo - $64,504 Grant (2017-18) | Canada Spends