Grant: Collaborative Research and Development Grants

Scientific and engineering documents typically contain both text and mathematics. However, when it comes to searching such documents, current commercial technology is able to match text only and so is not well-suited for tasks such as finding mathematical expressions; this must be extended to become math-aware. In this proposal we aim to search text and mathematical content together in order to get the best search results. It has been shown that search engines become more effective when they are able to infer some semantics for terms appearing in documents since, in such instances, terms with similar meanings can be matched. As a result we wish to ensure that math-aware search can infer semantics for mathematical formulas. In this way retrieval based on both text and mathematical formulas can become more effective. In order for a math-aware search engine to have broad impact, it is important that a user be able to enter their mathematical formulas in a natural handwritten manner rather than to use one-dimensional LaTeX-like commands. This can be natural on today's pen-based devices and so an important part of our project is to investigate how a pen-based interface can be used to retrieve documents that are rich in mathematical content. Converting handwritten mathematical expressions to be used in a math-aware search system has considerable difficulties. Challenges include recognition of individual characters, combining handwritten characters into math expressions, distinguishing among similar looking expressions, use of gestures, and determining wild cards. We plan to investigate data-driven methods to ensure good recognition of handwritten math. The implementation and evaluation of our math-aware retrieval system plays a central role in our work.