Falk Scholer - Measuring Search Effectiveness: Bringing the User Back


L'équipe MAGMA accueillera à partir du 1er avril prochain Falk Scholer, chercheur au RMIT de Melbourne en Australie pour une visite de 2 mois.

Il s'intéresse en particulier à l'évaluation des systèmes de recherche d'information. Les personnes intéressées pour discuter avec lui pendant son séjour peuvent d'ores et contacter Carole Adam pour organiser un planning.

Speaker Bio: Dr Falk Scholer is an Associate Professor at RMIT University in Melbourne, Australia. He is an active researcher in the field of information retrieval (IR), where his key interests include: the evaluation of search systems; relevance and user perceptions; interactive search and user modelling; document summarisation; and efficient indexing.

As computer scientists, many of the systems and applications that we build are ultimately intended to be used by people to complete real world tasks. However, the experimental evaluation of what makes a system "good" can be difficult, and is sometimes not related to the end user. This talk will examine the effectiveness measurement of information retrieval systems, and demonstrate that the main evaluation methodology, using test collections, can lead to conclusions that don't match task-based outcomes of end users.

I will then present recent work that investigates some potential causes of the mismatch between experimental evaluation and end users. First, we will consider user variability, and in particular the impact that different search query formulations of the same underlying information need can have on measurement accuracy. Second, we will examine the issue of relevance perceptions and the technique of magnitude estimation, an approach from the field of psychophysics that aims to directly measure subjective user experiences.