Data Labeling for Search Relevance Evaluation with Dmitry Ustalov

Dr. Dmitry Ustalov, Head of Research at Toloka shared his experience in building human-in-the-loop pipelines for information retrieval evaluation based on our full-day tutorial presented at the WWW’21 conference. There was an introduction of the ranking problem, a discussion on the commonly used ranking quality metrics, and then focus on a human-in-the-loop-based approach to obtain relevance judgments at scale. These judgments can be further used to improve the performance of search and recommender systems. Finally, he shared and discussed best practices and pitfalls from his own experience. 👉 Speaker Bio: Dr. Dmitry Ustalov is the Head of Research at Toloka, a global data labeling platform. He is responsible for enabling the state-of-the-art methods for quality control in Toloka and spreading the innovations made by the Toloka Research team. ========================= MLT (Machine Learning Tokyo) site: https://machinelearningtokyo