Directions in ML: Automating ML Performance Metric Selection

From music recommendations to high-stakes medical treatment selection, complex decision-making tasks are increasingly automated as classification problems. Thus, there is a growing need for classifiers that accurately reflect complex decision-making goals. One often formalizes these learning goals via a performance metric, which, in turn, can be used to evaluate and compare classifiers. Yet, choosing the appropriate metric remains a challenging problem. This talk will outline metric elicitation as a formal strategy to address the metric selection problem. Metric elicitation automates the discovery of implicit preferences from an expert or an expert panel using relatively efficient and straightforward interactive queries. Beyond standard classification settings, I will also outline early work on metric selection for group-fair classification. Learn more about the 2020-2021 Directions in ML: AutoML and Automating Algorithms virtual speaker series: ​