MPI-INF Logo
Campus Event Calendar

Event Entry

What and Who

Survey Equivalence: An Information-theoretic Measure of Classifier Accuracy When the Ground Truth is Subjective

Paul Resnick
University of Michigan, School of Information
SWS Distinguished Lecture Series

Paul Resnick is the Michael D. Cohen Collegiate Professor of Information and Associate Dean for Research at the University of Michigan School of Information. He was a pioneer in the fields of recommender systems and reputation systems. He recently started the Center for Social Media Responsibility, which encourages and helps social media platforms to meet their public responsibilities.
SWS, RG1, MMCI  
Public Audience
English

Date, Time and Location

Tuesday, 27 November 2018
10:30
90 Minutes
E1 5
002
Saarbrücken

Abstract

Many classification tasks have no objective ground truth. Examples include: which content or explanation is "better" according to some community? is this comment toxic? what is the political leaning of this news article? The traditional modeling approach assumes each item has an objective true state that is perceived by humans with some random error. It fails to account for the fact that people have greater agreement on some items than others. I will describe an alternative model where the true state is a distribution over labels that raters from a specified population would assign to an item. This leads to information gain (mutual information) as a theoretically justified and computationally tractable measure of a classifier's quality, and an intuitive interpretation of information gain in terms of the sample size for a survey that would yield the same expected error rate.

Contact

Claudia Richter
9303 9100
--email hidden

Video Broadcast

Yes
Kaiserslautern
G26
111
passcode not visible
logged in users only

Uwe Brahm, 09/12/2018 17:55
Claudia Richter, 09/11/2018 09:13 -- Created document.