Unsupervised Prediction Aggregation

prediction task, and one is interested in combining them with the goal
of obtaining a better prediction. Often, these sources are independent
systems, which are derived using different modeling assumptions or
training data. Examples in information retrieval and natural language
processing include aggregating the outputs of search engines, natural
languages parsers, or machine translation systems.

When producing an aggregate prediction, we prefer to rely more on the
signals with highest predictive accuracy. However, for these and many
other structured prediction tasks annotated data is scarce and
expensive to produce, so we cannot estimate their relative quality
directly.

In this talk, we propose an unsupervised learning framework to
prediction aggregation. The intuition behind our approach is that the
agreement between multiple signals can be exploited to estimate their
relative quality and, consequently, learn to effectively aggregate
their predictions. In many practical applications, the relative
quality of the constituent signals is unlikely to remain the same
across different domains, and the domain information is unspecified.
Therefore, we extend our aggregation formalism to explicitly model
latent variability in the quality of the constituent sources.

Finally, we experimentally demonstrate the effectiveness of our
approach for aggregating permutations, top-k lists, and dependency
parses.

This talk includes joint work with Dan Roth, Kevin Small, and Ivan Titov.



If you would like to be served a light lunch, please sign up on our
webpage until Saturday 9am-ish: http://feast.coli.uni-saarland.de/

Next week's special **TUESDAY** Feast talk will be from Alexandre
Klementiev, who is visiting from Johns Hopkins University. He will
present work on 'Unsupervised Prediction Aggregation'; the talk
includes joint work with Dan Roth, Kevin Small, and Ivan Titov.
(abstract below)

*** PLEASE NOTE: this is a special TUESDAY edition of FEAST ***

* Date: 5/25/2010 (** Tuesday **)
* Time: 12:00
* Location: Meeting room "Reuse" (-2.17), DFKI Building D 3 2
* Speaker: Alexandre Klementiev
* Title: Unsupervised Prediction Aggregation

If you would like to be served a light lunch, please sign up on our
webpage until Saturday 9am-ish: http://feast.coli.uni-saarland.de/

See you on Tuesday,

Josef, Caroline, Thomas, Alexis

(Added on behalf of Ivan Titov, MMCI)