AG 1, AG 3, AG 4, RG1, MMCI, AG 2, INET, AG 5, SWS   Info

Public Audience

English

90 Minutes

Saarbrücken

Computer vision systems deployed in the real-world will encounter inputs far from its training distribution. For example, a self-driving car might see a blue stop-sign it has not seen before. To ensure safe operation when faced with such out-of-distribution corner cases, it is vital to quantify the model robustness to such data before deployment. In this dissertation, we build generative models to create synthetic data variations at scale and leverage them to test the robustness of target computer vision systems to these variations. First, we build generative models which can controllably manipulate image and text data. This includes models to a) change visual context by removing objects, b) edit the appearance of an object in images, and c) change the writing style of text. Next, using these generative models we create model-agnostic and model-targeted input variations, to study the robustness of computer vision systems. While in the model-agnostic case, the input image is manipulated based on design priors, in the model-targeted case the manipulation is directly guided by the target model performance. With these methods, we measure and improve the robustness of various computer vision systems -- specifically image classification, segmentation, object detection and visual question answering systems -- to semantic input variations. Additionally, in the text domain, we deploy these generative models to improve diversity of image captioning systems and perform writing style manipulation to obfuscate private attributes of the user.

Connie Balzert

+49 681 9325 2000