Seeing What a GAN Cannot Generate

David Bau1,2, Jun-Yan Zhu1, Jonas Wulff1, William Peebles1, Hendrik Strobelt2, Bolei Zhou3, Antonio Torralba1,2
1MIT CSAIL, 2MIT-IBM Watson AI Lab, 3The Chinese University of Hong Kong


ICCV 2019
Preprint

Github
repo

Colab
notebook

The image quality output by Generative Adversarial Networks (GANs) continues to improve.

But what can a GAN not generate?

GAN reconstruction Real photo

We examine the omissions of a GAN in two ways:

  1. What does a GAN miss in its overall distribution?
  2. What does a GAN miss in each individual image?

Seeing Omissions in a GAN Distribution

To understand what the GAN's output distribution is missing, we gather segmentation statistics over the outputs, and compare the number of generated pixels in each output object class with the expected number in the training distribution.

A Progressive GAN trained to generate LSUN outdoor church images is analyzed below.

Of course the model is not perfect, but this graph reveals that the omissions of the model are specific. This model does not generate enough pixels of people, cars, palm trees, or signboards compared to the training distribution.

Instead of drawing such complex objects, it draws too many pixels of simple things like earth and rivers and rock.

Seeing Omissions in Individual GAN Images

Omissions in the distribution lead us to ask: how do these mistakes appear in individual images?

Seeing what a GAN does not generate requires us to compare the GAN's output with real photos. So instead of examining random images on their own, we use the GAN model to reconstruct real images from the training set. The differences reveal specific cases of what the GAN should ideally be able to draw, but cannot.

People

The GAN seems to avoid drawing people, synthesizing plausible scenes with the people removed.

GAN reconstruction Real photo
GAN reconstruction Real photo
GAN reconstruction Real photo
GAN reconstruction Real photo

Vehicles

A similar effect is seen for vehicles.

GAN reconstruction Real photo
GAN reconstruction Real photo
GAN reconstruction Real photo
GAN reconstruction Real photo

Signs

GAN reconstruction Real photo
GAN reconstruction Real photo
GAN reconstruction Real photo
GAN reconstruction Real photo

Monuments

GAN reconstruction Real photo
GAN reconstruction Real photo
GAN reconstruction Real photo