“Generative Models vs Discriminative Models”
On 28th February 2019 at Buildo, Data Science Milan has organized a meetup about Generative Models topic and how to use them beyond pictures.
“GANs beyond nice pictures: real value of data generation”, by Alexandr Honchar, AI Consultant
Let’s start with this question, what are Generative Models? Simply speaking they are a class of Unsupervised Machine Learning Models used to generate some data. GANs and VAEs belong to these class of models and Alex explained their characteristics.
Generative Adversarial networks or GANs are based on the game theory scenario in which a generator model must compete against a discriminator model. The discriminator model is a classifier that determines whether a given image looks like a real image from the dataset or like an artificially created image. It’s a binary classifier that will assume the form of a normal convolutional neural network (CNN). The generator model takes random input values from the training data and transform them into convincing images through a deconvolutional neural network. The generator model is trained to fool the discriminator, so you can think a GAN as a game with a policeman (discriminator) and a counterfeiter of fake moneys (generator). The counterfeiter learns to make cash that’s just slightly more realistic and ends up tricking the policeman. The two players are represented by two functions, each of which is differentiable both with respect to its inputs and with respect to its parameters.
Variational Autoencoder or VAE is a generative model as well as GAN and it differs from autoencoder, because this latest learn a “compressed representation” of input (images,texts..) automatically by first compressing the input (encoder) and then decompressing it back (decoder) to match the original input. The learning is aided by a mapping function. Instead, VAE learns the parameters of a probability distribution modelling training data. In this way if you sample points from this distribution, you can generate new input data samples.
The structure consists on an encoder network that turns the input samples into two parameters in a latent space (mean and variance). Then, are randomly sampled similar points from a latent normal distribution that is assumed to generate the data (prior) with noise values sampled from this distribution. At the end, a decoder network maps these latent space points back to the original input data. The parameters of the model are trained by a loss function that is a negative log-likelihood with a regularizer. The loss function can be split into two terms: the first one is the reconstruction loss with the goal forcing decoded samples to match initial inputs and the Kullback-Leibler divergence between the learned latent distribution (encoder distribution) and the prior distribution, acting as a regularization term.
Looking at the relationship between these two models, GANs framework can train some models that VAEs framework cannot and vice versa, but the two frameworks also have a large intersection. The most salient difference is: VAEs cannot have discrete variables at the input to the generator, while GANs cannot have discrete variables in the output of the generator. GANs have high-quality appealing results, but are difficult to train and the idea behind adversarial training can be applied in many other fields. VAEs are easy to train with nice formulation and an easy input prior to build but with blurry results based due to minimizing the MSE error. Alex has showed some examples where generative models can be applied beyond to generate images, such as anomaly detection, understanding data, domain adaption of not aligned-datasets and better embedding for further supervised learning.