dbkim_gan
no way to compare when less than two revisions
Differences
This shows you the differences between two versions of the page.
— | dbkim_gan [2019/07/11 11:29] (current) – created dongbinkim | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Basic Theory of Generative Adversarial Networks ====== | ||
+ | <!-- Replace the above line with the name of your "How To" Tutorial e.g. How to Laser cut Your Name in Wood --> | ||
+ | |||
+ | <!-- Everywhere you see <some sentence>, | ||
+ | |||
+ | **Author:** <DONGBIN KIM> Email: < | ||
+ | \\ | ||
+ | **Date:** Last modified on < | ||
+ | \\ | ||
+ | **Keywords: | ||
+ | \\ | ||
+ | |||
+ | <!-- Add a representative photo of your tutorial below. | ||
+ | |||
+ | {{: | ||
+ | \\ | ||
+ | Generative Adversarial Networks(GANs) was proposed as a new deep-learning framework | ||
+ | a mistake. This framework corresponds to a minimax | ||
+ | |||
+ | They have made an interesting example to help under- standing. The generative model can be thought of as anal- ogous to a team of counterfeiters, | ||
+ | The entire system can be trained with backpropagation in the case where G and D are defined by multilayer perceptrons. Their work has demonstrated the potential of the framework through qualitative and quantitative evaluation of the gener- ated samples. | ||
+ | Since its publication in 2014, GAN has been considered as the coolest idea in deep learning in the last 20 years. and there are many application for text, image, and video gener- ation with GANs. Furthermore, | ||
+ | Especially, DC-GAN does not only address weakness of original GAN, but also they address | ||
+ | Thus, GAN theory has been popularized in deep- learning(DL). The demand for studying GAN has in- creased as well. Those who has been studying on Deep- Learning(DL), | ||
+ | Therefore in this paper, the author presents an educational tutorial for original Generative Adversarial Networks. | ||
+ | |||
+ | \\ | ||
+ | ===== Motivation and Audience ===== | ||
+ | |||
+ | This tutorial' | ||
+ | |||
+ | <fc blue> | ||
+ | * Know how to python coding | ||
+ | \\ | ||
+ | * Perhaps also know statistics | ||
+ | \\ | ||
+ | * Perhaps additional background needed may include mathmatics | ||
+ | \\ | ||
+ | * This tutorial may also attract readers who wants to look at Generative Adversarial Networks | ||
+ | </fc> | ||
+ | \\ | ||
+ | The rest of this tutorial is presented as follows: | ||
+ | * Part List and Sources | ||
+ | * Installation Enviornment. | ||
+ | * Python Coding | ||
+ | * Final Words | ||
+ | |||
+ | ==== Parts List and Sources ==== | ||
+ | |||
+ | -Computer or Laptop | ||
+ | -Ubuntu 16.04 | ||
+ | -Python 3.5 | ||
+ | -Anaconda Package | ||
+ | -PyTorch(Non CUDA installed) | ||
+ | -Tensorflow | ||
+ | |||
+ | ==== Construction ==== | ||
+ | To learn Generative Adversarial Networks(GANs), | ||
+ | • Linux Ubuntu 16.04 | ||
+ | • Python 3.5 version | ||
+ | • Anaconda installer | ||
+ | • Pytorch | ||
+ | • Tensorflow | ||
+ | \\ | ||
+ | First of all, Linux 16.04 should be installed. Anyone can run on the Windows too, but Linux is highly recommended due to its compatibleness to other required software. From now on, the subsection below will describe the installation process of each software [5] | ||
+ | |||
+ | \\ | ||
+ | \\ | ||
+ | **Step 1 - Python** | ||
+ | \\ | ||
+ | \\ | ||
+ | Python is one of powerful computer language for deep- learning. All GANs are run by python package. For this GAN tutorial, we are using python version 3.5. Therefore it is very important to know python programming. Tutorials are on https:// | ||
+ | \\ | ||
+ | •python3 –version | ||
+ | \\ | ||
+ | •Python 3.5.X (Results) | ||
+ | \\ | ||
+ | Then the python 3.5 version is confirmed above | ||
+ | \\ | ||
+ | \\ | ||
+ | **Step 2 - Additional Python Package** | ||
+ | \\ | ||
+ | \\ | ||
+ | Before the big package installation, | ||
+ | \\ | ||
+ | • Numpy : the fundamental package for scientific comput- ing with Python, it has a powerful N-dimensional array object, sophisticated (broadcasting) functions, tools for integrating C/C++ and Fortran code, and useful linear algebra, fourier transform, random number capabili- tyies [6] | ||
+ | \\ | ||
+ | • Numpy install command : $pip install numpy | ||
+ | \\ | ||
+ | \\ | ||
+ | **Step 3 - Anaconda** | ||
+ | \\ | ||
+ | \\ | ||
+ | {{: | ||
+ | Anaconda is considered as one of the most popular python data science platform [7]. It helps install any python or PyTorch package for deep-learning organized. It accelerates streamline the data science workflows from data ingest through deployment. It also connects leverage and integrate all data sources to extract the most value from data. Thus, it’s very important to install the anaconda program. Check the link as following. | ||
+ | \\ | ||
+ | • http:// | ||
+ | \\ | ||
+ | When anacanda installation is finished, anaconda is automatically installed for python 3.7 version environment. As python 3.5 is compatible with all the other software package, it’s significant to change anaconda-python version from 3.7 to 3.5. To switch the python version to 3.5, please type the following in the terminal | ||
+ | \\ | ||
+ | • $conda install python=3.5 | ||
+ | \\ | ||
+ | Then anaconda command ’conda’ will switch the python version to 3.5. When the switch is done, move over to PyTorch installation. | ||
+ | \\ | ||
+ | \\ | ||
+ | |||
+ | **Step 4 - PyTorch** | ||
+ | \\ | ||
+ | \\ | ||
+ | {{: | ||
+ | PyTorch is an open source deep learning platform that provides a seamless path from research prototyping to pro- duction deployment [8]. Every PyTorch package | ||
+ | |||
+ | In fig, the preference is suggested on PyTorch website. As we choose linux, Python 3.5, and Conda. Please select fol- lowings. Lastly, mark ’None’ in CUDA selection. CUDA is another software package that uses graphic pro- cessing unit(GPU) provided by NVIDIA. It helps faster calculation speed. But, the user is not required to use CUDA at this point because it’s not necessary | ||
+ | \\ | ||
+ | \\ | ||
+ | **Step5 - Tensorflow** | ||
+ | \\ | ||
+ | \\ | ||
+ | {{: | ||
+ | Tensorflow is an open source software library for high performance numerical computation [9]. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Googles AI organization, | ||
+ | \\ | ||
+ | • https:// | ||
+ | \\ | ||
+ | The installation will be done by pip packages from Python. There are 4 available packages, and ’tensorflow’ should be chosen. Because GPU is not used, tensorflow-gpu should not be installed. Once again, this work focus to make any laptop available for GANs. When the installation is done, move over to Python Writer installation. | ||
+ | |||
+ | |||
+ | ==== Theory and Programming Description ==== | ||
+ | |||
+ | This section addresses the theory of original generative | ||
+ | \\ | ||
+ | {{: | ||
+ | \\ | ||
+ | Generative Adversarial Networks model is most straight- forward to apply when the models are both multilayer perceptrons. Input noise variables pz(z) is defined to learn the generator’s distribution pg over data x. Then, a data space | ||
+ | G(z; θg) is represented as to mapping these variables, where G is a differentiable function represented by a multilayer perceptron with parameters θg. We also define a second mul- tilayer perceptron D(x; θg) that outputs a single scalar. D(x) represents the probability that x came from the data rather than pg. We train D to maximize the probability of assigning the correct label to both training examples and samples from | ||
+ | G. We simultaneously train G to minimize: log(1-D(G(z))) | ||
+ | \\ | ||
+ | {{: | ||
+ | \\ | ||
+ | In other words, D and G play the following two-player minimax game with value function V (G, D) [1]. Figure describes the example of GAN framework. The framework pits two adversaries against each other in a game. Each player is represented by a differentiable function controlled by a set of parameters. Typically these functions are implemented as deep neural networks. The game plays out in two scenarios. In one scenario, training examples x are randomly sampled from the training set and used as input for the first player, the discriminator, | ||
+ | D. The goal of the discriminator is to output the probability that its input is real rather than fake, under the assumption that half of the inputs it is ever shown are real and half are fake. In this first scenario, the goal of the discriminator is for D(x) to be near 1. In the second scenario, inputs z to the generator are randomly sampled from the models prior over the latent variables. The discriminator then receives input G(z), a fake sample created by the generator. In this scenario, both players participate. The discriminator strives to make D(G(z)) approach 0 while the generative strives | ||
+ | Therefore, the usual desirable equilibrium point for the above defined GANs is that the Generator should model the real data and Discriminator should output the probability of | ||
+ | 0.5 as the generated data is same as the real data that is, it is not sure if the new data coming from the generator is real or fake with equal probability. | ||
+ | |||
+ | |||
+ | In this section, there are 3 parts. First of all, the simple data distribution is generated for simple GAN model. Second, generator(G) and discriminator(D) networks are written in python code. Lastly, the data distribution and the networks are used for training in an adversarial way. The objective of this implementation is to learn a new function that generates data from the same distribution as the training data. The expectation from the training is that the generator network should produce data which follows the data distribution. | ||
+ | \\ | ||
+ | 1) Training data generation: The dataset is generated as a quadratic function for clearness by numpy-generated random samples. The python code is below | ||
+ | \\ | ||
+ | <code c> | ||
+ | | ||
+ | |||
+ | | ||
+ | def get_y(x): | ||
+ | return 10 + x*x | ||
+ | def sample_data(n=10000, | ||
+ | data = [] | ||
+ | x = scale*(np.random.random_sample((n, | ||
+ | for i in range(n): | ||
+ | yi = get_y(x[i]) | ||
+ | data.append([x[i], | ||
+ | return np.array(data) | ||
+ | </ | ||
+ | \\ | ||
+ | \\ | ||
+ | The data distribution is depicted as below | ||
+ | \\ | ||
+ | {{: | ||
+ | \\ | ||
+ | 2) GAN python code build: First of all, 5 python packages are imported, tensorflow, numpy, seaborn, matplotlib.pyplot and training-data python created in previous section. import | ||
+ | * means that the python | ||
+ | \\ | ||
+ | <code c> | ||
+ | import tensorflow as tf | ||
+ | import numpy as np | ||
+ | from training_data import * | ||
+ | import seaborn as sb | ||
+ | import matplotlib.pyplot as plt | ||
+ | sb.set() | ||
+ | </ | ||
+ | \\ | ||
+ | \\ | ||
+ | Second, noise data z is randomly computed | ||
+ | \\ | ||
+ | <code c> | ||
+ | def noisesample_Z(m, | ||
+ | return np.random.uniform(-1., | ||
+ | </ | ||
+ | \\ | ||
+ | Then, the generator(G) and discriminator(D) | ||
+ | \\ | ||
+ | <code c> | ||
+ | def generator(Z, | ||
+ | with tf.variable_scope(" | ||
+ | hiddenlayer1 = tf.layers.dense(Z, | ||
+ | hiddenlayer2 = tf.layers.dense(hiddenlayer1, | ||
+ | out = tf.layers.dense(hiddenlayer2, | ||
+ | |||
+ | return out | ||
+ | </ | ||
+ | \\ | ||
+ | This function takes in the placeholder for random samples (Z), an array hsize for the number of units in the 2 hidden layers and a reuse variable which is used for reusing the same layers. Using these inputs it creates a fully connected neural network of 2 hidden layers with 16 nodes. The Leaky- Rectified Linear Unit(Leaky-ReLU) is used for activation function due to its stability. The output of this function is a 2-dimensional vector which corresponds to the dimensions of the real dataset that we are trying to learn. | ||
+ | The discriminator(D) network is implemented using the following function: | ||
+ | \\ | ||
+ | <code c> | ||
+ | def discriminator(X, | ||
+ | with tf.variable_scope(" | ||
+ | hiddenlayer1 = tf.layers.dense(X, | ||
+ | hiddenlayer2 = tf.layers.dense(hiddenlayer1, | ||
+ | hiddenlayer3 = tf.layers.dense(hiddenlayer2, | ||
+ | out = tf.layers.dense(hiddenlayer3, | ||
+ | |||
+ | return out, hiddenlayer3 | ||
+ | |||
+ | </ | ||
+ | \\ | ||
+ | This function takes input placeholder for the samples from the vector space of real dataset. The samples can be achieved from both real samples and fake sample by generator net- work. Similar to the Generator network above it also takes input hsize and reuse. Three hidden layers | ||
+ | Adversarial Training For the purpose of Adversarial train- ing, the following placeholders X and Z are defined for real samples and random noise samples respectively: | ||
+ | \\ | ||
+ | <code c> | ||
+ | X = tf.placeholder(tf.float32, | ||
+ | Z = tf.placeholder(tf.float32, | ||
+ | </ | ||
+ | \\ | ||
+ | In the placeholder, | ||
+ | \\ | ||
+ | <code c> | ||
+ | G_sample = generator(Z) | ||
+ | real_logits, | ||
+ | fake_logits, | ||
+ | </ | ||
+ | \\ | ||
+ | Using the logits for generated data and real data we define the loss functions for the Generator and Discriminator networks as follows: | ||
+ | \\ | ||
+ | <code c> | ||
+ | discriminator_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=real_logits, | ||
+ | generator_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=fake_logits, | ||
+ | |||
+ | </ | ||
+ | \\ | ||
+ | These losses are sigmoid cross entropy based losses. This is a commonly used loss function for discrete classification. It takes as input the logit given by the discriminator network and true labels for each sample. It then calculates the error for each sample. The optimized calculation is implemented by TensorFlow because it is more stable then directly taking calculating cross entropy. The equations | ||
+ | Next, the tf.get.collection fuction is used to get data from generator and discriminator. Then, the optimizers for the two networks using the loss functions defined above and scope of the layers defined in the generator and discriminator functions. RMSProp Optimizer is used for both the networks with the learning rate as 0.001. Using the scope we fetch the weights/ | ||
+ | \\ | ||
+ | <code c> | ||
+ | discriminator_variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, | ||
+ | generator_variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, | ||
+ | |||
+ | disc_step = tf.train.RMSPropOptimizer(learning_rate=0.001).minimize(discriminator_loss, | ||
+ | gen_step = tf.train.RMSPropOptimizer(learning_rate=0.001).minimize(generator_loss, | ||
+ | |||
+ | </ | ||
+ | \\ | ||
+ | {{: | ||
+ | Finally, the network training build is made following figure. [1] The difference is that the code used RMSProp optimizers than stochastic gradient for data update. Both the networks are trained in the required number of steps: | ||
+ | \\ | ||
+ | <code c> | ||
+ | for i in range(10001): | ||
+ | X_batch = sample_data(n=batch_size) | ||
+ | Z_batch = noisesample_Z(batch_size, | ||
+ | |||
+ | for _ in range(ndiscriminator_steps): | ||
+ | _, discriminatorloss = sess.run([disc_step, | ||
+ | |||
+ | for _ in range(ngenerator_steps): | ||
+ | _, generatorloss = sess.run([gen_step, | ||
+ | |||
+ | </ | ||
+ | \\ | ||
+ | The above code can be modified to include more complex training procedures such as running multiple steps of the discriminator and/or generator update, fetching the features of the real and generated samples and plotting the generated samples. Please refer to the code repository for such modifications. | ||
+ | The final code is presented below, more code lines added to get the visual image results. | ||
+ | \\ | ||
+ | *Full Code line* | ||
+ | <code c> | ||
+ | import tensorflow as tf | ||
+ | import numpy as np | ||
+ | from training_data import * | ||
+ | import seaborn as sb | ||
+ | import matplotlib.pyplot as plt | ||
+ | sb.set() | ||
+ | |||
+ | def noisesample_Z(m, | ||
+ | return np.random.uniform(-1., | ||
+ | |||
+ | def generator(Z, | ||
+ | with tf.variable_scope(" | ||
+ | hiddenlayer1 = tf.layers.dense(Z, | ||
+ | hiddenlayer2 = tf.layers.dense(hiddenlayer1, | ||
+ | out = tf.layers.dense(hiddenlayer2, | ||
+ | |||
+ | return out | ||
+ | |||
+ | def discriminator(X, | ||
+ | with tf.variable_scope(" | ||
+ | hiddenlayer1 = tf.layers.dense(X, | ||
+ | hiddenlayer2 = tf.layers.dense(hiddenlayer1, | ||
+ | hiddenlayer3 = tf.layers.dense(hiddenlayer2, | ||
+ | out = tf.layers.dense(hiddenlayer3, | ||
+ | |||
+ | return out, hiddenlayer3 | ||
+ | |||
+ | |||
+ | X = tf.placeholder(tf.float32, | ||
+ | Z = tf.placeholder(tf.float32, | ||
+ | |||
+ | G_sample = generator(Z) | ||
+ | real_logits, | ||
+ | fake_logits, | ||
+ | |||
+ | discriminator_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=real_logits, | ||
+ | generator_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=fake_logits, | ||
+ | |||
+ | discriminator_variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, | ||
+ | generator_variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, | ||
+ | |||
+ | disc_step = tf.train.RMSPropOptimizer(learning_rate=0.001).minimize(discriminator_loss, | ||
+ | gen_step = tf.train.RMSPropOptimizer(learning_rate=0.001).minimize(generator_loss, | ||
+ | |||
+ | |||
+ | ##To start the training, session is defined below. The batch ##size is that the number of data processed together. | ||
+ | |||
+ | sess = tf.Session() | ||
+ | tf.global_variables_initializer().run(session=sess) | ||
+ | |||
+ | batch_size = 256 | ||
+ | ndiscriminator_steps = 10 | ||
+ | ngenerator_steps = 10 | ||
+ | |||
+ | x_plot = sample_data(n=batch_size) | ||
+ | |||
+ | f = open(' | ||
+ | f.write(' | ||
+ | |||
+ | for i in range(10001): | ||
+ | X_batch = sample_data(n=batch_size) | ||
+ | Z_batch = noisesample_Z(batch_size, | ||
+ | |||
+ | for _ in range(ndiscriminator_steps): | ||
+ | _, discriminatorloss = sess.run([disc_step, | ||
+ | |||
+ | for _ in range(ngenerator_steps): | ||
+ | _, generatorloss = sess.run([gen_step, | ||
+ | |||
+ | ##The error and figure are saved by following codes | ||
+ | |||
+ | print (" | ||
+ | if i%10 == 0: | ||
+ | f.write(" | ||
+ | |||
+ | if i%1000 == 0: | ||
+ | plt.figure() | ||
+ | g_plot = sess.run(G_sample, | ||
+ | xax = plt.scatter(x_plot[:, | ||
+ | gax = plt.scatter(g_plot[:, | ||
+ | |||
+ | plt.legend((xax, | ||
+ | plt.title(' | ||
+ | plt.tight_layout() | ||
+ | plt.savefig(' | ||
+ | plt.close() | ||
+ | |||
+ | f.close() | ||
+ | |||
+ | </ | ||
+ | |||
+ | ==== Final Words ==== | ||
+ | The python GAN code has been tested with several changes. First of all, the generator and discriminator network are tested with different activation function. Second, the input in train-data python code is modified to give different order variable system input. All test is described in following. | ||
+ | \\ | ||
+ | • (1)Activation function : | ||
+ | \\ | ||
+ | • (2)Activation function : Rectified Linear Unit(ReLU), different input | ||
+ | \\ | ||
+ | • (3)Activation function : Sigmoid, different input | ||
+ | \\ | ||
+ | You will find that the lower order variable inputs works better in GAN training. The higher one affects the unstable train results. and the other shows that ReLU based generator and discriminator are more unstable than Leaky-ReLU. In actual test running with input x3, the system has stopped. Furthermore, | ||
+ | |||
+ | In this work, generative adversarial networks(GANs) educational tutorial is presented. Hardware and software environment installation is summarized. GAN code lines is given in python. Three test are implemented with different input and activation functions. The results demonstrated that Leaky- LeRU is the best activation function for GAN, however the train shows unstable result when it comes with higher order variable inputs. The expected work in the future is Deep- Convolutional Generative Adversarial Networks tutorial that solves instability from original GAN proposed by experi- ment results. [2] | ||
+ | |||
+ | |||
+ | |||
+ | ====Reference==== | ||
+ | |||
+ | [1] I. J. Goodfellow, et. al, ”Generative Adversarial Networks”, | ||
+ | \\ | ||
+ | \\ | ||
+ | [2] Unsupervised Representation Learning with Deep-Convolutional Gen- erative Adversarial Networks | ||
+ | \\ | ||
+ | [3] L. Metz, et. al, ”Unrolled GENERATIVE ADVERSARIAL NET- | ||
+ | WORKS”, International Conference on Learning Representation, | ||
+ | \\ | ||
+ | [4] D. J. Im, et. al, ”Generating Images with Recurrent Adversarial Networks”, | ||
+ | \\ | ||
+ | [5] T. Karras, et. al. ”Progressive Growing of GANs for Improved Quality, Stability, and Variation”, | ||
+ | \\ | ||
+ | [6] Linux Ubuntu, available : https:// | ||
+ | \\ | ||
+ | [7] Numpy, available : http:// | ||
+ | \\ | ||
+ | [8] Anaconda, | ||
+ | \\ | ||
+ | [9] PyTorch, | ||
+ | \\ | ||
+ | [10] Tensorflow, | ||
+ | \\ | ||
+ | [11] I. Goodfellow, ”Generative Adversarial Networks(GANs)”, | ||
+ | |||
+ | |||
+ | |||
+ | |||
dbkim_gan.txt · Last modified: 2019/07/11 11:29 by dongbinkim