10. Reconsturcting Inputs using Autoencoders/10.09 Building the Sparse Autoencoder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Sparse Autoencoder\n",
    "\n",
    "Building the sparse autoencoder is just as same as building the autoencoder except that here we use sparse regularizer in the encoder and decoder. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the libraries\n",
    "\n",
    "First, let us import the necessary libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "#modelling\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D\n",
    "from tensorflow.keras import backend as K\n",
    "from tensorflow.keras import regularizers\n",
    "import tensorflow as tf\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "#plotting\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "#dataset\n",
    "from keras.datasets import mnist\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the Dataset\n",
    "\n",
    "Let us load the MNIST dataset. We don't need the labels for autoencoder. Since we are reconstructing the given input we don't need the labels. So, we just load x_train for training and x_test for testing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, _), (x_test, _) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Normalize the data by dividing with max pixel value which is 255:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.astype('float32') / 255\n",
    "x_test = x_test.astype('float32') / 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Reshape the images as 2D array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))\n",
    "x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now the shape of data would become:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "((60000, 784), (10000, 784))\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape, x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Sparse Regularizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know that Sparse Regularizer is given as:\n",
    "\n",
    "\n",
    "$  \\beta (\\sum_{j=1}^{{l}^{(h)}} log \\frac{\\rho}{\\hat{\\rho_j}} + (1-\\rho) log \\frac{1-\\rho}{1-\\hat{\\rho_j}} ) $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the sparse regularizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sparse_regularizer(activation_matrix):\n",
    "    p = 0.01\n",
    "    beta = 3\n",
    "    p_hat = K.mean(activation_matrix) \n",
    "  \n",
    "    KL_divergence = p*(K.log(p/p_hat)) + (1-p)*(K.log(1-p/1-p_hat))\n",
    "    \n",
    "    sum = K.sum(KL_divergence) \n",
    "   \n",
    "    return beta * sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Encoder\n",
    "\n",
    "Define the encoder which takes the images as an input and returns the encodings.\n",
    "\n",
    "Define the size of the encodings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoding_dim = 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the value of lambda:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "lambda_ = 0.001 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the shape of input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_img = Input(shape=(784,))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the encoder which takes the images as the inputs and returns the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = Dense(encoding_dim, \n",
    "                activation='sigmoid',\n",
    "                kernel_regularizer=regularizers.l2(lambda_/2),activity_regularizer=sparse_regularizer)(input_img)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Define the Decoder\n",
    "\n",
    "Define the Decoder which takes the code returned by the encoder and returns the reconstructed input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder = Dense(784,\n",
    "                activation='sigmoid',\n",
    "                kernel_regularizer=regularizers.l2(lambda_/2),activity_regularizer=sparse_regularizer)(encoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the model\n",
    "\n",
    "Now that we defined encoder and decoder layers, we define the model which takes images as input and returns the output of the decoder layer which is the reconstructed image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(input_img, decoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at summary of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 200)               157000    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 784)               157584    \n",
      "=================================================================\n",
      "Total params: 314,584\n",
      "Trainable params: 314,584\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compile the model with loss as binary cross entropy and we minimize the loss using sgd optimizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='sgd', loss='mse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us train the model.\n",
    "\n",
    "Generally, we feed the data to the model as model.fit(x,y) where x is the input and y is the label. But since autoencoders reconstruct its inputs, the input and output to the model should be the same. So we feed the data to the model as model.fit(x_train, x_train):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 4s 67us/sample - loss: 0.5350 - val_loss: 0.5311\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 4s 58us/sample - loss: 0.5276 - val_loss: 0.5238\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 3s 56us/sample - loss: 0.5205 - val_loss: 0.5168\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 3s 55us/sample - loss: 0.5135 - val_loss: 0.5099\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 3s 57us/sample - loss: 0.5067 - val_loss: 0.5032\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 3s 55us/sample - loss: 0.5001 - val_loss: 0.4967\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 3s 55us/sample - loss: 0.4937 - val_loss: 0.4904\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 3s 55us/sample - loss: 0.4875 - val_loss: 0.4842\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 3s 56us/sample - loss: 0.4814 - val_loss: 0.4783\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 3s 57us/sample - loss: 0.4755 - val_loss: 0.4724\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f947d1c9690>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, x_train, epochs=10, batch_size=256, shuffle=True, validation_data=(x_test, x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next section, we will learn about another regularized variant of autoencoders called contractive autoencoders."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}