04. Generating Song Lyrics Using RNN/4.06 Generating Song Lyrics Using RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating Song Lyrics Using RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have learned enough about RNNs; now, we will look at how to generate song lyrics using RNNs. To do this, we simply build a character-level RNN, meaning that on every time step, we predict a new character.\n",
    "\n",
    "Let's consider a small sentence: _\"What a beautiful d.\"_ \n",
    "\n",
    "At the first time step, the RNN predicts a new character as a. The sentence will be updated to, _\"What a beautiful da.\"_\n",
    "\n",
    "At the next time step, it predicts a new character as y, and the sentence becomes, _\"What a beautiful day.\"_\n",
    "\n",
    "In this manner, we predict a new character at each time step and generate a song. Instead of predicting a new character every time, we can also predict a new word every time, which is called word level RNN. For simplicity, let's start with a character level RNN.\n",
    "\n",
    "How do RNNs predict a new character on each time step? Let's suppose that at a time step t=0, we feed an input character, let's say, x. Now, the RNN predicts the next character based on the given input character, x. To predict the next character, it predicts the probability of all the characters in our vocabulary to be the next character. Once we have this probability distribution, we randomly select the next character based on this probability. Confusing? Let us better understand this with an example. \n",
    "\n",
    "For instance, as shown in the following figure, let's suppose that our vocabulary contains four characters L, O, V, and E; when we feed the character l as an input, it computes the probability of all the words in the vocabulary to be the next character:\n",
    "\n",
    "\n",
    "![image](images/2.png)\n",
    "\n",
    "\n",
    "So, we have the probabilities as [0.0, 0.9, 0.0, 0.1], corresponding to the characters in the vocabulary [L,O,V,E]. With this probability distribution, we select O as the next character 90% of the time, and E as the next character 10% of the time. Predicting the next character by sampling from this probability distribution adds some randomness to the output. \n",
    "\n",
    "On the next time step, we feed the predicted character from the previous time step and the previous hidden state as an input to predict the next character, as shown in the following figure:\n",
    "\n",
    "\n",
    "![image](images/3.png)\n",
    "\n",
    "So, on each time step, we feed the predicted character from the previous time step and the previous hidden state as input and predict the next character:\n",
    "\n",
    "\n",
    "![image](images/4.png)\n",
    "\n",
    "As you can see in the above figure, at time step t=2, V is passed as an input, and it predicts the next character as E. But this does not mean that every time character V is sent as an input it should always return E as output. Since we are passing input along with the previous hidden state, the RNN has the memory of all the characters it has seen so far.\n",
    "\n",
    "So, the previous hidden state captures the essence of the previous input characters, which are L and O. Now, with this previous hidden state and the input V, the RNN predicts the next character as E."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing RNN in TensorFlow\n",
    "\n",
    "Now, we will look at how to build the RNN model in TensorFlow to generate song lyrics. The dataset used in this section available in the data folder as songdata.zip. You can unzip the dataset and place it in the data folder and start running the below code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let us import the required libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the downloaded input dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/songdata.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see few rows from our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artist</th>\n",
       "      <th>song</th>\n",
       "      <th>link</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ABBA</td>\n",
       "      <td>Ahe's My Kind Of Girl</td>\n",
       "      <td>/a/abba/ahes+my+kind+of+girl_20598417.html</td>\n",
       "      <td>Look at her face, it's a wonderful face  \\nAnd...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ABBA</td>\n",
       "      <td>Andante, Andante</td>\n",
       "      <td>/a/abba/andante+andante_20002708.html</td>\n",
       "      <td>Take it easy with me, please  \\nTouch me gentl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ABBA</td>\n",
       "      <td>As Good As New</td>\n",
       "      <td>/a/abba/as+good+as+new_20003033.html</td>\n",
       "      <td>I'll never know why I had to go  \\nWhy I had t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ABBA</td>\n",
       "      <td>Bang</td>\n",
       "      <td>/a/abba/bang_20598415.html</td>\n",
       "      <td>Making somebody happy is a question of give an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ABBA</td>\n",
       "      <td>Bang-A-Boomerang</td>\n",
       "      <td>/a/abba/bang+a+boomerang_20002668.html</td>\n",
       "      <td>Making somebody happy is a question of give an...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  artist                   song                                        link  \\\n",
       "0   ABBA  Ahe's My Kind Of Girl  /a/abba/ahes+my+kind+of+girl_20598417.html   \n",
       "1   ABBA       Andante, Andante       /a/abba/andante+andante_20002708.html   \n",
       "2   ABBA         As Good As New        /a/abba/as+good+as+new_20003033.html   \n",
       "3   ABBA                   Bang                  /a/abba/bang_20598415.html   \n",
       "4   ABBA       Bang-A-Boomerang      /a/abba/bang+a+boomerang_20002668.html   \n",
       "\n",
       "                                                text  \n",
       "0  Look at her face, it's a wonderful face  \\nAnd...  \n",
       "1  Take it easy with me, please  \\nTouch me gentl...  \n",
       "2  I'll never know why I had to go  \\nWhy I had t...  \n",
       "3  Making somebody happy is a question of give an...  \n",
       "4  Making somebody happy is a question of give an...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset consists of about 57,650 song lyrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "57650"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have song lyrics of about 643 artists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "643"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df['artist'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of songs from each artist is shown as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Donna Summer        191\n",
       "Gordon Lightfoot    189\n",
       "George Strait       188\n",
       "Bob Dylan           188\n",
       "Loretta Lynn        187\n",
       "Cher                187\n",
       "Alabama             187\n",
       "Reba Mcentire       187\n",
       "Chaka Khan          186\n",
       "Dean Martin         186\n",
       "Name: artist, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['artist'].value_counts()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On average, we have about 89 songs of each artist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "89.65785381026438"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['artist'].value_counts().values.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have song lyrics in the column text, so we combine all the rows of that column and save it as a text in a variable called data, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = ', '.join(df['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see a few lines of a song:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Look at her face, it's a wonderful face  \\nAnd it means something special to me  \\nLook at the way that she smiles when she sees me  \\nHow lucky can one fellow be?  \\n  \\nShe's just my kind of girl, she makes me feel fine  \\nWho could ever believe that she could be mine?  \\nShe's just my kind of girl, without her I'm blue  \\nAnd if she ever leaves me what could I do, what co\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[:369]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are building a char-level RNN, we will store all the unique characters in our dataset into a variable called chars; this is basically our vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chars = sorted(list(set(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Store the vocabulary size in a variable called vocab_size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vocab_size = len(chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Since the neural networks only accept the input in numbers, we need to convert all the characters in the vocabulary to a number.\n",
    "\n",
    "We map all the characters in the vocabulary to their corresponding index that forms a unique number. We define a char_to_ix dictionary, which has a mapping of all the characters to the index. To get the index by a character, we also define the ix_to_char dictionary, which has a mapping of all the indices to their respective characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "char_to_ix = {ch: i for i, ch in enumerate(chars)}\n",
    "ix_to_char = {i: ch for i, ch in enumerate(chars)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see in the following code snippet, the character 's' is mapped to an index 68 in the char_to_ix dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "68"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_to_ix['s']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, if we give 68 as an input to the ix_to_char, then we get the corresponding character, which is 's':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ix_to_char[68]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Once we obtain the character to integer mapping, we use one-hot encoding to represent the input and output in vector form. A one-hot encoded vector is basically a vector full of 0s, except for a 1 at a position corresponding to a character index.\n",
    "\n",
    "For example, let's suppose that the vocabSize is 7, and the character z is in the fourth position in the vocabulary. Then, the one-hot encoded representation for the character z can be represented as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 1., 0., 0.])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabSize = 7\n",
    "char_index = 4\n",
    "\n",
    "np.eye(vocabSize)[char_index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we have a 1 at the corresponding index of the character, and the rest of the values are 0s. This is how we convert each character into a one-hot encoded vector. \n",
    "\n",
    "In the following code, we define a function called one_hot_encoder, which will return the one-hot encoded vectors, given an index of the character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def one_hot_encoder(index):\n",
    "    return np.eye(vocab_size)[index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Network Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We need to define all the network parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#define the number of units in the hidden layer:\n",
    "hidden_size = 100  \n",
    " \n",
    "#define the length of the input and output sequence:\n",
    "seq_length = 25  \n",
    "\n",
    "#define learning rate for gradient descent is as follows:\n",
    "learning_rate = 1e-1\n",
    "\n",
    "#set the seed value:\n",
    "seed_value = 42\n",
    "tf.set_random_seed(seed_value)\n",
    "random.seed(seed_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining Placeholders\n",
    "\n",
    "Now, we will define the TensorFlow placeholders. The placeholders for the input and output are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "inputs = tf.placeholder(shape=[None, vocab_size],dtype=tf.float32, name=\"inputs\")\n",
    "targets = tf.placeholder(shape=[None, vocab_size], dtype=tf.float32, name=\"targets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the placeholder for the initial hidden state:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "init_state = tf.placeholder(shape=[1, hidden_size], dtype=tf.float32, name=\"state\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define an initializer for initializing the weights of the RNN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "initializer = tf.random_normal_initializer(stddev=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining forward propagation\n",
    "\n",
    "Let's define the forward propagation involved in the RNN, which is mathematically given as follows:\n",
    "\n",
    "$$ h_t =  \\operatorname{tanh}(U x_t + W h_{t-1} + bh) $$\n",
    "$$ \\hat{y} =  \\operatorname{softmax}(V h_t + by) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope(\"RNN\") as scope:\n",
    "    h_t = init_state\n",
    "    y_hat = []\n",
    "\n",
    "    for t, x_t in enumerate(tf.split(inputs, seq_length, axis=0)):\n",
    "        if t > 0:\n",
    "            scope.reuse_variables()  \n",
    "\n",
    "        #input to hidden layer weights\n",
    "        U = tf.get_variable(\"U\", [vocab_size, hidden_size], initializer=initializer)\n",
    "\n",
    "        #hidden to hidden layer weights\n",
    "        W = tf.get_variable(\"W\", [hidden_size, hidden_size], initializer=initializer)\n",
    "\n",
    "        #output to hidden layer weights\n",
    "        V = tf.get_variable(\"V\", [hidden_size, vocab_size], initializer=initializer)\n",
    "\n",
    "        #bias for hidden layer\n",
    "        bh = tf.get_variable(\"bh\", [hidden_size], initializer=initializer)\n",
    "\n",
    "        #bias for output layer\n",
    "        by = tf.get_variable(\"by\", [vocab_size], initializer=initializer)\n",
    "\n",
    "        h_t = tf.tanh(tf.matmul(x_t, U) + tf.matmul(h_t, W) + bh)\n",
    "\n",
    "        y_hat_t = tf.matmul(h_t, V) + by\n",
    "\n",
    "        y_hat.append(y_hat_t)       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply softmax on the output and get the probabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "output_softmax = tf.nn.softmax(y_hat[-1])  \n",
    "\n",
    "outputs = tf.concat(y_hat, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the cross-entropy loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=targets, logits=outputs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store the final hidden state of the RNN in hprev. We use this final hidden state for making predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hprev = h_t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining Backpropagation Through Time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will perform the BPTT, with Adam as our optimizer. We will also perform gradient clipping to avoid the exploding gradients problem.\n",
    "\n",
    "Initialize the Adam optimizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "minimizer = tf.train.AdamOptimizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the gradients of the loss with the Adam optimizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gradients = minimizer.compute_gradients(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the threshold for the gradient clipping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "threshold = tf.constant(5.0, name=\"grad_clipping\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clip the gradients which exceeds the threshold and bring it to the range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipped_gradients = []\n",
    "for grad, var in gradients:\n",
    "    clipped_grad = tf.clip_by_value(grad, -threshold, threshold)\n",
    "    clipped_gradients.append((clipped_grad, var))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update the gradients with the clipped gradients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "updated_gradients = minimizer.apply_gradients(clipped_gradients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start generating songs \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Start the TensorFlow session and initialize all the variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step by Step Explanation\n",
    "\n",
    "__First let us understand how RNN is generating song lyrics step by step, the complete code is given at the end of the notebook. __"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will look at how to generate the song lyrics using an RNN. What should the input and output to the RNN be? How does it learn? What is the training data? Let's see the explanation, along with the code, step by step. \n",
    "\n",
    "We know that in RNNs, the output predicted at a time step $t$ will be sent as the input to the next time step; that is, on every time step, we need to feed the predicted character from the previous time step as input. So, we prepare our dataset in the same way. \n",
    "\n",
    "For instance, look at the following table. Let's suppose that each row is a different time step; on a time step $t=0$ , the RNN predicted a new character, g, as the output. This will be sent as the input to the next time step $t=1$, . \n",
    "\n",
    "However, if you notice the input in the time step $t=1$, we removed the first character from the input o and added the newly predicted character g at the end of our sequence. Why are we removing the first character from the input? Because we need to maintain the sequence length. \n",
    "\n",
    "Let's suppose that our sequence length is eight; adding a newly predicted character to our sequence increases the sequence length to nine. To avoid this, we remove the first character from the input, while adding a newly predicted character from the previous time step.\n",
    "\n",
    "Similarly, in the output data, we also remove the first character on each time step, because once it predicts the new character, the sequence length increases. To avoid this, we remove the first character from the output on each time step, as shown in the following table: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image](images/5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will look at how we can prepare our input and output sequence similar to the preceding table. __The code is bolded. __\n",
    "\n",
    "Define a variable called pointer, which points to the character in our dataset. We will set our pointer to 0, which means it points to the first character:\n",
    "\n",
    "\n",
    "__pointer = 0__\n",
    "\n",
    "Define the input data:\n",
    "\n",
    "__input_sentence = data[pointer: pointer + seq_length]__\n",
    "\n",
    "\n",
    "What does this mean? With the pointer and the sequence length, we slice the data. Consider that the seq_length is 25 and the pointer is 0. It will return the first 25 characters as input. So, data[pointer:pointer + seq_length] returns the following output:\n",
    "\n",
    "_\"Look at her face, it's a \"_\n",
    "\n",
    "\n",
    "Define the output, as follows:\n",
    "\n",
    "__output_sentence = data[pointer + 1: pointer + seq_length + 1]__\n",
    "\n",
    "\n",
    "We slice the output data with one character ahead moved from input data. So, data[pointer + 1:pointer + seq_length + 1] returns the following:\n",
    "\n",
    "_\"ook at her face, it's a w\"_\n",
    "\n",
    "\n",
    "\n",
    "As you can see, we added the next character in the sentence and removed the first character. So, on every iteration, we increment the pointer and traverse the entire dataset. This is how we obtain the input and output sentence for training the RNN. \n",
    "\n",
    "As you have learned, an RNN only accepts numbers as input. Thus, once we sliced the input and output sequence, we get the indices of the respective characters, using the char_to_ix dictionary that we defined:\n",
    "\n",
    "__input_indices = [char_to_ix[ch] for ch in input_sentence]__\n",
    "\n",
    "__target_indices = [char_to_ix[ch] for ch in output_sentence]__ \n",
    "\n",
    "\n",
    "\n",
    "Convert the indices into one-hot encoded vectors by using the one_hot_encoder function we defined previously:\n",
    "\n",
    "__input_vector = one_hot_encoder(input_indices)__\n",
    "\n",
    "__target_vector = one_hot_encoder(target_indices)__\n",
    "\n",
    "\n",
    "This input_vector and target_vector become the input and output for training the RNN. Let's start training.\n",
    "\n",
    "The hprev_val variable stores the last hidden state of our trained RNN model. We use this for making predictions, and we store the loss in loss_val:\n",
    "\n",
    "__hprev_val, loss_val, _ = sess.run([hprev, loss, updated_gradients], feed_dict={inputs: input_vector,targets: target_vector,init_state: hprev_val})__\n",
    "\n",
    "\n",
    "\n",
    "We train the model for n iterations. After training, we start making predictions. Now, we will look at how to make predictions and generate song lyrics using our trained RNN. Set the sample_length, that is, the length of the sentence (song) we want to generate:\n",
    "\n",
    "__sample_length = 500__\n",
    "\n",
    "Randomly select the starting index of the input sequence:\n",
    "\n",
    "__random_index = random.randint(0, len(data) - seq_length)__ \n",
    "\n",
    "Select the input sentence with the randomly selected index:\n",
    "\n",
    "__sample_input_sent = data[random_index:random_index + seq_length]__\n",
    "\n",
    "As we know, we need to feed the input as numbers; convert the selected input sentence to indices:\n",
    "\n",
    "__sample_input_indices = [char_to_ix[ch] for ch in sample_input_sent]__\n",
    "\n",
    "Remember, we stored the last hidden state of the RNN in hprev_val. We used that for making predictions. Now, we will create a new variable called sample_prev_state_val by copying values from hprev_val.\n",
    "\n",
    "The sample_prev_state_val variable is used as an initial hidden state for making predictions:\n",
    "\n",
    "__sample_prev_state_val = np.copy(hprev_val)__\n",
    "\n",
    "\n",
    "Initialize the list for storing the predicted output indices:\n",
    "\n",
    "__predicted_indices = []__\n",
    "\n",
    "Now, for t in range of sample_length, we perform the following and generate the song for the defined sample_length. \n",
    "\n",
    "Convert the sampled_input_indices to the one-hot encoded vectors:\n",
    "\n",
    "__sample_input_vector = one_hot_encoder(sample_input_indices)__\n",
    "\n",
    "Feed the sample_input_vector, and also the hidden state sample_prev_state_val, as the initial hidden state to the RNN, and get the predictions. We store the output probability distribution in probs_dist:\n",
    "\n",
    "__probs_dist, sample_prev_state_val = sess.run([output_softmax, hprev],\n",
    " feed_dict={inputs: sample_input_vector,init_state: sample_prev_state_val})__\n",
    " \n",
    "Randomly select the index of the next character with the probability distribution generated by the RNN:\n",
    "\n",
    "__ix = np.random.choice(range(vocab_size), p=probs_dist.ravel())__\n",
    "\n",
    "Add this newly predicted index, ix, to the sample_input_indices, and also remove the first index from sample_input_indices to maintain the sequence length. This will form the input for the next time step:\n",
    "\n",
    "__sample_input_indices = sample_input_indices[1:] + [ix]__\n",
    "\n",
    "Store all the predicted chars indices in the predicted_indices list:\n",
    "\n",
    "__predicted_indices.append(ix)__\n",
    "\n",
    "Convert all the predicted_indices to their characters:\n",
    "\n",
    "__predicted_chars = [ix_to_char[ix] for ix in predicted_indices]__\n",
    "\n",
    "Combine all the predicted_Chars and save it as text:\n",
    "\n",
    "__text = ''.join(predicted_Chars)__\n",
    "\n",
    "Print the predicted text on every 50,000th  iteration:\n",
    "\n",
    "__print ('\\n')__\n",
    "\n",
    "__print (' After %d iterations' %(iteration))__\n",
    "\n",
    "__print('\\n %s \\n' % (text,))__\n",
    "\n",
    "__print('-'*115)__\n",
    "\n",
    "\n",
    "Increment the pointer and iteration:\n",
    "\n",
    "__pointer += seq_length__\n",
    "__iteration += 1__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complete code block for generating songs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pointer = 0\n",
    "iteration = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " After 0 iterations\n",
      "\n",
      " uhiq iUYo9ra)1FZTezk3FGF CoG)i,uzWKrmpmqJuo(94rKty5\"y-7t,1])zfwR2 FhFHcqdP2qy[!mw(1R[?xS?n(O-\"x5?\"k!efK MCnHSNA0h!SovpSppQ-(m,KBfn9\"j.95p86F?Mun0[qdJ-L7F.Wv!W.GunT9CnwfGobu\"WA?qAtfmREbZGCjDYvl:jN\"D7?iMisuv2hgH1((Z0XepXA7G.Z28znn'SaEzDzCUuM7Fr0)ahqaX7!sJf-B1a5gO!iW8LbnJO1q\" kjB jI2ZNd F-AX-hHhS-rM?RMy73ON[(Xu,N3T[AYD[?anzGVaBn4A Un0fOt!aDdNt7C)dSYapFz[!79\"B9Z5Y!KJLD (zQDZmjpnY7-546t?T ?shi8W BA8Yp.ghO7up[:p?zM7::tuU \"QDy97iWi:B:YBCVZ4)7NJnO[OW3zMvzc4liEs'CY,6-M,B[hBZ ?wtxdCRi-GlXtc[TmaB07L2Ui!hkk \n",
      "\n",
      "-------------------------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      " After 50000 iterations\n",
      "\n",
      " davest:]  \n",
      "[Brike and ener the clifes  \n",
      "Surs-  \n",
      "[Erd ring of me hroutse all what you  \n",
      "Y shrdie to thesp a Whone arive sraie  \n",
      "Take it herne  \n",
      "Mrme?  \n",
      "I come me hect try love:]  \n",
      "Onever love  \n",
      "What lefter. pripls  \n",
      "  \n",
      "[Vny Mvary-as in everythes arffre?  \n",
      "[Drel this Mry a claze  \n",
      "And drizes.  \n",
      "  \n",
      "[Ars.  \n",
      "Sark:]  \n",
      "Er a dide a can arcan to hars:]  \n",
      "Txctich,  \n",
      "[Kreves\n",
      "\n",
      "S F?fie yourse mun  \n",
      "Your hard: a dain, if a My er winh  \n",
      "[Erde stre madber  \n",
      "[Ursep: Sarning...:]  \n",
      "Want  \n",
      "That's lives  \n",
      "Make\n",
      "\n",
      ", f \n",
      "\n",
      "-------------------------------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "while True:\n",
    "    \n",
    "    if pointer + seq_length+1 >= len(data) or iteration == 0:\n",
    "        hprev_val = np.zeros([1, hidden_size])\n",
    "        pointer = 0  \n",
    "    \n",
    "    #select input sentence\n",
    "    input_sentence = data[pointer:pointer + seq_length]\n",
    "    \n",
    "    #select output sentence\n",
    "    output_sentence = data[pointer + 1:pointer + seq_length + 1]\n",
    "    \n",
    "    #get the indices of input and output sentence\n",
    "    input_indices = [char_to_ix[ch] for ch in input_sentence]\n",
    "    target_indices = [char_to_ix[ch] for ch in output_sentence]\n",
    "\n",
    "    #convert the input and output sentence to a one-hot encoded vectors with the help of their indices\n",
    "    input_vector = one_hot_encoder(input_indices)\n",
    "    target_vector = one_hot_encoder(target_indices)\n",
    "\n",
    "    \n",
    "    #train the network and get the final hidden state\n",
    "    hprev_val, loss_val, _ = sess.run([hprev, loss, updated_gradients],\n",
    "                                      feed_dict={inputs: input_vector,targets: target_vector,init_state: hprev_val})\n",
    "   \n",
    "       \n",
    "    #make predictions on every 500th iteration \n",
    "    if iteration % 500 == 0:\n",
    "\n",
    "        #length of characters we want to predict\n",
    "        sample_length = 500\n",
    "        \n",
    "        #randomly select index\n",
    "        random_index = random.randint(0, len(data) - seq_length)\n",
    "        \n",
    "        #sample the input sentence with the randomly selected index\n",
    "        sample_input_sent = data[random_index:random_index + seq_length]\n",
    "    \n",
    "        #get the indices of the sampled input sentence\n",
    "        sample_input_indices = [char_to_ix[ch] for ch in sample_input_sent]\n",
    "        \n",
    "        #store the final hidden state in sample_prev_state_val\n",
    "        sample_prev_state_val = np.copy(hprev_val)\n",
    "        \n",
    "        #for storing the indices of predicted characters\n",
    "        predicted_indices = []\n",
    "        \n",
    "        \n",
    "        for t in range(sample_length):\n",
    "            \n",
    "            #convert the sampled input sentence into one-hot encoded vector using their indices\n",
    "            sample_input_vector = one_hot_encoder(sample_input_indices)\n",
    "            \n",
    "            #compute the probability of all the words in the vocabulary to be the next character\n",
    "            probs_dist, sample_prev_state_val = sess.run([output_softmax, hprev],\n",
    "                                                      feed_dict={inputs: sample_input_vector,init_state: sample_prev_state_val})\n",
    "\n",
    "            #we randomly select the index with the probabilty distribtuion generated by the model\n",
    "            ix = np.random.choice(range(vocab_size), p=probs_dist.ravel())\n",
    "            \n",
    "            sample_input_indices = sample_input_indices[1:] + [ix]\n",
    "            \n",
    "            \n",
    "            #store the predicted index in predicted_indices list\n",
    "            predicted_indices.append(ix)\n",
    "            \n",
    "        #convert the predicted indices to their character\n",
    "        predicted_chars = [ix_to_char[ix] for ix in predicted_indices]\n",
    "        \n",
    "        #combine the predcited characters\n",
    "        text = ''.join(predicted_chars)\n",
    "        \n",
    "        #predict the predict text on every 50000th iteration\n",
    "        if iteration %50000 == 0:           \n",
    "            print ('\\n')\n",
    "            print (' After %d iterations' %(iteration))\n",
    "            print('\\n %s \\n' % (text,))   \n",
    "            print('-'*115)\n",
    "\n",
    "            \n",
    "    #increment the pointer and iteration\n",
    "    pointer += seq_length\n",
    "    iteration += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After training over several iterations, RNN will learn to generate better songs. In order to get better results, you can train the network with huge dataset for several iterations. \n",
    "\n",
    "\n",
    "\n",
    "In the next section, we will learn about the several different types of RNN architectures."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}