04. Generating Song Lyrics Using RNN/4.01 Hola Recurrent Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hola Recurrent Neural Nets!\n",
    "\n",
    "\n",
    "<center> The sun rises in the ____. </center>\n",
    "\n",
    "\n",
    "If we were asked to predict the blank in the above sentence, we might probably predict as\n",
    "'east'. How did we predict that the word 'east' would be the right word? Because we read\n",
    "the whole sentence, understood the context and predicted that the word 'east' would be an\n",
    "appropriate word here.\n",
    "\n",
    "If we use a feedforward neural network to predict the blank, it would not predict the right\n",
    "word. This is due to the fact that in feedforward network, each input is independent of each\n",
    "other and they make predictions only based on the current input and they don't remember\n",
    "previous inputs.\n",
    "\n",
    "Thus, input to the network will be just the word before the blank which is, 'the'. With this\n",
    "word alone as an input, our network cannot predict the correct word because it doesn't\n",
    "know the context of the sentence which means - it doesn't know the previous set of words\n",
    "to understand the context of the sentence and to predict an appropriate next word.\n",
    "\n",
    "Here is where we use Recurrent Neural networks. It predicts output not only based on the\n",
    "current input but also on the previous hidden state. But why does it have to predict the\n",
    "output based on the current input and the previous hidden state and why it can't just use\n",
    "the current input and the previous input?\n",
    "\n",
    "Because the previous input will store information just about the previous word while the\n",
    "previous hidden state captures the contextual information about all the words in the\n",
    "sentence that the network has seen so far. Basically, the previous hidden state acts like a\n",
    "memory and it captures the context of the sentence. With this context and the current input,\n",
    "we can predict the relevant word.\n",
    "\n",
    "For instance, let us take the same sentence, The sun rises in the ____. As shown in the\n",
    "following figure, we first pass the word 'the' as an input and then pass the next word 'sun'\n",
    "as input but along with this we also pass the previous hidden state $h_0$. So every time, we\n",
    "pass the input word - we also pass a previous hidden state.\n",
    "\n",
    "In the final step, we pass the word 'the' and also the previous hidden state $h_3$ which\n",
    "captures the contextual information about the sequence of words that the network has seen\n",
    "\n",
    "so far. Thus, $h_3$ acts as memory and stores information about all the previous words that\n",
    "the network has seen. With $h_3$ and the current input word 'the', we can now predict the\n",
    "relevant next word. \n",
    "\n",
    "![image](images/1.png)\n",
    "\n",
    "_In a nutshell, RNN uses previous hidden state as memory which captures and stores the\n",
    "information (inputs) that the network has seen so far._\n",
    "\n",
    "\n",
    "\n",
    "RNN is widely applied for use cases that involves sequential data like time series, text,\n",
    "audio, speech, video, weather and many more. It has been greatly used in various Natural\n",
    "Language Processing (NLP) tasks such as language translation, sentiment analysis, text\n",
    "generation and so on. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next section, we will learn about the difference between feedforward networks and RNN."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}