{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Introduction to Keras and TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## What's TensorFlow?"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## What's Keras?"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Keras and TensorFlow: A brief history"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Setting up a deep-learning workspace"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Jupyter notebooks: The preferred way to run deep-learning experiments"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Using Colaboratory"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### First steps with Colaboratory"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Installing packages with pip"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Using the GPU runtime"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## First steps with TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Constant tensors and variables"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**All-ones or all-zeros tensors**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"x = tf.ones(shape=(2, 1))\n",
"print(x)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"x = tf.zeros(shape=(2, 1))\n",
"print(x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Random tensors**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"x = tf.random.normal(shape=(3, 1), mean=0., stddev=1.)\n",
"print(x)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"x = tf.random.uniform(shape=(3, 1), minval=0., maxval=1.)\n",
"print(x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**NumPy arrays are assignable**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"x = np.ones(shape=(2, 2))\n",
"x[0, 0] = 0."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Creating a TensorFlow variable**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"v = tf.Variable(initial_value=tf.random.normal(shape=(3, 1)))\n",
"print(v)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Assigning a value to a TensorFlow variable**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"v.assign(tf.ones((3, 1)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Assigning a value to a subset of a TensorFlow variable**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"v[0, 0].assign(3.)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using `assign_add`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"v.assign_add(tf.ones((3, 1)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Tensor operations: Doing math in TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A few basic math operations**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"a = tf.ones((2, 2))\n",
"b = tf.square(a)\n",
"c = tf.sqrt(a)\n",
"d = b + c\n",
"e = tf.matmul(a, b)\n",
"e *= d"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### A second look at the GradientTape API"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using the `GradientTape`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"input_var = tf.Variable(initial_value=3.)\n",
"with tf.GradientTape() as tape:\n",
" result = tf.square(input_var)\n",
"gradient = tape.gradient(result, input_var)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using `GradientTape` with constant tensor inputs**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"input_const = tf.constant(3.)\n",
"with tf.GradientTape() as tape:\n",
" tape.watch(input_const)\n",
" result = tf.square(input_const)\n",
"gradient = tape.gradient(result, input_const)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using nested gradient tapes to compute second-order gradients**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"time = tf.Variable(0.)\n",
"with tf.GradientTape() as outer_tape:\n",
" with tf.GradientTape() as inner_tape:\n",
" position = 4.9 * time ** 2\n",
" speed = inner_tape.gradient(position, time)\n",
"acceleration = outer_tape.gradient(speed, time)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### An end-to-end example: A linear classifier in pure TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Generating two classes of random points in a 2D plane**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"num_samples_per_class = 1000\n",
"negative_samples = np.random.multivariate_normal(\n",
" mean=[0, 3],\n",
" cov=[[1, 0.5],[0.5, 1]],\n",
" size=num_samples_per_class)\n",
"positive_samples = np.random.multivariate_normal(\n",
" mean=[3, 0],\n",
" cov=[[1, 0.5],[0.5, 1]],\n",
" size=num_samples_per_class)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Stacking the two classes into an array with shape (2000, 2)**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = np.vstack((negative_samples, positive_samples)).astype(np.float32)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Generating the corresponding targets (0 and 1)**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"targets = np.vstack((np.zeros((num_samples_per_class, 1), dtype=\"float32\"),\n",
" np.ones((num_samples_per_class, 1), dtype=\"float32\")))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Plotting the two point classes**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"plt.scatter(inputs[:, 0], inputs[:, 1], c=targets[:, 0])\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Creating the linear classifier variables**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"input_dim = 2\n",
"output_dim = 1\n",
"W = tf.Variable(initial_value=tf.random.uniform(shape=(input_dim, output_dim)))\n",
"b = tf.Variable(initial_value=tf.zeros(shape=(output_dim,)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The forward pass function**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def model(inputs):\n",
" return tf.matmul(inputs, W) + b"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The mean squared error loss function**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def square_loss(targets, predictions):\n",
" per_sample_losses = tf.square(targets - predictions)\n",
" return tf.reduce_mean(per_sample_losses)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The training step function**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"learning_rate = 0.1\n",
"\n",
"def training_step(inputs, targets):\n",
" with tf.GradientTape() as tape:\n",
" predictions = model(inputs)\n",
" loss = square_loss(targets, predictions)\n",
" grad_loss_wrt_W, grad_loss_wrt_b = tape.gradient(loss, [W, b])\n",
" W.assign_sub(grad_loss_wrt_W * learning_rate)\n",
" b.assign_sub(grad_loss_wrt_b * learning_rate)\n",
" return loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The batch training loop**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"for step in range(40):\n",
" loss = training_step(inputs, targets)\n",
" print(f\"Loss at step {step}: {loss:.4f}\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"predictions = model(inputs)\n",
"plt.scatter(inputs[:, 0], inputs[:, 1], c=predictions[:, 0] > 0.5)\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"x = np.linspace(-1, 4, 100)\n",
"y = - W[0] / W[1] * x + (0.5 - b) / W[1]\n",
"plt.plot(x, y, \"-r\")\n",
"plt.scatter(inputs[:, 0], inputs[:, 1], c=predictions[:, 0] > 0.5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Anatomy of a neural network: Understanding core Keras APIs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Layers: The building blocks of deep learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### The base Layer class in Keras"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A `Dense` layer implemented as a `Layer` subclass**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"\n",
"class SimpleDense(keras.layers.Layer):\n",
"\n",
" def __init__(self, units, activation=None):\n",
" super().__init__()\n",
" self.units = units\n",
" self.activation = activation\n",
"\n",
" def build(self, input_shape):\n",
" input_dim = input_shape[-1]\n",
" self.W = self.add_weight(shape=(input_dim, self.units),\n",
" initializer=\"random_normal\")\n",
" self.b = self.add_weight(shape=(self.units,),\n",
" initializer=\"zeros\")\n",
"\n",
" def call(self, inputs):\n",
" y = tf.matmul(inputs, self.W) + self.b\n",
" if self.activation is not None:\n",
" y = self.activation(y)\n",
" return y"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"my_dense = SimpleDense(units=32, activation=tf.nn.relu)\n",
"input_tensor = tf.ones(shape=(2, 784))\n",
"output_tensor = my_dense(input_tensor)\n",
"print(output_tensor.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Automatic shape inference: Building layers on the fly"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import layers\n",
"layer = layers.Dense(32, activation=\"relu\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import models\n",
"from tensorflow.keras import layers\n",
"model = models.Sequential([\n",
" layers.Dense(32, activation=\"relu\"),\n",
" layers.Dense(32)\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([\n",
" SimpleDense(32, activation=\"relu\"),\n",
" SimpleDense(64, activation=\"relu\"),\n",
" SimpleDense(32, activation=\"relu\"),\n",
" SimpleDense(10, activation=\"softmax\")\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### From layers to models"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The \"compile\" step: Configuring the learning process"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([keras.layers.Dense(1)])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"mean_squared_error\",\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model.compile(optimizer=keras.optimizers.RMSprop(),\n",
" loss=keras.losses.MeanSquaredError(),\n",
" metrics=[keras.metrics.BinaryAccuracy()])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Picking a loss function"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Understanding the fit() method"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Calling `fit()` with NumPy data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"history = model.fit(\n",
" inputs,\n",
" targets,\n",
" epochs=5,\n",
" batch_size=128\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"history.history"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Monitoring loss and metrics on validation data"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using the `validation_data` argument**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([keras.layers.Dense(1)])\n",
"model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=0.1),\n",
" loss=keras.losses.MeanSquaredError(),\n",
" metrics=[keras.metrics.BinaryAccuracy()])\n",
"\n",
"indices_permutation = np.random.permutation(len(inputs))\n",
"shuffled_inputs = inputs[indices_permutation]\n",
"shuffled_targets = targets[indices_permutation]\n",
"\n",
"num_validation_samples = int(0.3 * len(inputs))\n",
"val_inputs = shuffled_inputs[:num_validation_samples]\n",
"val_targets = shuffled_targets[:num_validation_samples]\n",
"training_inputs = shuffled_inputs[num_validation_samples:]\n",
"training_targets = shuffled_targets[num_validation_samples:]\n",
"model.fit(\n",
" training_inputs,\n",
" training_targets,\n",
" epochs=5,\n",
" batch_size=16,\n",
" validation_data=(val_inputs, val_targets)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Inference: Using a model after training"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"predictions = model.predict(val_inputs, batch_size=128)\n",
"print(predictions[:10])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Summary"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter03_introduction-to-keras-and-tf.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}