ResNet

Residual Network is a deep Learning model used for computer vision applications.
The ResNet (Residual Neural Network) architecture was introduced by Kaiming He, Xiangyu
Zhang, Shaoqing Ren and Jian sun in their paper titled. Resnet achieved unprecedented results
by addressing the challenges associated with training very deep neural networks.
Resnet outperformed other architectures by winning winning the image classification task in
ILVRSC 2015 by a substantial margin with top 5 error rate of 3.57 % and achieved remarkable
results.

Challenges faced by Deep Neural Networks

Vanishing/Exploding Gradient Problem:

As the number of layers in the neural network increases the gradients of the loss function with
respect to the weight may become extrememly small during backpropogation. This makes it
difficult for the latter layers to learn any meaningful pattern as the updates of the weights are
almost negligible to have an impact. Conversely in some cases the gradients can become very
large during backpropogation leading to instability in the training process by causing the
weights to be updated by large amount, making the optimization process difficult to control.

Degradation Problem in Neural Network: The degradation problem in neural networks refers to
the phenomenon where, as the depth of a neural network increases ,the performance of the
network on the training data saturates and then starts to degrade. The degradation problem is
particularly problematic because it goes against the intuition that deeper networks are more
able to extract intricate and abstract features. We can see it as twofold problem

Performance Plateau: As the number of layers in the neural network increases the training
error tends to saturate and stops improving. It means that the additional layers are not having
any significant benefit on the reduction of training error.

Accuracy Degradation: After the addition of more layers surprisingly the error on the validation
set starts increasing and the performance on the unseen data becomes poor.

In this residual network there are total 16 residual blocks.

The first set consists of 3 residual blocks. Each residual block consists of 2 convolution layers
where each convolution layer consists of 64 filters of size 3x3 and a skip connection which
performs identity mapping.

Second set consists of 4 residual blocks. Each residual block contains 2 convolutional layers
where each layer consists of 128 kernels of of size 3x3 and a skip connection .

Conclusion :
 1. Residual Networks offer several advantages including improved training of very deep
networks,faster convergence and reduced model complexity.
2. ResNets facilitates the training of deep neural network without suffering from
degradation in performance which is the main issue in plain networks.
3. The skip connections in ResNets allow the network to learn identity mappings making it
easier for the model to learn the identity function when needed.
4. Skip connections helps in optimization process which leads to faster convergence and
less training time.
5. The skip connections in the residual networks leads to better generalization on the
unseen data as the network can skip unnnecesary or irrelevant information.
6. REsNets have consistently demonstrated state of the art performance on the various
datasets and their flexibility have made them useful in various other task including
object detection,segmentation .Because of these attributes they have been widely
adopted for many deep learning task.

Residual Networks (ResNet) and ResNeXt

ResNet block with and without 1×1 convolution, which transforms the input into the desired
shape for the addition operation.

ResNet Model
The first two layers of ResNet are the same as those of the GoogLeNet we described before:
the 7×7 convolutional layer with 64 output channels and a stride of 2 is followed by
the 3×3 max-pooling layer with a stride of 2. The difference is the batch normalization layer
added after each convolutional layer in ResNet.

GoogLeNet uses four modules made up of Inception blocks. However, ResNet uses four modules
made up of residual blocks, each of which uses several residual blocks with the same number of
output channels. The number of channels in the first module is the same as the number of input
channels. Since a max-pooling layer with a stride of 2 has already been used, it is not necessary
to reduce the height and width. In the first residual block for each of the subsequent modules,
the number of channels is doubled compared with that of the previous module, and the height
and width are halved.

Then, we add all the modules to ResNet. Here, two residual blocks are used for each module.
Lastly, just like GoogLeNet, we add a global average pooling layer, followed by the fully
connected layer output.

There are four convolutional layers in each module (excluding the 1×1 convolutional layer).
Together with the first 7×7 convolutional layer and the final fully connected layer, there are 18
layers in total. Therefore, this model is commonly known as ResNet-18. By configuring different
numbers of channels and residual blocks in the module, we can create different ResNet models,
such as the deeper 152-layer ResNet-152. Although the main architecture of ResNet is similar to
that of GoogLeNet, ResNet’s structure is simpler and easier to modify. All these factors have
resulted in the rapid and widespread use of ResNet. Fig. depicts the full ResNet-18.

The ResNet-18 architecture.

At last, at the ILSVRC 2015, the so-called Residual Neural Network (ResNet) by Kaiming He et al
introduced anovel architecture with “skip connections” and features heavy batch normalization.
Such skip connections are also known as gated units or gated recurrent units and have a strong
similarity to recent successful elements applied in RNNs. Thanks to this technique they were
able to train a NN with 152 layers while still having lower complexity than VGGNet. It achieves a
top-5 error rate of 3.57% which beats human-level performance on this dataset.
AlexNet has parallel two CNN line trained on two GPUs with cross-connections, GoogleNet has
inception modules ,ResNet has residual connections.