ISSN 2278-3091

Volume
Ritik Gandhi, International Journal of Advanced 10,inNo.5,
Trends September
Computer - October
Science and 2021
Engineering, 10(5), September - October 2021, 2947 – 2953
International Journal of Advanced Trends in Computer Science and Engineering
Available Online at http://www.warse.org/IJATCSE/static/pdf/file/ijatcse031052021.pdf
https://doi.org/10.30534/ijatcse/2021/031052021

Deep Learning Based Car Damage Detection,

Classification and Severity
Ritik Gandhi1
1
Shri Govindram Seksaria Institute of Technology and Science, Indore, India, ritikgandhi21@gmail.com

Received Date : August 06, 2021 Accepted Date : September 15, 2021 Published Date : October 06, 2021

 of the images were also done [15]. Since the dataset is small,
ABSTRACT we used Data Augmentation Techniques to synthetically
enlarge the dataset. We consider the common car damages
In the accident insurance industry, settling the claim is a such as bumper dent, bumper scratch, door dent, door scratch,
time-consuming process since it is a manual process and there glass shattered, scratch, smash and some images of no
is a gap between the optimal and the actual settlement. Using damaged cars.
deep learning models, we are not only trying to speed up the A number of techniques were tried such as directly training a
process but also provide better customer service and increase CNN, using a pre-trained CNN model, using transfer
the profitability of insurance companies. In this paper we are learning from large CNNs and building an ensemble
using various pretrained models such as VGG 16, VGG 19, classifier. Out of all the techniques tried we observed that
Resnet50 and Densenet and based on these models, selecting transfer learning works the best. The damaged part
the best performing models. We initially check whether the classification and its localization was done using the
car is damaged or not using the Resnet50 model and if it’s a YOLOv3 model which classifies an image and draws the
damaged one we use the WPOD-net model to detect the bounding boxes around them. To add more to it, YOLOv3
license plate. To identify the damaged region, we use the framework is significantly faster than R-CNN models because
YOLO model. At last, comes the damage severity which is it has a bigger network and residual networks are added by
implemented using the Densenet model. After implementing adding various shortcut connections. To extract and read the
various models, we find out that transfer learning gives better license plate we used the WPOD-net model which detects the
results than fine-tuning. In addition to that we propose a license plate no matter how different the distortion is and
framework that integrates all of this into one application and further rectifies the license plate area to a rectangular shape so
in turn helps in the automation of the insurance industry. that the detections can be further fed to the OCR network. The
damage severity of the car was done on 3 parameters: Minor,
Key words: Deep Learning, Damage assessment (detection, Moderate, Severe.
classification and severity), Pre-trained CNN Models, YOLO Although a lot many minor factors were taken into account so
as to make the model as much best performing as possible but
1. INTRODUCTION along the way the focus was on the influence of certain
hyper-parameters and searching theoretically defined ways to
The Global Auto Insurance market is projected to reach $1.06 adapt them [2].
trillion by 2027 and still a lot of money is being wasted when Since Deep Learning is one of the best techniques when it
it comes to claims. The traditional method involves a tedious comes to image processing related tasks, a major challenge
process wherein the customer submits the claim documents to was to reduce the model training time since a traditional CNN
the agent who in turn submits the claim documents to the model can be very time-consuming to perform image
company and then an external evaluator inspects the unit and classification tasks and identify the correct weights for the
correspondingly prepares his reports. The company reviews it network by multiple forward and backward iterations.
and issues the LOA and then sends the car to the shop for
repairs. This has forced the insurance firms to look out for 2. RELATED WORK
solutions that include fair assessment and faster agreement of
claims. Whenever object detection comes into play, deep learning has
In this paper we try to employ different Convolutional Neural always shown promising results. The most popular detection
Network (CNN) models such as VGG16, VGG19, Resnet50, algorithms include the Convolutional Neural Networks
Densenet ,etc and based on the accuracy, select the models (CNN), since they perform well for many computer vision
that work best for us. We couldn’t find any publicly available tasks such as visual object recognition and detection [3][4].
dataset for the same and therefore created our own dataset by With computing resources based on transfer learning
web scraping different sites. Manual filtering and annotating solutions and extensive use of data, deep learning has been
outstanding in image classification [5][6].

2947
 Ritik Gandhi, International Journal of Advanced Trends in Computer Science and Engineering, 10(5), September - October 2021, 2947 – 2953

To perform different tasks of localization and detection 3. DATASET DESCRIPTION

different models are proposed, however according to [7], they
tried to implement a complete system with transfer learning There were a couple of datasets related to the car damage
based on CNN models but could not calculate the damage classification but none of them served the purpose we wanted
severity. to get a proper architecture; hence we created our own dataset
that contained images. We considered images of cars that
Pre-trained CNN models are very complicated to understand were damaged and undamaged. If the cars were damaged,
because of their intense variance but they can still be used as a then the damaged part was considered such as bumper dent,
feature extractor. Their weights can be freely downloaded and bumper scratch, door dent, door scratch, windshield damage,
applied via transfer learning. head lamp broken, tail lamp broken, smash. There was
Structural damages have also been identified and studied another dataset that was specifically used for predicting the
using the CNNs in [8] where the authors propose a deep damage severity of car that was classified as minor, moderate
learning-based method to characterize the cracks on a and severe. For the license plate dataset, a dataset of the
composite material. Indian car license plate images was used.
 Dataset 1- Training and validation sets of damaged
In case of small number of labeled samples autoencoders have
and undamaged cars.
improved the performance of the classifier. Multi sensor-data
fusion techniques have been used to solve vehicle body  Dataset 2-Training and validation sets of Damage on
damage problems [9]. Unsupervised pre-training techniques Front, Rear and Side.
have improvised the general performance of the classifier as
compared to the supervised techniques. CNN models also  Dataset 3-Training and validation sets of Damage
have tremendous applications in ship-target detection as Severity Minor, Moderate, Severe.
stated by Wang et al [10] which solves the problem of closely  Dataset 4- Training and validation sets of different
aligned targets and multi-scale targets. Based on RCNN, damaged parts such as door dent, door scratch,
building target detection algorithms have been proposed bumper dent, bumper scratch etc.
which remote sense the images of different scenes [11]. To
handle automatic vehicle damage detection via photographs  Dataset 5- Training and validation sets of number
3D CAD Models were used [12]. The YOLO Object Detection plate with different distortions.
model was applied in [13] although the results weren’t quite Since we had a small dataset, we used Data Augmentation to
satisfactory and up-to the mark. In [14] the team collected enlarge the dataset. To improve the execution of models and
different images and sorted the dataset into many classes and expand small sized datasets data augmentation gives an ideal
since the dataset containing the images were less, they solution as explained in [15]. Although there are a couple of
synthetically enlarged the dataset 5 times. Since thy couldn’t approaches for the same, we enlarged the dataset twice using
achieve an appropriate accuracy, they went with predefined horizontal flip transformations and random rotations between
models and from output of pre-trained models, trained a -30 and 30. The YOLOv3 models are trained on the COCO
linear SVM. Based on the experiments, using SoftMax Dataset but the images had to be annotated using third party
Classifier is better than Linear SVM. For images, tools called labelImg. 3 YOLO models developed individually
Convolutional Auto Encoders (CAE) have shown good identify various damaged parts of the car. The YOLOv3
results. models are trained on the COCO Dataset but the images had
to be annotated using third party tools called labelImg. 3
Table 1. Test accuracy with CNN training YOLO models developed individually identify various
damaged parts of the car.
Table 2. Description of the dataset

Most of the papers majorly concentrate on CNN models to

detect the damaged part with techniques via transfer learning
and some researchers use a better segmentation algorithm
with the camera type image for analysis.
Related to the ALPR (Automatic License Plate Recognition)
systems are Scene Text Spotting (STS) which find and read
text/numbers in natural scenes. Many systems proposed
typically use image binarization or gray-scale analysis to find
candidate proposals.

2948
 Ritik Gandhi, International Journal of Advanced Trends in Computer Science and Engineering, 10(5), September - October 2021, 2947 – 2953

After Data Augmentation for each dataset the number of files

were as follows:
 Dataset 1- Original Data + Data Augmentation 2
(3980 Train files, 490 Test files)
 Dataset 2- Original Data + Data Augmentation 2
(1996 Train files, 193 Test files)
 Dataset 3- Original Data + Data Augmentation 2
(1968 Train files, 189 Test files)

The CNN was trained on both the original dataset and

augmented dataset.

Fig 2. A flowchart depicting the overall process of the car

damage assessment.

Stage 1: Detecting whether the car is damaged or not

There are different pretrained models like VGG16, VGG19,

Densenet but Resnet50 turned out to be the most
accurate model to validate whether the car is damaged or not.
The dataset contains train and validation sets such as Bumper
Dent, Bumper Scratch, Door Dent, Door Scratch, Glass
Shattered, Head Lamp, Tail Lamp, Undamaged, etc. If the car
Fig 1. Sample images for car damage types is undamaged then it simply detects it and if it’s a damaged
one, then there are further localizations made by the YOLO
4. EXPERIMENTS AND IMPLEMENTATIONS models. The model shows an accuracy close to 89% on the
validation set.
Initially, a CNN was trained with random initializations and
for every convolutional layer a RELU non-linearity is used.
Furthermore, the results also showed that data augmentation
improves the performance and generalization as compared
with training on the original dataset. The pretrained models
were better than the models implemented from scratch and
therefore VGG16, VGG19, Densenet and Resnet50 were
imported without fully connected layers. To compare all the
models Logistic Regression was chosen with features
extracted from this model. Two models were trained by
keeping Logistic Regression as the baseline model wherein
the first model had layers as non-trainable and the second
model had layers as trainable. Hyperparameter tuning of
logistic regression was done and using best alpha the models Fig 3. Images in train data folder
were created. Stage 1 was compiled using Binary cross
entropy loss whereas, Stage 2 and Stage 3 were compiled
using Categorical Cross entropy. Stochastic gradient
descent optimizer (SGD) and accuracy were used as the
metric. Each model was trained for 50 epochs and the best
model was saved using Model Checkpoint.The overall
application is divided into 4 stages:
The first stage involves in detecting whether the car is
damaged or not. The second stage involves extracting and
reading the license number plate of the damaged car. Next
stage involves the localization of the damaged part and
figuring out which part of the car is damaged using the YOLO
model. The last stage classifies the severity of the damaged
car. Figure 2 depicts a flowchart of developing car damage
assessment architecture
Fig 4. Images in test data folder

2949
 Ritik Gandhi, International Journal of Advanced Trends in Computer Science and Engineering, 10(5), September - October 2021, 2947 – 2953

The images for the damaged and not damaged classes are
equal leading to no class imbalance in both the training and
testing data folder.

Stage 2: Extracting and Reading the license plate

The major aim in this stage is to extract the cropped image of

the license plate and read the same using the WPOD-net
model (Warped Planar Object Detection Network). To
decrease the computational cost, it is better to convert the
cropped image to the grey image and then to enhance the
contrast and differentiate between license plate and other
parts of image a grey level processing is applied. To highlight
the difference between the background and the license plate
frontier, the edges are detected using Roberts’ operator.
An approach using the CNN model involves character
segmentation done on the binary image of the preprocessed
Fig 6. A flowchart depicting two different approaches for the
license plate and the extracted one. The CNN model is trained
number plate extraction
over a dataset of alphanumeric characters to recognize the
segmented characters efficiently with an accuracy of about
93%.

In different distortions the WPOD-net model detects the

license plates and regresses the coefficients that actually
unwraps the license plate into a proper rectangular shape.

Fig 7. Predictions on each segmented image

Fig 5. Mechanism of the WPOD-net model to extract the
license plate.

The second approach uses Pytesseract which is an optical

character recognition tool. The preprocessed image is passed
to the Pytesseract OCR engine and we get the predicted
license plate number from the image.

The license plate reader was tested over a dataset of the Indian
car license plate images and the score was determined by
comparing the predicted sequences of both Pytesseract and
CNN with the actual value, using a Sequence Matcher.
The best of the two was taken as the score of that image. The
accuracy was close to 80.34%.

Fig 8. CNN Architecture

2950
 Ritik Gandhi, International Journal of Advanced Trends in Computer Science and Engineering, 10(5), September - October 2021, 2947 – 2953

Stage 3: Damaged Part Classification and Localization Since there is no proper data, LabelImg tool was used for
using YOLO creating bounding boxes and giving classes. Certain specific
files were created for training yolo. Since we had 3 classes the
The damaged part classification is done using the YOLO YOLO was trained for 7000 epochs and the weights were
model, if the car is damaged then then this model is used to saved for multiples of 1000. The observation showed that
localize the damaged part of the car. YOLO refers to “YOU YOLO custom 5000 weights gave better results than other
LOOK ONLY ONCE” and is one of the most versatile models.
models when it comes to object detection. It classifies and YOLO divides all the input images into the SxS grid system
finds damaged part of a car in an image and draws the and each grid is responsible for object detection. The grid
bounding boxes around them. cells are actually responsible for prediction the boundary
boxes for the detected objects. For every box there are 5 main
attributes that are to be considered x and y for coordinates, w
and h for width and height of the object, and a confidence
score for the probability that the box containing the object. 9
classes are used for classification such as bumper dent,
bumper scratch, door dent, door scratch, windshield damaged
etc.

Fig 9. Door scratch Fig 10. Windshield damage

Table 3. Accuracy of Resnet50 damage classification

model

The results mentioned in the above table were from a

validation set of 500+ images with 9 classes and the accuracy
of the overall model were close to 88.99%.
Fig 11. Window damage Fig 12. Bumper scratch
Stage 4: Car Damage Severity

The classification of car damage severity is as follows:

 Minor Damage – It typically involves slight damage
to the vehicle that does not impede the vehicle to
cause severe injuries. It includes the headlight
scratches, dents and digs in the hood or windshield,
from gravel or debris, scratches in the paint.
 Moderate Damage - Any kind of damage that
Fig 13. Bonnet damage Fig 14. Head light damage
impairs the functionality of the vehicle in any way is
moderate damage. It involves large dents in hood,
fender or door of a car. Even if the airbags are
deployed during collision, then it comes under
moderate damage.
 Severe Damage – Structural damages such as bent or
twisted frames, broken/bent axels, missing pieces of
the vehicles and in some cases even the destruction
of airbags. These types of damages are a big threat to
Fig 15. Tail light damage Fig 16. Door dent the human life.

The densenet model was chosen to ensure the maximum flow

between the layers in the network. In addition to that, the
dense connectivity pattern requires fewer training parameters

2951
 Ritik Gandhi, International Journal of Advanced Trends in Computer Science and Engineering, 10(5), September - October 2021, 2947 – 2953

than the traditional models such as VGG16, VGG 19. For

training purposes, the Adam optimizer was chosen due to its
fast convergence. The accuracy of the overall model was close
to 70%.
Categorical cross entropy was used as the loss function and
the model was trained for 100 iterations.
Training - {Minor-278, Moderate-315, Severe-386,
Total-979}
Testing - {Minor-48, Moderate-55, Severe-68, Total-171}

Fig 20. Original Data

Fig 17. Minor damage On the original data for stage 1 that is to check whether the
car is damaged or not we can observe that Densenet trained on
all layers is performing better than other models. The
accuracy of this model is 96.3%, precision of 94.9% and
recall of 97.8%.
For stage 2 which is the Damage Localization again Densenet
outperformed the rest of the models with an accuracy of
76.5%, precision of 76.8% and recall of 74.4%. For stage 3
Resnet perfromed better than rest of the models with an
accuracy of 67.8%, precision of 68.5% and recall of 67.3%.

Fig 18. Moderate damage

Fig 19. Severe damage

5. MODEL SELECTION Fig 21. Original Data + Data Augmentation 1

Accuracy, Precision and Recall are chosen as the three On the original data with the first data augmentation, we
different metrics to estimate the performance of our different observe that resnet and densenet trained on all layers
transfer learning models such as VGG16, VGG19, Resnet50
performed better than other models. On the resent model we
and Densenet. The higher the matrices the better our model
get an accuracy of 96.1% but precision is lower than densenet
performs.
model.For densenet model the accuracy is 95.9%, precision is
94.9% and recall of 97.8%. For damage localization

2952
 Ritik Gandhi, International Journal of Advanced Trends in Computer Science and Engineering, 10(5), September - October 2021, 2947 – 2953

Densenet performs better than other models with an accuracy REFERENCES

of 80.4%, precision of 80.7% and recall of 78.9%. Resnet 1. “https://www.irmi.com/articles/expert-commentary/
perfromed better than other models for Damage Severity with controlling-claims-leakage-through-technology.”
an accuracy of 69.6%, precision of 69.2% and recall of 2. Jeffrey de Deijn. 2018. Automatic Car Damage
68.5%. Recognition using Convolutional Neural Networks.
(2018).
3. B. Y. Lecun Y., Bottou L. and H. P.,
“Gradient-based learning applied to document
recognition,” Proceedings of IEEE, vol. 86, no. 11,
1998.
4. A. Krizhevsky, I. Sutskever, and G. E. Hinton,
“Imagenet classification with deep convolutional
neural networks,” in Advances in Neural
Information Processing Systems 25, F. Pereira, C. J.
C. Burges, L. Bottou, and K. Q. Weinberger, Eds.
Curran Associates, Inc., 2012, pp. 1097–1105.
5. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and
Jian Sun. 2016. Deep residual learning for image
recognition. In Proceedings of the IEEE conference
on computer vision and pattern recognition.
770–778.
6. Karen Simonyan and Andrew Zisserman. 2014.
Very deep convolutional network for large-scale
image recognition arXiv preprint
arXiv:1409.1556(2014).
Fig 22. Original Data + Data Augmentation 2
7. Ranjodh Singh, Meghna P Ayyar, Tata Sri Pavan,
Sandeep Gosain, and Rajiv Ratn Shah. 2019.
With the second augmentation we observe that the Resnet and
Automating Car Insurance Claims Using Deep
Densenet trained on all layers performed better than the other
Learning Techniques. In2019 IEEE Fifth
models. The accuracy of the Resnet model is 95.4% but the International Conference on Multimedia Big Data
precision is lower than the densenet model. The Densenet (BigMM).IEEE, 199–207.
model performs better for the damage localization with an 8. M. G. M. R. G. Soumalya Sarkar, Kishore K. Reddy,
accuracy of 77.7%, precision of 77.8% and recall of 76.6%. “Deep learning for structural health monitoring: A
Resnet performs better than other models with an accuracy of damage characterization application,” in Annual
68.4% on damage severity. Conference of the Prognostics and Health
Management Society, 2016.
6. CONCLUSION 9. S. Gontscharov, H Baumgartel, A.Kneifel, and K.-L.
We started by exploring the applicable deep learning Krieger, Algorithm development for minor damage
algorithms for the car damage detection and also created new identification in vehicle bodies using adaptive sensor
datasets which provided us to explore the detection, data processing," Procedia Technology, vol. 15, pp.
classification and the severity of the damaged cars. The 586 { 594,2014. 2nd International Conference on
pre-trained models were experimented by fine-tuning and System-Integrated Intelligence: Challenges for
Product and Production Engineering.
transfer learning with certain regularization techniques.
10. G. Wang and S. Liang, ‘‘Ship object detection based
From the above models we can safely conclude that Resnet
on mask RCNN,’’ in Proc. Radio Eng., 2018, pp.
model works best to detect whether a car is damaged or not,
947–952
YOLO models to identify the car damage classification and 11. J. Li and W. He, ‘‘Building target detection
the densenet model to check the severity of the car damage. algorithm based on mask RCNN,’’ in Proc. Sci.
Regarding the proposed models there are still overfitting Surv. Mapping, Apr. 2019, pp. 1–13.
issues but there is still room for improvements in terms of 12. S. Jayawardena, Image based automatic vehicle
accuracy. In addition to that if we have a proper high-quality damage detection. PhD thesis, College of
dataset with adequate features and labels we can also try to Engineering and Computer Science (CECS), 12
predict the cost of repairing for the damaged car part and that 2013.
would help the auto-insurance industry to make better and 13. Mahavir Dwivedi, Malik Hashmat Shadab, SN
cost-effective solutions. Omkar, Edgar Bosco Monis, Bharat Khanna, and
Satya Ranjan. Deep Learning Based Car Damage
Classification and Detection.

2953