Results in Engineering 18 (2023) 101107

Contents lists available at ScienceDirect

Results in Engineering
journal homepage: www.sciencedirect.com/journal/results-in-engineering

Research paper

Automatic image captioning combining natural language processing and

deep neural networks
Antonio M. Rinaldi ∗ , Cristiano Russo, Cristian Tommasino
Department of Electrical Engineering and Information Technologies, IKNOS-LAB Intelligent and Knowledge Systems (LUPT), University of Naples Federico II, 80125 Via
Claudio, 21, Napoli, Italy

A R T I C L E I N F O A B S T R A C T

Keywords: An image contains a lot of information that humans can detect in a very short time. Image captioning aims to
Object detection detect this information by describing the image content through image and text processing techniques. One of
Image captioning the peculiarities of the proposed approach is the combination of multiple networks to catch as many distinct
Deep neural networks
features as possible from a semantic point of view. In this work, our goal is to prove that a combination
Semantic-instance segmentation
strategy of existing methods can efficiently improve the performance in the object detection tasks concerning the
performance achieved by each tested individually. This approach involves using different deep neural networks
that perform two levels of hierarchical object detection in an image. The results are combined and used by
a captioning module that generates image captions through natural language processing techniques. Several
experimental results are reported and discussed to show the effectiveness of our framework. The combination
strategy has also improved, showing a gain in precision over single models.

1. Introduction mon valid approach includes classification that assigns semantic classes
to an image given in input visual features extracted from the image.
The image captioning topic has recently received great attention in Generating a meaningful image content description in a natural lan-
the computer vision community. The goal of image captioning is to de- guage requires a high level of recognized object understanding. An im-
scribe analyzed image content. This issue is interesting for its important portant task in this context is recognizing the detected objects’ semantic
practical applications and because it is a great challenge for computer meanings. For these reasons, classification approaches have been used
vision to understand image contents. to assign recognized objects to semantic classes based on their visual
Computer Vision (CV) and Natural Language Processing (NLP) were features. The use of natural language processing techniques attempts to
considered and treated as separate research areas in the past. Nowa- give a possible solution to this problem by trying to reduce the seman-
days, with the huge amount of available data in multiple formats tic gap [2,4,30,32] defined in [34] as “the lack of coincidence between
produced from several sources, such as mobile devices, IoT, and mul- the information that can be extracted from visual data and the interpre-
timedia social networks [19], researchers can apply both techniques tation that these same data have for a user in a given situation”.
in conjunction to better understand multimedia contents and to ex- There are two general paradigms in image captioning: top-down and
tract knowledge more than information from a given visual content. bottom-up. The top-down paradigm starts from the information given
Indeed, a fundamental task in image retrieval and computer vision is by an image and converts it into words; on the other hand, the bottom-
the automatic annotation of images, whose goal is to find proper, hence up approach considers words that describe various aspects of an image
meaningful, words or phrases for images. The more such annotations and combines them to generate a sentence. In both approaches, linguis-
are semantically rich, the more such mapping between the annotated tic models are used to generate coherent sentences. Recent top-down
text and the visual content is consistent. In this context, approaches approaches follow an encoder-decoder framework to generate image
based on artificial intelligence, which often exploits pre-trained models, captions. Convolutional neural networks are developed to encode visual
are used to learn the mapping between low-level and semantic features images, and recurrent neural networks to decode this information. This
and then generate annotations or captions for a given image. A com- paper presents a framework based on a top-down approach with a set

* Corresponding author.
E-mail addresses: antoniomaria.rinaldi@unina.it (A.M. Rinaldi), cristiano.russo@unina.it (C. Russo), cristian.tommasino@unina.it (C. Tommasino).

https://doi.org/10.1016/j.rineng.2023.101107
Received 5 September 2022; Received in revised form 6 April 2023; Accepted 16 April 2023
Available online 19 April 2023
2590-1230/© 2023 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

of convolutional neural networks combined to encode the input images

and a decoder of the detected information that generates a descrip-
tion of image contents. A distinctive feature of the proposed approach
is the combination strategy. In our study, we consider multiple neural
network models to exploit the best traits of each model in the final cap-
tioning task. This is a common and well-established technique used in
machine learning under different conditions and use cases. The remain-
der of this paper is organized as follows. In Section 2 an overview of the
related research is presented; Section 3 describes the architecture of the Fig. 1. System architecture.
implemented system for image captioning together with our approach;
In Section 4, we report several experimental results and, eventually,
the system dictionary. All visual features are sent to an RNN for caption
Section 5 concludes the paper and proposes future directions of our
generation. Karpathy et al. [12] describe a multimodal Recurrent Neu-
works.
ral Network architecture that uses the deep visual-Semantic alignments
to learn to generate novel descriptions of image regions. They show
2. Related works
their method can overcome state-of-the-art results. Anderson et al. [3]
proposed an architecture that combines a bottom-up approach based on
A large number of methods and techniques based on different ap-
Faster R-CNN that recognizes image regions associated with a feature
proaches have been applied to image captioning during years [14,35]
vector with a top-down approach to measuring the feature weights. The
and, the improvement of computer vision techniques [33,38] has led
bottom-up approach returns a feature vector composed of the location
to the design and implementation of more efficient image captioning
of each object in an image together with the class to which the object
systems [9].
belongs. This vector is weighed using two LSTMs to generate the image
The current state of the art in image captioning can be divided
caption.
into three categories: (i) top-down, (ii) bottom-up, and (iii) mixed ap-
A more recent approach based on the Transformer model is proposed
proaches.
in [42]. A Transformer is a deep learning model which allows using
The bottom-up approach can be viewed as a classic methodology
attention mechanisms without using RNNs. The authors use a knowl-
for image captioning in which the process starts with combining visual edge graph to help the model generate captions and add a Kullback-
concepts, objects, attributes, words, and phrases into sentences using Leibler (KL) divergence term to discriminate incorrect predictions. An-
linguistic models. A model called Baby Talk is proposed in [13]. This other attention-based mechanism for image captioning is presented in
name derives from the similarity with children’s language used to gen- [11]. Such a bio-inspired approach relies on the concept of divergent-
erate image descriptions. It is based on detection modules used to detect convergent attention, combining visual and semantic information as a
objects. A set of classifiers processes each candidate object, and a Con- multi-perspective input to their model.
ditional Random Field (CRF) is built with graph labeling to generate Likely in [23,39,15], the authors proposed to train a deep neural
captions. Fang et al. [7] proposed an architecture based on an image network by integrating textual and visual features.
dataset and related descriptions. The system uses a weakly supervised In this work, we proposed a framework for image captioning based
learning approach to create detectors of common words used in the on an encoder step performed by combining three CNNs for image ob-
descriptions of the analyzed image. After this step, sentences are gener- ject detection of images. The single results of these CNNs are combined
ated from this set of words and classified according to their similarity with having the best result without losing information due to the CNN
to image contents. characteristics. Moreover, a classification on two hierarchical levels is
Top-down approaches define image captioning as a machine lan- proposed to assign to the recognized objects a class or a “superclass”
guage translation problem. Instead of translating contents among dif- (a more generic concept from a semantic perspective) depending on
ferent languages, these approaches translate visual information into the highest result scores. Finally, a decoder step is implemented, con-
natural language. The visual representation is derived from a CNN sidering a combining strategy based on feature extraction to perform
trained on large datasets; conversely, the caption is generated by an a natural language processing step for having a clear and exhaustive
RNN that processes natural language. The differences between the var- description of an image.
ious existing implementations often concern the type of RNN used. In
[21], a multimodal RNN to generate image captions is used. It models 3. The implemented system and approach
a probability distribution to generate a previous word related to an im-
age, and the captions are generated according to this distribution. The The system architecture is shown in Fig. 1. From a general point of
model consists of two subnetworks: a deep recurrent neural network for view, It is organized around three main modules: a detection, a combi-
sentences and a deep convolutional neural network for images. In addi- nation, and a captioning module that generates the image caption. In
tion, the model is also applied for retrieval tasks able to find images or this section, we describe the implemented module, the chosen DNN ar-
sentences. Vinyals et al. [37] present a model that uses RNN in a neural chitectures, the combination strategy, and the techniques used for the
and probabilistic framework to generate image descriptions. Fixed-size, image captioning generation.
The description probability is modeled with an RNN, where the vari-
able number of words is expressed as a fixed length memory updated 3.1. Detection module
after a new input using a non-linear function generated by a Long-Short
Term Memory (LSTM) network. This type of model uses RNNs to en- The detection module is based on three instance segmentation mod-
code the variable input length into a fixed-size vector, and it uses this els: Mask R-CNN [8], YOLOv3 [26], and RetinaNet [16]. The choice of
representation to decode the desired output sentence. these models is due to their quasi-real-time performances as shown in
In recent years, there has been a transition to mixed approaches that [22,10], their relevance in several applications and subdomains such
involve selecting and merging results generated by bottom-up and top- as point-based instance segmentation [25], single-pixel reconstruction
down approaches. An approach based on a semantic attention model is [41], and ball detection [5]. In this work, we propose a two levels
proposed in [40]. The first task analyzes the responses from a classifica- hierarchical classification. The instance segmentation is divided into re-
tion step by a CNN to build a global visual description. Later, attribute gression for object bounding boxes and classification to recognize the
detectors are run to have a list of attributes or visual concepts that ap- object class. This strategy allows assigning a superclass for the recog-
pear more often in the image. Each attribute corresponds to an entry in nized object if the model fails to determine a more specific subclass.

2
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

dimensions expressed as width and height. If the cell starts from the
top left corner of the image (𝑐𝑥 , 𝑐𝑦 ) and the previous bounding box has
width and height 𝑝𝑤 , 𝑝ℎ , then the prediction is:
( )
𝑏𝑥 = 𝜎 𝑡𝑥 + 𝑐𝑥
( )
𝑏𝑦 = 𝜎 𝑡𝑦 + 𝑐𝑦
𝑏𝑤 = 𝑝𝑤 𝑒𝑡𝑤
𝑏ℎ = 𝑝ℎ 𝑒𝑡ℎ
The width and height are predicted as offsets from the centroids of
the cluster, while the center coordinates relative to the allocation of the
filter application are defined using a sigmoid function.
The sum squared error is the loss function used for the training as
shown in Equation (3):

∑
𝑆𝑆𝐸 = _𝑖 = 1𝑛 (𝑥𝑖 − 𝑥)
̄ 2 (3)

Where 𝑛 is the number of observations, 𝑥𝑖 is the value of the i-th

observation, and 𝑥̄ is the real value.
The ground truth of the predicted coordinates is 𝑡̂ ∗, and the gradient
is the value of the ground truth minus the prediction: 𝑡̂ ∗ −𝑡 ∗. The value
of the ground truth can be easily calculated by reversing the previous
equations. YOLOv3 predicts a score for each bounding box using logis-
tic regression. This score is one if the bounding box overlaps a ground
truth object more than any other previous bounding box. If the consid-
ered bounding box is not the best but overlaps a ground truth object
Fig. 2. Detection Module Architecture.
for no more than a threshold, the prediction is ignored. The thresh-
old usually used is 0.5. YOLO assigns a bounding box for each ground
As seen in Fig. 2, each model is duplicated using one model for class
truth object. Each box predicts the classes that the bounding box might
detection and one for superclass detection. Each of these six modules
contain using a multilabel classification. For this purpose, independent
receives in input a training set, a validation set, and a test set, and
logistic classifiers are used.
it returns as output a vector containing the features of the classified
The third used architecture is RetinaNet. The RetinaNet architec-
objects in the image test set.
ture is composed of three parts: a backbone, a feature pyramid network
Mask R-CNN is an extension of Faster R-CNN [27] for instance, seg-
mentation problems. This DNN architecture adds a third branch that (FPN), and a back-end detection. An image is processed by the back-
allows the mask prediction of an instance. This operation is performed bone. Usually, ResNet and, together with the FPN, a network similar to
in parallel with the bounding box regressor and the classifier in the an encoder-decoder is implemented. The benefit of this architecture is
Faster R-CNN. Specifically, Mask R-CNN generates a mask with size that the features of consecutive layers are merged from the rawest to
28x28 for each region of interest (RoI) which will be later expanded to the finest layer with an effective feature propagation to different lay-
fit the corresponding bounding box size. For each identified instance, ers at different scales to the next layer. After this step, the multi-scale
Mask R-CNN will return as output the recognized class, its bounding pyramid features move data to the back end, and it switches on two
box, and a binary mask superimposed on it. branches for the bounding box regression and object classification. In
The presence of masks adds to the loss function of each extracted these last two branches, there is no weight sharing; the weights of each
RoI an additional term as shown in Equation (1): branch are shared through the pyramid features. The anchors used have
an area ranging from 322 to 5122 on the pyramid levels P3 to P7 respec-
 = 𝑐𝑙𝑠 + 𝑏𝑜𝑥 + 𝑚𝑎𝑠𝑘 (1) tively and 3 proportions {1 ∶ 2, 1 ∶ 1, 2 ∶ 1} are used. For a denser scale
1 2
Where 𝑐𝑙𝑠 and 𝑏𝑜𝑥 are equal to the loss functions related to the coverage, anchors of magnitude {20 , 2 3 , 2 3 } are added to each level of
bounding box classifier and regressor used in Faster R-CNN and Fast the pyramid having nine anchors for each level. For each anchor, there
R-CNN. For 𝑚𝑎𝑠𝑘 only one ground truth mask is associated to each is a one-dimensional vector of length 𝐾 (number of classes) of ranking
RoI and a sigmoid activation function is to be applied to each pixel targets and a 4-dimensional vector of box regression targets. Anchors
of the mask. The branch associated with the generated mask prediction are assigned to the ground-truth object boxes with an Intersection over
produces a binary mask with 𝑚 ×𝑚 size for each of the possible K classes. union (IoU) threshold of 0.5 and to the background if the IoU is in the
We ave 𝐾 × 𝑚2 possible masks, each of them associated with a different
range [0, 0.4). Each anchor is assigned to a maximum of one object and
class. 𝑚𝑎𝑠𝑘 is defined as the average between the binary cross-entropy
sets the corresponding entry of the class to 1 and the other entries to 0
loss functions which includes the k-th mask if the region is associated
in the length vector 𝐾. If the anchor has an IoU in the range [0.4, 0.5),
with the k-th ground truth mask as shown in Equation (2):
it is not assigned and ignored during the training. The regression box is
1 ∑
𝑚𝑎𝑠𝑘 = − [𝑦𝑖𝑗 𝑙𝑜𝑔 𝑦̂𝑘𝑖𝑗 + (1 − 𝑦𝑖𝑗 )𝑙𝑜𝑔(1 − 𝑦̂𝑘𝑖𝑗 )] (2) calculated as the offset between the anchor and the assigned object or
𝑚2 1≤𝑖,𝑗≤𝑚 omitted if there is no assignment.
The second convolutional neural network used in this work is YOLO The classification subnet predicts the probability of the presence of
(acronym of You Only Look Once), in its latest version. This network is an object in each spatial position for each of the 𝐴 anchors and each 𝐾
based on the Darknet53 image classifier, consisting of 53 convolutional object class. The used network is a Fully Convolutional Network (FCN)
layers, allowing the network to achieve good performance in a short that applies 4 3x3 convolutional layers. Each layer has 𝐶 filters and
time [26]. is followed by a ReLU activation and a convolutional layer with 𝐾 × 𝐴
YOLO defines the bounding boxes using dimensional clusters like filters. (K classes, A=9 anchors, and C=256 filters). The subnet used for
anchor boxes. It predicts 𝑡𝑥 , 𝑡𝑦 , 𝑡𝑤 , 𝑡ℎ information for each bounding rep- the box regression is an FCN for each of the pyramid levels to decrease
resented as x and y coordinates at the beginning of the box and its the offset from each anchor box to the nearest ground-truth object.

3
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

3.3. Captioning module

Image captioning is the process of generating textual descriptions

of images using artificial intelligence techniques. The goal is to cre-
ate a system that can accurately understand the contents of an image
and produce a natural language description that captures the most im-
portant aspects of the image. The image captioning process typically
involves a deep learning model that uses a combination of convolu-
tional neural networks (CNNs) and recurrent neural networks (RNNs)
to analyze the visual features of an image and generate a corresponding
caption. The CNNs are used to extract relevant features from the image,
while the RNNs are used to generate a sequence of words that form a
Fig. 3. Architecture of the combiner module.
coherent caption. Image captioning has a wide range of applications, in-
cluding helping visually impaired individuals understand the contents
3.2. Combiner module of images, assisting with image retrieval and organization, and enhanc-
ing the user experience on social media platforms and e-commerce sites.
This module combines the results obtained from the models in the This module inputs the combined results and describes the recognized
detection module. objects in natural language to generate a caption of the whole image.
As we can see in Fig. 3, the first three nodes perform a combination The caption has the following syntax: “The image X contains list of objects
of the results generated by each detection model to determine the object on the left; list of object on the right”. Such rigid structure represents a lim-
class or superclass. Later, the three results determined by these combi- itation in the caption generation. However, we point out that a smooth and
nations are the input of a fourth combiner. The combiner first performs dynamic generation of captions is not the main goal of this study, which is fo-
the combination between the results of Mask R-CNN and YOLOv3. The cused instead on evaluating the combination strategy over individual neural
results of this combination are then compared with the results of Reti- network models. In the actual implementation, it uses the spatial feature
naNet, and the output vector containing the final results is provided, of the bounding boxes in the feature vector from the combiner mod-
which then includes the combination of the results of all three models ule to identify the object’s position in the image and understand if they
on two hierarchical levels of classes and superclasses. are on the right or left side of the image, dividing them into two sub-
The combiner output is a single feature vector that will be used to vectors. After this step, the module recognizes the class of the objects
generate the caption of the analyzed image. The elements of the feature using the predicted class in the combined vector feature, and it counts
vector are: the number of occurrences of the same class in each subvector. The ob-
ject name is derived from the class id (see Section 4). If we have more
• Image Id occurrences of the same object assigned to the same class, its name is
• Bounding boxes of identified objects pluralized, and the verbs are accordingly conjugated.
• Class assigned to the object
• Probability of belonging to the class (i.e., score) 4. Experimental results

The combination step is performed by comparing all the elements In this section, we propose and discuss several experimental results
of one vector with all the elements of the other one looking for the obtained from various executions of the presented system to put into
intersections between the bounding boxes. If one or more intersections evidence the performance of the proposed framework for image cap-
are found, the maximum is chosen, and if it exceeds a threshold of 0.8 tioning. The neural networks used for the image object segmentation
(i.e., overlap), we evaluate which of the two elements generating this have been trained by means of a transfer learning approach starting
intersection has the highest score of assignment to a predicted class, from already trained weights. In the training phase, an NVIDIA GeForce
and we chose it as the best candidate. The elements of each vector that 980 4 Gb GPU and NVIDIA CUDA and CUDNNN tools [24] have been
does not have a maximum intersection value will also be added to the used with the Python PyCharm development environment. The analysis
final vector. of various models obtained with the different configurations defined in
The metric we use for the evaluation is the well-established Inter- the training phase has been validated in two steps: (i) Graphic Analy-
section over Union, shown in Equation (4): sis: using the Tensorboard tool [20], the loss function trends have been
𝐴∩𝐵 analyzed. The aim is to have a loss function related to the training data
𝐼𝑜𝑈 = (4)
𝐴∪𝐵 decreasing as much as possible at any epoch and a validation loss with-
Let’s clarify with an example of the combiner job. We suppose to out a rich growing to avoid overfitting; (ii) Evaluation Metrics: when
have a vector with two elements, A and B, and the second vector with reasonable values for the loss functions have been obtained, we proceed
only one element C, with the following IoUs: with the related metrics calculation. The used metrics are IoU, Average
𝐼𝑜𝑈 𝐴,𝐶 = 0.95 Precision, and Average Recall. In the following of this section, we will
𝐼𝑜𝑈 𝐵,𝐶 = 0.45 present the used data set and our strategy for tuning an efficient model
It means that the elements A and C refer to the same object in the to have a good capacity to learn correctly the instances in training set
image, having a very high IoU. We proceeded to compare the score of having a good capacity for generalization of the learned knowledge.
the assignment to the predicted class. Assuming that: The results obtained from our framework and some examples of image
𝑠𝑐𝑜𝑟𝑒𝐴 = 0.85 caption are presented at the end of the section.
𝑠𝑐𝑜𝑟𝑒𝐶 = 0.95
our combiner will choose C. 4.1. Dataset description
Since element B has an IoU lower than the IoU between the other
two elements and lower than the threshold, it is not considered super- The entire COCO dataset [18] has been used. This dataset consists
imposed to the object defined in element C. However, it is added to of 118287 images for the training set, 5000 for the validation set, and
the final vector because it identifies another object in the image. In this 40671 for the test set. It contains 80 object classes divided into 12 su-
way, no information is lost during the different combination steps. perclasses, as shown in Table 1.

4
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Table 1 Table 4
COCO dataset classes. MASK-RCNN parameters.

Superclass Classes Parameter Description

Person Person BACKBONE Resnet 101 or resnet 50, where 50 and 101 are
Vehicle bicycle, car, motorcycle, airplane, bus, train, truck, boat the number of layers of the network
Outdoor Traffic light, fire hydrant, stop sign, parking meter, bench GPU_COUNT Number of used GPUs
Animal Bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe IMAGES_PER_GPU Number of images to analyze on each GPU
Accessory Backpack, umbrella, handbag, tie, suitcase BATCH_SIZE Not modifiable, equal to the product of the two
Sports Frisbee, skis, snowboard, sports ball, kit, baseball bat, previous ones
baseball glove, skateboard, surfboard, tennis racket DETECTION_MIN_CONFIDENCE Minimum score threshold to consider a
Kitchen Bottle, wine glass, cup, fork, knife, spoon, bowl classification
Food Banana, apple, sandwich, orange, broccoli, IMAGE_MIN_DIM Minimum size of the scaled image
carrot, hotdog, pizza, donut, cake IMAGE_SHAPE Image shape in [W H D] format
Furniture Chair, couch, potted plant, bed, dining table, toilet LEARNING_RATE Step size at each iteration
Electronic Tv, laptop, mouse, remote, keyboard, cell phone MAX_GT_INSTANCES Maximum number of ground truth instances to
Appliance Microwave, oven, toaster, sink, refrigerator use in an image
Indoor Book, clock, vase, scissors, NAME Configuration name
teddybear, hair drier, toothbrush NUM_CLASSES Number of classes to be classified
NUMBER OF_EPOCHS Number of epochs
STEPS_PER_EPOCH Steps for each epoch
Table 2 TRAIN_ROIS_PER_IMAGE RoIs trained for each image
VALIDATION_STEPS Number of steps for validation
Statistics of COCO classes used in the
WEIGHT_DECAY Term in the weight update rule that causes the
training set.
weights to exponentially decay to zero
Class Number of Images

Person 64115
Chair 12774 • IMAGE_SHAPE=[512 512 3].
Car 12251 • IMAGE_PER_GPU=1 (hence BATCH_SIZE=1)
Dining Table 11837

Different values have been used for the tuning of other parameters.
In particular:
Table 3
Statistics of COCO classes used in the
validation set. • LEARNING_RATE: between 0.001 and 0.01 in steps of 0.001
• WEIGHT_DECAY: between 0.0001 and 0.01 in steps of 0.0001
Class Number of Images
• DETECTION: between 0.0 and 0.9 in steps of 0.1
Person 2693 • TRAINED_ROIS_PER_IMAGE: between 10 and 80 in step of 10
Chair 580 • STEPS: between 10 and 100 in steps of 10
Car 535
Dining Table 501
• VALIDATION_STEPS: between 10 and 80 in steps of 10
• DETECTION_MAX_INSTANCES: between 10 and 100 in steps of 10
• MAX_GT_INSTANCES: between 10 and 100 in steps of 10
For our purposes, we have chosen to analyze the four most pop- • Number of training images: 1000, 10000 and all images in the
ulated classes of images in the dataset. The statistics about the used dataset.
classes for the training set and the validation set are shown in Table 2
and 3, respectively. Moreover, we have to consider which network layers to train: (i)
Heads: weights related only to fully-connected layers; (ii) 3+, 4+, 5+:
4.2. Mask-RCNN weights for the first 3/4/5 layers of ResNet; (iii) All: weights for all
layers. In this regard, several hybrid configurations have been tested
The implementation of Mask R-CNN used for the creation and eval- with different strategies as a learning rate variation every 𝑥 steps, a
uation of the model [1] is implemented in Python language, and it variation of the involved layers in the weight training process, and a
uses TensorFlow and Keras libraries. It has been re-engineered for in- combination of the above two strategies. The following pipelines have
tegration into our segmentation problem. Mask R-CNN has numerous been used for image augmentation:
parameters whose variation has a considerable impact on the final re-
sult. We have manually tested several combinations of parameters for • Horizontal rotation
each model. In this paper, we show the results for the ones which gave • Vertical rotation
us the best accuracy. These parameters are shown in Table 4. • One of the two previous random choices
The parameters listed above are related to the model configuration, • One of the two rotations chosen randomly followed by a 60, 120
together with the number of training epochs and the used data augmen- or 180 degree rotation always chosen randomly
tation pipeline. From a general point of view, we will have a decrement • One of the two rotations randomly chosen followed by a Gaussian
in the loss functions related to the training steps when epochs increase. or an average blur
Moreover, it is also necessary that we haven’t an increment of the loss • Linear contrast
function related to the validation set (validation loss) if the epochs will • From 0 to 2 operations random chosen between one of the two
increase. If this increment occurs, the model will be in overfitting. rotations, one of the three rotations of 60, 120 or 180 degrees and
In our implementation, the Mask-RCNN basic parameters have been one of the two blurs
set at the following maximum values:
In general, with the same parameters, the ResNet101 backbone
• NUMBER OF_EPOCHS=40 shows a better performance in terms of loss than ResNet50, as shown in
• STEPS_PER_EPOCH=100 Fig. 4. We decided to use a ResNet101 backbone for further experiments
• VALIDATION_STEPS=80 on image captioning.
• TRAIN_ROIS_PER_IMAGE=80 The best performance was obtained with the following parameters:

5
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Fig. 4. Comparison between ResNet50 backbone (in red) and Resnet101 backbone (in blue).

Fig. 5. Loss and Validation Loss Function for model 1 and 2.

• DETECTION MAX. 100 ten on 4+, and 20 on all. Model 2 has been trained for five epochs on
• DETECTION_MIN_CONFIDENCE 0.8 heads, five on 4+ and 30 on all; (ii) Learning rate: it is set to 0.0001
• LEARNING RATE 0.001 value after 20 epochs in model 1, it is set to 0.0001 after ten epochs
• MAX_GT_INSTANCES 100 for model 2. Due to the similarities in the loss trends, we calculate the
• STEPS_FOR_EPOCH 100 reference metrics to choose the best model. The results are shown in
• TRAIN FOR IMAGE 80 Fig. 6. The best model is the number 1, and some examples of object
• VALIDATION_STEPS 50 segmentation using Mask-RCNN are in Fig. 7, where the three images
• WEIGHT_DECAY 0.0001 shown are named “img16”, “img345” and “img1” respectively.
• All images of the dataset
• Pipeline 1
4.3. RetinaNet
We will show some examples of models resulting from different pa-
rameter settings. We analyze only the two best configurations to explain The used RetinaNet implementation for the creation and evaluation
our evaluation process. This process has been used with all the other of the model is the version present in [17]. It has been re-engineered to
DNN models. be integrated into our proposed instance segmentation pipeline.
In Fig. 5 we show a comparison of model 1 and 2. This network has a simpler implementation compared with Mask-
Model 1 is the one in blue, while model 2 is the one in red. We notice RCNN, and the tuning parameters for the training step are:
how they have a similar loss function at the end of the training epochs,
while the validation loss in the case of model 1 decreases more quickly, • Network Depth
and for model 2, it remains more or less constant for all the training • Learning Rate
epochs. The differences in configuration between these two models are • Number of epochs
(i) Epoch division: Model 1 has been trained for ten epochs on heads, • Batch size

6
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Fig. 6. Evaluation metrics.

This configuration has been chosen by experimental results obtained

with different parameter values, and the evaluation results are in Fig. 9.
Some examples of object segmentation are in Fig. 10.
In the first figure, the baseball player is detected as a person with
a very high score and a very precise bounding box. In the second fig-
ure, only 4 of the children present and one of the three chairs in the
foreground are detected, with variable scores and bounding boxes. In
the third figure, the yellow car in the background is detected with a
very high score (0.999) and a very accurate bounding box. On the other
hand, the car in the foreground has a lower score with a partial bound-
ing box.

4.4. YOLOv3

The implementation of YOLOv3 used for the creation and evaluation

of the model is the version present in the repository [36] written in
Python language, with Pytorch extension, and then reengineered the
original project to the proposed instance segmentation problem.
The training parameters are similar to the ones listed in the previous
models:

• Number of epochs
• Learning rate
• Learning momentum
• Weight decay
• Image size
• Multiscale (changing the image size during training)

The image augmentation has been performed with 3 operations:

Fig. 7. Examples of object segmentation using Mask R-CNN.
• contrast variation
• saturation variation
• brightness variation
ResNet101 has been chosen as the backbone due to the optimal
trade-off between execution speed and mAP. The figure The experimen- After several experiments, the best model for this network has been
tal results obtained using different parameters values Fig. 8 shows the defined with the following configuration and its loss trend is shown in
trend of the best model loss function trained with the following param- Fig. 11.
eter values.
• 100 epochs
• 50 epochs • Learning rate = 0.001
• Learning rate=0.0001 • Learning Momentum = 0.9
• Batch size=4 • Weight decay = 0.005

7
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Fig. 8. Loss trend of used Retinanet model.

Fig. 9. Model Evaluation.

Fig. 10. Examples of object detection using Retinanet.

• Image size = 512x512 4.5. Classes/superclass combiner

• Multiscale not used
In some cases, the object’s visual similarity in a first-class classi-
The evaluation results for the model are in Fig. 12. fication step could induce a misclassification because they represent
Some examples of object segmentation are in Fig. 13. different objects but in the same hierarchy. We try to solve this fine-
We can notice that the model gives optimal performances on person grain classification error by implementing a strategy to combine the
class. In fact, in the first image, it detects the baseball player, while in first results with the ones obtained from a direct superclass. For the
the second image, are recognized four persons where 3 of them have a detection of superclasses, the pre-trained weights used to perform the
very high score. In the third image, the car hasn’t a good detection both transfer learning have been used. In the following of this subsection,
from the score value and bounding boxes. we show the results performed using this strategy for the three used

8
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Fig. 11. Loss trend of used model for YOLOv3.

Fig. 12. Model evaluation.

Fig. 13. Examples of object detection using YOLOv3.

DNNs. The performance evaluation of the combined results obtained by chairs, previously classified as dining table, are now identified as furni-
the best-trained model on the classes and superclasses for Mask-RCNN ture. In the first object classification of the third image, the truck has
is shown in Fig. 14. been recognized as car while it is now classified as a vehicle superclass
As we can see, better results have been obtained in terms of mAP. with a higher score than the previous classification.
Fig. 15 shows the improvement of MASK-RCNN classification using su- Fig. 16 shows the performance of the combined results obtained by
perclasses. the best-trained model on classes and superclasses for Retinanet.
In fact, in addition to the two classified people in the first image, Also in this case we obtained better results in terms of mAP.
the baseball bat has been classified as “sports” superclass improving The improvements of Retinanet classification using superclasses are in
the overall detection. In the second image, instead, we notice that the Fig. 17.

9
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Fig. 14. Combined superclasses results with MASK-RCNN.

Fig. 15. Combined segmentation with MASK-RCNN.

Fig. 16. Combined superclasses results with RetinaNet.

In the first image, in addition to the baseball player identified as per- as outdoor superclass, have been added. The YOLOv3 performance for
son, the ball has been detected and classified as sports. In the second one, the best model trained on classes and superclasses is in Fig. 18.
the detections of three other persons, a book on the right side identified Also in this case we have obtained better results in terms of mAP.
as indoor, and the armchair on the top left side identified as furniture The classification results of this combination are shown in Fig. 19.
have been added to the previous detections. In the third image, the de- We observe that in the first image, another person is added to the
tection of the yellow car in the background has been kept, while the detection results together with the baseball identified as “sports”. In the
one in the foreground has been replaced by a detection of the vehicle second image, two more persons, two chairs classified as furniture, the
superclass more generic than the first classification but with a higher notebook classified as electronic, and the book classified as indoor have
score and a much more precise bounding box. In addition, the detec- been added to the objects previously detected. In the third image, the
tion of the truck identified as a vehicle and the parking meter, identified detected cars have been replaced by the superclass vehicle.

10
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Fig. 17. Combined segmentation with RetinaNet.

Fig. 18. Combined Metrics of YOLO model.

4.6. Final results and captioning Table 5

Captions for test images.

The image img16 contains 1 person and 1 sports

The previous examples show that each model can have different pre-
object on the left; 1 person and 1 sports object on the right.
cision in the object detection task. For this reason, instead of generating
The image img345 contains 2 electronic objects,
an image caption from the best result obtained by the models, we pro-
3 people, 1 furniture and 1 chair on the left; 3 people, 1 furniture
pose an approach based on the combination of results from these models and 3 indoor objects on the right.
in a single results vector using the strategy discussed in the subsection The image img1 contains 1
3.2. The overall model evaluation results are shown in Fig. 20 outdoor object and 2 cars on the left; 1 vehicle on the right.
Fig. 21 clearly state that we have better results using our combined
strategy with respect to each single model.
Some detection examples using this strategy are in Fig. 22. 5. Conclusion and future work
We explicitly point out that all objects have been detected and clas-
sified using our combined approach. For example, in the first image, In this work, we have proposed a system for the automatic gen-
the baseball is detected and classified with a maximum score of 0.856 eration of image captions. Our main contribution is the development
given by YOLOv3. The same object has been detected only by two of of a framework based on the combined use of several convolutional
the three used models. neural networks (CNN). We have also exhaustively investigated such
After the detection of the combiner module, the image caption is networks by evaluating several models based on them. The weaknesses
generated according to the approach presented in subsection 3.3. and strengths of each model have been highlighted and discussed. The
The image captions generated for our examples are shown in Ta- classification results have been used by means of a captioning module to
ble 5. generate a description of the image content in natural language, exploit-

11
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Fig. 19. Combined segmentation with YOLOv3.

Fig. 20. Metrics for the combination of the three architectures.

effectiveness of the combined task. A straight research direction would

look at comparing our framework with previous state-of-the-art works.
Further improvements might be obtained from an ablation study to bet-
ter evaluate each model contribution since several neural network ar-
chitectures are combined. Focused crawlers [6] could be integrated into
our framework for collecting images of specific domains of knowledge.
Further improvements can be considered to improve the performance
of the caption module using multimedia-semantic networks [29,28] or
knowledge graphs [31]. Different image features can be used to have
a better description of images together with semantic relations among
the detected objects.

CRediT authorship contribution statement

Fig. 21. Combined strategy comparison. All authors who contributed substantially to the study’s concep-
tion and design were involved in the preparation and review of the
manuscript until the approval of the final version. Antonio M. Ri-
ing both the spatial and lexical information provided by the networks. naldi, Cristiano Russo and Cristian Tommasino were responsible for the
The combination of the results gives us an impressive improvement literature search, manuscript development, and testing. Furthermore,
regarding the classifications and detection steps, as shown by experi- Antonio M. Rinaldi, Cristano Russo and Cristian Tommasino actively
ments, which indicates the goodness of our proposed approach and the contributed to all parts of the article, including interpretation of results,

12
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

Fig. 22. Combined strategy detection.

review and approval. In addition, all authors contributed to the develop- [8] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE
ment of the system for the performance of the system tests. All authors International Conference on Computer Vision, 2017.
[9] M.Z. Hossain, F. Sohel, M.F. Shiratuddin, H. Laga, A comprehensive survey of deep
have read and agreed to the published version of the manuscript.
learning for image captioning, ACM Comput. Surv. 51 (6) (2019) 1–36.
[10] P. Hurtik, V. Molek, J. Hula, M. Vajgl, P. Vlasanek, T. Nejezchleba, Poly-yolo: higher
Declaration of competing interest speed, more precise detection and instance segmentation for yolov3, arXiv preprint,
arXiv:2005.13243.
The authors declare that they have no known competing financial [11] J. Ji, Z. Du, X. Zhang, Divergent-convergent attention for image captioning, Pattern
Recognit. 115 (2021) 107928.
interests or personal relationships that could have appeared to influence
[12] A. Karpathy, L. Fei-Fei, Deep visual-semantic alignments for generating image de-
the work reported in this paper. scriptions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2015.
Data availability [13] G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A.C. Berg, T.L. Berg, Baby Talk:
Understanding and Generating Image Descriptions Proceedings of the 24th Cvpr,
The used data set is public and reference has been cited during the 2011.
[14] A. Kumar, S. Goel, A survey of evolution of image captioning techniques, Int. J.
text manuscript. Hybrid Intell. Syst. 14 (3) (2017) 123–139.
[15] C.-W. Kuo, Z. Kira, Beyond a pre-trained object detector: cross-modal textual and
References visual context for image captioning, in: Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, 2022.
[1] W. Abdulla, Mask r-cnn for object detection and instance segmentation on keras and [16] Y. Li, F. Ren, Light-weight retinanet for object detection, arXiv preprint, arXiv:1905.
tensorflow, https://github.com/matterport, Mask _RCNN. 10011.
[2] M.W. Akram, M. Salman, M.F. Bashir, S.M.S. Salman, T.R. Gadekallu, A.R. Javed, [17] T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollàr, Pytorch implementation of retinanet
A novel deep auto-encoder based linguistics clustering model for social text, Trans. object detection, https://github.com/yhenon/pytorch-retinanet, 2020.
Asian Low-Resource Lang. Inf. Process. (2022). [18] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C.L.
[3] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, L. Zhang, Bottom- Zitnick, Microsoft coco: common objects in context, in: European Conference on
up and top-down attention for image captioning and visual question answering, in: Computer Vision, Springer, 2014.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, [19] K. Madani, A.M. Rinaldi, C. Russo, A semantic-based strategy to model multime-
2018. dia social networks, in: Transactions on Large-Scale Data-and Knowledge-Centered
[4] M.F. Bashir, H. Arshad, A.R. Javed, N. Kryvinska, S.S. Band, Subjective answers Systems XLVII, Springer, 2021, pp. 29–50.
evaluation using machine learning and natural language processing, IEEE Access 9 [20] D. Mané, et al., Tensorboard: Tensorflow’s Visualization Toolkit, 2015.
(2021) 158972–158983. [21] J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, A. Yuille, Deep captioning with multi-
[5] M. Buric, M. Pobar, M. Ivasic-Kos, Ball detection using yolo and mask r-cnn, in: 2018 modal recurrent neural networks (m-rnn), arXiv preprint, arXiv:1412.6632.
International Conference on Computational Science and Computational Intelligence [22] E. Mohamed, A. Shaker, H. Rashed, A. El-Sallab, M. Hadhoud, Insta-yolo: real-time
(CSCI), IEEE, 2018. instance segmentation, arXiv preprint, arXiv:2102.06777.
[6] A. Capuano, A.M. Rinaldi, C. Russo, An ontology-driven multimedia focused crawler [23] V.-Q. Nguyen, M. Suganuma, T. Okatani, Grit: faster and better image captioning
based on linked open data and deep learning techniques, in: Multimedia Tools and transformer using dual visual features, in: Computer Vision–ECCV 2022: 17th Euro-
Applications, Springer, 2019, pp. 1–22. pean Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI,
[7] H. Fang, S. Gupta, F. Iandola, R.K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Springer, 2022.
Mitchell, J.C. Platt, et al., From captions to visual concepts and back, in: Proceedings [24] NVIDIA, P. Vingelmann, F.H. Fitzek, Cuda, release: 10.2.89, https://developer.
of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. nvidia.com/cuda-toolkit, 2020.

13
A.M. Rinaldi, C. Russo and C. Tommasino Results in Engineering 18 (2023) 101107

[25] L. Qi, Y. Wang, Y. Chen, Y.-C. Chen, X. Zhang, J. Sun, J. Jia, Pointins: Point-based [34] A.W. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain, Content-based image
instance segmentation, IEEE Trans. Pattern Anal. Mach. Intell. (2021). retrieval at the end of the early years, IEEE Trans. Pattern Anal. Mach. Intell. 22 (12)
[26] J. Redmon, A. Farhadi, Yolov3: an incremental improvement, arXiv preprint, arXiv: (2000) 1349–1380.
1804.02767. [35] G. Srivastava, R. Srivastava, A survey on automatic image captioning, in: Interna-
[27] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: towards real-time object detection tional Conference on Mathematics and Computing, Springer, 2018.
with region proposal networks, in: Advances in Neural Information Processing Sys- [36] Ultralytics, Pytorch implementation of yolov3, “YOLOv3 in PyTorch” https://
tems, 2015. github.com/ultralytics/yolov3, 2020.
[28] A.M. Rinaldi, C. Russo, K. Madani, A semantic matching strategy for very large [37] O. Vinyals, A. Toshev, S. Bengio, D. Erhan, Show and tell: a neural image caption
knowledge bases integration, Int. J. Inf. Technol. Web Eng. 15 (2) (2020) 1–29, generator, in: Proceedings of the IEEE Conference on Computer Vision and Pattern
publisher: IGI Global. Recognition, 2015.
[29] A.M. Rinaldi, C. Russo, C. Tommasino, A knowledge-driven multimedia retrieval [38] C. Wang, K. Huang, How to use bag-of-words model better for image classification,
system based on semantics and deep features, Future Internet 12 (11) (2020) 183, Image Vis. Comput. 38 (2015) 65–74.
publisher: Multidisciplinary Digital Publishing Institute. [39] Y. Wang, B. Xiao, A. Bouferguene, M. Al-Hussein, H. Li, Vision-based method for
[30] C. Russo, K. Madani, A.M. Rinaldi, Knowledge construction through semantic in- semantic information extraction in construction by integrating deep learning object
terpretation of visual information, in: International Work-Conference on Artificial detection and image captioning, Adv. Eng. Inform. 53 (2022) 101699.
Neural Networks, Springer, 2019. [40] Q. You, H. Jin, Z. Wang, C. Fang, J. Luo, Image captioning with semantic attention,
[31] C. Russo, K. Madani, A.M. Rinaldi, Knowledge acquisition and design using seman- in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
tics and perception: a case study for autonomous robots, Neural Process. Lett. (2020) 2016.
1–16. [41] J. Yu, J. Yao, J. Zhang, Z. Yu, D. Tao, Sprnet: single-pixel reconstruction for one-
[32] C. Russo, K. Madani, A.M. Rinaldi, An unsupervised approach for knowledge con- stage instance segmentation, IEEE Trans. Cybern. 51 (4) (2020) 1731–1742.
struction applied to personal robots, IEEE Trans. Cogn. Dev. Syst. 13 (1) (2020) [42] Y. Zhang, X. Shi, S. Mi, X. Yang, Image captioning with transformer and knowledge
6–15. graph, Pattern Recognit. Lett. 143 (2021) 43–49.
[33] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, T. Poggio, Robust object recognition
with cortex-like mechanisms, IEEE Trans. Pattern Anal. Mach. Intell. 29 (3) (2007)
411–426.

14