Maintenance 4.

0: Intelligent and Predictive

Maintenance System Architecture
Ana Cachada∗ , José Barbosa∗ , Paulo Leitão∗ , Carla A. S. Geraldes†‡ , Leonel Deusdado† , Jacinta Costa†
Carlos Teixeira§ , João Teixeira§ , António H.J. Moreira¶ , Pedro Miguel Moreirak , Luís Romerok
∗
Research Centre in Digitalization and Intelligent Robotics (CeDRI), Instituto Politécnico de Bragança,
Campus de Santa Apolónia, 5300-253 Bragança, Portugal
Email: {acachada, jbarbosa, pleitao}@[Link]
† Polytechnic Institute of Bragança, Campus Sta Apolónia, 5300-253 Bragança, Portugal

Email: {carlag, leodeus, jcosta}@[Link]

‡ Centro ALGORITMI - University of Minho, Campus Azurém, 4800-058 Guimarães, Portugal
§ Catraport, Lda, Zona Industrial de Mós, Lote n.o 1, 5300-692 Mós, Bragança

Email: {[Link], [Link]}@[Link]

¶ 2Ai – Polytechnic Institute of Cávado and Ave, Barcelos, Portugal

Email: amoreira@[Link]
k ARC4DigiT - Applied Research Center for Digital Transformation – Instituto Politécnico de Viana do Castelo

Av do Atlântico, 4900-348 Viana do Castelo, Portugal

Email:{pmoreira, romero}@[Link]

Abstract—In the current manufacturing world, the role of extremely significant, but unfortunately necessary to ensure the
maintenance has been receiving increasingly more attention while required productivity levels. For example, considering a press
companies understand that maintenance, when well performed, machine that has short stoppages of five minutes per hour,
can be a strategic factor to achieve the corporate goals. The latest
trends of maintenance leans towards the predictive approach, leading to 40 minutes of downtime per shift (considering a
exemplified by the Prognosis and Health Management (PHM) eight hour shift), if we consider that we have a production
and the Condition-based Maintenance (CBM) techniques. The rate of 14 pieces/minute, at the end of the shift the loss can
implementation of such approaches demands a well structured go up to 560 pieces. Assuming that the machine is producing
architecture and can be boosted through the use of emergent ICT an exit cone of the catalyst piece for Jaguar, for which the sale
technologies, namely Internet of Things (IoT), cloud computing,
advanced data analytics and augmented reality. Therefore, this price is about 1,26 e per piece, the total loss will be bigger
paper describes the architecture of an intelligent and predictive than 705,60 e per shift, since this value does not consider the
maintenance system, aligned with Industry 4.0 principles, that correspondent maintenance additional cost incurred to recover
considers advanced and online analysis of the collected data the equipment.
for the earlier detection of the occurrence of possible machine Nowadays, industrial maintenance is mainly reactive and
failures, and supports technicians during the maintenance inter-
ventions by providing a guided intelligent decision support. preventive, being the predictive strategy only applied for
critical situations. Traditionally, these maintenance strategies
Keywords: Industry 4.0, industrial maintenance, predictive main- are not taking into consideration the huge amount of data
tenance, data analysis, augmented reality. being generated in the shop floor and the available emergent
Information and Communications Technology (ICT), e.g.,
I. I NTRODUCTION Internet of Things (IoT), Big data, advanced data analytics,
Throughout the years, the manufacturing world has evolved, cloud computing and augmented reality. However, the main-
forcing all intrinsic and related departments to evolve as well. tenance paradigm is changing and industrial maintenance is
One of the areas that has evolved significantly is maintenance. now understood as a strategical factor and a profit contributor
Historically, industrial maintenance started as a necessary evil, to ensure productivity in industrial systems [1], [2]. This
meaning that maintenance operations were executed only when shift in the maintenance paradigm has led to the research
strictly necessary. However, in industrial manufacturing envi- and development of new ways to execute maintenance by
ronments, often characterized as being stochastic, dynamic and considering the operational state of assets and enabled the
chaotic, maintenance is a crucial issue to ensure production development of new maintenance approaches, such as the
efficiency, since the occurrence of unexpected disturbances Prognostic and Health Management (PHM), the condition-
leads to a degradation of the system performance, causing based maintenance (CBM), amongst others [3]. In its broad
the loss of productivity and business opportunities, which are sense, these approaches apply data analysis techniques to the
crucial roles to achieve competitiveness. In fact, the costs information produced in the shop floor processes to detect
associated to maintenance tasks in a production company are anomalies in the assets’ behavior.

978-1-5386-7108-5/18/$31.00 ©2018 IEEE 139

Authorized licensed use limited to: Thammasat University. Downloaded on September 24,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
 Thus, having in mind the improvement of the performance process where algorithms are used to detect anomalies, di-
of the production process, this work aims to develop an agnose faults and predict Remaining Useful Lifetime (RUL).
intelligent and predictive approach for the industrial mainte- Although the main goal of PHM is to provide the health state
nance, aligned with the Industry 4.0 principles, that considers and estimate the RUL of the components or equipments, also
advanced analysis of the data collected from the shop floor financial benefits such as operational and maintenance cost
to monitor and earlier detect the occurrence of disturbances reductions and extended lifetime are achieved [6]. A PHM
and consequently the need to implement maintenance ac- analysis involves a variety of steps including the collection
tions. This approach extends PHM and CBM maintenance of data and data characterization, the extraction of features
approaches by considering machine learning and augmented from collected data, and finally the diagnosis and prognosis.
reality technologies to support maintenance technicians during Essential steps for implementing a PHM system are discussed
the maintenance interventions by providing a guided intelligent in detail in [7].
decision support articulated by the use of human-machine According to Alaswad and Xiang [8], CBM is “a main-
interaction technologies. tenance strategy that collects and assesses real-time infor-
The rest of the paper is organized as follows: Section mation, and recommends maintenance decisions based on
II presents an overview on the most relevant concepts and the current condition of the system”. In summary, CBM
techniques associated to predictive maintenance. Section III policies predict the remaining useful life based on the cur-
describes the overall architecture for an intelligent and pre- rently observed system state, and suggests system inspection
dictive maintenance system. A detailed description of the and maintenance actions. Furthermore, it allows end users
architecture modules related to the data collection and analysis to perform better-planned maintenance, reduce or eliminate
for monitoring and early identification of maintenance needs unnecessary inspections, and decrease time-based maintenance
is presented in Section IV. Section V describes the decision intervals with confidence.
support system, detailing the module related to the intelligence There are various international standards related to the
of the decision support engine and how augmented reality will CBM approach, for example the ISO 13374 [9] addresses the
be used to support the execution of maintenance operations. Open System Architecture for Condition-Based Maintenance
Finally, Section VII rounds up the paper with the conclusions. (OSA-CBM), held by MIMOSA [10], representing formats
and methods for communicating, presenting, and displaying
II. R ELATED W ORK relevant information and data. Initially, OSA-CBM comprised
In modern manufacturing systems, the role of maintenance seven generic layers to attain a well constructed system [11],
is becoming more important contributing to the organization but currently considers six functional blocks [10]:
profit, clearly bringing the need for the maintenance operation 1) Data Acquisition: provides the access to digitized sensor
to be in harmony with the corporate objectives [2]. or transducer data and records this data.
Usually, the maintenance management is categorized into 2) Data Manipulation: may perform single and/or multi-
different policies: (i) corrective or run-to-failure maintenance, channel signal transformations and may apply special-
(ii) preventive maintenance, and (iii) predictive maintenance. ized feature extraction algorithms to the gathered data.
Corrective maintenance is an unscheduled repair, where equip- 3) State Detection: performs condition monitoring by com-
ments are allowed to operate until they fail, moment in paring features against expected values or operational
which a maintenance intervention is performed. Preventive limits and returning conditions indicators and/or alarms.
maintenance, probably the most popular maintenance policy, 4) Health Assessment: determines if the system’s health is
is a regularly performed set of actions on an equipment to suffering degradation by considering trends in the health
lessen the likelihood of it failing. This type of maintenance is history, operational status and maintenance history.
performed while the equipment is still working and is planned 5) Prognostics Assessment: projects the current health state
so that any required resources are available. Finally, predictive of the asset into the future by considering an estimation
maintenance is a philosophy or attitude that uses the actual of future usage profiles.
operating condition of the plant equipments and systems to 6) Advisory Generation: provides recommendations related
optimize the plant operations and/or processes [4]. Predictive to maintenance actions and modification of the asset
maintenance concerns the application of sensor technology and configuration, by considering operational history, current
analytical tools to predict when equipments’ failures might and future mission profiles and resource constraints.
occur and to prevent the occurrence of the failures by per- In the OSA-CBM architecture, data flow usually occurs
forming maintenance. The failures’ prediction can be done by between adjacent functional blocks. Nevertheless, if required,
applying, e.g., vibration monitoring or thermography, and must each block may be able to request data from non adjacent
be effective at predicting failures and also provide sufficient functional blocks [11].
warning time for the upcoming maintenance. When this policy In the recent years many efforts have been done in order
is working effectively, maintenance is only performed on to implement predictive maintenance in the factories. For
equipments when it is required. example, the Senseye company [12] provides a system that
The PHM concept is often used with other approaches like gathers data from several sources, analyzes this data and
predictive maintenance and CBM [5]. PHM is an engineering sends a notification to a designated person every time a

140

Authorized licensed use limited to: Thammasat University. Downloaded on September 24,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
abnormality is detected or failure is predicted. This solution The Visualization component allows to compare Key Per-
uses machine learning to perform condition monitoring and formance Indicators (KPIs) against the expected operational
prognosis analysis. Other example, is the Watchdog Agent- limits. In order to determine these operational limits the facts
based Real-time Remote Machinery Prognostics and Health resultant from the Off-line Data Analysis are considered and
Management (R2MPHM) platform presented and detailed in the raw data is displayed in a graphic format to facilitate
[13]. The Watchdog Agent consists of embedded compu- its interpretation. On the other hand, the Early Detection of
tational prognostic algorithms and a software toolbox for Failures component processes the facts and rules through the
predicting degradation of devices and systems. The Watchdog use of an inference engine, and triggers a maintenance warning
Agent-based R2MPHM platform receives data, processes it when an anomaly is detected in an earlier stage. Depending on
and extracts features that allows to detect possible failure the detected anomaly, the maintenance warnings can lead to
occurrence and supports the estimation of the remaining useful different maintenance actions, namely corrective, preventive
life. For this purpose, the Watchdog Agent Toolbox makes or predictive. Once the need for a maintenance intervention
use of several techniques and algorithms, such as Neural is detected, this information is sent to the scheduling tool,
Networks, Bayesian Belief Network, Fuzzy Logic Prediction. that will schedule the intervention according to the current
In spite of the referred benefits, the application of data production state and the maintenance resources availabilities.
acquisition, data processing, fault diagnostics, prognostics, and In spite of the importance of the scheduling system, this is out
decision reasoning are still not mature enough [14], neither of scope of this work and consequently will not be detailed in
working in an integrated manner. For this reason, companies this paper.
do not have yet enough trust on these techniques and technolo- The execution of scheduled maintenance interventions is
gies to implement to handle intelligent and predictive mainte- guided and supported by a decision support system that
nance tasks. Another reason for the lack of implementation is selects the appropriate maintenance procedure and translates
related to the costs associated with integration of an intelligent it into a language understandable by the human. The In-
system with the existing systems as well with the reluctance telligent Decision Support module is also able to adapt or
of change from the human players. create new maintenance procedures in cases that there are
Another key aspect, for this maintenance innovations be no known maintenance procedures for the detected anomaly.
fully exploitable, is taking place during the maintenance proce- The maintenance procedure is provided to the maintenance
dure phase, i.e. when the actual repair actions are taking place. technician, while performing the required maintenance actions,
Here, machine learning algorithms allied with augmented by using advanced Human-Machine Interfaces (HMI), e.g.,
reality will have a symbiotic connection aiming to provide head mounted devices.
an intelligent, interactive, simple and effortless maintenance Comparing the proposed system architecture with the OSA-
repair operation. In fact, machine learning algorithms will CBM architecture, it is possible to verify that both present
dynamically guide the maintenance repair technician while the similar functional blocks. The Data Collection module of the
virtual reality environment will provide a comfortable human- proposed architecture is equivalent to the Data Acquisition
system interaction. functional block of the OSA-CBM. Also, the Data Collection
module fits, partially, within the Data Manipulation func-
III. S YSTEM A RCHITECTURE tional block of the OSA-CBM. The Off-line Data Analysis
module is aligned with the Data Manipulation and the State
A proper system architecture for condition-based mainte- Detection functional blocks since the knowledge needed to
nance should present specific modules such as those described perform condition monitoring is generated in this module. The
in the previous section. The proposed system architecture Dynamic Monitoring of the proposed architecture matches the
integrates all the aforementioned referred modules to create a State Detection, Health Assessment and Prognosis Assessment
functional system that allows the implementation of intelligent functional blocks of the OSA-CBM architecture. The Decision
and predictive maintenance, taking advantage of a broad spec- Support module is completely aligned with the Advisory
trum of technologies, such as IoT, machine learning, expert Generation functional block. Figure 2 illustrates how the
systems, among others. The developed architecture is depicted OSA-CBM architecture was adapted to the Maintenance 4.0
in Figure 1. approach.
The system functionality is initiated with the Data Collec-
tion module, where the data from several sources is collected IV. DATA A NALYSIS FOR M ONITORING AND E ARLY
and stored in a database. This database will feed the Off- I DENTIFICATION OF M AINTENANCE N EEDS
line Data Analysis module, where advanced data analytics,
machine learning and cloud technologies are used to perform This section concerns the Data Collection, Off-line Data
the knowledge generation. The outputs of this module are the Analysis and Dynamic Monitoring modules, which are able
generation or adjustment of rules, procedures and facts, which to perform the analysis of the collected data to perform the
will be used by the Dynamic Monitoring functional block. monitoring of the assets’ condition and the early detection of
The Dynamic Monitoring module is divided into two com- failures, and consequently detects the need for maintenance
ponents, the Visualization and the Early Detection of Failures. interventions.

141

Authorized licensed use limited to: Thammasat University. Downloaded on September 24,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
 Figure 1. System architecture for the intelligent and predictive maintenance 4.0.

This predictive capability contributes for the reduction of the consideration relevant features, such as the information pre-
production process downtime, the optimization in planning the sentation and the type of interface. The overall effects of the
maintenance interventions, and transforms the traditional “fail HMI appliance could not be totally predictable or even mea-
and recover” practices into “predict and prevent” practices. surable since they do not depend only on the system design.
The design goals and the consequent application should take
A. Interfaces Design for Data Collection into account the hierarchy of needs, such as 1) substantial
The Data Collection module considers different ways to procedures and advices, 2) continuous performance efficiency
collect data, namely automatic, semi-automated and manual checklist, 3) ratings that come from data assessments, 4) spec-
data collection. The data collection is considered automatic ifications concerning reliability and validity, and 5) usability
when is automatically acquired and stored in the database and efficacy of the system.
that will feed the rest of the system, semi-automatic when the HMIs should be designed to involve users in the definition
information is automatically recorded in a database but has to of the display and customization of the right assistance by
be manually transferred to the system’s database. Finally, the applying the User Centered Design approach [15]. Also, the
manual data collection occurs through the use of interfaces, user interfaces must follow ergonomic guidelines established
where the information is manually inserted by a worker. for the well-design of software and hardware systems. The
Thus, this module can receive information from multiples following guidelines are some examples of the broad spectrum
sources, i.e. from several machines or assets, or from several of guidelines that must the considered for the interfaces design:
departments or services. The collection of the required data is • The interface should never give rise to potentially haz-
performed considering the IoT technologies and the ergonomic ardous behavior for the user (ISO 4040).
standards of HMI. • The place of the system should never damage the user
The design principles, to create user interfaces for data responsiveness and must remain compatible with the
collection in industrial environments, should always take into hierarchy of attention demand, e.g., closeness, line of

142

Authorized licensed use limited to: Thammasat University. Downloaded on September 24,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
 Collecting non-automated information during the produc-
tion process is a complex process, since the operator will
have a new focus of attention. The type of information to
be collected would be the classification of the defects (and
the corresponding timestamp) of the components produced,
marked at the end of the production line, when manually
inspected one by one. Hand-held devices, such as tablets
interfaces, will assist workers during the quality inspection
procedure to the task of data collection in real time production,
and also by specialists from the quality department, in a
subsequent statistical analysis of the data collected. The layout
of the developed interface app for the manual data collection
is depicted in Figure 3, and operates as a bridge between
human intentions and interactions to collect important data so
that they can be stored and available in a real time automatic
system machine.

Figure 2. Adaptation of the interaction between the functional blocks of the

OSA-CBM for the Maintenance 4.0 project.
Figure 3. User interface for the defects’ data collection.

sight and comfortable position. In a technological perspective, HTML, AngularJS, PHP and
• The interface should be simple and intuitive, simplifying CSS were used to guarantee portability and transversality of
to understand the information and allowing to be focused the application in different mobile operating systems, as well
in the execution of other tasks, e.g., visual inspection. as the web connection to collect and record data in real time,
• The interaction should consider the ergonomic perfor- as illustrated in Figure 4. HTML and CSS were used to
mance of the user dialogue commutation - range of configure the appearance of the several interface elements to
movement, point-click movement, controls obstruction be displayed on a web page of any browser. AngularJS was
and display. essential in the application of dynamic functions, so that all
• The developed menus should always consider simple data was shown to the operator automatically, and updated in
structures that promote grouping guidelines of functions real time. PHP, as a server language, established the connection
to give good ‘visuality’ to the interaction system. between the web interface and the database, allowing to send
• The total alignment dialogue trough all elements regard- and receive information.
ing visual accuracy, aesthetic, and order should have a
proper language that will not compromise performance.
Considering the case study considered in this work, that
is related to an industrial metal stamping machine producing
components for the automotive industry, this module receives
inputs from several sources, namely from a stamping machine,
operators and the quality department. The information received
from the machine consists of a buffer that registers the
actions of the machine including the failures occurred, and the
information related to some sensors. Regarding the operator, Figure 4. Architecture of the app for the collection of data related to defects.
the information collected refers to the defect occurrence and
is collected through the use of a simple and intuitive interface After the conceptual analysis of the involved information
that registers the type of defect detected and the time of flow, the related set of classes were defined to store the
occurrence. elemental information to the collection of data structures.

143

Authorized licensed use limited to: Thammasat University. Downloaded on September 24,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
The class entities resulting from the conceptual model are
presented in the Database, and related to the registration of
the information concerning Operators, Defects, Production
Batches and the respective evaluated Produced Pieces. All
this information is collected, interconnected, and stored in
a MySQL database, communicating with the interface page
through SQL commands present in PHP.
This type of information, associated to the defects and
timestamps signaled by the workers at the end of the assembly
line during the inspection task, will be presented in real time
to the quality department. In the quality department, with the
Figure 5. Proposed RNN model architecture based on a LSTM layer to predict
help of a tablet with a dashboard interface, real-time statistical the next failure/event type.
information will be updated from the data collected to predict
future changes in quality of pieces at the production lines.
C. Dynamic Monitoring
The Dynamic Monitoring module comprises two compo-
B. Off-line Data Analysis
nents, the Visualization and the Early Detection of Failures.
This module will take advantage of several technologies, The Visualization receives inputs from the Off-line Data
namely advanced data analytics, machine learning and cloud Analysis module that provides information about the normal
technologies to extract knowledge from the collected data in functioning of assets operational limits, and data from the Data
order to create new monitoring rules and procedures or update Collection module, which allows a real-time monitoring of
the existing ones. For this purpose, a deep machine learning specific variables. The real-time parameters will be graphi-
approach with supervised learning for the early fault prediction cally represented in a chart and when a parameter exits its
and predictive maintenance was developed. The algorithm was operational interval, or the trend shows that it will eventually
trained from the input data of previous events/faults, labeled exit its operational interval, the data point is labeled in the
accordingly to the type of event (fault or operation), rather graphical interface. Some examples of the parameters that will
than being explicitly programmed by a set of static rules. Due be visualized are described in Table I.
to the large dataset (more than 43000 events from 10 working
days), the deep machine learning approach has the advantage Table I
E XAMPLE OF COLLECTED DATA PARAMETERS .
of detecting underling patterns that may not be detected by a
human operator/programmer [16], [17]. Parameter Description
The evaluation of this machine learning approach was tested Type of defect Type of defect detected while the visual inspection
on two types of recurrent neural networks (RNN), the long is performed
short-term memory (LSTM) and the gated recurrent unit Date Date and time when the defect is detected
(GRU) [18]. These networks are especially attractive to the Part reference Identification of the part where the defect is de-
predictive maintenance domain since they can learn from past tected
sequences and forecast the next probable event. The input Pressure Pressure of the hydraulic piston during the oper-
data consists in more than 43000 previous events, each of ation
which is represented by a preprocessed encoded feature vector Temperature Temperature in the surroundings of the equipment
that encodes the event type (e.g. sensor activation, axis error, Humidity Humidity in the surroundings of the equipment
missing parts, etc.), the day of week, the time of day (in 5
Vibration Vibration of some components of the equipment
min blocks) and the time from a past event, in a total of 147
features. The data was separated into a training and validation Operational noise Characteristic noise produced during the operation
of the equipment
set and a testing set by 70% and 30%, respectively.
The overall model architecture, represented in Figure 5,
consists in a LSTM/GRU layer feed with all sequences of On the other hand, the Early Detection of Failures is
events through each internal layer, the final LSTM state performed by a inference engine that uses a set of rules to
encodes the machine behavior, including relations between match the existing facts, which can be retrieved from the
past and recent events. The final LSTM state is then input in a database where the collected data is being stored. The rules are
fully connected output layer which determines the probability generated in the Off-line Data Analysis module through the
of a specific fault type in the next 5 minutes block. Training detection of the correlation between different parameters. The
was accomplished using an LSTM network with 200 cells, rules follow a simple structure based on the If Condition
Adam optimizer and binary cross entropy as loss function then Action syntax, which are processed by the infer-
through 50 epoch. ence engine. When a rule is fired, a correspondent action

144

Authorized licensed use limited to: Thammasat University. Downloaded on September 24,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
is triggered, namely a warning for the need of maintenance the engine will search for maintenance procedures applied to
interventions. failures with similar features and show it to the operator.
Both Visualization and Early Detection of Failure compo-
nents may exchange data. B. Augmented reality to support interactive maintenance op-
When the need for maintenance interventions is detected, erations
the information is sent to the Decision Support module through Augmented Reality (AR) is not a new technology, but
the scheduling tool. instead it has been an active research area since almost three
decades. AR is an interesting technology as it enhances the
V. I NTELLIGENT D ECISION S UPPORT FOR M AINTENANCE
user’s interaction and the perception of the real world by
I NTERVENTIONS
supplementing it with virtual things that coexist in the same
An important piece in this intelligent and predictive mainte- space as the real ones. Therefore, AR supplements reality,
nance architecture is the decision support system for mainte- rather than replacing it [20], [21]. Recent interest is being
nance technicians during the execution of maintenance inter- driven by enhancements of graphics capabilities, in particular
ventions. This intelligent decision support system, articulated in mobile and wearable devices, and in the plentifulness
with human-machine interaction technologies, e.g. augmented of wide assortment of sensors and, as well as, the support
reality, contributes for a faster and more efficient reaction and of advances on tracking combined with the availability of
recovery of the failure occurrence when compared to paper affordable AR enabled hardware, such as Microsoft HoloLens
procedures [19]. [22], DAQRI smart glasses [23] or Atheer AiR Glasses, made
A. Intelligence of decision support engine AR an emergent topic of applied research.
All of this is also supported by the growth in the number
The Intelligent Decision Support module is composed by a of things with a virtual counterpart. Information (3D models
database, which can be shared by the database used in the other and animations, real time information, documentation, etc.)
modules, an engine that processes the maintenance procedures, regarding every thing with a virtual/digital existence, feeds
a tool that allows to update or create maintenance procedures applications that can supplement the user with a multitude of
and a human-machine visualization tool. relevant, up-to-date and contextualized knowledge. Thus, AR
The database contains, amongst others, the maintenance technologies are interesting to many sectors, including indus-
procedures expressed in a formal logical representation, e.g., trial applications, from design through maintenance processes.
BPMN (Business Process Model and Notation) as the example Since its origination, several AR applications have been
depicted in Figure 6, which defines the sequences of single envisioned and conceived to the manufacturing sector, such
maintenance tasks. This database is connected to an inference as assembly, maintenance, and repair of machinery [20], [21],
engine that selects the proper maintenance procedure, and [24], [25]. There have been several systems demonstrating
translates the procedure into a language understandable by the the ability and usefulness of AR to act as an instruction and
human that will be applied in the maintenance intervention. guidance tool [26]. Recently, the AR adoption by key players
on industrial innovation, such as Boeing and General Electrics,
resulted in the following main benefits: improved productivity,
higher product and process quality (decreasing error rates), and
better ergonomics.
This augmented reality component is being implemented
with a hololens head mounted display and a tablet display. In
both cases, the operator sees the real scene (the machinery) and
sees the maintenance operation augmented over the real scene.
The first offers an immersive experience where the interaction
is done with hand movements to make selections. The second
follows a traditional approach through the tablet display and
the interaction is achieved through touch. The task is entrusted
to a operator that follows the instructions presented on the AR
display. In a first approach the system will be aimed for the
operator training.
Each maintenance operation has several steps to follow
Figure 6. Example of a maintenance procedure expressed in BPMN format. in order to successfully achieve the task. The augmented
reality process is supported by text, audio and 3D models
For each case, if the necessary maintenance intervention and animations. The text describes the operation step together
does not have a correspondent maintenance procedure in the with an audio description which can be controlled during
database, the engine enriched with machine learning, and the process. The 3D models and animations represent the
particularly learning by analogy algorithms, will attempt to machinery and explains the operation with a 3D animation of
adapt or create a new maintenance procedure. This means that the process to be done. It may assume that pieces are involved

145

Authorized licensed use limited to: Thammasat University. Downloaded on September 24,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
 in the maintenance process, in which case, they are identified [3] A. Garg and S. G. Deshmukh, “Maintenance management: literature
by its 3D model and factory reference. review and directions,” Journal of Quality in Maintenance Engineering,
vol. 12, no. 3, pp. 205–238, 2006.
At each maintenance step, the operator must confirm its [4] R. K. Mobley, An Introduction to Predictive Maintenance, 2nd edition.
completion before continuing, by interacting with the display Elsevier Science (USA), 2002.
interface. The interface includes buttons to confirm steps, to [5] J.-H. Shin and H.-B. Jun, “On condition based maintenance policy,”
Journal of Computational Design and Engineering, no. 2, pp. 119–127,
retrieve steps, to cancel the whole operation or to open a 2015.
communication channel with a senior operator. The operator [6] T. Sutharssan, S. Stoyanov, C. Bailey, and C. Yin, “Prognostic and
may be assisted by a more experienced operator during the health management for engineering systems: a review of the data-driven
approach and algorithms,” The Journal of Engineering, no. 7, pp. 215–
maintenance process. For such purpose, the operator may start 222, 2015.
a video or audio conference with an assistant to make any [7] S. Das, R. Hall, and S. Herzog, “Essential steps in prognostic health
questions concerning doubts or problems one may have during management,” in 2011 IEEE Conference on Prognostics and Health
Management (PHM). IEEE, 2011, pp. 1–9.
the maintenance process. [8] S. Alaswad and Y. Xiang, “A review on condition-based maintenance
optimization models for stochastically deteriorating system,” Reliability
VI. C ONCLUSION Engineering and System Safety, vol. 157, pp. 54–63, 2017.
[9] ISO, “ISO 13374, Condition monitoring and diagnostics of machines —
Nowadays maintenance is considered as an integral part of Data processing, communication and presentation,” 2012.
the manufacturing process that contributes to the product qual- [10] Mimosa - An Operations and Maintenance Information Open
ity, plant availability and ability to meet delivery schedules. System Alliance, “MIMOSA OSA-CBM,” 2010. [Online]. Available:
[Link]
This is particularly important in manufacturing companies [11] M. Lebold, K. Reichard, C. S. Byington, and R. Orsagh, “OSA-CBM
that adopt modern management philosophies. The growing architecture development with emphasis on XML implementations,”
importance of maintenance management has generated an Maintenance and Reliability Conference (MARCON), pp. 6–8, 2002.
[12] Senseye, “Forecast machine failure, scalable predictive maintenance.”
increasing interest in the development and implementation [Online]. Available: [Link]
of efficient maintenance strategies that are able to improve [13] L. Liao, H. Wang, and J. Lee, “A reconfigurable watchdog Agent R
the system reliability, prevent system failures, and reduce for machine health prognostics,” International Journal of COMADEM,
vol. 11, no. 3, pp. 2–15, 2008.
maintenance costs. [14] B. Sun, S. Zeng, R. Kang, and M. G. Pecht, “Benefits and challenges
In this context, predictive maintenance policies, together of system prognostics,” IEEE Transactions on Reliability, vol. 61, no. 2,
with, e.g., PHM and CBM approaches for maintenance man- pp. 323–335, 2012.
[15] G. A. Boy, The Handbook Of Human-Machine Interaction A Human-
agement, can be considered important tools to industries in Centered Design Approach. Ashgate Publishing Limited, 2011.
the era of emerging ICTs. This paper introduces the archi- [16] D. Dong, X.-Y. Li, and F.-Q. Sun, “Life prediction of jet engines based
tecture of an intelligent and predictive maintenance system, on lstm-recurrent neural networks,” pp. 1–6, 2017.
[17] O. Aydin and S. Guldamlasioglu, “Using lstm networks to predict engine
aligned with Industry 4.0 principles and following the struc- condition on large scale data processing framework,” pp. 281–285, 2017.
ture of functional blocks of the OSA-CBM. This system [18] A. Karpathy, J. Johnson, and L. Fei-Fei, “Visualizing and understanding
architecture considers advanced and online analysis of the recurrent networks,” arXiv preprint arXiv:1506.02078, 2015.
[19] V. Havard, D. Baudry, X. Savatier, B. Jeanne, A. Louis, and B. Mazari,
collected data for the earlier detection of the occurrence of “Augmented industrial maintenance (AIM): A case study for evaluating
possible machine failures, and supports technicians during the and comparing with paper and video media supports,” in Lecture Notes
maintenance interventions by providing a guided intelligent in Computer Science (including subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics), vol. 9768, 2016, pp.
decision support. The proposed system extends the basic CBM 302–320.
functionalities with the integration of decision support systems [20] R. Azuma, “A survey of augmented reality,” Presence, vol. 6, no. 4, pp.
and augmented reality technologies to enhance the interaction 355–385, 1997.
[21] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. Mac-
among humans and machines, improving the performance of Intyre, “Recent advances in augmented reality,” IEEE Comput. Graph.
the execution of maintenance interventions. Appl., vol. 21, no. 6, pp. 34–47, Nov. 2001.
Future work will be devoted to the further development [22] B. C. Kress and W. J. Cummings, “Towards the ultimate mixed reality
experience: Hololens display architecture choices,” SID Symposium
of the several system architecture modules, and the posterior Digest of Technical Papers, vol. 48, no. 1, pp. 127–131, 2017.
deployment in a real industrial production unit focusing the [23] P. Greenhalgh, B. Mullins, A. Grunnet-Jepsen, and A. K. Bhowmik,
metal stamping for the automotive sector. “Industrial deployment of a full-featured head-mounted augmented-
reality system and the incorporation of a 3d-sensing platform,” SID
ACKNOWLEDGMENT Symposium Digest of Technical Papers, vol. 47, no. 1, pp. 448–451,
2016.
The work reported in this paper was supported by the FCT [24] S. Henderson and S. Feiner, “Exploring the benefits of augmented
(Fundação para a Ciência e a Tecnologia) Project SAICT- reality documentation for maintenance and repair,” IEEE Transactions
on Visualization and Computer Graphics, vol. 17, no. 10, pp. 1355–
POL/23725/2016. 1368, Oct 2011.
[25] A. Nee, S. Ong, G. Chryssolouris, and D. Mourtzis, “Augmented reality
R EFERENCES applications in design and manufacturing,” CIRP Annals, vol. 61, no. 2,
[1] M. Faccio, A. Persona, F. Sgarbossa, and G. Zanin, “Industrial main- pp. 657 – 679, 2012.
tenance policy development: A quantitative framework,” International [26] S. Werrlich, E. Eichstetter, K. Nitsche, and G. Notni, “An overview
Journal of Production Economics, vol. 147, no. PART A, pp. 85–93, of evaluations using augmented reality for assembly training tasks,”
2014. International Journal of Computer, Electrical, Automation, Control and
[2] A. Sharma, G. S. Yadava, and S. G. Deshmukh, “A literature review and Information Engineering, vol. 11, no. 10, pp. 1142 – 1148, 2017.
future perspectives on maintenance optimization,” Journal of Quality in
Maintenance Engineering, vol. 17, no. 1, pp. 5–25, 2011.

146

Authorized licensed use limited to: Thammasat University. Downloaded on September 24,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
Powered by TCPDF ([Link])