See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/318297570

Blockchain Based Distributed Control System for Edge Computing

Conference Paper · May 2017

DOI: 10.1109/CSCS.2017.102

CITATIONS READS
237 1,193

1 author:

Alexandru Stanciu
Institutul Naţional de Cercetare-Dezvoltare în Informatică
37 PUBLICATIONS   376 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

PEGAF - Grid enabled experimental platform for the development of applications oriented on workflows with dynamic allocation of ressources View project

GridMOSI - Grid enabled Virtual Organisation for High Performance Modelling, Simulation and Optimisation View project

All content following this page was uploaded by Alexandru Stanciu on 01 April 2023.

The user has requested enhancement of the downloaded file.

 Blockchain based distributed control system
for Edge Computing

Alexandru Stanciu
National Institute for Research and Development in Informatics
Bucharest, Romania
alex@ici.ro

Abstract—Edge computing proposes a novel model for There are similar concepts which have some overlap with
providing computational resources close to end devices that are edge computing, like mobile cloud computing, which is
connected to the network. It has numerous applications in Internet expected to have an external infrastructure (usually available as
of Things, as well as smart grids, healthcare, smart home, etc. This cloud services) for data processing and storage for mobile
paper presents ongoing research regarding the use of blockchain applications.
technology as a platform hierarchical and distributed control
systems based on IEC 61499 standard. Hyperledger Fabric was Another term used interchangeably with edge computing is
selected as the blockchain solution, where function blocks are to fog computing. It extends the cloud computing paradigm to the
be implemented as smart contracts on a supervisor level. The edge of the network, thus enabling a new breed of applications
integration with the edge nodes that perform on the executive level and services. Its main characteristics are: a) low latency and
responsible for actual process control is based on a micro-services location awareness; b) wide-spread geographical distribution; c)
architecture where Docker containers implement function blocks, mobility; d) very large number of nodes, e) predominant role of
and Kubernetes platform is used for orchestrating the execution wireless access, f) strong presence of streaming and real time
of containers across the edge resources. applications, g) heterogeneity. These characteristics make the
fog the appropriate platform for a number of critical Internet of
Keywords—edge computing; IoT; blockchain; smart contracts; Things (IoT) services and applications, namely, connected
hierarchical distributed control system
vehicle, smart grid, smart cities, and, in general, wireless sensors
I. INTRODUCTION and actuators networks [2].
The emergence of Internet connected smart devices which We have investigated the blockchain technology as a
can be accessed and controlled remotely via computer networks, platform for edge computing in order to implement a distributed
has raised the expectations for new and enhanced intelligent control system based on IEC 61499 standard.
computing services. The rest of the paper is organized as follows. Section II
However, security and privacy of user’s data, and specific presents an overview of the edge computing architecture. IEC
requirements regarding personal data protection, have 61499 standard for distributed control system is reviewed in
challenged the effectiveness of the centralized computing model Section III. In Section IV is discussed the blockchain
available on the public cloud infrastructures. A new computing technology, and the current research regarding implementation
model, with the emphasis on decentralization, and the of function blocks as smart contracts and the integration within
collaboration of individual work units to achieve a common the distributed control system. Related work is discussed in
goal, has been devised for Internet of Things (IoT) applications, Section V, and the conclusion are presented in the final section.
and for other domains such as the smart grid, healthcare,
connected vehicles, etc. II. EDGE COMPUTING ARCHITECTURE OVERVIEW
For many IoT applications which require mobility support,
The main characteristic of this model is that data processing
location awareness and low latency, there is a need of a new
occurs at the edge of the network, as close as possible to the
platform, one which can provide computational resources to
smart device which both produce and consume the data, and the
both large-scale sensor networks which monitor the
need of powerful computing resources is limited [1].
environment, as well as intelligent services based on data
Edge computing is a new paradigm dedicated to applications processing and cloud resources integration.
that require minimal latency, and which must process large
Edge computing nodes can be seen as members of a
quantity of data that is difficult to transfer over the network. As
decentralized network which provides compute, storage and
the cloud services are not suitable for such use cases because of
networking services to end devices.
the limited network bandwidth, security and privacy of data,
various solutions have been implemented to address computing Since smart devices are usually inadequate in computation
needs at the edge of the network. power, battery, storage and bandwidth, IoT applications and
services are usually backed up by strong server backends, which
are mostly deployed in the cloud, since cloud computing is
considered as a promising solution to deliver services to end
Hierarchical
users and provide applications with elastic resources at low cost Blockchain
Cloud services distributed control
[3]. system
Edge computing has not replaced the need for cloud
computing services. Sometimes, it is necessary to integrate
cloud resources in a three-layer architecture which is based on Edge nodes
devices, a mesh of edge nodes, and has cloud services on top. It
has several properties, such as its close distance to end users.
For latency sensitive applications, the computations must be Physical processes and devices
kept as close to the data as possible. Location-awareness is
another important property because intelligent services need to Fig. 1. Hierarchical distributed control system model for edge computing
use the context of the user to provide the best experience. For
data intensive applications, edge nodes can provide the first III. IEC 61499 STANDARD FOR DISTRIBUTED CONTROL SYSTEMS
steps for data processing, thus limiting the volume of data that
The IEC 61499 standard outlines a generic model for
should be transferred to the central cloud services.
distributed control systems, defining the architecture and the
It can be argued that while the edge nodes provide compliance requirements for software tools.
localization with context and low latency for data processing,
The promising application areas of IEC 61499 include
the cloud provides a central point of references which can
flexible material handling systems, in particular airport baggage
coordinate the edge nodes.
handling, flexible reconfigurable manufacturing automation,
This three-tier model for edge computing is necessary intelligent power distribution networks and smart grid, as well
especially for data intensive applications by reducing the data as the wide range of embedded networked systems.
transfer and storage requirements. Data processing in the edge
There were various attempts to create distributed control
nodes is thus extremely important for applications that produce
systems, first by using Programmable Logic Controllers (PLCs)
huge volumes of data, and which should provide real-time
to centrally process collected data from field area networks.
response to end users.
After that, it was experimented the integration of PLCs via
Another interesting application domain for edge computing networks, and then, by genuinely distributed automation
is the smart grid. In order to measure and control the distribution development, where the intelligence is designed from the very
network, smart meters are deployed on the field. There is a beginning as decentralized and embedded into software
central system to control the grid infrastructure which gathers components, which can be freely distributed across networked
and analyses the data received, and sends commands to adjust hardware devices [4].
any modifications with respect to supply and demand of
The IEC 61499 standard has been designed to support
resources.
distributed automation control systems. The main component is
With edge computing, this central automation system, can be the function block (FB), which has been designed like a process
integrated with a network of intelligent devices, in a hierarchical abstraction which is used in distributed computing systems - an
control structure. This covers the geographical distribution of independent computational activity with its own set of variables
the power grid and provides advanced services based on (context) and communication with other processes via
integration between the central and local resources. The central messages.
control structure on the higher level has a strategic role, whereas
Function blocks which are the atomic units of execution in
the local control systems on the lower level are responsible for
IEC 61499 based systems, consists of two parts, a function block
the real-time operation of field resources.
interface and an execution control chart that operates over a set
This type of role separation in a hierarchical distributed of events and variables. The execution of a FB entails accepting
control system is used in the context of edge computing model inputs from its interface, processing the inputs using the
as a criterion for the selection of the level on which its algorithms selected by the execution control chart, and emitting
components are deployed. For example, as presented in Fig. 1, outputs.
the principal components that are involved in strategy decision
Typically, an algorithm consists of loops, branching and
are placed on the higher level, and are implemented as smart
update statements, which are used to consume inputs and
contracts in a blockchain provided as a cloud service. The rest
generate outputs. The IEC 61499 standard allows algorithms to
of the components are part of the lower level of the hierarchical
be specified in a variety of implementation-dependent
distributed control system, and are implemented on the edge
languages.
nodes (which are close to the processes to be controlled).
Another function block type is the Service Interface
We will review in the next section the IEC 61499 standard
Function Block (SIFB). This represents the interface to low level
for distributed control systems, and will present our ongoing
services provided by the operating system or hardware of the
work related to the usage of blockchain technology in order to
embedded device, such as:
execute a critical part of distributed control algorithms in a
secure environment.
 • Graphical User Interface (GUI) elements such as a input Blockchains allow us to have a distributed peer-to-peer
fields and controls, network where non-trusting members can interact with each
other without a trusted intermediary, in a verifiable manner.
• Communication services,
Within the blockchain context, smart contracts are scripts
• Interfaces to hardware devices. stored on the blockchain. (They can be thought of as roughly
Service interface function blocks (SIFB) can be considered analogous to stored procedures in relational database
as device drivers that connect the external environment with management systems). Since they reside on the chain, they have
function block applications. IEC 61499 compliant software tools a unique address.
and their associated runtime packages can provide a large A smart contract can be triggered by addressing a transaction
selection of GUI and communications SIFBs. to it. It then executes independently and automatically in a
Device and resource models are used in the IEC 61499 prescribed manner on every node in the network, according to
standard to emulate the physical components (e.g. controllers, the data that was included in the triggering transaction. (This
sensors, actuators, etc.) as logical elements of the process implies that every node in a smart contract enabled blockchain
automation system. A conceptual model of the main entities and is running a virtual machine (VM), and that the blockchain
the relationship between them is presented in Fig. 2. network acts as a distributed VM) [7].

A device model is the functional definition of a physical A. Hyperledger Fabric

component in a larger distributed system. Each device may Hyperledger Fabric (github.com/hyperledger/fabric) is an
contain some inherent behaviour owing to its physical implementation of a distributed ledger platform for running
subcomponents. In order to manage the complexity of devices, smart contracts, leveraging familiar and proven technologies,
the concept of resource models is used. A device may contain with a modular architecture allowing pluggable
zero or more resources encapsulating independent function or implementations of various functions. It is one of multiple
tasks. projects currently in incubation under the Hyperledger Project.
A resource represents an independent task executing on a The distributed ledger protocol of the fabric is run by peers.
device. Such tasks are segregated from each other in such a way The fabric distinguishes between two kinds of peers: A
that a particular system resource (e.g. a sensor or an actuator) validating peer is a node on the network responsible for running
may only be accessed and operated upon by a single resource. consensus, validating transactions, and maintaining the ledger.
Due to the absence of shared variables, resources and devices On the other hand, a non-validating peer is a node that functions
communicate using communication function blocks in order to as a proxy to connect clients (issuing transactions) to validating
perform the coordination between tasks. peers. A non-validating peer does not execute transactions but it
may verify them.
resource The Hyperledger Fabric is a permissioned blockchain
platform aimed at business use. It is open-source and based on
FB FB standards, runs user-defined smart contracts, supports strong
security and identity features, and uses a modular architecture
with pluggable consensus protocols [8].
FB
SIFB SIFB resource B. Experimental testbed specifications
The use case that is the subject of our investigation on the
network topic of distributed control systems is related to the PID control
of dynamic systems.
Fig. 2. A basic overview of IEC 61499 distributed control systems
The three-tier edge computing model can be used to design
Each device is capable of performing a set of different tasks the components of the PID controller on the lower level, and
that coordinate by means of a communication network and, thus, include on the higher-level elements that are responsible for the
constitute a distributed system [5]. selection of algorithms and methods dedicated to the fine tuning
of the controller parameters.
IV. USING BLOCKCHAIN BEYOND BITCOIN TRANSACTIONS
A blockchain is essentially a distributed database of records, In order to experiment the execution of a distributed control
or public ledger of all transactions or digital events that have system based on IEC 61499 standard on edge computing nodes
been executed and shared among participating parties. Each an adapted environment from the Calculos project [9] was
transaction in the public ledger is verified by consensus of a proposed. The aim of Calculos was to design and implement an
majority of the participants in the system. Once entered, open platform for cloud services which can provide modeling
information can never be erased. The blockchain contains a and optimization with complex algorithms standardized as
certain and verifiable record of every single transaction ever function blocks, according to the IEC 61499 model, in order to
made. Bitcoin, the decentralized peer-to-peer digital currency, is be compatible, and usable in any control system.
the most popular example that uses blockchain technology [6]. The high-level architecture of the distributed control system
is presented in Fig. 3. As Function Blocks are implemented in
Docker containers, Kubernetes platform is used for applications are the equivalent of the SIFB in the IEC 61499
orchestrating the execution of containers across the edge nodes. standard.
Each FB has its control algorithms and execution logic We have evaluated the performance of Hyperledger Fabric
implemented in a Python program which is packaged together v0.6 regarding the number of transactions (invoke and query
with its dependencies in a container image. FBs communicate transaction types) executed per second (tps). For this
(events and data) using a publish/subscribe mechanism via a benchmarking, we have used the following Google Cloud
broker which is based on Redis message broker. Platform resources:
IEC 61499 resources which can contain several FBs are • n1-standard-4, a machine type with 4 vCPUs and 15 GB
implemented as Pods - a Kubernetes specific abstraction memory, which was able to execute 186 tps (query
dedicated to a group of linked containers, where inside a Pod, transactions) and 291 tps (invoke transactions).
each container can access any other containers.
• n1-standard-2, a machine type with 2 vCPUs and 7.5 GB
Pod Pod memory, which executed 170 tps (query transactions)
and 255 tps (invoke transactions).
FB FB FB FB The performance benchmarking was performed with
ub ub
b/s b/s hyperledger-py tool against a cluster of 4 validating nodes that
pu pu
Redis FB Redis FB
use Practical Byzantine Fault Tolerance consensus.
These numbers show that higher level of the distributed
SIFB Resource SIFB Resource
control system which is based on the blockchain technology has
a clear limitation regarding the load that can be effectively
REST REST processed in real time. However, as the role of this layer is to
Device
monitor and supervise the lower level, it is reasonable to expect
Service Service that number of function block that are executed as smart
Fig. 3. High level architecture of IEC 61499 components deployed as Docker contracts to be a small fraction of the total number of the
containers with Kubernetes platform function blocks that compose the automation system.
In addition, as the strategic decisions are based on the
The process interface and the user interface of the IEC 61499
evaluation of expected performance and related costs, the
control system are designed as REST APIs implemented by
importance of a secure medium to keep and execute these
Service Interface Function Blocks (SIFB). An IEC 61499
transactions is underlined by the possibility to automatically
Device contains one or more Resources which should be able to
enforce operational rules specified by users, and encoded in
communicate among each other via SIFBs.
smart contracts.
Kubernetes allows the definition of services as objects which
expose Pods resources. These Kubernetes services provide the V. RELATED WORK
REST interface to Resources, and they are the basis for a micro- Moving IoT components from the cloud onto edge hosts
service architecture. Each Resource implements a specific helps in reducing overall network traffic and thus minimizes
functionality of the distributed control system, and all the latency. However, provisioning IoT services on the IoT edge
Resources are loosely coupled in a message based architecture. devices presents new challenges regarding system design and
The Docker containers are deployed on edge nodes in maintenance. One possible approach is the use of software
addition to Hyperledger Fabric validating nodes. This defined IoT components in the form of virtual IoT resources.
architecture emulates the three tier edge computing model: This, in turn, allows exposing the thing/device layer and the core
physical devices and processes, edge nodes and cloud services. IoT service layer as collections of micro services that can be
The blockchain is deployed on the top level to ensure that distributed to a broad range of hosts. In [10] was investigated the
transactions are secured and properly validated. The blockchain concept of software-defined IoT components called virtual
can execute smart contracts which are implemented in Go resources and the use permission-based blockchains as a means
language, and thus the function blocks are designed as smart for distribution.
contracts that enforce the user decisions concerning high-level A new secure, private, and lightweight architecture for IoT,
strategies employed in the process automation system. based on blockchain technology that eliminates the overhead
One very important aspect which must be considered is that while maintaining most of its security and privacy benefits was
smart contracts cannot access outside data, such as remote APIs investigated on a smart home application as a representative case
or services. In order to be able to achieve consensus among study for broader IoT applications. The proposed architecture
blockchain nodes, the execution must be deterministic. was hierarchical, and consists of smart homes, an overlay
network and cloud storages coordinating data transactions with
The workaround this problem is based on special blockchain to provide privacy and security [11].
decentralized blockchain application which are used to push
data into the blockchain, so that other smart contracts can An approach for developing a campus-wide sensor network
consume it, and transfer data to the outside world. These special using commodity single board computers (Raspberry Pi) was
presented in [12]. Edge computation is made of per-node event
triggers, that are defined and monitored on each sensor device
directly. Further compute tasks might include local aggregation [4] V. Vyatkin, "IEC 61499 as enabler of distributed and intelligent
of data such as map/reduce or per-node user queries executing automation: State-of-the-art review," IEEE Transactions on Industrial
Informatics, vol. 7, no. 4, pp. 768-781, 2011.
on-device.
[5] L. H. Yoong, P. S. Roop, Z. E. Bhatti, and M. M. Kuo, Model-Driven
A framework supporting context-aware sensing, computing Design Using IEC 61499. Springer, 2015.
and communication capabilities into industrial applications was [6] M. Crosby, P. Pattanayak, S. Verma, and V. Kalyanaraman, "Blockchain
introduced in [13]. The solution provides context information technology: Beyond bitcoin," Applied Innovation, vol. 2, pp. 6-10, 2016.
sensing and processing, an access mechanism for interfacing [7] K. Christidis and M. Devetsikiotis, "Blockchains and Smart Contracts for
the Internet of Things," IEEE Access, vol. 4, pp. 2292-2303, 2016.
sensor networks with IoT and cloud services and a four-level
architecture to perform industrial process control that includes [8] C. Cachin, "Architecture of the Hyperledger blockchain fabric," in
Workshop on Distributed Cryptocurrencies and Consensus Ledgers, 2016.
the modules of a context-aware control platform.
[9] O. Chenaru, G. Florea, A. Stanciu, V. Sima, D. Popescu, and R. Dobrescu,
In [14] was presented an evaluation of Docker as an edge "Modeling Complex Industrial Systems Using Cloud Services," in
Control Systems and Computer Science (CSCS), 2015 20th International
computing platform based on four requirements: deployment Conference on, 2015, pp. 565-571: IEEE.
and termination; resource & service management; fault
[10] M. Samaniego and R. Deters, "Using Blockchain to push Software-
tolerance and caching. Due to its small footprint, good Defined IoT Components onto Edge Hosts," in Proceedings of the
performance and fast deployments, it was assessed that Docker International Conference on Big Data and Advanced Wireless
could be a viable Edge computing platform. Technologies, 2016, p. 58: ACM.
[11] A. Dorri, S. S. Kanhere, and R. Jurdak, "Blockchain in internet of things:
VI. CONCLUSIONS Challenges and Solutions," arXiv preprint arXiv:1608.05187, 2016.
Edge computing proposes a novel model for providing [12] K. Hentschel, D. Jacob, J. Singer, and M. Chalmers, "Supersensors:
Raspberry Pi Devices for Smart Campus Infrastructure," in Future
computational resources close to end devices that are connected Internet of Things and Cloud (FiCloud), 2016 IEEE 4th International
to the network. It has numerous applications in Internet of Conference on, 2016, pp. 58-62: IEEE.
Things, as well as smart grids, healthcare, smart home, etc. One [13] D. Merezeanu, G. Vasilescu, and R. Dobrescu, "Context-aware Control
important feature of edge computing is that the centralized Platform for Sensor Network Integration in IoT and Cloud," STUDIES IN
computing model based on cloud services is augmented with a INFORMATICS AND CONTROL, vol. 25, no. 4, pp. 489-498, 2016.
decentralized network of nodes that create an intermediate layer [14] B. I. Ismail et al., "Evaluation of docker as edge computing platform," in
between the sensors and devices and cloud services. For data Open Systems (ICOS), 2015 IEEE Confernece on, 2015, pp. 130-135:
IEEE.
intensive applications, it has the benefit of reduced data transfers
and increased responsiveness.
For many IoT applications, a distributed automation system
can be implemented as a hierarchical structure with two tiers,
with the higher level performing supervision and strategic
decisions, and the lower level having direct control of devices
and processes.
We have investigated the IEC 61499 standard for distributed
control systems, and have presented the ongoing research
regarding the implementation of function blocks as smart
contracts executed by the blockchain on a supervision level, as
well as the integration with the edge nodes that perform the
executive level responsible for process control.
ACKNOWLEDGMENT
This work was partly supported by the Romanian National
Research Programme PNII project “Cloud Architecture for an
open Library of Complex re-Usable Logical function blocks for
Optimized Systems – CALCULOS”, and by Nucleu project PN
16090401 “Evaluation and experimentation of Internet of
Things platforms”.
REFERENCES
[1] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, "Edge computing: Vision and
challenges," IEEE Internet of Things Journal, vol. 3, no. 5, pp. 637-646,
2016.
[2] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, "Fog computing and its
role in the internet of things," in Proceedings of the first edition of the
MCC workshop on Mobile cloud computing, 2012, pp. 13-16: ACM.
[3] S. Yi, Z. Hao, Z. Qin, and Q. Li, "Fog computing: Platform and
applications," in Hot Topics in Web Systems and Technologies (HotWeb),
2015 Third IEEE Workshop on, 2015, pp. 73-78: IEEE.

View publication stats