Advances in Intelligent Systems and Computing 926

Leonard Barolli
Makoto Takizawa
Fatos Xhafa
Tomoya Enokido   Editors

Advanced
Information
Networking and
Applications
Proceedings of the 33rd International
Conference on Advanced Information
Networking and Applications (AINA–2019)
Advances in Intelligent Systems and Computing

Volume 926

Series editor
Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences,
Warsaw, Poland
e-mail: [email protected]
The series “Advances in Intelligent Systems and Computing” contains publications on theory,
applications, and design methods of Intelligent Systems and Intelligent Computing. Virtually all
disciplines such as engineering, natural sciences, computer and information science, ICT, economics,
business, e-commerce, environment, healthcare, life science are covered. The list of topics spans all the
areas of modern intelligent systems and computing such as: computational intelligence, soft computing
including neural networks, fuzzy systems, evolutionary computing and the fusion of these paradigms,
social intelligence, ambient intelligence, computational neuroscience, artificial life, virtual worlds and
society, cognitive science and systems, Perception and Vision, DNA and immune based systems,
self-organizing and adaptive systems, e-Learning and teaching, human-centered and human-centric
computing, recommender systems, intelligent control, robotics and mechatronics including
human-machine teaming, knowledge-based paradigms, learning paradigms, machine ethics, intelligent
data analysis, knowledge management, intelligent agents, intelligent decision making and support,
intelligent network security, trust management, interactive entertainment, Web intelligence and multimedia.
The publications within “Advances in Intelligent Systems and Computing” are primarily proceedings
of important conferences, symposia and congresses. They cover significant recent developments in the
field, both of a foundational and applicable character. An important characteristic feature of the series is
the short publication time and world-wide distribution. This permits a rapid and broad dissemination of
research results.
** Indexing: The books of this series are submitted to ISI Proceedings, EI-Compendex, DBLP,
SCOPUS, Google Scholar and Springerlink **

Advisory Board
Chairman
Nikhil R. Pal, Indian Statistical Institute, Kolkata, India
e-mail: [email protected]
Members
Rafael Bello Perez, Faculty of Mathematics, Physics and Computing, Universidad Central de Las Villas, Santa
Clara, Cuba
e-mail: [email protected]
Emilio S. Corchado, University of Salamanca, Salamanca, Spain
e-mail: [email protected]
Hani Hagras, School of Computer Science & Electronic Engineering, University of Essex, Colchester, UK
e-mail: [email protected]
László T. Kóczy, Department of Information Technology, Faculty of Engineering Sciences, Győr, Hungary
e-mail: [email protected]
Vladik Kreinovich, Department of Computer Science, University of Texas at El Paso, El Paso, TX, USA
e-mail: [email protected]
Chin-Teng Lin, Department of Electrical Engineering, National Chiao Tung University, Hsinchu, Taiwan
e-mail: [email protected]
Jie Lu, Faculty of Engineering and Information, University of Technology Sydney, Sydney, NSW, Australia
e-mail: [email protected]
Patricia Melin, Graduate Program of Computer Science, Tijuana Institute of Technology, Tijuana, Mexico
e-mail: [email protected]
Nadia Nedjah, Department of Electronics Engineering, University of Rio de Janeiro, Rio de Janeiro, Brazil
e-mail: [email protected]
Ngoc Thanh Nguyen, Wrocław University of Technology, Wrocław, Poland
e-mail: [email protected]
Jun Wang, Department of Mechanical and Automation, The Chinese University of Hong Kong, Shatin,
Hong Kong
e-mail: [email protected]

More information about this series at http://www.springer.com/series/11156

Leonard Barolli Makoto Takizawa
• •

Fatos Xhafa Tomoya Enokido

•

Editors

Advanced Information
Networking and Applications
Proceedings of the 33rd International
Conference on Advanced Information
Networking and Applications (AINA-2019)

123
Editors
Leonard Barolli Makoto Takizawa
Department of Information Department of Advanced Sciences
and Communication Engineering Hosei University
Fukuoka Institute of Technology Koganei-Shi, Tokyo, Japan
Fukuoka, Japan

Fatos Xhafa Tomoya Enokido

Department of Computer Science Faculty of Business Administration
Technical University of Catalonia Rissho University
Barcelona, Barcelona, Spain Tokyo, Japan

ISSN 2194-5357 ISSN 2194-5365 (electronic)

Advances in Intelligent Systems and Computing
ISBN 978-3-030-15031-0 ISBN 978-3-030-15032-7 (eBook)
https://doi.org/10.1007/978-3-030-15032-7

Library of Congress Control Number: 2019933217

© Springer Nature Switzerland AG 2020

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publisher remains neutral with regard
to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Welcome Message from AINA-2019 Steering
Committee Co-chairs

Welcome to the 33rd International Conference on Advanced Information

Networking and Applications (AINA-2019). It is our great pleasure and honor to
hold AINA-2019 at Kunibiki Messe, Matsue, Japan, from March 27 to March 29,
2019. On behalf of the AINA Steering Committee and AINA-2019 Organizing
Committee, we would like to express to all participants our cordial welcome and
high respect.
AINA is an international forum, where scientists and researchers from academia
and industry working in various scientific and technical areas of networking and
distributed computing systems can demonstrate new ideas and solutions in dis-
tributed computing systems.
AINA was born in Asia, but it is now an international conference with high
quality thanks to the great help and cooperation of many international friendly
volunteers. AINA is a very open society and is always welcoming international
volunteers from any country and any area in the world. In conjunction with
AINA-2019 main conference, we have also 14 international workshops.
An international conference can be organized by support and great voluntary
efforts of many people and organizations. Our main responsibility is to coordinate
various tasks carried out with other willing and talented volunteers.
We would like to thank AINA-2019 general co-chairs, PC co-chairs, workshops’
co-chairs, track area chairs, PC members, and workshops’ organizers for their great
efforts to make AINA-2019 a very successful event. We have special thanks to the
finance chair and web administrator co-chairs.
We would like to take opportunity to thank all members of the Organization
Committee and Program Committee as well as all reviewers for their hard work to
make the reviews on time and authors for submitting the papers. We would like to
thank Local Arrangement Team for the technical support and good local arrange-
ment for the conference.

v
vi Welcome Message from AINA-2019 Steering Committee Co-chairs

Finally, we would like to thank: Matsue City, Shimane Prefecture, The

Telecommunications Advancement Foundation (TAF), Japan, for their financial
support.
We do hope that you will have a great time in Matsue, Japan.

Makoto Takizawa
Leonard Barolli
AINA Steering Committee Co-chairs
Welcome Message from AINA-2019 General
Co-chairs

It is our great pleasure to welcome you all at the 33rd International Conference on
Advanced Information Networking and Applications (AINA-2019), which will be
held at Kunibiki Messe, Matsue, Japan, from March 27 to March 29, 2019.
AINA International Conference is a forum for sharing ideas and research work in
the emerging areas of information networking and their applications. The area of
advanced networking has grown very rapidly, and the applications around it have
experienced an explosive growth especially in the area of pervasive and mobile
applications, sensor networks, ad hoc networks, vehicular networks, multimedia
computing and social networking, semantic collaborative systems, as well as grid,
P2P, IoT, and cloud computing. This advanced networking revolution is trans-
forming the way people live, work, and interact with each other and is impacting
the way business, education, entertainment, and health care are operating. The
papers included in the proceedings cover theory, design, and application of com-
puter networks, distributed computing, and information systems.
Each year AINA receives a lot of paper submissions from all around the world. It
has maintained high-quality accepted papers and is aspiring to be one of the main
international conferences on the information networking in the world. In con-
junction with AINA-2019 conference, there are 14 workshops, which also accepted
good-quality papers.
An international conference of this size requires the support and help of many
people. A lot of people have helped and worked hard to produce a successful
AINA-2019 technical program and conference proceedings. First, we would like to
thank all authors for submitting their papers, the session chairs, and distinguished
keynote speakers. We are indebted to program area chairs, Program Committee
members, and reviewers, who carried out the most difficult work of carefully
evaluating the submitted papers.
We would like to give our special thanks to Prof. Makoto Takizawa and Prof.
Leonard Barolli, the co-chairs of the Steering Committee, for their strong encour-
agement, guidance, and insights, and for spending a lot of energy for conference
organization and shaping the conference program. We would like to thank PC
co-chairs and workshops’ co-chairs of AINA-2019 for their great contribution to

vii
viii Welcome Message from AINA-2019 General Co-chairs

the success of the conference. Our special thanks go to the finance chair and web
administrator co-chairs.
Finally, we would like to thank the Local Arrangement Team for the support and
good local arrangement for the conference. We do hope that you will have a great
time in Matsue, Japan.

Tomoya Enokido
Farookh Hussain
Alireza Shahrabi
AINA-2019 General Co-chairs
Welcome Message from AINA-2019 Program
Committee Co-chairs

Welcome to the 33rd International Conference on Advanced Information

Networking and Applications (AINA-2019), which will be held at Kunibiki Messe,
Matsue, Japan, from March 27 to March 29, 2019.
The purpose of AINA conference is to bring together researchers, developers,
and industrial experts to share new ideas and recent research results in the emerging
areas of information networking and their applications. The papers included in the
proceedings cover all aspects of theory, design, and application of computer net-
works and distributed computing systems. Most of the papers deal with new trends
in information networking, such as wireless sensor networks, ad hoc networks,
cloud computing, peer-to-peer systems, grid computing, pervasive and ubiquitous
systems, multimedia systems, security, multi-agent systems, IoT, and web-based
systems.
This edition AINA received many paper submissions from all over the world.
Each submission was peer-reviewed by Program Committee members and invited
external reviewers. Finally, the Program Committee accepted 112 papers (about
25% acceptance ratio), which will be presented during the conference days.
Unfortunately, many interesting and good papers could not be accepted in
AINA-2019 due to the limited number of time slots allocated for presentations at
the conference.
We are very proud and honored to have two distinguished keynote talks by
Dr. Markus Aleksy, ABB AG, Germany, and Naohiro Hayashibara, Kyoto Sangyo
University, Japan, who will present their recent work and will give new insights and
ideas to the conference participants.
Organizing an international conference of this size is of course a team effort.
Therefore, we gladly admit that we had the help of many very professional people.
First of all, we would like to thank all the authors for their interesting contributions
since they shape the program and make it interesting for the audience. Moreover,
we would like to express our thankfulness to all program vice chairs for their great
efforts. Additionally, we would like to thank all Program Committee members and
reviewers who carried out the most important work to evaluate the submitted
papers. We also thank the workshops’ co-chairs for organizing many excellent

ix
x Welcome Message from AINA-2019 Program Committee Co-chairs

workshops and symposiums, which enrich the conference and provide additional
opportunities for discussions and future co-operations.
The great success of the AINA conference series would not be possible without
the enormous commitment and support of the Steering Committee co-chairs
Prof. Makoto Takizawa and Prof. Leonard Barolli. Therefore, we would like to
thank them for their strong encouragement and guidance.
The general coordination of an event such as AINA conference requires a lot of
coordination effort as well as many other activities related to the conference
organization. Here, we thank the general co-chairs for their great support and
invaluable suggestions. We give special thanks to the finance chair and web
administrator co-chairs for their great efforts and efficient work to deal with many
conference matters.
We hope you will enjoy the conference and readings and have a great time in
Matsue, Japan.

Akimitsu Kanzaki
Flora Amato
Omar Hussain
AINA-2019 Program Committee Co-chairs
Welcome Message from AINA-2019 Workshops’
Co-chairs

Welcome to AINA-2019 workshops to be held in conjunction with the 33rd

International Conference on Advanced Information Networking and Applications
(AINA-2019) at Kunibiki Messe, Matsue, Japan, from March 27 to March 29,
2019. The goal of AINA workshops is to provide a forum for international
researchers and practitioners to exchange and share their new ideas, research
results, and ongoing work on leading-edge topics in the different fields of infor-
mation networks and their applications. Some of the accepted workshops deal with
topics that open up perspectives beyond the ordinary, thus enriching the topics
usually addressed by the AINA conference.
For this edition, the following 14 symposiums and workshops will be held with
AINA-2019.
1. The 15th International Symposium on Frontiers of Information Systems and
Network Applications (FINA-2019)
2. The 15th International Workshop on Heterogeneous Wireless Networks
(HWISE-2019)
3. The 12th International Symposium on Mining and Web (MAW-2019)
4. The 12th International Workshop on Bio and Intelligent Computing
(BICom-2019)
5. The 12th International Workshop on Telecommunication Networking,
Applications and Systems (TeNAS-2019)
6. The 10th International Workshop on Disaster and Emergency Information
Network Systems (IWDENS-2019)
7. The 7th International Workshop on Collaborative Emerging Systems
(COLLABES-2019)
8. The 6th International Workshop on Security Intricacies in Cyber-Physical
Systems and Services (INTRICATE-SEC-2019)
9. The 5th International Workshop on Engineering Energy Efficient
InternetWorked Smart seNsors (E3WSN-2019)

xi
xii Welcome Message from AINA-2019 Workshops’ Co-chairs

10. The 4th International Workshop on Innovative Technologies in Informatics and

Networking (WITIN-2019)
11. The 4th International Workshop on Big Data Processing in Online Social
Network (BOSON-2019)
12. The 2nd International Workshop on Internet of Everything and Machine
Learning Applications (IOEMLA-2019)
13. The 1st International Workshop on Multi-Clouds and Mobile Edge Computing
(M2EC-2019)
14. The 1st Workshop on Artificial Intelligence and Machine Learning
(AIMAL-2019)
We would like to thank the community for their great response to AINA-2019
workshops. The excellent technical program of the workshops was the result of a
professional work from workshop chairs, workshop program committees,
reviewers, and authors.
We would like to give our special thanks to Prof. Makoto Takizawa
and Prof. Leonard Barolli, the Steering Committee chairs of AINA International
Conference, for their strong encouragement and guidance to organize the AINA-2019
workshops and symposiums. We would like to thank AINA-2019 general co-chairs
for their advices to make possible organization of AINA-2019 workshops and
symposiums. We are thankful to AINA-2019 program co-chairs for their support and
help to prepare the technical program of AINA-2019 workshops and symposiums.
We wish all of you entertaining and rewarding experience in all workshops and
AINA-2019 International Conference.

Hui-Huang Hsu
Omid Ameri Sianaki
Rubem Pereira
AINA-2019 Workshops’ Co-chairs
AINA-2019 Organizing Committee

General Co-chairs

Tomoya Enokido Rissho University, Japan

Farookh Hussain University of Technology Sydney, Australia
Alireza Shahrabi Glasgow Caledonian University, UK

Program Committee Co-chairs

Akimitsu Kanzaki Shimane University, Japan

Flora Amato University of Naples Federico II, Italy
Omar Hussain University of New South Wales, Australia

Workshops’ Co-chairs

Hui-Huang Hsu Tamkang University, Taiwan

Omid Ameri Sianaki Victoria University, Australia
Rubem Pereira Liverpool John Moores University, UK

International Special Issue Journal Co-chairs

Fatos Xhafa Technical University of Catalonia, Spain

David Taniar Monash University, Australia
Isaac Woungang Ryerson University, Canada

xiii
xiv AINA-2019 Organizing Committee

Award Co-chairs

Marek Ogiela AGH University of Science and Technology,

Poland
Kin Fun Li University of Victoria, Canada
Markus Aleksy ABB AG, Germany
Fang-Yie Leu Tunghai University, Taiwan

Publicity Co-chairs

Arjan Durresi IUPUI, USA

Akio Koyama Yamagata University, Japan
Wenny Rahayu La Trobe University, Australia
Lidia Ogiela AGH University of Science and Technology,
Poland

International Liaison Co-chairs

Nadeem Javaid COMSATS University Islamabad, Pakistan

Minoru Uehara Toyo University, Japan
Hsing-Chung Chen Asia University, Taiwan

Local Arrangement Co-chairs

Elis Kulla Okayama University of Science, Japan

Keita Matsuo Fukuoka Institute of Technology, Japan

Finance Chair

Makoto Ikeda Fukuoka Institute of Technology, Japan

Web Chairs

Donald Elmazi Fukuoka Institute of Technology, Japan

Yi Liu Fukuoka Institute of Technology, Japan
Miralda Cuka Fukuoka Institute of Technology, Japan
Kevin Bylykbashi Fukuoka Institute of Technology, Japan
AINA-2019 Organizing Committee xv

Steering Committee Chairs

Makoto Takizawa Hosei University, Japan

Leonard Barolli Fukuoka Institute of Technology, Japan

Tracks and Program Committee Members

1. Network Protocols and Applications

Track Co-chairs

Bhed Bahadur Bista Iwate Prefectural University, Japan

Francesco Palmieri University of Salerno, Italy

TPC Members

Jiahong Wang Iwate Prefectural University, Japan

Shigetomo Kimura University of Tsukuba, Japan
Chotipat Pornavalai King Mongkut’s Institute of Technology
Ladkrabang, Thailand
Danda B. Rawat Howard University, USA
Gongjun Yan University of Southern Indiana, USA
Akio Koyama Yamagata University, Japan
Sachin Shetty Old Dominion University, USA
Sasthi C. Ghosh Indian Statistical Institute, India
Arcangelo Castiglione University of Salerno, Italy
Raffaele Pizzolante University of Salerno, Italy
Ugo Fiore Parthenope University of Naples, Italy
Massimo Ficco University of Campania Luigi Vanvitelli, Italy
Mauro Iacono University of Campania Luigi Vanvitelli, Italy
Sergio Ricciardi Barcelonatech, Spain

2. Next Generation Wireless Networks

Track Co-chairs

Li-Der Chou National Central University, Taiwan

Isaac Woungang Ryerson University, Canada
Tuleen Boutaleb Glasgow Caledonian University, UK
xvi AINA-2019 Organizing Committee

TPC Members

Yoshiyuki Kido Osaka University, Japan

Susumu Date Osaka University, Japan
Te-Lung Liu National Center for High-performance
Computing, Taiwan
Yuh-Shyan Chen National Taipei University, Taiwan
Chiu-Ching Tuan National Taipei University of Technology,
Taiwan
Guey-Yun Chang National Central University, Taiwan
Luca Caviglione National Research Council of Italy (CNR), Italy
Andrea Ceccarelli University of Firenze, Italy
Telex Magloire Ngatched Memorial University of Newfoundland, Canada
Nkouatchah
Marcello Luiz Brocardo Universidade do Estado de Santa Catarina, Brazil
Glaucio Carvalho Ryerson University, Canada
Gaurav Indra University of Delhi, India
Marcello Trovati University of Derby, UK
Petros Nicopolitidis Aristotle University of Thessaloniki, Greece
Goutam Mali Xavier University Bhubaneswar, India
Lei Shu Guangdong University of Petrochemical
Technology, China
Xiaoping Che Beijing Jiaotong University, China
Marco Antonio To Galileo University, Guatemala
Felipe Lalanne Inria Chile, Chile
Hwee Pink Tan Singapore Management University, Singapore
Victor Ramiro EasyMile, France
Chihi Houda Laboratoire de recherche Innov’COM à
Sup’Com, Tunisia
Chun-Wei Tsai National Chung Hsing University, Taiwan
Leonardo Montecchi University of Campinas, Brazil
R. K. Pateriya MANIT Bhopal, India
Manas Khatua Indian Institute of Technology Jodhpur, India
Sherali Zeadally University of Kentucky, USA
Yasir Malik Concordia University of Edmonton, Canada
Danda B. Rawat Howard University, USA
Weichao Wang University of North Carolina at Charlotte, USA
Petros Nicopolitidis Aristotle University of Thessaloniki, Greece
Ilsun You Soonchunhyang University, Korea
Luca Caviglione Institute for Applied Mathematics
and Information Technologies, National
Research Council of Italy (CNR), Italy
Hamed Aly Acadia University, Canada
Rohit Ranchal IBM Watson Health Cloud, USA
AINA-2019 Organizing Committee xvii

Sherif Saad Ahmed University of Windsor, Canada

Wei Lu Keene State College, USA
Marcelo Luiz Brocardo Universidade do Estado de Santa Catarina, Brazil
Juggapong Natwichai Chiang Mai University, Thailand
Neeraj Kumar Thapar University, India
Brian Stewart University of Strathclyde, UK
Roberto Ramirez-Iniguez Glasgow Caledonian University, UK
Huan Nguyen Middlesex University London, UK
Quoc-Tuan Vien Middlesex University London, UK
Imed Romdhani Edinburgh Napier University, UK

3. Multimedia Systems and Applications

Track Co-chairs

Hiroaki Nishino Oita University, Japan

Markus Aleksy ABB AG, Germany

TPC Members

Yong-Moo Kwon Korea Institute of Science and Technology,

Korea
Yoshihiro Okada Kyushu University, Japan
Makoto Nakashima Oita University, Japan
Nobukazu Iguchi Kinki University, Japan
Kenzi Watanabe Hiroshima University, Japan
Philipp Bauer ABB AG, Germany
Norman Franchi Technical University of Dresden, Germany
Fabian Mackenthun NXP Semiconductors Germany GmbH, Germany
Andreas Müller Robert Bosch GmbH, Germany
Michael Schneider Bosch Rexroth AG, Germany
Hans Schotten DFKI, Germany

4. Pervasive and Ubiquitous Computing

Track Co-chairs

Chih-Lin Hu National Central University, Taiwan

Neil Yen The University of Aizu, Japan
Vamsi Paruchuri University of Central Arkansas, USA
xviii AINA-2019 Organizing Committee

TPC Members

Hong Va Leong Hong Kong Polytechnic University, Hong Kong

Ling-Jyh Chen Academia Sinica, Taiwan
Yao-Hua Ho National Taiwan Normal University, Taiwan
Jiun-Long Huang National Chiao Tung University, Taiwan
Somchoke Ruengittinun Kasetsart University, Thailand
Tanapat Anusas-Amornkul King Mongkut’s University of Technology
North Bangkok, Thailand
Xin-Mao Huang Aletheia University, Taiwan
Jen-Wei Huang National Cheng Kung University, Taiwan
Hui Lin Tamkang University, Taiwan
Xiaokang Zhou Shiga University, Japan
Moloud Abdar The University of Aizu, Japan
Jason Hung Overseas Chinese University, Taiwan
Chun-Chia Wang City University, Taiwan
Sriram Chellappan University of South Florida, USA
Yu Sun University of Central Arkansas, USA
Qiang Duan Penn State University, USA
Jessie Walker Jackson State University, USA
Han-Chieh Wei Dallas Baptist University, USA

5. Internet Computing and E-Learning

Track Co-chairs

Santi Caballe Open University of Catalonia, Spain

Akio Koyama Yamagata University, Japan

TPC Members

Nicola Capuano University of Salerno, Italy

Thanasis Daradoumis University of the Aegean, Greece
Elisabetta Fersini University of Milan-Bicocca, Italy
Angelo Gaeta University of Salerno, Italy
Giuseppe Guarino University of Salerno, Italy
Krassen Stefanov Sofia University “St. Kliment Ohridski,” Bulgaria
Jordi Conesa Open University of Catalonia, Spain
David Gañan Open University of Catalonia, Spain
Jorge Miguel Grupo San Valero, Spain
Minoru Uehara Toyo University, Japan
Fumiaki Sato Toho University, Japan
Tomoyuki Nagase Hirosaki University, Japan
Bhed Bista Iwate Prefectural University, Japan
AINA-2019 Organizing Committee xix

Makoto Ikeda Fukuoka Institute of Technology, Japan

Keita Matsuo Fukuoka Institute of Technology, Japan
Shingo Yamaguchi Yamaguchi University, Japan
Kazuyuki Kojima Saitama University, Japan
Shinji Sugawara Chiba Institute of Technology, Japan
Noritaka Shigei Kagoshima University, Japan
Yukinobu Fukushima Okayama University, Japan
Khin Khin Zaw Okayama University, Japan
Nobuo Funabiki Okayama University, Japan

6. Distributed Systems Database and Data Mining

Track Co-chairs

Eric Pardade La Trobe University, Australia

Naohiro Hayashibara Kyoto Sangyo University, Japan

TPC Members

Asm Kayes La Trobe University, Australia

Ronaldo dos Santos Mello Universidade Federal de Santa Catarina, Brazil
Saqib Ali Sultan Qaboos University, Oman
Hong Quang Nguyen Ho Chi Minh City International University,
Vietnam
Md Saiful Islam Griffith University, Australia
Salimur Choudhury Lakehead University, Canada
Kanwalinderjit Gagneja Florida Polytechnic University, USA
Toshihiro Yamauchi Okayama University, Japan
Franco Frattolillo University of Sannio, Italy
Mohammad Shojafar University of Padova, Italy
Ji Zhang University of Southern Queensland, Australia
Xiaofeng Ding HUST, China
Lucian Podan Polytechnic University of Timisoara, Romania

7. Grid, P2P, and Scalable Computing

Track Co-chairs

Fumiaki Sato Toho University, Japan

Tomoki Yoshihisa Osaka University, Japan
xx AINA-2019 Organizing Committee

TPC Members

Akira Kanaoka Toho University, Japan

Yoshiaki Terashima Soka University, Japan
Atsushi Takeda Tohoku Gakuin University, Japan
Tomoya Kawakami NAIST, Japan
Yusuke Gotoh Okayama University, Japan
Yuuichi Teranishi NICT, Japan
Akimitsu Kanzaki Shimane University, Japan

8. Agents, Semantics and Intelligent Computing

Track Co-chairs

Salvatore Vitabile University of Palermo, Italy

Farookh Hussain University of Technology Sydney, Australia
Takahiro Uchiya Nagoya Institute of Technology, Japan

TPC Members

Vincenzo Conti University of Enna Kore, Italy

Minoru Uehara Toyo University, Japan
Mauro Migliardi University of Padua, Italy
Roberto Pirrone University of Palermo, Italy
Massimo Cossentino Italian National Research Council (CNR), Italy
Hai Dong RMIT University, Australia
Mohammad Younas Oxford Brookes University, UK
Geoff Skinner Newcastle University, Australia
Mukesh Prasad University of Technology Sydney, Australia
Salem Alkhalaf Qassim University, Saudi Arabia
Atsuko Mutoh Nagoya Institute of Technology, Japan
Shinsuke Kajioka Nagoya Institute of Technology, Japan
Ryota Nishimura Tokushima University, Japan
Shohei Kato Nagoya Institute of Technology, Japan

9. Cloud, Data Centers, and Services Computing

Track Co-chairs

Asm Kayes La Trobe University, Australia

Salvatore Venticinque University of Campania Luigi Vanvitelli, Italy
AINA-2019 Organizing Committee xxi

TPC Members

Alex Ng La Trobe University, Australia

Alan Colman Swinburne University of Technology, Australia
Lianhua Chi La Trobe University, Australia
Iqbal H. Sarker Swinburne University of Technology, Australia
Stanley Shanapinda La Trobe University, Australia
Iman Avazpour Deakin University, Australia
Mohamed Abdelrazek Deakin University, Australia
Amani Ibrahim Deakin University, Australia
Giancarlo Fortino University of Calabria, Italy
Massimiliano Rak University of Campania Luigi Vanvitelli, Italy
Jason J. Jung Chung-Ang University, Korea
Dimosthenis Kyriazis University of Piraeus, Greece

10. Internet of Things and Social Networking

Track Co-chairs

Nadeem Javaid COMSATS University Islamabad, Pakistan

Shusuke Okamoto Seikei University, Japan

TPC Members

Safdar Hussain Bouk Daegu Gyeongbuk Institute of Science

and Technology (DGIST), Korea
Kamran Munir University of the West of England, UK
Syed Hassan Ahmed Georgia Southern University, USA
Muhammad Imran King Saud University (KSU), Saudi Arabia
Sohail Iqbal National University of Sciences
and Technology (NUST), Pakistan
Hiroki Sakaji The University of Tokyo, Japan
Jun Iio Chuo University, Japan
Masaru Kamada Ibaraki University, Japan
Masaki Kohana Ibaraki University, Japan
Shinji Sakamoto Seikei University, Japan

11. Security, Privacy, and Trust Computing

Track Co-chairs

Ren-Junn Hwang Tamkang University, Taiwan

Omar Hussain University of New South Wales, Australia
Hiroaki Kikuchi Meiji University, Japan
xxii AINA-2019 Organizing Committee

TPC Members

Yongfeng Huang Tsinghua University, China

Hung-Min Sun National Tsing Hua University, Taiwan
Chun-I Fan National Sun Yat-sen University, Taiwan
Nai-Wei Lo National Taiwan University of Science
and Technology, Taiwan
Raylin Tso National Chengchi University, Taiwan
Hui Tian National Huaqiao University, China
Sazia Parvin UNSW, Australia
Saqib Ali Sultan Qaboos University, Oman
Farookh Hussain University of Technology Sydney, Australia
Adil Hammadi Curtin University, Australia
Asif Gill University of Technology Sydney, Australia
Kouichi Sakurai Kyushu University, Japan
Takamichi Saito Meiji University, Japan
Shou-Hsuan Stephen Huang University of Houston, USA
Isao Echizen National Institute of Informatics, Japan
Sushmita Ruj Indian Statistical Institute, India

12. Cognitive and Neural Computing with Application

Track Co-chairs

Lidia Ogiela AGH University of Science and Technology,

Poland
Hoon Ko Chosun University, Korea

TPC Members

Goreti Marreiros Institute of Engineering of Polytechnic of Porto,

Portugal
Urszula Ogiela AGH University of Science and Technology,
Poland
Kitae Bae Seoul Media Institute of Technology, Korea
Jongsun Choi Soongsil University, Korea
Libor Měsíček J.E. Purkyně University, Czech Republic
Francesco Pascale University of Salerno, Italy
AINA-2019 Organizing Committee xxiii

AINA-2019 Reviewers

Adil Hammadi Jiahong Wang

Admir Barolli Jiun-Long Huang
Ahmed Dawoud Jörg Domaschka
Ahmed Elmokashfi Kazuyuki Kojima
Akio Koyama Keita Matsuo
Amir Taherkordi Kyriakos Kritikos
Andrea Ceccarelli Lei Shu
Angelo Gaeta Leonard Barolli
Arcangelo Castiglione Leonardo Montecchi
Arjan Durresi Li-Der Chou
Ashkan Yousefi Makoto Nakashima
Asm Kayes Makoto Takizawa
Bhed Bista Marco Antonio To
Chiu-Ching Tuan Masaki Kohana
Chun-Chia Wang Massimo Ficco
Chun-Il Fan Mauro Iacono
Chun-Wei Tsai Minoru Uehara
Danda Rawat Mukesh Prasad
Dimitris Apostolou Nadeem Javaid
Elis Kulla Nai-Wei Lo
Eric Pardede Nicola Capuano
Fabian Mackenthun Nobukazu Iguchi
Farookh Hussain Noritaka Shigei
Farshid Hajati Norman Franchi
Fatos Xhafa Omer Rana
Feroz Zahid Omid Ameri Sianaki
Francesco Palmieri Øyvind Ytrehus
Fumiaki Sato Paresh Saxena
Gongjun Yan Pawel Skrzypek
Goutam Mali Philip Moore
Hai Dong Philipp Bauer
Hans Schotten Pornavalai Chotipat
Hiroaki Kikuchi Qiang Duan
Hiroaki Nishino Quentin Jacquemart
Hong Quang Nguyen Raffaele Pizzolante
Houda Chihi Rajesh Pateriya
Hung-Min Sun Ralph Holz
Hwee Pink Tan Raylin Tso
Isao Echizen Ren-Junn Hwang
Jawwad Shamsi Roberto Pirrone
Jen-Wei Haung Ronaldo dos Santos Mellos
Jesús Escudero-Sahuquillo Rubem Pereira
xxiv AINA-2019 Organizing Committee

Sabita Maharjan Thomas Dreibholz

Sachin Shetty Tomoya Enokido
Salman Taherizadeh Tor Skeie
Santi Caballé Ugo Fiore
Saqib Ali Victor Ramiro
Sasthi Ghosh Vincenzo Conti
Sergio Ricciardi Vlado Stankovski
Shigetomo Kimura Wang Xu An
Shingo Yamaguchi Xiaoping Che
Shinji Sakamoto Xiaoyi Lu
Shinji Sugawara Xing Zhou
Shinsuke Kajioka Xuejun Cai
Shou-Hsuan Huang Yiannis Verginadis
Sushmita Ruj Yongfeng Huang
Susumu Date Yoshihiro Okada
Takamichi Saito Yoshiyuki Kido
Te-Lung Liu Yuh-Shyan Chen
Telex M. N. Ngatched
AINA-2019 Keynote Talks
Utilizing Wireless Communication to Enable
New Types of Industrial Applications

Markus Aleksy

ABB AG Corporate Research Center, Ladenburg, Germany

Abstract. The recent progress in the development of wireless communication

technologies enables new types of industrial applications. Traditional industrial
systems usually rely on wired communication technologies to connect sensors and
actuators. However, these type of communication needs to be replaced by wireless
technologies in future to address new developments, such as mixed reality appli-
cations, automated guided vehicles, moving robots and drones or achieving higher
flexibility required by increasing demand for highly customized products and
adaptable production facilities. In this talk, we will address and discuss represen-
tative use cases and concepts focusing on the usage of wireless technologies in an
industrial setting. Moreover, we will present the related challenges and require-
ments of communication networks in such environments and discuss the applica-
bility of 5th generation wireless communication systems.

xxvii
Lévy Walk on Graphs: Message Dissemination
and Uninformed Search

Naohiro Hayashibara

Kyoto Sangyo University, Kyoto, Japan

Abstract. Random walks play an important role in computer science, spreading a

wide range of topics in theory and practice, including networking, distributed
systems, and optimization. Particularly, Lévy walk, a family of random walks, has
attracted attention in recent years in the context of mobile ad-hoc networks,
delay-tolerant networks, opportunistic communication, and global optimization
problems. It is also used as a model of various things not only in informatics but
also in biology and environmental science. Lévy walk is a mathematical fractal
which is characterized by long segments followed by shorter hops in random
directions. More precisely, the step distance obeys the power law distribution. The
pattern has been found by Paul Lévy, but the similar pattern has also been evolved
as a naturally selected strategy that gives animals and insects an edge in the search
for sparse targets to survive. In fact, this movement pattern has been observed in the
molecular machinery operating in cells, bacteria, the behavior of honeybees,
mussels, mud snails, wandering albatross and shearwaters. In the area of computer
science, it is most likely to be used as the mobility model in mobile ad-hoc net-
works because of the statistical similarity of human mobility. Most of the research
work, however, assumes a continuous plane and hardly any results on graphs are
available. The goal of this keynote is introducing Lévy walk and its variants and
presenting the challenge on Geometric graphs, especially Unit disk graphs,
regarding message dissemination and uninformed search by using Lévy walk. The
results on both message dissemination and uninformed search show that Lévy walk
is quite efficient compared to random walks because of its ballistic trajectory. They
also clarify the relationship between the efficiency of message dissemination and
uninformed search, and the average degree/the diameter of the graphs.

xxix
Contents

Electricity Load Forecasting in Smart Grids Using Support

Vector Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Nasir Ayub, Nadeem Javaid, Sana Mujeeb, Maheen Zahid,
Wazir Zada Khan, and Muhammad Umar Khattak
Optimization of Response and Processing Time for Smart Societies
Using Particle Swarm Optimization and Levy Walk . . . . . . . . . . . . . . . 14
Ayesha Anjum Butt, Zahoor Ali Khan, Nadeem Javaid, Annas Chand,
Aisha Fatima, and Muhammad Talha Islam
On Maximizing User Comfort Using a Novel Meta-Heuristic
Technique in Smart Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Sajjad Khan, Zahoor Ali Khan, Nadeem Javaid, Waleed Ahmad,
Raza Abid Abbasi, and Hafiz Muhammad Faisal
Towards Efficient Energy Management in a Smart Home
Using Updated Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Hafiz Muhammad Faisal, Nadeem Javaid, Zahoor Ali Khan,
Fahad Mussadaq, Muhammad Akhtar, and Raza Abid Abbasi
A New Memory Updation Heuristic Scheme for Energy Management
System in Smart Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Waleed Ahmad, Nadeem Javaid, Sajjad Khan, Maria Zuraiz,
Tayyab Awan, Muhammad Amir, and Raza Abid Abbasi
Towards Efficient Scheduling of Smart Appliances for Energy
Management by Candidate Solution Updation Algorithm
in Smart Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Sahibzada Muhammad Shuja, Nadeem Javaid,
Muhammad Zeeshan Rafique, Umar Qasim, Raja Farhat Makhdoom Khan,
Ayesha Anjum Butt, and Murtaza Hanif

xxxi
xxxii Contents

Minimizing Daily Electricity Cost Using Bird Chase Scheme

with Electricity Management Controller in a Smart Home . . . . . . . . . . 82
Raza Abid Abbasi, Nadeem Javaid, Shujat ur Rehman, Amanulla,
Sajjad Khan, Hafiz Muhammad Faisal, and Sajawal Ur Rehman Khan
Optimal Power Flow with Uncertain Renewable Energy Sources
Using Flower Pollination Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Muhammad Abdullah, Nadeem Javaid, Inam Ullah Khan,
Zahoor Ali Khan, Annas Chand, and Noman Ahmad
Efficient Information Flow Control by Reducing Meaningless
Messages in P2PPSO Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Shigenari Nakamura, Tomoya Enokido, Leonard Barolli,
and Makoto Takizawa
Comparison of Intelligent Algorithms with FACTS Devices
for Minimization of Total Power Losses . . . . . . . . . . . . . . . . . . . . . . . . . 120
Shohreh Monshizadeh, Gunne John Hegglid, and Svein Thore Hagen
Energy-Efficient Recovery Algorithm in the Fault-Tolerant
Tree-Based Fog Computing (FTBFC) Model . . . . . . . . . . . . . . . . . . . . . 132
Ryuji Oma, Shigenari Nakamura, Dilawaer Duolikun, Tomoya Enokido,
and Makoto Takizawa
Energy-Efficient Group Migration of Virtual Machines in a Cluster . . . 144
Dilawaer Duolikun, Tomoya Enokido, and Makoto Takizawa
A Secure and Trustworthy Intelligent System for Clustering
in VANETs Using Fuzzy Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Kevin Bylykbashi, Yi Liu, Donald Elmazi, Keita Matsuo,
Makoto Ikeda, and Leonard Barolli
Terminal Access Data Anomaly Detection Based on Random Forest
for Power User Electric Energy Data Acquisition System . . . . . . . . . . . 166
Xiaobing Liang, Bing Zhao, Qian Ma, Bang Sun, and Baojiang Cui
Multi-level Authentication Protocols Using Scientific
Expertise Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Marek R. Ogiela and Lidia Ogiela
Biometric-Based Linguistic Solutions for Data Encryption
and Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Urszula Ogiela, Makoto Takizawa, and Lidia Ogiela
Dependability Analysis for On-Demand Computing Based
Transaction Processing System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Dharmendra Prasad Mahato, Jasminder Kaur Sandhu,
Nagendra Pratap Singh, and Kamlesh Dutta
Contents xxxiii

A Comparative Analysis of Trust Requirements in Decentralized

Identity Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Andreas Grüner, Alexander Mühle, Tatiana Gayvoronskaya,
and Christoph Meinel
Combined Methods Based Outlier Detection for Water Pipeline
in Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Oussama Ghorbel, Aya Ayadi, Rami Ayadi, Mohammed Aseeri,
and Mohamed Abid
The Improved Energy-Efficient Quorum Selection with Excluding
Meaningless Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Tomoya Enokido, Dilawaer Duolikun, and Makoto Takizawa
A Fuzzy-Based System for Actor Node Selection in WSANs
Considering Level of Received Signal . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
Donald Elmazi, Miralda Cuka, Makoto Ikeda, Leonard Barolli,
and Makoto Takizawa
Selection of IoT Devices in Opportunistic Networks: A Fuzzy-Based
Approach Considering IoT Device’s Selfish Behaviour . . . . . . . . . . . . . . 251
Miralda Cuka, Donald Elmazi, Makoto Ikeda, Keita Matsuo,
Leonard Barolli, and Makoto Takizawa
Detecting Mental Health Illness Using Short Comments . . . . . . . . . . . . 265
Takahiro Baba, Kensuke Baba, and Daisuke Ikeda
On the Weakest Failure Detector for Read/Write-Based
Mutual Exclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Carole Delporte-Gallet, Hugues Fauconnier, and Michel Raynal
An Active Signaling Mechanism to Reduce Access Collisions
in a Distributed TDMA Based MAC Protocol
for Vehicular Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
Fouzi Boukhalfa, Mohamed Hadded, Paul Muhlethaler,
and Oyunchimeg Shagdar
Estimation of Tags Using Various Data for Online Videos . . . . . . . . . . . 301
Hiroki Sakaji, Akio Kobayashi, Masaki Kohana, Yasunao Takano,
and Kiyoshi Izumi
Framework for Feature Selection in Health Assessment Systems . . . . . . 313
Ayesha Ubaid, Fan Dong, and Farookh Kadeer Hussain
A Framework to Achieve Full Waypoint Enforcement in Hybrid
SDN Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
Sandhya Rathee, T. Dinesh Ram Kumar, K. Haribabu,
and Ashutosh Bhatia
xxxiv Contents

A New Fuzzy Logic Based Model for Location Trust Estimation

in Electric Vehicular Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
Ilhem Souissi, Nadia Ben Azzouna, Tahar Berradia, and Lamjed Ben Said
Angular Histogram-Based Visualisation of Network Traffic Flow
Measurement Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
Adrian Pekar, Mona B. H. Ruan, and Winston K. G. Seah
Performance Analysis of WMNs by WMN-PSODGA Simulation
System Considering Weibull and Chi-square Client Distributions . . . . . 366
Admir Barolli, Shinji Sakamoto, Leonard Barolli, and Makoto Takizawa
An NSH-Enabled Architecture for Virtualized Network
Function Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
Vinícius F. Garcia, Leonardo C. Marcuzzo, Giovanni V. Souza,
Lucas Bondan, Jéferson C. Nobre, Alberto E. Schaeffer-Filho,
Carlos R. P. dos Santos, Lisandro Z. Granville, and Elias P. Duarte Jr.
Personal Book Management Application on Blockchain . . . . . . . . . . . . . 388
Hiryu Kawaguchi, Yasunao Takano, and Hiroshi Sakuta
Performance Evaluation of WMNs by WMN-PSOSA System
Considering Chi-square and Exponential Client Distributions . . . . . . . . 397
Shinji Sakamoto, Leonard Barolli, and Shusuke Okamoto
Realization and Preliminary Evaluation of MPI Runtime
Environment on Android Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
Masahiro Nissato, Hiroki Sugiyama, Kanemitsu Ootsu, Takeshi Ohkawa,
and Takashi Yokota
Accelerating the Digital Transformation of Business and Society
Through Composite Business Ecosystems . . . . . . . . . . . . . . . . . . . . . . . . 419
Shridhar Choudhary, Ian Thomas, Mehdi Bahrami, and Motoshi Sumioka
An IP Multimedia Subsystem Service Discovery and Exposure
Approach Based on Opinion Mining by Exploiting
Twitter Trending Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
Armielle Noulapeu Ngaffo, Walid El Ayeb, and Zièd Choukair
A Hybrid Cross-Layer Protocol for Wireless Sensor Networks
Based on Backbone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
Piercarlo Fermino Soares, João Carlos Giacomin, and Tales Heimfarth
Implementing Lightweight IoT-IDS on Raspberry Pi
Using Correlation-Based Feature Selection
and Its Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
Yan Naung Soe, Yaokai Feng, Paulus Insap Santosa, Rudy Hartanto,
and Kouichi Sakurai
Contents xxxv

Quantifying the Limitations of Learning-Assisted

Grammar-Based Fuzzing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
Yuma Jitsunari, Yoshitaka Arahori, and Katsuhiko Gondow
Predicting Elephant Flows in Internet Exchange Point
Programmable Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
Marcus Vinicius Brito da Silva, Arthur Selle Jacobs,
Ricardo José Pfitscher, and Lisandro Zambenedetti Granville
ISDI: A New Window-Based Framework for Integrating IoT
Streaming Data from Multiple Sources . . . . . . . . . . . . . . . . . . . . . . . . . 498
Doan Quang Tu, A. S. M. Kayes, Wenny Rahayu, and Kinh Nguyen
Improving Document Similarity Calculation Using
Cosine-Similarity Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
Yasunao Takano, Yusuke Iijima, Kou Kobayashi, Hiroshi Sakuta,
Hiroki Sakaji, Masaki Kohana, and Akio Kobayashi
An Efficient Event-Based Protocol for Emergency Situations
in Smart Cities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
Sediane C. L. Hernandes, Marcelo E. Pellenz, Alcides Calsavara,
and Manoel C. Penna
Gossip Message Dissemination Protocols in the Presence
of Link Instability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
Takumu Hirooka and Naohiro Hayashibara
Analysis of Ethereum Smart Contracts and Opcodes . . . . . . . . . . . . . . . 546
Stefano Bistarelli, Gianmarco Mazzante, Matteo Micheletti,
Leonardo Mostarda, and Francesco Tiezzi
Towards 802.11g Signal Strength Estimation in an Industrial
Environment: A Practical Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559
Dalton Cézane Gomes Valadares,
Joseana Macêdo Fechine Régis de Araújo, Ângelo Perkusich,
Marco Aurélio Spohn, Elmar Uwe Kurt Melcher,
and Natália Porfírio Albuquerque
Revealing Storage and Speed Transmission Emerging Technology
of Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
Heru Susanto, Fang-Yie Leu, Didi Rosiyadi, and Chin Kang Chen
A Web-Based Artwork Editing System Empowered
by Neural Style Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584
Kenta Goto and Hiroaki Nishino
Mobile App and Malware Classifications by Mobile
Usage with Time Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
Yong Zheng and Sridhar Srinivasan
xxxvi Contents

Improvement of Self Position Estimation of Electric Wheelchair

Combining Multiple Positioning Methods . . . . . . . . . . . . . . . . . . . . . . . . 607
Fumiai Sato
A Roadside Unit Placement Scheme for Vehicular Ad-hoc Networks . . . 619
Seif Ben Chaabene, Taoufik Yeferny, and Sadok Ben Yahia
Personality-Aware Collaborative Learning: Models
and Explanations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
Yong Zheng and Archana Subramaniyan
Interfacer: A Model-Driven Development Method
for SDN Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
João Eurípedes Pereira Júnior, Flávio de Oliveira Silva,
João Henrique de Souza Pereira, and Pedro Frosi Rosa
Reducing the IEEE 802.11 Beacon Overhead in Low
Mobility Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655
Gabriel de Carvalho Ferreira, Priscila Solis Barreto, and Eduardo Alchieri
AXARPS: Scalable ARP Snooping Using Policy-Based Mirroring
of Core Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667
Motoyuki Ohmori, Naoki Miyata, and Ichiroh Suzuta
DCS-MAC: A Distributed Cross-Layer Communication Protocol
for Directional Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677
Shamanth Nagaraju, V. Sreejith, Nipun Sood, Lucy J. Gudino,
Mehul Kasliwal, and Rajas Kejriwal
An MQTT-SN-Based QoS Dynamic Adaptation Method
for Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 690
Helbert da Rocha, Tania L. Monteiro, Marcelo Eduardo Pellenz,
Manuel C. Penna, and Joilson Alves Junior
Centrality Based Geocasting for Opportunistic Networks . . . . . . . . . . . 702
Jagdeep Singh, Sanjay K. Dhurandher, Isaac Woungang,
and Makoto Takizawa
An Efficient Data Transmission Technique for Big Video Files
over HetNet in Emerging 5G Networks . . . . . . . . . . . . . . . . . . . . . . . . . 713
Richa Siddavaatam, Isaac Woungang, and Sanjay Kumar Dhurandher
Backhaul-Based Cooperative Caching in Small Cell Network . . . . . . . . 725
Yu-Ting Wang, Yun-Zhan Cai, Lo-An Chen, Sian-Jhe Lin,
and Meng-Hsun Tsai
A MEC-Assisted Method for Early Handover Using the Fully
Distributed Mobility Management (MEC-F-DMM) Architecture . . . . . . 737
Chung-Ming Huang, Duy-Tuan Dao, and Meng-Shu Chiang
Contents xxxvii

Link Level Capacity Analysis of Precoded FBMC/OQAM Systems . . . . 751

Jihed Ghodhbane, Noureddine Boujnah, and Ridha Bouallegue
A QoS-Based Flow Assignment for Traffic Engineering
in Software-Defined Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 762
Lakshmi Priya Thiruvasakan, Quoc-Tuan Vien, Jonathan Loo,
and Glenford Mapp
A UAV-Collaborative Sensing Method for Efficient Monitoring
of Disaster Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775
Akimitsu Kanzaki and Hideyuki Akagi
A Scheme to Improve Stream Transaction Rates for Real-Time
IoT Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 787
Chaxiong Yukonhiatou, Tomoki Yoshihisa, Tomoya Kawakami,
Yuuichi Teranishi, and Shinji Shimojo
A Video Data Distribution Method for Flexible Bandwidth Allocation
in Hybrid Broadcasting Environments . . . . . . . . . . . . . . . . . . . . . . . . . . 799
Satoru Matsumoto and Tomoki Yoshihisa
Multilingual Entity Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 810
Ilgiz Mustafin, Marius-Cristian Frunza, and JooYoung Lee
Evaluating DASH Player QoE with MPTCP in Presence
of Packet Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 821
Sunit Kumar Nandi, Pranav Kumar Singh, and Sukumar Nandi
VoIP Traffic Management Using a Software-Defined
Networking Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834
Paulo Vieira Jr. and Adriano Fiorese
A Self Healing Microservices Architecture: A Case Study in Docker
Swarm Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 846
Basel Magableh and Muder Almiani
Transparent State Machine Replication for Kubernetes . . . . . . . . . . . . . 859
Felipe Borges, Luis Pacheco, Eduardo Alchieri, Marcos F. Caetano,
and Priscila Solis
An Optimal Travel Route Recommendation System for Tourists’
First Visit to Japan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 872
Chen Yuan and Minoru Uehara
Simulation of Secure Volunteer Computing by Using Blockchain . . . . . 883
Johjima Shota, Kaneko Kosuke, Subodh Sharma, and Sakurai Kouichi
A Novel Movie Recommendation System Based on Collaborative
Filtering and Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 895
Chu-Hsing Lin and Hsuan Chi
xxxviii Contents

Response to Co-resident Threats in Cloud Computing

Using Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 904
Chu-Hsing Lin and Hsiao-Wen Lu
Automatic Graph-Based Clustering for Security Logs . . . . . . . . . . . . . . 914
Hudan Studiawan, Christian Payne, and Ferdous Sohel
Retrieving Text-Based Surrounding Objects in Spatial Databases . . . . . 927
Bojie Shen, Md. Saiful Islam, David Taniar, and Junhu Wang
Forecasting Crypto-Asset Price Using Influencer Tweets . . . . . . . . . . . . 940
Hirofumi Yamamoto, Hiroki Sakaji, Hiroyasu Matsushima,
Yuki Yamashita, Kyohei Osawa, Kiyoshi Izumi, and Takashi Shimada
Challenges and Strategies for Developing Decentralized Applications
Based on Blockchain Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 952
Thanh Chung Dao, Binh Minh Nguyen, and Ba Lam Do
Design of Remote Heart Monitoring System for Cardiac Patients . . . . . 963
Afef Benjemmaa, Hela Ltifi, and Mounir Ben Ayed
An Efficient Virtual Machine Placement via Bin Packing
in Cloud Data Centers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 977
Aisha Fatima, Nadeem Javaid, Tanzeela Sultana, Mohammed Y. Aalsalem,
Shaista Shabbir, and Durr-e-Adan
Commodore: Fail Safe Container Scheduling in Kubernetes . . . . . . . . . 988
Christos Christodoulopoulos and Euripides G. M. Petrakis
Dynamic Ranking System of Cloud SaaS Based on Consumer
Preferences - Find SaaS M2NFCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1000
Mohammed A. Ikram, Nabin Sharma, Muhammad Raza,
and Farookh Khadeer Hussain
A Methodology for Automating the Cloud Data Center
Availability Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1011
Guto Leoni Santos, Daniel Rosendo, Demis Gomes, Leylane Ferreira,
Andre Moreira, Djamel Sadok, Judith Kelner, Glauco Goncalves,
Mattias Wilderman, and Patricia Takako Endo
Job Scheduling Simulator for Assisting the Mapping Configuration
Between Queue and Computing Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . 1024
Yuki Matsui, Yasuhiro Watashiba, Susumu Date, Takashi Yoshikawa,
and Shinji Shimojo
MVMM: Data Center Scheduler Algorithm for Virtual
Machine Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1034
Nawel Kortas and Habib Youssef
Contents xxxix

A Large-Scale Wired Network Energy Model

for Flow-Level Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1047
Loic Guegan, Betsegaw Lemma Amersho, Anne-Cécile Orgerie,
and Martin Quinson
An Inter-slice Management Solution for Future
Virtualization-Based 5G Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1059
Borja Bordel, Ramón Alcarria, Diego Sánchez-de-Rivera,
and Álvaro Sánchez
IoT Security Viewer System Using Machine Learning . . . . . . . . . . . . . . 1071
Yuya Kunugi, Hiroyuki Suzuki, and Akio Koyama
Service Oriented Architecture for Interconnecting LoRa Devices
with the Cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1082
Konstantinos Tsakos and Euripides G. M. Petrakis
Self-adaptive RFID Authentication for Internet of Things . . . . . . . . . . . 1094
Bacem Mbarek, Mouzhi Ge, and Tomas Pitner
Limiting the Influence to Vulnerable Users in Social Networks:
A Ratio Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1106
Huiping Chen, Grigorios Loukides, Jiashi Fan, and Hau Chan
IoT Based Wide Area Road Surface State Sensing
and Communication System for Future Safety Driving . . . . . . . . . . . . . 1123
Yoshitaka Shibata, Akira Sakuraba, Goshi Sato, and Noriki Uchida
Expressing Trust with Temporal Frequency of User Interaction
in Online Communities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1133
Ekaterina Yashkina, Arseny Pinigin, JooYoung Lee, Manuel Mazzara,
Akinlolu Solomon Adekotujo, Adam Zubair, and Luca Longo
Hot Spot Tracking by Time-Decaying Bloom Filters
and Reservoir Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1147
Kai Cheng
A Fuzzy Logic Based Trust-ABAC Model
for the Internet of Things . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1157
Hamdi Ouechtati, Nadia Ben Azzouna, and Lamjed Ben Said
Decision Support System Using Trust Planning Among
Food-Energy-Water Actors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1169
Suleyman Uslu, Davinder Kaur, Samuel J. Rivera, Arjan Durresi,
and Meghna Babbar-Sebens
iHome: Smart Home Management as a Service in the Cloud
and the Fog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1181
George Myrizakis and Euripides G. M. Petrakis
xl Contents

Design of Robot Service Functions for a Framework Establishing

Human-Machine Trust . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1193
Fumi Ito, Eri Ozawa, and Yuka Kato
A Proposal for a Dynamic Digital Map to Prevent Heatstroke
Using IoT Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1205
Kanae Matsui and Keiya Sakai
Analysis of Consumers Perceptions of Food Safety Risk
in Social Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1217
Alessandra Amato, Walter Balzano, Giovanni Cozzolino,
and Francesco Moscato
Implicit and Continuous Authentication of Smart Home Users . . . . . . . 1228
Noureddine Amraoui, Amine Besrour, Riadh Ksantini,
and Belhassen Zouari
Supporting Internet-Based Location for Location-Based Access
Control in Enterprise Cloud Storage Solution . . . . . . . . . . . . . . . . . . . . 1240
Muhammad I. H. Sukmana, Kennedy A. Torkura, Hendrik Graupner,
Ankit Chauhan, Feng Cheng, and Christoph Meinel
Exploiting Multiple Paths in Multi-hop Co-operative Ad-Hoc
Networks for Providing Security without Trusted Third Party . . . . . . . 1254
T. Dinesh Ram Kumar, Ashutosh Bhatia, and R. C. Hansdah
Deniable Secret Handshake Protocol - Revisited . . . . . . . . . . . . . . . . . . 1266
Somnath Panja, Sabyasachi Dutta, and Kouichi Sakurai
Detection of Algorithmically Generated Domain Names in Botnets . . . . 1279
Deepak Kumar Vishvakarma, Ashutosh Bhatia, and Zdenek Riha
How Important Are Logs of Ordinary Operations? Empirical
Investigation of Anomaly Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1291
Akinori Muramatsu and Masayoshi Aritsugi
Access Control Based Dynamic Path Establishment for Securing
Flows from the User Devices with Different Security Clearance . . . . . . . 1303
Uday Tupakula, Vijay Varadharajan, and Kallol Karmakar
Interactive Aggregate Message Authentication Scheme
with Detecting Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1316
Shingo Sato and Junji Shikata
Basic Study on Targeted E-mail Attack Method Using OSINT . . . . . . . 1329
Kota Uehara, Kohei Mukaiyama, Masahiro Fujita, Hiroki Nishikawa,
Takumi Yamamoto, Kiyoto Kawauchi, and Masakatsu Nishigaki
Contents xli

Network Location Privacy Protection with Multi-access

Edge Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1342
Ping Zhang, Mimoza Durresi, and Arjan Durresi
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1353
 Electricity Load Forecasting in Smart
Grids Using Support Vector Machine

Nasir Ayub1 , Nadeem Javaid1(B) , Sana Mujeeb1 , Maheen Zahid1 ,

Wazir Zada Khan2 , and Muhammad Umar Khattak3
1
COMSATS University Islamabad, Islamabad 44000, Pakistan
[email protected]
2
Farasan Networking Research Laboratory,
Department of Computer Science and Information System, Jazan University,
Jazan 82822-6694, Saudi Arabia
3
Bahria University, Islamabad 44000, Pakistan
http://www.njavaid.com

Abstract. One of the key issues in the Smart Grid (SG) is accurate
electric load forecasting. Energy generation and consumption have highly
varying. Accurate forecasting of electric load can decrease the fluctuating
behavior between energy generation and consumption. By knowing the
upcoming electricity load consumption, we can control the extra energy
generation. To solve this issue, we have proposed a forecasting model,
which consists of a two-stage process; feature engineering and classifi-
cation. Feature engineering consists of feature selection and extraction.
By combining Extreme Gradient Boosting (XGBoost) and Decision Tree
(DT) techniques, we have proposed a hybrid feature selector to mini-
mize the feature redundancy. Furthermore, Recursive Feature Elimina-
tion (RFE) technique is applied for dimension reduction and improve
feature selection. To forecast electric load, we have applied Support Vec-
tor Machine (SVM) set tuned with three super parameters, i.e., kernel
parameter, cost penalty, and incentive loss function parameter. Electric-
ity market data is used in our proposed model. Weekly and months ahead
forecasting experiments are conducted by proposed model. Forecasting
performance is assessed by using RMSE and MAPE and their values
are 1.682 and 12.364. The simulation results show 98% load forecasting
accuracy.

1 Introduction
Smart Grid (SG) is an intelligent power system that efficiently manages gener-
ation, distribution and consumption of energy by introducing new technologies
in power grids and enable two-way communication between consumer and util-
ity [1]. Energy is the necessity and most valuable asset. The new generation is
attracted towards SG due to the extensive shortage of energy during the sum-
mer. SG manages the generation, distribution and consumption by implementing
different techniques on the power grid, utility and demand side. Efficient energy
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1–13, 2020.
https://doi.org/10.1007/978-3-030-15032-7_1
2 N. Ayub et al.

utilization can reduce the shortage of energy, minimize the electricity cost. Many
works are performed on different problems of SG [2]. At DSM, the home appli-
ances are scheduled using meta heuristic techniques to reduce the electricity cost,
peak to average ratio and achieve an optimal tradeoff between electricity cost
and user comfort [3].
SG facilitates consumer in reliability and sustainability by providing efficient
energy management. The Smart Meter (SM) made easy to get enough infor-
mation about future energy generation by providing real time-sharing of data
between consumer and utility. It will create a balance between energy generation
and consumption of energy. The consumer takes part in the operations of SG by
shifting the load from on peak hours to off peak hours and energy preservation
to lessen their power consumption cost [4,5].
With the help of Demand Side Management (DSM), consumers can manage
their energy utilization in an economical fashion. DSM is a program in which
consumer is able to manage their energy consumption pattern, according to the
price declared by the utility. Market competitors have more benefit from the
load forecasting. Several decisions are based on upcoming load prediction, such
as demand, supply management, power generation scheduling, reliability analysis
and maintenance planning [6].
Efficient generation and consumption of energy is another issue of the energy
sector. Utility maximization is the ultimate goal of user and utility. With the
help of accurate load forecast, energy producers will maximize their cost and
consumer will take benefit of low cost price of purchasing electricity. There is
no proper energy generation strategy in SG. To avoid extra generation, a per-
fect balance is required between the produced and consumed energy. Therefore,
precise load forecast holds more importance for market set-up management [7].
New England Control Area Independent System Operator (ISO-NE) is a
regional transmission organization, which is managed by independent system
operator. It is responsible for wholesale energy market operations. It supplies
energy to the different states of England, including Massachusetts, Maine, Con-
necticut, New Hampshire, Vermont, and Rhodes Island. The analytics in the
paper are formed on large data set of ISO NE. Price is not the only parameter
that affects the load, however, there are some other parameters that also effect
on the electrical load such as temperature, weather conditions etc.
The quantity of real world data is quite large [8]. SG data is surveyed in
detail [9]. The large amount of data provides information to utility to perform
analysis, which leads to more improvement in the markets operation planning
and management. To optimize the demand side of SG, a decision-making method
is needed. A proper decision-making results in the minimization of power loss,
reduction in the electricity cost and PAR in end user [10]. Keeping in mind these
problems, researchers mainly focus on the power scheduling problem. Different
optimization techniques are used to solve the power scheduling issue [11,12].
There is a very huge amount of electricity load data referred as big data. Big
data are very complex and large amount of data. Big data analytics make the
extraction of hidden patterns easy, market trends and other valuable information.
 Electricity Load Forecasting in Smart Grids Using Support Vector Machine 3

In literature, many techniques are used for load forecasting. The size of data is
very complex and huge, which creates difficulty in the training data. Deep Neural
Network (DNN) has the computational power to handle the training of big data.
DNN has the advantage to forecast accurately and handle huge amount of data.
Many forecasting techniques are discussed in the literature. Forecasting tech-
niques can be categorized in three groups, which are data driven, classical and
artificially intelligent. To forecast, classifier based techniques are used such as
Random forest, naive bayes and ARIMA etc. Artificial intelligence techniques are
Deep Neural Network (DNN), Artificial Neural Network (ANN), Shallow Neu-
ral Network (SNN), Particle Swam Optimization (PSO) etc. Aforementioned
techniques are used for forecasting the load or price. Due to automatic feature
extraction and training processes, neural network has an advantage over other
techniques.
In paper [13–15], SNN has the worst results and have over fit problem. DNN
performs better in forecasting price and load than SNN. In [16], the author
implemented Restricted Boltzmann Machine (RBM) and Rectified Linear Unit
(ReLU) for forecasting. RBM is used for data processing and training the data,
while ReLU performs load forecasting. KPCA is used for extraction of features
and DE based SVM for price forecasting in [17]. To forecast cooling load, Deep
Auto Encoders (DAE) are used [18]. DAE performs better in achieving accuracy
and learning the data. DAE is an unsupervised learning method and outperforms
in attaining the good accuracy results.
Gated Recurrent Units (GRU) technique is implemented for forecasting the
price in [19]. In [20], Parameter Estimation Method (PEM) is applied to detect
the abnormal behavior of load. GRU beats the Long Short Term Memory
(LSTM) technique in achieving accuracy in price forecasting. Two deep neu-
ral network techniques; Convolutional Neural Network (CNN) and LSTM are
combined for forecasting load [17]. The Hybrid of LSTM and CNN outperforms
in results than CNN and LSTM separately and other several models. DNN mod-
els show better performance in achieving accuracy in the results of forecasting.
SG big data help to find the trend of load and cost. It gives help to utility in
making a demand, supply and maintenance plan, which is the basic requirement
for demand supply balance.
Feature engineering is one of the application of the classifier. Two popular
operation are used in feature engineering; selection and extraction. Several meth-
ods are used for feature engineering in electric load. In article [21–24], author
study about the existing techniques of feature engineering to gain suitable fea-
tures from the data. Forecasting accuracy can be improved by the involvement
of big data.

2 Contributions

In this article, we highlight the electricity load forecasting problem. The objective
of our work is to predict the accurate electric load forecasting using electricity
load data set. To solve this problem, we have applied SVM classifier to predict
4 N. Ayub et al.

the electricity load. SVM is a classifier that divides the data into appropriate
categories by making a hyperplane between them. The SV part of the classifier
has the advantage to define the hyperplane between that classes. SVM is a
proficient method, however, the following challenges need to be answered for
better accuracy of electricity load forecasting.

• High computational complexity: SVM has high computational complex-

ity and weak in processing the uncertain data [11]. In electricity load fore-
casting, redundant features in data increase the computational complexity of
SVM in its training processes and also reduces the prediction accuracy.
• Hard to tune parameters: Super parameters of SVM has an effect on
the performance of SVM in forecasting. Those parameters are Cost penalty,
kernel parameter and incentive loss function. It is difficult to find the exact
values of these parameters for higher accuracy.

To address the challenges mentioned above, we have proposed a forecasting

model called Hybrid Feature Selection, Extraction and Classification (HFSEC).
The HFS part of the model is based on Hybrid XGboost and DTC, feature
extraction process based on RFE and classification is based on SVM classifier.
The proposed model implements feature engineering by selecting features regard-
ing time period and dimensionally reduced in of electricity load data features.
The hybrid feature selector uses the combination of two techniques XGboost
and DTC, rather than using one to give a selection of features. Further remov-
ing redundancy in the data, RFE is applied. The actual contribution of this
paper is:

• A forecasting model is implemented to achieve accurate load forecasting by

using the big data in SG. We have integrated selection, extraction, and clas-
sification in our proposed model to solve the addressed problem.
• To implement this model, a hybrid selector is proposed by the combination
of XGboost and DTC, which gives us the feature importance and feature
selection control. RFE is used to remove the redundancy from the selected
features. We have also tuned the parameters of SVM to make forecasting
accuracy better.
• The forecasting performance of our proposed model performs better. The real
world electricity load data are used in this paper. Extensive simulations are
performed, which shows 98% accurate results.

3 System Model

Our proposed system model consists of four parts: normalizing the data by pre-
processing, training the data, testing the data, an SVM classifier with tuned
parameters and forecasting load data as shown in Fig. 1.
 Electricity Load Forecasting in Smart Grids Using Support Vector Machine 5

3.1 Preprocessing Data

Daily system load data are acquired from ISO-NE. The three years system load
data, i.e. January 2015 to December 2017 are used in this article. The data are
divided month wise and similar months load data, i.e. January 2015, January
2016 and January 2017, first three weeks of the month are used in the training
process and the last week for testing. All the data are arranged in the same
manner. The data is normalized with maximum values. Data is categorized into
three parts train, test and validate data.

3.2 Training and Forecasting of Data

After preprocessing of data, we obtained training, validation and testing data.
The obtained data are given to SVM for training. The SVM has three layers; the
input layer, hidden layers, and output layer. The tuning parameters, i.e. kernel
parameter set as Radial Basis Function (RBF), cost penalty and gamma values
are set to 27 and 38. These values are finalized after extensive simulations and
tuning the parameter values. The network predicts step ahead values at each time
step during the training process of SVM. The SVM acquires every arrangement
and updates the network until the preceding time step. The first training of a
network of training data is called an initial network. The initial network formed
is tested on validation data. The initial network gains a forecasted value of a
step ahead result. After gaining the forecasting results, the forecasting network
relearns and tunes the network on validation data until the forecasting errors
are reduced to a minimum value. After all, the tuned and final network is used
for load forecasting. The steps of implementing model are listed below:
1. The load data are normalized as (F/Max (F)). Load data are divided month
wise and split into categories, train, validation and test.
2. Training data are used for network training and tested on validation data.
Forecasting errors are calculated on validation data.
3. Network is tuned and validation data actual are updated with new data.
4. The network tests on the test data, and weekly ahead load and month ahead
load are forecasted. The forecasting performance evaluation is performed by
RMSE, MAPE, MAE, and MSE.

3.3 Proposed Model

The issue of the load forecasting is accuracy. Many factors affect the electricity
load and makes the classifier training difficult. To improve the accuracy, we have
proposed a network consists of a hybrid feature selector, RFE based extraction
and SVM based classifier as shown in Fig. 1. Parts of the model are listed below.

3.3.1 Hybrid Feature Selector

This section describes the feature selection process of our model. We proposed a
hybrid feature selector, by combing the XGboost, DTC and defined threshold,
6 N. Ayub et al.

i.e. µ to control feature selection. HFS consists of two feature evaluators i.e.
X and D. These two evaluators calculate the feature importance separately. In
the feature selection process, the features are selected by joining the feature
importance generated by the two evaluators. Feature selection is based W X and
W D , which can be normalized by

W X = W X /max(W X ), (1)

W D = W D /max(W D ). (2)
Then the feature selection perform as

reserve W X [Tk ] + W D [Tk ] > µ
Fs = (3)
drop, W X [Tk ] + W D [Tk ] ≤ µ,

W B [Tk ] represents the feature importance calculated by evaluator XGBoost,

W D [Tk ] shows feature importance given by the DT. µ is the threshold controlling
the feature selection. Features have also redundancy among them. To remove
further redundancy and dimension reduction, they sent to RFE.

3.3.2 Feature Extraction-RFE

The feature extraction process is described in this section. The features selected
by HFS are considered to have no irrelevant features, however, it contains redun-
dant features. To reduce dimension and redundancy of features, Recursive Fea-
ture Eliminator (RFE) is applied for removing redundancy. To find a suitable
low dimensional embedding, data needs non-linear mapping in electricity load
forecasting. Thus RFE is applied to reduce nonlinear dimension.

Fig. 1. System model

 Electricity Load Forecasting in Smart Grids Using Support Vector Machine 7

4 Simulations
In order to simulate the performance of our proposed work, we have performed
the simulation with python. The simulator runs on the system Intel Core i3,
4GB RAM and 500GB storage. Daily electric load data of ISO New England
Control Area (ISO NECA) from January 2015 to December 2017 are taken as
input data for the simulator. Simulation results are as follows.

4.1 Feature Selection Using XGBoost and DTC

XGBoost and Decision Tree Classifier (DTC) is applied to calculate the impor-
tance of features with respect to the target, i.e. load system. In feature selection,
every feature sequence has a form as a vector. Every element of the sequence rep-
resents the feature values of different time stamps. However, our objective is to
predict the electricity load, which is named “system load” in the data. Features
that have a small effect on the target are removed. XGBoost technique finds the
importance of features, i.e. importance in numeric values and also the dimension
of features, i.e. true or false and DTC technique shows the grade of the features.
We select the features by taking hybrid of XGBoost and DTC (XGDTC). We
set a threshold for selection of features in hybrid XGDTC i.e. features having
a grade greater than 0.7 in DTC and having a false dimension in XGBoost are
selected as best features. With the increase in threshold, more features will drop
which leads to maximize the training speed and minimize the accuracy. Best
features have the highest value of importance and also have a high effect on the
target feature. The best features are then given to the forecasting engine for load
prediction. Figures 2 and 3 shows the XGBoost and DTC importances.

Fig. 2. Feature importance using DTC

8 N. Ayub et al.

Fig. 3. Feature importance using XGBoost

4.2 Load Forecasting

The normalized load of three years is shown in Fig. 4, which shows different
variations among different days. This is the description of load data of years
2015 to 2017. Data are split into training and testing, in which training and
testing days are 822 and 274 and given to the forecasting engine for prediction.

24000

22000
Normalload(MW)

20000

18000

16000

14000

12000
0 200 400 600 800 1000 1200

Days

Fig. 4. Normalize load of ISO NE January 2015 to December 2017

 Electricity Load Forecasting in Smart Grids Using Support Vector Machine 9

Figure 5, describes a load of the month December 2016, December 2017,

January 2016, and January 2017.

24000
Dec 2017
Dec 2016
22000 Jan 2017
Jan 2016

20000
Load (MW)

18000

16000

14000

12000
5 10 15 20 25 30

Days

Fig. 5. Similar months load

There is less change in the load pattern of similar months, i.e. January 2016,
January 2017 and December 2016, December 2017. There is high variation in the
load values of different months, i.e. January 2017 and December 2017. Therefore,
first three weeks of January 2015, January 2016 up to January 2017 are used to

18500
Act ual
18000 Predict ion

17500

17000
Load (MW)

16500

16000

15500

15000

14500
1 2 3 4 5 6
Days

Fig. 6. January 2017 last week prediction

10 N. Ayub et al.

train the forecasting engine and test on the last week of January 2017. All the
similar months of data are trained in the same pattern.
The forecasted load of the last week of January 2017 is shown in Fig. 6.
Figure 7, illustrates the actual and forecasted load of the month December 2017.
Load forecasting of 9 months is shown in Fig. 8 (Table 1).

20000
Act ual
Predict ion
19000

18000
Load (MW)

17000

16000

15000

14000

13000
5 10 15 20 25 30

Days

Fig. 7. One month prediction (Dec 2017)

24000
Actual
Prediction
22000

20000
Load(MW)

18000

16000

14000

12000
50 100 150 200 250 300
Days

Fig. 8. Nine months prediction

 Electricity Load Forecasting in Smart Grids Using Support Vector Machine 11

Table 1. Attributes of data

Attributes Description
DA Day-Ahead Cleared Demand, comprised of price-sensitive
demand and demand
RT demand Real time demand
DA LMP Day ahead local marginal price
DA EC Energy consumption of day ahead
DA CC Congestion demand of day ahead
DA MLC Day Ahead marginal loss component
RT LMP Real time location marginal price
RT CC Real time congestion component
RT EC Energy component of real time
RT MLC Real time marginal loss component
Dry bulb Dry bulb temperature for weather station corresponding to
the load zone
Dew point Dew point temperature for weather station corresponding
to the load zone

4.3 Performance Evaluation

To evaluate the performance, two evaluators are used; Root Mean Square Error
(RMSE), Mean Average Percentage Error (MAPE). MAPE has the lowest error
value, i.e., 1.682. RMSE has the highest error value, which is not a good result.
The formulas of MAPE and RMSE is given in Eqs. 4 and 5.

Table 2. Performance evaluators

Evaluator Value
RMSE 12.364
MAPE 1.682

TM
1  Av
M AP E = 100 | | (4)
T tm=1 Fv

 TM
1 
RM SE =  (Av − Fv )2 (5)
T tm=1

where Av is the observed test value at time tm and Fv is the forecasted value at
time tm (Table 2).
12 N. Ayub et al.

5 Conclusions and Future Work

In this work, SVM classifier is used to solve the load forecasting accuracy prob-
lem. Forecasting model is based on feature engineering and classifier adjustment.
The forecasting model consists of two stages; feature engineering and SVM clas-
sifier. A hybrid of two techniques (XGBoost and DTC) is applied for feature
selection to select the best features among features in input data. After selec-
tion of features by feature engineering, features include some redundancy. New
features are selected after removing the redundancy by using RFE technique,
which has a positive effect on SVM classifier speed and forecasting accuracy.
The performance error metrics are calculated using MAPE and RMSE. SVM
classifier is tuned with three super parameters until the accuracy is achieved.
SVM classifier has 98% accuracy. In the future, other methods can be applied
to improve forecasting accuracy.

References
1. Kailas, A., Cecchi, V., Mukherjee, A.: A survey of communications and networking
technologies for energy management in buildings and home automation. J. Comput.
Netw. Commun. 2012, 12 (2012)
2. Iqbal, Z., Javaid, N., Iqbal, S., Aslam, S., Khan, Z.A., Abdul, W., Almogren,
A., Alamri, A.: A domestic microgrid with optimized home energy management
system. Energies 11(4), 1002 (2018)
3. Iqbal, Z., Javaid, N., Mohsin, S., Akber, S., Afzal, M., Ishmanov, F.: Performance
analysis of hybridization of heuristic techniques for residential load scheduling.
Energies 11(10), 2861 (2018)
4. Rahim, M.H., Javaid, N., Shafiq, S., Iqbal, M.N., Khalid, M.U., Memon, U.U.:
Exploiting heuristic techniques for efficient energy management system in smart
grid. In: 2018 14th International Wireless Communications and Mobile Computing
Conference (IWCMC), pp. 54–59. IEEE (2018)
5. Xiang-ting, C., Yu-hui, Z., Wei, D., Jie-bin, T., Yu-xiao, G.: Design of intelli-
gent demand side management system respond to varieties of factors. In: 2010
China International Conference on Electricity Distribution (CICED), pp. 1–5.
IEEE (2010)
6. Khan, M., Javaid, N., Naseem, A., Ahmed, S., Riaz, M., Akbar, M., Ilahi, M.:
Game theoretical demand response management and short-term load forecasting
by knowledge based systems on the basis of priority index. Electronics 7(12), 431
(2018)
7. Ahmad, A., Javaid, N., Mateen, A., Awais, M., Khan, Z.: Short-term load forecast-
ing in smart grids: an intelligent modular approach. Energies 12(1), 164 (2019)
8. Wang, K., Yu, J., Yu, Y., Qian, Y., Zeng, D., Guo, S., Xiang, Y., Wu, J.: A survey
on energy internet: architecture, approach and emerging technologies. IEEE Syst.
J. (2017)
9. Jiang, H., Wang, K., Wang, Y., Gao, M., Zhang, Y.: Energy big data: a survey.
IEEE Access 4, 3844–3861 (2016)
10. Alam, M.R., Reaz, M.B.I., Ali, M.A.M.: A review of smart homes—past, present,
and future. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 42(6), 1190–1203
(2012)
 Electricity Load Forecasting in Smart Grids Using Support Vector Machine 13

11. Zhu, Q., Han, Z., Başar, T.: A differential game approach to distributed demand
side management in smart grid. In: 2012 IEEE International Conference on Com-
munications (ICC), pp. 3345–3350. IEEE (2012)
12. Soares, J., Silva, M., Sousa, T., Vale, Z., Morais, H.: Distributed energy resource
short-term scheduling using signaled particle swarm optimization. Energy 42(1),
466–476 (2012)
13. Zhu, Z., Tang, J., Lambotharan, S., Chin, W.H., Fan, Z.: An integer linear pro-
gramming based optimization for home demand-side management in smart grid,
pp. 1–5 (2012)
14. Liu, J., Li, C.: The short-term power load forecasting based on sperm whale algo-
rithm and wavelet least square support vector machine with DWT-IR for feature
selection. Sustainability 9(7), 1188 (2017)
15. Ghasemi, A., Shayeghi, H., Moradzadeh, M., Nooshyar, M.: A novel hybrid algo-
rithm for electricity price and load forecasting in smart grids with demand-side
management. Appl. Energy 177, 40–59 (2016)
16. Wang, K., Xu, C., Zhang, Y., Guo, S., Zomaya, A.: Robust big data analytics for
electricity price forecasting in the smart grid. IEEE Trans. Big Data (2017)
17. Ryu, S., Noh, J., Kim, H.: Deep neural network based demand side short term load
forecasting. Energies 10(1), 3 (2016)
18. Fan, C., Xiao, F., Zhao, Y.: A short-term building cooling load prediction method
using deep learning algorithms. Appl. Energy 195, 222–233 (2017)
19. Kuo, P.-H., Huang, C.-J.: An electricity price forecasting model by hybrid struc-
tured deep neural networks. Sustainability 10(4), 1280 (2018)
20. Moghaddass, R., Wang, J.: A hierarchical framework for smart grid anomaly detec-
tion using large-scale smart meter data. IEEE Trans. Smart Grid 9(6), 5820–5830
(2018)
21. Zhao, J.H., Dong, Z.Y., Li, X.: Electricity price forecasting with effective feature
preprocessing. In: IEEE Power Engineering Society General Meeting, p. 8-pp. IEEE
(2006)
22. Qiu, Z.-W.: Mutivariable mutual information based feature selection for electric-
ity price forecasting. In: 2012 International Conference on Machine Learning and
Cybernetics (ICMLC), vol. 1, pp. 168–173. IEEE (2012)
23. Abedinia, O., Amjady, N., Zareipour, H.: A new feature selection technique for load
and price forecast of electrical power systems. IEEE Trans. Power Syst. 32(1), 62–
74 (2017)
24. Qian, H., Qiu, Z.: Feature selection using C4.5 algorithm for electricity price pre-
diction. In: 2014 International Conference on Machine Learning and Cybernetics
(ICMLC), vol. 1, pp. 175–180. IEEE (2014)
 Optimization of Response and Processing
Time for Smart Societies Using Particle
Swarm Optimization and Levy Walk

Ayesha Anjum Butt1 , Zahoor Ali Khan2 , Nadeem Javaid1(B) , Annas Chand3 ,
Aisha Fatima1 , and Muhammad Talha Islam1
1
COMSATS University Islamabad, Islamabad 44000, Pakistan
[email protected]
2
Computer Information Science, Higher Colleges of Technology, Fujairah 4114, UAE
3
COMSATS University Islamabad, Abbotabad 22010, Pakistan
http://www.njavaid.com

Abstract. Reducing delay and latency in the cloud computing environ-

ment is a challenge for the present research community. This study per-
formed a rigorous, comparative analysis of the fog computing paradigm
and the conventional cloud computing paradigm in the context of the
Smart Grid (SG). To meet the consumers’ demand and optimize cloud
services to achieve service level objectives is of great importance. The
fog is introduced to enhance the efficiency of the cloud and to fulfill
the consumer requests at the edge of the network. When the requests
of Smart Societies (SSs) are huge on fog, the increased demand for real-
time response is becoming a challenge for the SG. In this study, Par-
ticle Swarm Optimization is implemented and compared with the pro-
posed techniques: Improved PSO with Lewy Walk (IPSOLW). These
load balancing algorithms are compared on the basis of Closest Data
Center (CDC) and Optimize Response Time (ORT). These proposed
algorithms handle the load of SS on fog. The proposed IPSOLW han-
dles more requests because of LW, the requests are directly allocated to
best DC.

1 Introduction
Recently world technology has been evolved to the conceptualization, develop-
ment, and implementation of cloud computing system. The way of traditional
Information Technology (IT) ridding towards the cloud. The way using hard-
ware, computers or even software spin into cloud computing, due to its adaptive
and widespread nature. Cloud computing depends on the network of Data Cen-
tric Network, which are the exclusive monopolized hubs which are responsible for
computation and storage. To diminish the restrain of cloud, most requests are
processing within the Data Centers (DCs). However, due to increase of the num-
ber of internet connections the number of connected smart devices also increased
over the internet. This thing turns up towards the emerging technology light of
the Internet of Things (IoT).
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 14–25, 2020.
https://doi.org/10.1007/978-3-030-15032-7_2
 Optimization of Response and Processing Time for Smart Societies 15

The cloud computing facilitates the operation of SG, by case of this the
energy utilization of SG becomes intelligent, efficient and automated integration
that are indirectly connected to the distributed power generation. When the huge
amount of data is collected by Smart Meters (SMs) then this data is required to
be processed and stored. The cloud provides this storage facility to the incoming
request of the SHs.
Cloud also facilitates about optimal workload allocation that how to control
load in a heterogeneous environment [1]. For the sake of this purpose, multi-
tenant framework is proposed, a load balancing cloud max-min algorithm is pro-
posed to schedule the consumers’ requests. The priority of incoming requests is
calculated on the basis of stochastic probability distribution. To meet the service
level architecture, two scenarios are implemented by selecting performance and
execution time of the balancing algorithm. The two scenarios are: (i) implemen-
tation of VMs, and divides consumer requests on them. (ii) implementation of
vertical scaling VMs with higher speed CPUs to maximize the consumer requests.
The Monte Carlo simulations are performed to execute the consumer requests.
In [2,3], the integration, distribution of power becomes larger and complex. It is
because the rate of growth of requests is increased from the consumer side day
by day. So, it becomes difficult for the grids and SGs to facilitate the consumer
according to their requirements. To make the system scalable and efficient to
handle the consumers’ requests, cloud platform are introduced in the SG envi-
ronment to reduce the load on grid side. The implemented cloud platform also
enhanced the performance of the grid stations. In [2], grid cloud is implemented
on the cloud platform to handle the bulk data, for this purpose online analy-
sis and forensic exploration is used. Resilient Distributed Data Sets (RDDs) by
using direct acyclic to solve the low performance problem is used in [3]. It is
because these implemented techniques monitor the consumers requests and then
operate the services and provide a validated platform to the requests on the
cloud.
There is another issue of cloud service provider is that maximum operational
cost. To deal this type of problem parallel distributed load balancing algorithm
is proposed in [4]. The implemented algorithm low cost complexity is based
on proximal Jacobian alternating direction method of multipliers. To check the
result of this implemented algorithm extensive simulation is done, by using fog
devices the Computational cost of cloud is also reduced. In [5] Ant Colony Opti-
mization is proposed to calculate cost, RT and PT on the basis of ORT policy.
Summarizing the previous work, these things reveal how the concept of cloud
and fog is introduced in SG. How these concepts made homes to SHs in IoT. To
keeping in mind the previous studies, in this paper, we implement cloud and fog
based integrated environment. In the direction of efficiency and performance of
implementing the system, the contributions of this paper could be summarized
below (Fig. 1):

• Fog and cloud based integrated platform for six regions of the world.
• Implementation of PSO and our proposed algorithm IPSOLW.
16 A. A. Butt et al.

Fig. 1. Proposed smart environment

• Comparison of both implemented and proposed algorithm on the basis of two

service broker policies CDC and ORT.
• On the basis of above mentioned algorithms and broker policies, we will cal-
culate average RT, PT and Cost.
• This paper proposed IPSOLW performs better than existing PSO.

The remainder of the paper is organized as follows: related studies are presented
in Sect. 1. System model with proposed methodology are demonstrated in Sect. 4.
The Sect. 5 defines the proposed algorithm of this Sect. 6 describes the simulation
results of our proposed schemes. The Sect. 7 describes the conclusion of this
paper.

2 Related Work

Nikolaos et al. in [6], proposed a two layer architecture. Their aim is to antagonist
issues inside the data centers. They want to tackle the problems of resource uti-
lization, VMs allocations and load balancing of numerous incoming results and
their placement in physical servers. To resolve these tackled issues and main-
tain the QoS metrics and to minimize energy consumption, they also discuss
fuzzy Takagi-sugeno modeling in their work. By implementing two layer proposed
 Optimization of Response and Processing Time for Smart Societies 17

model they achieve reduction of energy. Even so, they carry out their employ-
ment for lower stage or single host. In future, they can implement or apply this
for multiple servers. To guarantee the VM performance and how to place VMs
to reduce the consumption of PM is one of major challenge is discussed by Zhao
et al. in [7]. To resolve this challenge first they consider the problem of CPU uti-
lization and power consumption, second check the trend of VM degradation and
then formulates the bi-objective problem. For the sake of solution, they design
a power aware and performance guaranteed VMP (PPVMP) model. They also
propose a meta-heuristic algorithm in their work named as Ant Colony Opti-
mization (ACO) to reduce the energy consumption. The cloud sim and open
stack platform is used to evaluate the result of their designed model and pro-
posed algorithm. The results that, PPVMP can reduce PM power consumption
and guarantee the performance of VM placement over the cloud. However, they
can also consider fog in their work to enhance the performance of their model.
The QoS during task scheduling and to minimize energy consumption is
another challenge. To overcome this challenge energy aware task scheduling
method named as QET is introduced. The introduced model is based on QoS
in cloud environment to minimize the energy consumption. PM selection energy
consumption is also done through QoS based model. However, this implemented
model cannot handle solution when there are heterogeneous PMs. They can also
implement this model in a fog environment by resolving the shortcomings of
their model [8].
In [9], Zoltan discusses about two optimization problems, that influence each
other significantly. He is also examining the problem of VM placement and the
utilization of PMs. To propose the solution of these problems, the uses of pri-
vate cloud and fog DC of a private organization. The different algorithms such
as; dedicated black box, determine the candidates of VMs and PMs, etc., are
predicted to solve these problems. These implements algorithms gives different
results on different solutions. However, performance and cost of implementing
algorithms are not discussed.
Yu and Jiang in [4], intend to reduce the cost of cloud data centers by using
fog devices. It is because the revenue loss of these networks is less by allocating
the services to near cloud data centers due to huge network. To fulfill their
aim, they formulate the operational minimization cost of fog devices which is
provided by cloud service providers. The provided cost consists of four parts
that are: energy cost of cloud data centers, network bandwidth cost, due to delay
revenue loss, and paid allowances to fog devices. In their work, they also proposed
parallel and distributed algorithm which is based on PJ-ADM. They get less
computational cost with low complexity. Nevertheless, they can also compute the
performance of their network and implementation cost of the network can also be
considered. When there is large network with multiple servers, it consumes lot’s
of energy. Due to this the energy cost is also very high [10]. To investigate about
this Lia and Sun discuses in their study. According to them, the deployment
of cloud networks with Software Define Network (SDN) it becomes easier to
implement and adjustment of different types of network in the cloud. They also
18 A. A. Butt et al.

discuss about the Virtual Contents Distributed Networks (VCDN) and their
energy efficiency issue. To overcome all aforementioned issues and challenges,
they proposed an off-line optimal algorithm which provides the optimal solution
of studies problems. For the sake of improvement in energy efficiency, they design
a time efficient approximate algorithm to predict incoming traffic to improve
energy efficiency. To check the performance of their proposed approach they
perform multiple simulations. After performing the simulations, they save the
energy in VCDN based network. There is a tradeoff between numerous incoming
traffic and QoS in VCDN network.
Marco et al. discuss about the evolution of power systems towards the grids in
[11]. According to their study, new targets and technologies are required because
it totally depends on the modernization of the distributed grids. To achieve the
new targets their work presents smart metering infrastructure. Which unlocks
the set of services and automation management of the distributed grid. Their
proposed algorithm gives the opportunity to communicate with smart meter.
This is one way communication. In this communication the requirements of con-
sumers is monitored and than provides services according to that requirements.
It is because the proposed algorithm is real time distribution algorithm, which
provide services over the cloud by using automation of grid. Their aim is also
to provide the key features such as; interoperability, scalability to the cloud. By
using their implemented real time distribution algorithm they fulfill their aim.
On the other hand, if smart metering infrastructure facilitates with two way
communication it gives better results.

3 Problem Statement
The proposed work in [12–14] is focused on different problems: Virtual Machine
(VM) allocation problems, Variable Size Bin Packing Problem (VSBPP), multi
objective problems and energy management related to cloud and fog based com-
puting. In [12], Improved Levy Whale Optimization Algorithm (ILWOA) is pro-
posed. However, the multi objective problems such as processing and response
time of the cloud are not handled by the proposed algorithm. Multi Objective
Grey Wolf Optimizer (MOGWO) is proposed in [13]. Although, it is not possible
for it to find satisfactory solutions in larger search space, when optimum result is
required. When requests need VMs to store and process the data, so it is difficult
for it to find appropriate platform in the search space. In [6,7], the fog based
architecture is implemented by using day-ahead, Particle Swarm Optimization
(PSO), Binary Particle Swarm Optimization (BPSO) and BAT algorithm. How-
ever, there is more delay in finding a best solution and search space because of
their premature convergence. In [14], Levy Flight Particle Swarm Optimization
(LFPSO) is proposed. Because it searches both locally and globally. The solution
is given at current time due to randomness in it’s nature.
This work is to devise an integrated model of cloud and fog based comput-
ing by using Improved PSO by using Levy Walk (IPSOLW). The Micro Grid
(MG) and Smart Meter (SM) are attached to the SHs that are connected to
 Optimization of Response and Processing Time for Smart Societies 19

the implemented architecture. MG is used as a small grid known as renewable

energy resources. MG also keeps the record of SHs. SM is used to keep the
energy consumption tracks of the SH. We formulate this scheduling problem as
a multi objective problem to schedule the incoming requests to cloud and fog
based architecture.

4 Proposed System Model

The cloud computing services is extended by fog computing and the services at
the edge of the network where data is created and acted upon. The term fog is
introduced by “cisco” in recent few years. In Fig. 2 shows the proposed system
model which introduces SM and MG architecture based on fog computing and
efficient management of energy. As this proposed system model shows the three
tiers of the proposed architecture. In first tier, which is the bottom-most layer of
the proposed model is known as consumer layer. The SHs for which this energy
management model is proposed are located in the different regions of the world.
These are the SSs because of direct communication to fog and cloud. These SHs
in SSs can send request for the services to the cloud and fog of the architecture.
When their services is processed according to the requirements they can also
get response. SHs can also communicate with each other and share their energy
resources with each other. The saved energy is provided to the servers at the
bottom levels of the architecture, because SHs can manage their own energy.
SMs are also attached with the SHs, which records and monitors the services.
SM provides the record of energy consumption of the SHs or the Smart building
through which it is attached.
MG plays an vital role between bottom layer and the middle layer of this
proposed architecture. Each region and fog in the proposed model are connected
with its own MG. It keeps the record of MG. When any of the SHs requires
services from the servers of the top two layers, they requests for that. If required
services are not fulfilled by the upper servers, than MG provides services to that
SH. As the aforementioned scenario, then it is another possibility that if MG
able to fulfill the requirements of the consumers, then MG gives the services to
them.
In second tier, which is middle layer of the proposed system model is known as
fog computing layer. The VMs and inside them PMs are the primary components
of this layer. The fog is located at the edge of the network, then it provide services
to the consumers with less latency and delay. Fog provides its services locally
in this proposed system. It also act as intermediary layer between cloud and
consumer layer. Because of regional location, maximum number of requests is
processed by fog easily with less latency and delay. This fog layer also overcomes
the deficiencies of the cloud.
In third tier, which is the upper most layer of the proposed model. The
high level servers and DCs comprise this tier. Cloud always provides it’s services
remotely. It processed and serves large number of incoming requests. It also
overcomes the shortcomings of the fog, that is less storage and processing of
20 A. A. Butt et al.

Fig. 2. Cloud and fog based integrated system model

requests. This layer provides the services to the consumers in terms of storage.
As same like the other cloud, the proposed model cloud also provides three
types of services which are: Infrastructure as a Services (IaaS), consumers of
 Optimization of Response and Processing Time for Smart Societies 21

this proposed model can used the services any time. They have to paid for these
services to the servers, benefit of these services is that they donot need to pay
extra charges for any other services. Platform as a Service (Paas), the consumers
of SHs can easily used any type of software and applications on the server systems
of DCs without installing them on their personal computers.

5 Proposed Scheme

The novel meta heuristic algorithm that depends on the nature of the swarm
and random behavior of levy is known as IPSOLW. The algorithm works same
as the nature of the swarm, i.e., particle search for best position, where the best
position is found, particle best position is updated. In this proposed algorithm,
the velocity of particle swarm is updated with levy walk, because of it’s prema-
ture convergence. In this paper, IPSOLW Algorithm 1 performs load balancing.
Initializes VMs, fogs in the case of load balancing. Besides, the probability fitness
is calculated with respect to DC. Our environment is fog based, so the fogs act as
a source of best position. Therefore, the implemented load balancing algorithm
provides help to VMs to find the best feasible solution until they find the best
fitness value or best optimization solution value.

6 Simulation Results and Discussion

In this section, we have performed simulation by using Cloud analyst to check
the efficiency of PSO and proposed algorithm IPSOLW. To do so, performance of
these algorithms is compared on the basis of RT, PT, and cost. The cost, RT, and
PT is compared on the basis of service broker policies, i.e., Closest Data Center
(CDC) and Optimize Response Time (ORT). The clusters RT is also calculated
on the basis of these policies. The simulation results of average RT of this paper
implemented load balancing algorithms are shown in Fig. 4a. In this figure the
Avg. RT of PSO and IPSOLW are compared on the basis of CDC and ORT.
The avg. RT of PSO on the basis of CDC is 13.24%, and on the basis of ORT
is 7.09%. The avg. RT on the basis of CDC is 7.031% and on the basis of ORT
is 4.37% for IPSOLW. In this fig. it is clearly shown that the RT of IPSOLW
is maximum than PSO. It simply means that IPSOLW handles large number
of requests of consumers of SHs rather than PSO. The avg. RT of IPSOLW is
maximum to handle large number of incoming requests from consumer side.
22 A. A. Butt et al.

Algorithm 1. IPSOLW
1: Start
2:
3: Initialize the position of a Particles of P Swarms in the search space randomly;
4:
5: Search for list of VM and Fog;
6:
7: j = F og;
8:
9: i = V M;
10:
11: for t = 1:24 do
12: Let Y is a random position of search space;
13:
14: Evaluate the position of the PVM;
15:
16: Initialize the memory of each PVM;
17:
18: while iter. ≤ itermax do
19: Determine the VM Size;
20:
21: Compute the processing time using equation 1;
22:
23: Calculate Transmission delay using equation 2;
24:
25: Compute the response time using equation 3;
26:
27: for i = 1 : C do
28: Randomly get a PVM j to follow i;
29:
30: Define awareness probability;
31:
32: if rj ≥ AP j,iter then
33: xi,iter1 = xi,iter + ri X(mj,iter − xi,iter )
34:
35: else
36: xi,iter+1 = Y ;
37:
38: end if
39: end for
40: end while
41: Evaluate the usefulness of new fog;
42:
43: Check the new position of the PVM;
44:
45: end for
46: End
 Optimization of Response and Processing Time for Smart Societies 23

IPSOLW is also higher than PSO, it is because IPSOLW also has maxi-
mum RT. The reason to have maximum PT is that IPSOLW handles number of
incoming requests of SHs consumers. The PSO also handles numerous requests of
consumer side, it gives nimble solution as compared to other algorithms. Some-
times, these solutions are not feasible to the consumers, it is because PSO has
premature convergence nature. In this fig. the PT of PSO is 1.15% and 5.381%
on the basis of CDC and ORT. The PT of IPSOLW is 2.12% and 2.77% on the
basis of CDC and ORT.
The formula which is used to calculate the processing time is given in below:
P T = T R/V Mspeed (1)
where TR is the total requests of the consumers and V Mspeed is the speed of
VMs. The transmission delay is computed as given below:
T D = T otall + T otalT D (2)
where l is latency and TD is the transmission delay.
The formula which is used to calculate the response time is:
RT = F T − AT + T D (3)
where F T is the finish time, A T is the arrival time and TD is the transmission
delay.

Fig. 3. Proposed scenario world division map

The cost of PSO and IPSOLW is shown in Fig. 5a and b. These figures repre-
sent the VM cost, Data Transfer (DT) cost and Total Cost of these implemented
load balancing algorithms. In this Fig. 5a, it is clearly shown that VM cost and
DT cost of PSO is minimum on the basis of service broker policies. The VM
and DT cost of IPSOLW is also minimum on the basis of service broker policies
by using IPSOLW Fig. 5b. The VM Cost of PSO is 1.11% and 1.15%, which is
based on CDC and ORT. DT cost of PSO is 98.88% and 98.84%. The VM and
DT cost of IPSOLW which is based on CDC and ORT is 98.6% (Fig. 3).
24 A. A. Butt et al.

300 250
PSO PSO
250 IPSOLW IPSOLW
200

Average PT (ms)
Average RT(ms)

200
150
150
100
100

50
50

0 0
CDC ORT CDC ORT
Service Broker Policies Service Broker Policies

(a) Average Response Time (b) Average Processing Time

Fig. 4. Average RT and PT

7000 8000

6000
6000
5000
PSO.VM Cost
IPSOLW.VM Cost
PSO.DT Cost
Cost ($)

Cost ($)

4000 IPSOLW.DT Cost

PSO.TCost
4000 IPSOLW.TCost
3000

2000
2000
1000

0 0
CDC ORT CDC ORT
Service Broker Policies Service Broker Policies

(a) PSO Cost Comparsion (b) IPSOLW Cost Comparison

Fig. 5. PSO and IPSOLW cost comparison

7 Conclusion

Considering the numbers of SHs applications that are running on numerous

things, as well as integration of cloud and fog based SSs such application are
processed at the edge of the network. In this paper, we bring down the size
of VMs to minimize the utilization of energy. To calculate the results of our
proposed scenario on the basis of six SSs on the basis of CDC and ORT, nature
inspired algorithm are implemented and proposed. The Proposed algorithm in
this work is IPSOLW, which is compared with implementing. In the end, this
thing is clearly concluded that proposed IPSOLW has 7.031% and 4.37% RT
on the basis of CDC and ORT. The PT of IPSOLW on the basis of CDC is
2.12% and ORT is 2.77%. However, IPSOLW has a maximum cost as compared
to PSO. In future we will consider bin packing problem with the different meta
heuristics algorithm.
 Optimization of Response and Processing Time for Smart Societies 25

References
1. Wang, Z., Hayat, M.M., Ghani, N., Shaban, K.B.: Optimizing cloud-service per-
formance: efficient resource provisioning via optimal workload allocation. IEEE
Trans. Parallel Distrib. Syst. 28(6), 1689–1702 (2017)
2. Anderson, D., Gkountouvas, T., Meng, M., Birman, K., Bose, A., Hauser, C.,
Litvinov, E., Luo, X., Zhang, F.: GridCloud: infrastructure for cloud-based wide
area monitoring of bulk electric power grids. IEEE Trans. Smart Grid, 1–10 (2018)
3. Wang, W., Zhou, F., Li, J.: Cloud-based parallel power flow calculation using
resilient distributed datasets and directed acyclic graph. J. Mod. Power Syst. Clean
Energy, 1–13 (2018)
4. Yu, L., Jiang, T., Zou, Y.: Fog-assisted operational cost reduction for cloud data
centers. IEEE Access 5, 13578–13586 (2017)
5. Buksh, R., Javaid, N., Fatima, I.: Towards efficient resource utilization exploiting
collaboration between HPF and 5G enabled energy management controllers in
smart homes. Sustainability 10(10), 3592 (2018). 3–24
6. Leontiou, N., Dechouniotis, D., Denazis, S., Papavassiliou, S.: A hierarchical control
framework of load balancing and resource allocation of cloud computing services.
Comput. Electr. Eng. 67, 235–251 (2018)
7. Zhao, H., Wang, J., Liu, F., Wang, Q., Zhang, W., Zheng, Q.: Power-aware and
performance-guaranteed virtual machine placement in the cloud. IEEE Trans. Par-
allel Distrib. Syst. 29(6), 1385–1400 (2018)
8. Xue, S., Zhang, Y., Xiaolong, X., Xing, G., Xiang, H., Ji, S.: QET: a QoS-
based energy-aware task scheduling method in cloud environment. Cluster Comput.
20(4), 3199–3212 (2017)
9. Mann, Z.Á.: Resource optimization across the cloud stack. IEEE Trans. Parallel
Distrib. Syst. 29(1), 169–182 (2018)
10. Liao, D., Sun, G., Yang, G., Chang, V.: Energy-efficient virtual content distribution
network provisioning in cloud-based data centers. Future Gener. Comput. Syst. 83,
347–357 (2018)
11. Pau, M., Patti, E., Barbierato, L., Estebsari, A., Pons, E., Ponci, F., Monti, A.:
A cloud-based smart metering infrastructure for distribution grid services and
automation. Sustain. Energy Grids Netw. 15, 14–25 (2017)
12. Abdel-Basset, M., Abdle-Fatah, L., Sangaiah, A.K.: An improved Lévy based whale
optimization algorithm for bandwidth-efficient virtual machine placement in cloud
computing environment. Cluster Comput., 1–16 (2018)
13. Mirjalili, S., Saremi, S., Mirjalili, S.M., dos Coelho, L.: Multi-objective grey wolf
optimizer: a novel algorithm for multi-criterion optimization. Expert Syst. Appl.
47, 106–119 (2016)
14. Jensi and Wiselin Jiji: An enhanced particle swarm optimization with levy flight
for global optimization. Appl. Soft Comput. 43, 248–261 (2016)
 On Maximizing User Comfort
Using a Novel Meta-Heuristic Technique
in Smart Home

Sajjad Khan1 , Zahoor Ali Khan2 , Nadeem Javaid1(B) , Waleed Ahmad1 ,

Raza Abid Abbasi1 , and Hafiz Muhammad Faisal1
1
COMSATS University Islamabad, Islamabad 44000, Pakistan
[email protected]
2
Computer Information Science,
Higher Colleges of Technology, 4114, Fujairah, UAE
http://www.njavaid.com

Abstract. The day by day increase in population is producing a gap

between the demand and supply of electricity. Installation of new elec-
tricity generation system is not a good solution to tackle the high demand
of electricity. To get the most out of the existing system, several demand
response schemes have been presented by researchers. These schemes try
to schedule the appliances in such a way that electricity consumption
cost and peak-to-average ratio are minimized along with maximum user
comfort. However, there exists a trade-off between user comfort and elec-
tricity consumption cost. In this paper, a novel scheme is developed for
the home energy management system to schedule the home appliances
in such a way that comforts the consumers economically. To evaluate the
effectiveness of our proposed scheme, comparison is performed with two
well known meta-heuristic techniques namely Flower Pollination Algo-
rithm (FPA) and Jaya Optimization Algorithm (JOA). Experimental
results shows that the proposed scheme outperforms FPA and JOA in
appliances waiting time reduction. Furthermore, the proposed scheme
reduced the electricity consumption cost and peak to average ratio by
58% and 56% respectively as compared to unscheduled scenario.

Keywords: Smart home · Smart grid · Meta-heuristic techniques ·

Home energy management · Demand response

1 Introduction
The gap between supply and demand is increased due to increasing electric-
ity demand in residential, commercial, and industrial sectors. To deal with the
increasing electricity demand, there exist two possible solutions: (i) installation
of new electricity generation systems and: (ii) efficient utilization of the available
energy resources. The former approach is in efficacious due to high expenses and
it may cause for global warming. The later approach is economical and highly
recommended, because it avoids to install new generation plants.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 26–38, 2020.
https://doi.org/10.1007/978-3-030-15032-7_3
 On Maximizing User Comfort Using a Novel Meta-Heuristic Technique 27

Traditional Grid (TG) is unable to solve the increasing electricity demand

problem. Electricity generation in TG relies on large electricity producing plants
that operates by conventional fossil fuels such as coal, oil and gas. This leads to
the emission of carbon dioxide gas into the Earth’s atmosphere which is believed
to be the main cause of green house effect. Furthermore, TG is confronted with
one-way power flow, electricity loses due to long transmission lines and power
in-stability. Smart Grid (SG) upgrades the in efficient TG to a quick responsive
electricity network by incorporating Information and Communication Technol-
ogy (ICT) based solutions [1]. These ICT based solutions not only enhances
the stability of power system but also enables the SG to incorporate Distributed
Energy Sources (DESs). DESs tackle the transmission losses in the SG by produc-
ing energy locally. SG manages the electricity demand of consumers via Demand
Side Management (DSM).
In DSM, consumers minimize their electricity consumption cost and Peak
to Average Ratio (PAR) by scheduling their electric appliances from high price
to low price intervals [2]. For this purpose, Demand Response (DR) strategies
are adopted which refer to the changes made in electricity load requirements of
consumers in response to the policies implemented by the utility. To minimize
the high electricity demand, SG encourages consumers to reduce their electricity
consumption temporarily in high price intervals. For this purpose, DSM intro-
duced two types of DR programs that motivate the consumers to actively reduce
their load in high price intervals. These schemes are (i) price base DR schemes
(ii) incentive base DR schemes. In price base DR schemes, utility dynamically
change the buying and selling prices of electricity. Some of the well-known pricing
schemes [3] are Time of Use pricing (ToU), Inclined Block Rate (IBR) pricing,
Real time Pricing (RTP), peak time rebate pricing and critical peak pricing.
Whereas, in incentive base DR scheme, utility incentivize consumers in the form
of low priced bills. In this scheme, home appliances are wirelessly turned-OFF
by utility due to peak creation.
DR strategies not only solve the DR problem but also helps the consumers
to reduce their electricity consumption cost. However, these schemes cannot
solve the demand and supply problem without involvement of the consumers.
Whereas, the consumers are neither economists nor power analysts to schedule
and monitor their electricity consumption. Furthermore, due to lack of informa-
tion about electricity tariffs, electricity demand may increased. Therefore, the
need of a Home Energy Management System (HEMS) to intelligently switch the
home appliances ON or OFF with varying electricity demand in SG is inevitable.
Our work focuses on the energy management of a smart home. The basic reason
for home energy management is that residential user consumes 30–40% of the
overall energy [4]. A smart home comprises of several components such as Smart
Meter (SM), smart scheduler, smart electric appliances and smart storage devices
etc. These appliances are able to react, in response to the signals received in order
to shift or reduce their electricity load. The SM provides two-way communica-
tion between smart home and utility via advanced metering infrastructure. In
28 S. Khan et al.

this way, consumers are informed about the varying prices and power failures
that may occurred due to natural disasters or equipment failures.
Home appliances scheduling problem in HEMS can either be solved using
predictive energy management or real-time energy management [5]. The former
approach requires HEMS to schedule the home appliances on the basis of histori-
cal energy consumption pattern [6]. The later approach uses real-time algorithm
to shift electrical appliance to other times in order to reduce their electricity con-
sumption during peak hours. Furthermore, the real time approach to schedule
home appliance is divided into two subcategories (1) analytic approach and (2)
meta-heuristic approach. The first approach gives the prominent results; how-
ever, this approach is computationally expensive and converges slowly. Whereas,
the second approach is computationally in expensive and provides near to opti-
mal results [7].
In this work, our aim is to develop a HEMS to schedule the home appli-
ances using real time management. Price base DR strategies with meta-heuristic
techniques will be used to evaluate the performance of our scheme. The rest
of the paper is organized as follows: Sect. 2 is the related work followed by the
motivation and problem statement in Sect. 3. Sections 4 and 5 describes the sys-
tem model and optimization techniques. Section 6 discusses the results, and the
paper is concluded in Sect. 7.

2 Related Work
Many DR schemes have been proposed by different authors in last few years.
These schemes mainly focused on minimizing PAR, energy consumption cost
and to maximize User Comfort (UC).
The work in [8] design a scheme for minimizing cost and PAR. The proposed
scheme use two well known meta-heuristic techniques namely Cuckoo Search
Optimization Algorithm (CSOA) and Strawberry Algorithm (SA). The work
considered that each home is equipped with an energy production system, i.e.,
Micro Grid (MG). MG in the proposed work includes a wind turbine along
with PV system. MG for electricity generation is an intermittent source. This
intermittent nature of MG affects the power system stability. In order to make
the power system stable, Energy Storage System (ESS) is also considered in this
work. Experimental results demonstrate that the proposed scheme reasonably
minimizes cost and PAR using CSOA and SA. Moreover, CSOA is effective in
minimizing consumption cost and maximizing profit as compare to SA. A hybrid
technique for Home Energy Management Controller (HEMC) is developed to
schedule home appliances according to user preferences [9]. The scheme used
Lightening Search Algorithm (LSA) with Artificial Neural Network (ANN) to
compute learning rate along with the best case values used in the hidden layers of
ANN in the HEMC. Experimental results demonstrate that the proposed hybrid
scheme minimizes the electricity bill. The efficiency of the proposed system is
further enhanced by incorporating Renewable Energy Sources (RES). However,
PAR alleviation and power system stability are ignored in this work. The work
 On Maximizing User Comfort Using a Novel Meta-Heuristic Technique 29

in [10] propose a scheme to minimize the load demand using GA. In this scheme
the electricity consumption pattern (of industrial, commercial enterprises and
domestic users) from multiple data-sets is evaluated. Performance of the scheme
is evaluated by examining the energy consumption pattern through GA, i.e.,
GA-DSM and DSM without GA. Experimental results demonstrate that the
scheme achieves the desired results, i.e., minimization of electricity consumption.
Moreover, GA-DSM performs better as compared to DSM without GA. However,
UC and PAR are ignored in this work.
The work in [11] presents a scheme to tackle the problem of load demand in
critical hours using appliance scheduling and smart charging. A community based
scheduling algorithm is proposed to make sure that the electricity consumers are
always connected to electricity source. In this scheme, the consumers electricity
load and associated batteries are merged together to acquire the charging as
well as consumption patterns of the consumers appliances. Home appliances are
scheduled on the basis of day-ahead electricity pricing. In this work, the authors
combine the RTP scheme with the ToU scheme. So that high electricity load
is shifted to off-peak hours. Experimental results demonstrate that this scheme
achieves minimum cost and PAR by merging RTP with ToU. Furthermore, the
work also achieves an acceptable trade-off in terms of UC and cost minimization.
In [12], a scheme is developed for load scheduling and power trading by incor-
porating RES. The scheme uses dynamic programming to shift the appliances
to different time intervals. In this work, the consumers not only produces the
electricity from RES but also sells surplus electricity to the utility or neighbor-
ing homes. Experimental results demonstrate that the proposed work reduces
electricity cost of the consumers. It is due to the reason that competing users
exist in the neighborhood who are willing to sell their surplus electricity at
low price. The work adopts a game theory-based approach for power trad-
ing between consumers and prosumers. The authors in [13] propose a scheme
to schedule the electric appliances by incorporating RES along with storage
resources. In this scheme, the appliances scheduling problem is first formulated
as MKP. Then using meta-heuristic techniques, namely GA, Hybrid GA-PSO
(HGPSO), Wind Driven Optimization (WDO), Bacterial Foraging Optimization
(BFO) and Binary PSO (BPSO) a balance schedule of home appliances is gen-
erated. Experimental results demonstrate that HGPSO outperforms GA, WDO,
BFO and BPSO in-terms of cost and PAR minimization.
The work in [14] proposed HEMS using meta heuristic techniques to plan
the operational time of electric appliances. This work uses GA, CSOA and
Crow Search Algorithm (CSA) to minimize cost, PAR and appliances waiting
time. This work incorporates ESS for stable grid operation. Experimental results
demonstrate that CSOA outperforms GA and CSA in terms of PAR reduction,
user waiting time and electricity cost minimization. However, this work did not
consider the electricity consumption cost of ESS.
The authors in [15] present a scheme for scheduling home appliance on the
basis of time and device based priorities. The priorities are based on user pref-
erences. On the basis of the time and device-based priority, an absolute comfort
30 S. Khan et al.

level of consumers is computed. The HEMC then schedule the electric appliance
using the absolute comfort level, consumers budget and the total power available
to the consumer. GA is used to generate the schedule of the home appliances
based on maximum UC. Experimental results demonstrate that the proposed
scheme reasonably maximizes the UC in a small budget. Huang et al. in [16]
propose DR scheme for peak load reduction. A multi-agent minority game based
strategy is adopted to reduce peak load and fairly allocate solar power for man-
agement of load. Supervised learning techniques are employed to predict the
demand of energy. However, this scheme considered a similar profile and energy
consumption pattern for all consumers. Experimental results demonstrate that
the scheme reasonably minimizes the peak load. However, this work is affected
by the changing weather conditions.

3 Motivation and Problem Statement

DR optimization techniques play a key role to overcome the increasing demand of
electricity supply problem. Many techniques have been proposed to balance the
varying electricity demand. However, there exists a trade off between conflicting
objectives. Normally, PAR and electricity cost is minimized at the expense of
UC. The work proposed in [14] minimizes cost and PAR with an acceptable
trade off in user waiting time by incorporating ESS. However, this work did
not consider the operational and maintenance cost of the ESS. The work in
[11] and [13] minimizes electricity consumption cost. However, UC is completely
ignored in this work. The work in [15] present an absolute user comfort using
time and device based priority. However, in this work PAR reduction is not taken
into account. In the aforementioned work, no scheme is proposed to minimize
appliances waiting time.

4 System Model
This work investigates energy consumption in a smart home. The smart home
is comprised of numerous smart appliances. Each appliance has a distinct power
rating and Length of Operational Time (LOT). Figure 1 represents the proposed
system model. The appliances considered in this paper are taken from [17]. The
smart home is directly connected with an electric grid. The electric grid has
the feature of a smart grid (bi-directional communication between the utility
and consumers). The use of smart meter allows the consumer to extract the
pricing signals and maintenance schedule from the utility. Generally appliances
scheduling is done for a single day. In this paper, a day is divided into twenty
four hour time slots.
According to the energy utilization pattern and operational properties,
the home appliances are categorized into three classes. Class 1 contains the
Deferrable Interruptible (AD I ), Class 2 is the Deferrable Non-interruptible
(AD N ) whereas, class 3 represents Non-Deferrable Appliances (AN D ). AD I can
be flexibly adjusted in any time interval. AD N can also be adjusted in any time
 On Maximizing User Comfort Using a Novel Meta-Heuristic Technique 31

Utility

Smart meter

Fig. 1. System model

intervals however, these appliances cannot be interrupted during their opera-

tions. AD N are the appliances that should be continuously ON without any
interruption in their respective time intervals. Table 1 represents the set of appli-
ances used in this work. It is assumed that each appliance has the capability to
communicate with the local EMC using a home area network. The EMC controls
these appliances by switching them from high to low price intervals.

Table 1. Appliances classification

Class Name Power (kWh) LOT(h)

ADI Vacuum cleaner 0.7 6
Electric car 2.5 14
Laptop 0.1 8
Dish washer 1.8 8
Desktop 0.3 6
ADN Washing machine 1 3
Cloth dryer 3 5
AN D Cooker top 2.15 2
Cooker oven 5 2
Microwave 1.7 2
Refrigerator 0.3 24
Interior lights 0.84 6

Let ω be the time when the consumer intended to switch an appliance ON, σ
represents the LOT of an appliance and Ω represents when the HEMS initiates
the appliance. The Waiting Time (WT) of an appliance is computed as the
difference between Ω and ω. The objective of this work is to determine an optimal
operational schedule of home appliances such that the overall waiting time of
appliances and electricity cost is minimized. Hence, the objective function can
32 S. Khan et al.

be stated as in Eq. 1. In this paper, reducing the appliances waiting time and
maximizing UC are used alternatively. Figure 2 represents the execution pattern
of an appliance.

minimize WT s.t. CostScheduled < CostU nscheduled (1)

Fig. 2. Execution pattern of an appliance

The cost (in terms of money) that is paid by the consumer to the utility is
computed as the power consumed by an appliance in one hour weighted with
the electricity price announced by the utility in that hour. According to previous
studies, the well known pricing schemes are CPP, IBR, RTP and ToU. In this
work, the pricing scheme used is CPP. The per hour electricity cost announced
by the utility is shown in Fig. 3. From this figure, it can be seen that the per hour
cost during peak hours is very high. It means that high energy consumption cost
during these hours will result in high cost. Therefore, our goal is to minimize
electricity consumption during these hours. The total electricity cost and load
of the smart appliances for one hour and complete day is computed using Eqs. 2
and 3 respectively.
24
LoadT = ρ(t) ∗ η (2)
t=1


24
CostT = ρ(t) ∗ η ∗ β (3)
t=1

140
CPP
120
Price (cent/kWh)

100

80

60

40

20

0
1 4 8 12 16 20 24
Time (hour)

Fig. 3. Pricing signal

 On Maximizing User Comfort Using a Novel Meta-Heuristic Technique 33

Here LoadT presents the total load, CostT is the total electricity cost, ρ is
the power ratings of an appliance, η is ON or OFF status of an appliance and β
is the electricity price announced by the utility. PAR is computed using Eq. 4

max(LoadH )
P AR = (4)
LoadT
Here LoadH represents the maximum scheduled load in a particular hour.

5 Optimization Techniques
In this section, the existing and proposed optimization techniques are discussed.

5.1 FPA
In [18], Yang et al. develop a nature inspired algorithm on the basis of flower
pollination process. Typically pollination occurs due to transfer of pollens. This
transfer is mainly linked with pollinators such as honeybees and insects etc.
Pollination in flowers can either be abiotic or biotic. Abiotic pollination in flowers
occurs due to water diffusion or wind blowing. From the view point of biological
studies, flower pollination is termed as survival of the fittest. Therefore, it is
considered as an optimization process in plants reproduction where the objective
is to minimize the reproduction of weak flowers. FPA is idealized using the
following rules.
• Cross and biotic pollination is termed as global pollination. It occurs due to
Levy flights of pollinators.
• Local pollination is termed as self and abiotic pollination.
• Flower constancy is termed as the probability of reproduction in similar flow-
ers. It is directly proportional to the similarity of the flowers involved in
reproduction.
• Switch probability decides the status of local or global pollination. It is
directly affected by physical proximity and environmental factors such as
wind blowing.

5.2 Jaya Algorithm

Keeping in view, the parameters specific nature of the meta-heuristic techniques
R. Venkata Rao proposed Jaya Optimization Algorithm (JOA) [19]. JOA has
no algorithm specific parameters. It is based on the idea that the algorithm
proposed to solve a particular problem must be victorious in acquiring the best
solution and has to move away from the worst solution. JOA is idealized using
the following rules.
• Given a population, identify the best and worst solutions.
• Modify the candidate solutions based on the best and worst solutions.
34 S. Khan et al.

5.3 User Comfort Maximization

This section discusses the proposed scheme User Comfort Maximization (UCM).
UCM is a population based meta-heuristic scheme. It starts by initializing a ran-
dom candidate solutions. Every candidate solution is evaluated according to user
preferences. The best and worst candidate solutions are recorded so that the per-
formance of our scheme is enhanced. In this scheme, probabilistic approach is
employed to iteratively transform the candidate solutions. To transform the can-
didate solutions, the proposed scheme uses local or global transformation. Levy
flights are used for global transformation. The status of local or global transfor-
mation is determined randomly. After every transformation, the proposed scheme
evaluate the candidate solution. If the new solution meets consumer’s require-
ments, the population is updated. However, if the solution does not accord with
the consumers requirements, then the candidate solution is modified on the basis
of the best and worst candidate solution. A mutation policy is adopted for this
purpose. This policy to update candidate solutions ensures that the proposed
solution moves towards achieving the best solution. Moreover, this policy also
helps to reduce the number of iterations required to compute an optimal solution.
The pseudo code of the proposed scheme is given below.
Algorithm 1 Algorithm for UCM
1: Objective min(W T ), Cost
2: Input:Random solutions, termination criteria,
M aximum Iteration
3: Determine the best and worst candidate solutions in the initial population
4: Output: Optimal schedule of home appliances
5: ittr = 0, transf ormation probability
6: while ittr < M aximum Iteration do
7: for i = 1 to P opulation do
8: flag= rand
9: if f lag < p then
10: transf orm local candidate solution ;
11: else
12: transf orm global candidate solution ;
13: end if
14: end For
15: Identif y the best and worst candidate solutions
16: if Solution moves towards the best then
17: U pdate solution ;
18: else
19: M odif y solution via mutation ;
20: end if
21: if N ew solution moves towards worst solutions then
22: U pdate solution ;
23: else
24: Discard new solution and keep old solution ;
25: end if
26: end while
 On Maximizing User Comfort Using a Novel Meta-Heuristic Technique 35

6 Simulation Results and Discussion

In this section, we illustrate the effectiveness of our proposed work. Simulations
are performed for a single home with multiple appliances. The purpose of this
work is to minimize the appliances waiting time with minimum consumption
cost. The power ratings, operational hours and class of the appliances used in
this work are shown in Table 1. The proposed work was simulated using MAT-
LAB 2017a installed on Intel(R) Core(TM) i5-6430 CPU. In meta-heuristic tech-
niques, one of the most important features is randomization. To deal with it, we
have taken average results of ten experiments.
The performance of our proposed work is evaluated in terms of minimiz-
ing appliance waiting time with minimum cost. To illustrate that our proposed
scheme is able to minimize the appliances waiting time, the performance of
our scheme is compared with two well-known meta-heuristic techniques namely
FPA and JOA. According to Fig. 4, the proposed scheme effectively minimized
the waiting time as compare to FPA and JOA.

6
Waiting time (hour)

5
4
3
2
1
0
JOA FPA UCM

Fig. 4. Appliances waiting time

According to Fig. 3, time slots 11–17 are high price intervals. Therefore,
increasing the load in these hours will increase the overall cost. In this work, we
reduced the hourly electricity consumption cost in the high price intervals. This
ultimately minimized the total electricity cost. Figure 8 represents the hourly
electricity consumption cost of the home appliances. In this figure, it is clearly
seen that the proposed scheme effectively minimized the electricity consumption
cost. Time slots 14–17 are recorded as the highest price intervals in our proposed
scheme. However, it is due to the use of AD N in these time slots. These appli-
ances do not play any role in minimizing electricity consumption cost. The overall
electricity consumption cost for a single day is shown in Fig. 5. The proposed
work minimizes electricity cost by 58.02% whereas, JOA and FPA minimized
the cost by 43% and 67% as compared to unscheduled cost. However, FPA has
a higher waiting time.
Figure 7 shows the per hour load consumed by home appliances. In this work,
we minimized the load consumption during high price intervals. The load con-
sumption in the unscheduled scenario creates PAR in the high price intervals. It
36 S. Khan et al.

Fig. 5. Total cost

6
PAR

0
Unscheduled JOA FPA UCM

Fig. 6. PAR

is due to the irregular consumption pattern of unscheduled appliances. This is

the main reason of high electricity cost for unscheduled case. From this figure,
it is clear that the proposed scheme successfully shift the unscheduled load from
high to low price intervals. FPA also shows a similar pattern. However, JOA
scheduled load during these hours. For this reason, the performance of JOA in
cost consumption is not good. Shifting load from high to low price intervals
creates peak during low price intervals. As it can be seen from Fig. 7 the load
consumption in time slots 21–24 is high. This is the trade-off in our scheme.
Figure 6 represents the PAR of all the schemes. The figure shows that JOA
outperforms the proposed scheme and FPA. PAR reduction in JOA, FPA and
proposed scheme is 70%, 56% and 34% respectively. Moreover, the trade-off in
all schemes is concluded from this figure. JOA minimized PAR however, its per-
formance in waiting time and cost minimization is not good. Similarly, FPA
minimized the total cost, however, the performance in waiting time and PAR is
not good. The proposed work efficiently minimized the cost with an acceptable
cost.
 On Maximizing User Comfort Using a Novel Meta-Heuristic Technique 37

Fig. 7. Per hour load consumption of consumer

2000
Unscheduled
Per hour cost (cent)

JOA
1500 FPA
UCM

1000

500

0
1 4 8 12 16 20 24
Time (hour)

Fig. 8. Per hour cost consumption of consumer

7 Conclusion
HEMS possess the potential to solve the increasing demand of electricity. In this
paper, a novel scheme for HEMS is proposed to reduce the appliances waiting
time. The proposed scheme successfully shifted the starting time of home appli-
ances from high to low price intervals, taking into account the appliances wait-
ing time. Performance of the proposed scheme is compared with the well known
population based meta-heuristic techniques namely JOA and FPA. Experimen-
tal results demonstrate that the proposed scheme outperform JOA and FPA in
minimizing the total waiting time of home appliances. Moreover, the proposed
scheme is effective in minimizing the total electricity cost and PAR by 58% and
56% respectively as compared to unscheduled scenario. In future, we will employ
renewable energy sources at consumers end to reduce the appliance waiting time.
Moreover, this work can also be extended using various pricing schemes such as
day ahead RTP and ToU pricing schemes.

References
1. Farhangi, H.: The path of the smart grid. IEEE Power Energy Mag. 8(1) (2010)
2. Samuel, O., Javaid, S., Javaid, N., Ahmed, S., Afzal, M., Ishmanov, F.: An efficient
power scheduling in smart homes using Jaya based optimization with time-of-use
and critical peak pricing schemes. Energies 11(11), 3155 (2018)
38 S. Khan et al.

3. Economics, Frontier, and Sustainability First. Demand side response in the domes-
tic sector-a literature review of major trials. Final Report, London, August (2012)
4. Lior, N.: Sustainable energy development: the present (2009) situation and possible
paths to the future. Energy 35(10), 3976–3994 (2010)
5. Shakeri, M., Shayestegan, M., Reza, S.S., Yahya, I., Bais, B., Akhtaruzzaman, M.,
Sopian, K., Amin, N.: Implementation of a novel home energy management sys-
tem (HEMS) architecture with solar photovoltaic system as supplementary source.
Renew. Energy 125, 108–120 (2018)
6. Pilloni, V., Floris, A., Meloni, A., Atzori, L.: Smart home energy management
including renewable sources: a QoE-driven approach. IEEE Trans. Smart Grid
9(3), 2006–2018 (2018)
7. Shareef, H., Ahmed, M.S., Mohamed, A., Al Hassan, E.: Review on home energy
management system considering demand responses, smart technologies, and intel-
ligent controllers. IEEE Access 6, 24498–24509 (2018)
8. Aslam, S., Javaid, N., Khan, F.A., Alamri, A., Almogren, A., Abdul, W.: Towards
efficient energy management and power trading in a residential area via integrating
a grid-connected microgrid. Sustainability 10(4), 1245 (2018)
9. Ahmed, M.S., Mohamed, A., Homod, R.Z., Shareef, H.: Hybrid LSA-ANN based
home energy management scheduling controller for residential demand response
strategy. Energies 9(9), 716 (2016)
10. Bharathi, C., Rekha, D., Vijayakumar, V.: Genetic algorithm based demand side
management for smart grid. Wirel. Pers. Commun. 93(2), 481–502 (2017)
11. Adika, C.O., Wang, L.: Smart charging and appliance scheduling approaches to
demand side management. Int. J. Electr. Power Energy Syst. 57, 232–240 (2014)
12. Samadi, P., Wong, V.W.S., Schober, R.: Load scheduling and power trading in
systems with high penetration of renewable energy resources. IEEE Trans. Smart
Grid 7(4), 1802–1812 (2016)
13. Ahmad, A., Khan, A., Javaid, N., Hussain, H.M., Abdul, W., Almogren, A.,
Alamri, A., Azim Niaz, I.: An optimized home energy management system with
integrated renewable energy and storage resources. Energies 10(4), 549 (2017)
14. Aslam, S., Iqbal, Z., Javaid, N., Khan, Z.A., Aurangzeb, K., Haider, S.I.: Towards
efficient energy management of smart buildings exploiting heuristic optimization
with real time and critical peak pricing schemes. Energies 10(12), 2065 (2017)
15. Khan, A., Javaid, N., Khan, M.I.: Time and device based priority induced comfort
management in smart home within the consumer budget limitation. Sustainable
Cities Soc. (2018)
16. Huang, H., Cai, Y., Hang, X., Hao, Y.: A multiagent minority-game-based demand-
response management of smart buildings toward peak load reduction. IEEE Trans.
Comput.-Aided Des. Integr. Circ. Syst. 36(4), 573–585 (2017)
17. Pooranian, Z., Abawajy, J.H., Conti, M.: Scheduling distributed energy resource
operation and daily power consumption for a smart building to optimize economic
and environmental parameters. Energies 11(6), 1348 (2018)
18. Yang, X.-S.: Flower pollination algorithm for global optimization. In: International
Conference on Unconventional Computing and Natural Computation, pp. 240–249.
Springer, Heidelberg (2012)
19. Rao, R.: Jaya: a simple and new optimization algorithm for solving constrained and
unconstrained optimization problems. Int. J. Industr. Eng. Comput. 7(1), 19–34
(2016)
 Towards Efficient Energy Management
in a Smart Home Using
Updated Population

Hafiz Muhammad Faisal1 , Nadeem Javaid1(B) , Zahoor Ali Khan2 ,

Fahad Mussadaq3 , Muhammad Akhtar3 , and Raza Abid Abbasi3
1
Comsats University Islamabad, Islamabad 44000, Pakistan
[email protected]
2
Computer Information Science, Higher Colleges of Technology, Fujairah 4114, UAE
3
NCBA&E, Multan, Pakistan
http://www.njavaid.com

Abstract. Energy management using demand side management (DSM)

techniques plays a key role in smart grid (SG) domain. Smart meters and
energy management controllers are the important components of the SG.
A lot of research has been done on energy management system (EMS)
for scheduling the appliances. The aim of current research is to organize
the power of the residential units in an optimized way. Intelligent energy
optimization techniques play a vital role in reduction of the electric-
ity bill via scheduling home appliances. Through appliance’s scheduling,
consumer gets feasible cost in response to the consumed electricity. The
utility provides the facility for consumers to schedule their appliances for
the reduction of electricity bill and peak demand reduction. The utility
company is allowed to remotely shut down their appliances in emer-
gency conditions through direct load control programs. A lot of research
has been done on energy management system (EMS) for scheduling the
appliances. In this work, an efficient EMS is proposed for controlling the
load in residential units. Meta-heuristic algorithms have been used for
the optimization of the user energy consumption schedules in an efficient
way. Our proposed scheme is used to minimize the user waiting time.
User waiting time is inversely proportional to the total cost and peak to
average ratio (PAR). Simulation result shows the minimum user waiting
time, however, the total cost is compromised due to the high demand
of the load. In the end, our proposed scheme will be validated through
simulations.

Keywords: Smart grid · Home energy management system ·

Real time price

1 Introduction
Conventional energy production systems cannot fulfill the user’s need. It gives
rise to a number of different challenges that are faced by the electric power
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 39–52, 2020.
https://doi.org/10.1007/978-3-030-15032-7_4
40 H. M. Faisal et al.

industry. Approximately, 10–30% energy is wasted in the methods used to sup-

ply the electric power from the source to end consumers. As we know the power
generation and power utilization is a one-way process so, the power generation
system is unable to control and manage the electricity consumption. Tradition-
ally, power generation systems and its demand side management (DSM) methods
are centrally distributed and only focused on industrial customers. Smart grid
(SG) solve maximum problems that are present in conventional energy produc-
tion systems. The bi-directional flow of power between the electricity generation
system and the end consumer allow many factors to be controlled to play a vital
role in energy wastage. The bi-directional communication is not only concerned
by the consumers for electricity price and maintenance schedules of the distri-
bution network, however, also motivate the providers to monitor and analyze
the real time power utilization data. Distributed energy system and information
technology are involved in the SG. Using SG, communication can be handled
and monitored with the proper mechanism. Recent developments of the SG allow
load management methodologies to be implemented more efficiently by permit-
ting them to utilize new technologies. DSM and demand response (DR) are
important components of SG. Between the power grid and consumers, DR play
a vital role. DR schemes are used to reduce the electricity consumption without
affecting the user’s comfort. Two types of DR programs are considered:

• Incentive base DR
• Time based DR
Using the incentive-based DR program, utility can turn ON and OFF the con-
sumers appliances with the term notice. Whereas Time based DR requires the
consumers participation for scheduling their according to the change in price sig-
nals. DR is very helpful in reduction of the consumers electricity bills based their
participation. The core objective of using multiple DR and DSM methodologies
is to alleviate the burden of the electricity bill and energy usage. The objec-
tive of sellers is to increase their profits whereas consumers want their personal
incentives. Smart meters (SM) are installed in the residential area. The rela-
tionship between SM and DSM consist of many different and connected parts
with many features and perspectives to consider. Without regard to exceptions,
the target of SM usage is to minimize energy consumption as well as peak for-
mation without losing the consumers comfort. SM provides the user’s complete
information of demand, supply and price signal. There are some methodologies
including mixed integer linear programming (MILP), linear programming, and
mixed integer non-linear programming (MINLP), etc., which can be applied for
minimization of the energy consumption expanses while managing the efficient
schedules of electricity. DSM itself is not a technique; it is more generally a col-
lection of strategies used to change the user’s energy patterns for obtaining their
suitable power distribution. The collection of strategies (i.e. DSM) encourage
the consumers to monitor and control multiple factors (e.g. power load, appli-
ance management) of power consumption which results in lower power wastage.
This can be achieved by applying various artificial intelligence algorithms. In
 Towards Efficient Energy Management 41

DSM strategy consumers perform load management and shifting the load into
different time duration.

2 Related Work

Using MILP in [1] as a design technique, the main objective of this method
is to minimize the peak to average ratio (PAR) value and the total electricity
bill paid by consumers. The authors also focuses on balance load management
but the user comfort is not considered in MILP. There is a trade-off between
conventional systems and today’s renewable energy sources (RES). SG is a two-
way communication of utility and consumers and by saving this 10–30%, we cover
a lot of energy wastage problem. Cost minimization is a big challenge for today’s
researchers in SG. Using a genetic algorithm (GA) technique in the research
article [2], cost minimization is achieved at a low level with the integration of RES
and stored energy. When price and demand are higher than the stored energy
is helpful in specific time changes. Deployment and maintenance cost of storage
devices and RES has been ignored in this technique. Majority of researchers
have focused on residential areas only. Balancing the load in commercial and
residential areas is a big problem. However, using GA-DSM [3] algorithm in peak
hour, electricity consumption is reduced by 21% in an industry which is very
remarkable. The authors neglected the PAR value and user comfort feasibility.
The authors in [4] proposed scheme of MINLP is used to solve the cost min-
imization under the price tariff ToU. Although the cost minimization at the
peak hour is achieved, however, the author disregarded the PAR value which is
another important factor in SG. Dynamic programming [5] technique was used
for cost minimization and by scheduling the gadgets for various duration. This
will be done by the integration of RES and energy storage systems (ESS) with
SG. Residents have the capability to produce the electricity from RES. A con-
sumer can sell the additional electricity to the neighbours. The important factor
of installation and maintenance has been ignored in RES. The novel schedul-
ing model [6] of the combination of GA and binary partial swarm optimization
(BPSO) algorithms. The goal of this technique is electricity bill and PAR min-
imization. In DSM, the user can manage their home appliances by shifting the
load to another time so the load demand is a key factor in this regard. By shifting
the load and by using cuckoo search algorithm (CSA) [7] algorithm peak load
has been reduced by 22%. Balanced load curve generated by the CSA algorithm
shows the user preference for appliance usage, load shifting can then be done
by using this curve. For real-time schedule controller, new binary backtracking
search algorithm (BBSA) [8] was proposed. By using the load limit home appli-
ances are shifted from peak hour and electricity price reduced 21% per day in
comparison to PSO algorithm. Huang et al. [9] developed the two point esti-
mation method embedded with PSO method for reducing the computational
complexity in a home energy management systems (HEMS). This scheme is
intelligent enough in comparison to GPSO-LHA in the context of computational
burden. Author [9] did not consider the cost of electricity and PAR value which
42 H. M. Faisal et al.

are the important factor in HEMS. The author in [4] proposed using MINLP
technique cost minimization is achieved under the price tariff of ToU however,
PAR is not considered in this technique. Utilizing DSM, 30% power utiliza-
tion can be minimized without knowing the usage on the user side. The overall
objective of load management is to schedule a load during high demand to low
demand intervals. This can be done by the combination of the GA algorithm and
bacterial foraging (BF) in [10]. Hybrid technique [10] minimizes the electricity
bill and PAR using the load management shifting. Hybrid scheme uses the RTP
signal for reducing the electricity bills and PAR.
In HEMS, SG has a significant functionality to minimize the users’ cost using
DSM. Meta heuristic scheme is designed in [11] for the reduction of cost and PAR
value. Applying the combination of GA and CSA, we achieved the minimized
cost and PAR value as well, under the RTP price tariff, as compared to other
techniques with the desirable user waiting time. In DSM user performed pri-
orities that are set to schedule the appliances in HEMS. The authors proposed
[12] evolutionary accretive comfort algorithm (EASA) which is comprised of four
postulations. These postulations are defined according to the time and device
bases priorities. Based on the input values EASA generates optimal energy solu-
tion pattern which satisfies the user budget. The author defines three different
user budgets to find the absolute solutions. Ma, Yao, et al. [13] defines discom-
fort function for two different type of gadgets. First category is flexible starting
time and the other is flexible power devices. Authors in [13] considered a multi-
objective function for user comfort and cost parameters.
The proposed bat algorithm in [14] can be applied to obtain the optimum
result. By applying this algorithm energy consumption can be reduced which
is simply a non-linear optimization problem. The main goal of current work is
to decrease the power usage and increase consumer comfort standards in the
residential area. In Al Hasib [15], the author considered bidirectional energy
exchange between SG and small residential building. The main goal of this paper
is to maintain the balance between electricity cost and user comfort. Here the
appliances load was categorized into three categories. Based on a declining block
rates (DBR), the author proposed a comfort demand function. The authors in
[16] recommended a min max load scheduling (MMLS) algorithm used to reduce
the PAR while optimizing the operational comfort level (OCL) of user’s. It is
important to note the difficulty faced by user’s, under the control of HEMS when
reducing power consumption. For residential demand response, the author has
proposed an intelligent algorithm which analyze the effect of HEMS operations
on the use’r comfort [18].
The authors in [19], smart homes are integrated with SG to purchase/sell
electricity in peak load demand. The proposed scheme objective is to minimize
the cost and PAR along with the increase in earning profits. In this model two
optimization techniques, CSA and strawberry algorithm (SA) are used with RES:
wind turbine (WT), photovoltaic panels (PV) and ESS. The simulations results
show that the proposed scheme efficiently reduce cost and PAR with maximiz-
ing earning’s. CSA optimization technique outperforms than SA to minimizing
 Towards Efficient Energy Management 43

cost and PAR during peak load demand. Many techniques and models in [20–25]
were addressed using SG and micro SG with standalone and connected-grid with
HEMS are the emerging research areas in the last few years. In [26,27], many
authors have proposed scholastic programming models however, dynamic pro-
gramming schemes in [28,29] were proposed. These static and dynamic models
need precise tuning in their algorithm to manage the parameters and to control
them. The authors in [30] proposed DSM model, where RESs are connected.
The proposed model consist of three layers: the utility, the customer and the
DR aggregator. The role of the DR aggregator has been defined as a mediator
that communicate with both customers and the utility. The experimental out-
comes demonstrate that consumers can get profit from the proposed design: the
DR aggregator can make the profit by providing DR services; the utility can
reduce the generation cost; customers can save money on their monthly elec-
tricity bill. Evolutionary algorithms are used for load shifting in order to reduce
the cost of the customers [32]. All service sides have data sets, where schedul-
ing problem have been managed to solve the efficiency problem, the industry
faced more problem because of big power consumption appliances. Due to high
load user’s need to use the energy more intelligently in both residential and
commercial sector.

3 Problem Formulation
DSM techniques are proposed to handle the irregular consumption of electric-
ity which is the complex task to tackle. Consumers require more electricity at
certain time intervals, so, there is a possibility for peak formation and electric-
ity blackouts. In this situation, intelligent algorithms are required for EMS to
help user’s for scheduling the power from high demand intervals to low demand
intervals in an effective manner. Most of the techniques have been designed to
reduce the peak formation, electricity cost and user’s discomfort [1,33,35]. How-
ever, there is always a trade-off between PAR, electricity bill and user comfort
standards. RES integration is lacking in [1] for enhancing the comfort standards
of the residents. In [33], appliance priorities are not considered in an automatic
fashion. The study in [35] prioritizes the appliances manually; however, they need
automatic priority specifications for controlling the whole system efficiently. So,
there is need to design an EMS which can optimize the energy consumption of
the residential sector consumers efficiently. Meta-heuristic algorithms are used
for the optimization of the energy consumption schedules defined by user’s.

4 System Model
We have developed a home energy management (HEM) method for controlling
the energy consumption load and price of the smart homes. Initially, this algo-
rithm starts with a single smart home and 15 appliances in it. These appliances
are categorized into two main categories: schedulable and non-schedulable appli-
ances. Smart meter decides the operation time of the appliances according to
44 H. M. Faisal et al.

their power rating and defined pricing tariff from the utilities. The power rating
varies for each appliance and scheduling of these appliances is done in such a
way to achieve the optimum solution from the designed objective function. In
this work, two pricing tariffs are used: RTP and CPP for checking their impact
on the customers electricity bills. DR and DSM used in SG provide more sta-
bility and reliability in grid operation. The aim of this work is to reduce the
PAR, energy consumption and cost, and to enhance the consumers preferences
according the consumers standards. Main architecture of this system is visual-
ized in Fig. 1. The electricity bill is conveyed to consumers through the smart
meter. HEM controller decides which appliances should be turned on using the
defined pricing signals during the peak and off-peak hours. The core objective
of this study is to reduce the power utilization, PAR and electricity cost while
maximizing the user comfort. However, there is always a trade-off which occurs
between electricity bill and consumers preferences. Total energy utilization is
formulated using Eq. 1. Equations 2 and 3 is used to calculate the PAR and cost
respectively.


24
Load = (P R ∗ S(t)), S(t) = [1/0] (1)
t=1
P AR = (M ax(Load)/Avg(Load)) (2)

24
Cost = (P P ∗ P R ∗ S(t)), S(t) = [1/0] (3)
t=1

Fig. 1. System model

 Towards Efficient Energy Management 45

5 Optimization Algorithms
Traditional optimization algorithms, which belongs to mathematical techniques,
are not working satisfactorily if a large number of devices exists. Computa-
tionally power is also slow and time consuming. Behind this reason, we apply
heuristic schemes grey wolf optimization (GWO) and Jaya algorithm to obtain
our objectives. We proposed a JGO algorithm, which is discussed in details in
the subsection below.

5.1 GWO
GWO is a novel meta-heuristic algorithm. It consists of four types of wolves:
alpha, beta, delta and omega. There are three main phases of hunting in GWO.
Alpha is used as the most fittest solution between beta, delta and omega. List
of main steps of GWO is given.
1. Encircling the prey
2. Hunting
3. Exploitation: It is also called attacking the grey
4. Exploration: It is also known as search for prey.

5.2 Jaya Algorithm

Suppose f(x) is the target function to be minimized or maximized. Assume that
there are “m” number of design variable and “n” number of candidate solutions,
at any iteration i, where k = 1, 2, 3, 4. . ...n. Let the best value of f(x) (i.e.
f(x)best) is obtained by the best candidate names as “best” and vice versa for
the worst candidate (i.e. f(x)worst). If the value for jth variable during the ith
iteration for kth candidate is Xj,k,i then this value is modified as per the following
equation given below:
´ = Xj,k,i + r1,j,i (Xj , best, i..........)
Xj,k,i (4)
Where the value of j variable, for the “best” candidate is Xj, best, i and vice
versa for the worst candidate, where the updated value of Xj,k,i and r1,j,i and
r2,j,i which are the two random number for the jth variable is X’j,k,i in the range
of [0,1].

5.3 Updated Population

In this section, we described our proposed scheme. The population update is
performed in GWO, the updating is totally dependent on the placement of the
primary three accurate candidates. So in GWO, we tend to note initial 3 good
solutions, oblige the other wolves or search agents to change and update their
locations on the idea of the placement of the most effective search agent. Here,
Eq. 5 is used to find the effective search agent location.
X1 + X2 + X3
X(t + 1) = (5)
3
46 H. M. Faisal et al.

Hence, we are able to say that Jaya population update strategy is good as
compared to the GWO as a result of here, our aim is to search out the more
effective result as possible and also the a lot of optimized result may be possible
by having a large random and various population. Where initialization of GWO
is better than Jaya initialization. So, we selected Jaya based population update
strategy and GWO based initialization strategy, so that, we proposed a new
proposed algorithm.

6 Pseudo Code of the Proposed Scheme

1. Generate initial search agents

Gi (i = 1, 2, ...., n) (6)

2. Initialize the vector’s

3. Calculate the fitness value
4. Iter = 1
5. repeat
6. for i = 1:
7. end for
8. Calculate the fitness value of all solutions
9. Update the value
10. Update the vectors
11. Iter = Iter+1
12. until Iter ≥ maximum number of iterations
13. output best solution.

7 Simulation and Reasoning

In this section, we demonstrate simulation results and evaluate the performance
of the proposed algorithm. The load, cost and user’s waiting time for each appli-
ance are represented in terms of hours, cents and kWh. By applying RTP signal
in a smart home, we achieve maximum user comfort time, however, at some level
cost and PAR values are maximized. There is always a trade-off between PAR,
cost and user comfort. In RTP price signal tariff, electricity price varies during
different time slots of a single day. Figure 2 shows the complete details of price
per hour in 24 h. In afternoon, the price rate is two times higher. Per hour cost is
increased due to unbalancing of the price at peak time. This is shown in Fig. 3.
Figure 4 shows the hourly based energy consumption pattern in both scheduled
and unscheduled scenarios.
Figure 5 represents the PAR value of GWO, Jaya and proposed algorithm. It
demonstrates that proposed technique result is better than the GWO algorithm,
however, Jaya algorithm is better than our proposed algorithm. PAR value of
proposed algorithm is 50% less than the GWO so as compared to GWO we
achieves PAR goal in a smart home, however; Jaya technique is suitable for the
 Towards Efficient Energy Management 47

28
RTP
26

24

22
Price (cent/kWh)

20

18

16

14

12

10

8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Time (hour)

Fig. 2. Total cost

250
Unscheduled
GWO
JGO
Jaya
200
Per hour cost (cent)

150

100

50

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Time (hour)

Fig. 3. Per hour cost

reduction of PAR in RTP signal. Figure 6 shows the total cost of GWO, Jaya and
proposed algorithm. It is shown that the overall cost of the algorithm is higher
as compared to GWO and Jaya. This effects the overall cost per day in the RTP
signal. Reduction of electricity cost is the core objective of DSM in SG. With
the comparison of Jaya and GWO, cost of our proposed technique is higher due
to higher user demand. Figure 6 shows the total cost of three different schemes
in terms of cents. Complete load details of unscheduled load, Jaya algorithm,
GWO algorithm and proposed algorithm are shown in Fig. 7. It can be observed
that all values are approximately equal. Graphical representation of the load is
useful for calculating the overall cost in the RTP signal.
The user waiting time (hour) is shown in Fig. 8. The consumer satisfaction
level is measured in terms of waiting time. In our proposed scheme, user’s satis-
48 H. M. Faisal et al.

12
Unscheduled
GWO
JGO
10 Jaya
Per hour load (kWh)

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Time (hour)

Fig. 4. Per hour load

8

5
PAR

0
Unscheduled GWO JGO Jaya

Fig. 5. PAR

1600

1400

1200
Total cost (cent)

1000

800

600

400

200

0
Unscheduled GWO JGO Jaya

Fig. 6. Total cost

 Towards Efficient Energy Management 49

120

100

80
Total load (kWh)

60

40

20

0
Unscheduled GWO JGO Jaya

Fig. 7. Total load

5
Waiting time (hour)

0
GWO JGO Jaya

Fig. 8. User waiting time

faction is the time limit a user waits for a particular appliance to turn on. So user
waiting time is inversely propositional to cost and PAR. We achieved minimal
user waiting time for high load and cost in DSM. Our proposed scheme achieved
minimum user waiting time however, cost and PAR values are compromised.
Proposed scheme of user waiting time is low as compared to GWO and Jaya.
While, increasing the cost and PAR, the waiting time increases, which is shown
in Fig. 8. Our proposed scheme tries to achieve the maximum trade off between
user waiting time and cost.

8 Conclusion
In this paper, we evaluate a load management problem in a smart home for
different electrical appliances. These appliances are scheduled using the meta
heuristics techniques according to their consumption pattern. We evaluate two
50 H. M. Faisal et al.

meta heuristic algorithms performance on the parameters of cost, PAR and user
comfort. Simulation result shows the tradeoff of cost and user waiting time.
Results show proposed updated population scheme is effective as compare to
GWO and Jaya in term of waiting time. In future, we will combine RES into SG
for PAR reduction.

References
1. Bradac, Z., Kaczmarczyk, V., Fiedler, P.: Optimal scheduling of domestic appli-
ances via MILP. Energies 8(1), 217–232 (2014)
2. Mary, G.A., Rajarajeswari, R.: Smart grid cost optimization using genetic algo-
rithm. Int. J. Res. Eng. Technol. 3(07), 282–287 (2014)
3. Bharathi, C., Rekha, D., Vijayakumar, V.: Genetic algorithm based demand side
management for smart grid. Wireless Pers. Commun. 93(2), 481–502 (2017)
4. Setlhaolo, D., Xia, X., Zhang, J.: Optimal scheduling of household appliances for
demand response. Electr. Power Syst. Res. 116, 24–28 (2014)
5. Samadi, P., Wong, V.W.S., Schober, R.: Load scheduling and power trading in
systems with high penetration of renewable energy resources. IEEE Trans. Smart
Grid 7(4), 1802–1812 (2016)
6. Ullah, I., Javaid, N., Khan, Z.A., Qasim, U., Khan, Z.A., Mehmood, S.A.: An
incentive-based optimal energy consumption scheduling algorithm for residential
users. Procedia Comput. Sci. 52, 851–857 (2015)
7. Cakmak, R., Altas, I.H.: Scheduling of domestic shiftable loads via Cuckoo Search
optimization algorithm. In: 2016 4th International Istanbul Smart Grid Congress
and Fair (ICSG), pp. 1–4. IEEE, April 2016
8. Ahmed, M.S., Mohamed, A., Khatib, T., Shareef, H., Homod, R.Z., Ali, J.A.: Real
time optimal schedule controller for home energy management system using new
binary backtracking search algorithm. Eng. Build. 138, 215–227 (2017)
9. Huang, Y., Wang, L., Guo, W., Kang, Q., Wu, Q.: Chance constrained optimization
in a home energy management system. IEEE Trans. Smart Grid (2016). https://
doi.org/10.1109/TSG.2016.2550031
10. Khalid, A., Javaid, N., Mateen, A., Khalid, B., Khan, Z.A., Qasim, U.: Demand side
management using hybrid bacterial foraging and genetic algorithm optimization
techniques. In: 2016 10th International Conference on Complex, Intelligent, and
Software Intensive Systems (CISIS), pp. 494–502. IEEE (2016)
11. Aslam, S., Javaid, N., Khalid, A.: An efficient home energy management scheme
using cuckoo search. In: International Conference on P2P, Parallel, Grid, Cloud
and Internet Computing 3PGCIC 2017: Advances on P2P, Parallel, Grid, Cloud
and Internet Computing, pp 167–178 (2017)
12. Khan, A., Javaid, N., Khan, M.I.: Time and device based priority induced comfort
management in smart home within the consumer budget limitation
13. Ma, K., Yao, T., Yang, J., Guan, X.: Residential power scheduling for demand
response in smart grid. Int. J. Electr. Power Energy Syst. 78, 320–325 (2016)
14. Mohan, N., Sivaraj, R., Pyira, R.D.: A comprehensive reviews of bat algorithm and
its application to various optimization problems. Asian J. Res. Soc. Sci. Humanti.
6, 676–690 (2016)
15. Al Hasib, A. Nikitin, N., & Natvig, L. (2015) Cost comfort balancing in a smart
Residential building with bidirectional energy trading. Sustainable interent and
ICT for sustainability ( SustainIT), 2015 (pp, 1-6)
 Towards Efficient Energy Management 51

16. Chen, Y., Lin, R.P., Wang, C., Groot, M.D., Zeng, Z.: Consumer operational com-
fort level based power demand management in the smart grid. In: Proceedings of
the 3rd IEEE PES Innovative Smart Grid Technologies Europe (ISGT Europe),
Berlin, Germany, 14–17 October 2012, pp. 1–6 (2012)
17. Hassan, N.U., Khalid, Y.I., Yuen, C., Huang, S., Pasha, M.A., Wood, K.L., Kerk,
S.G.: Framework for minimum user participation rate determination to achieve
specific demand response management objectives in the residential smart grids.
Int. J. Electri. Power Energy Syst. 74, 91–103 (2016)
18. Kuzlu, M.: Score-based intelligent home energy management (HEM) algorithm for
demand response applications and impact of HEM operation on customer comfort.
IET Gener. Transm. Distrib. 9, 627–635 (2015)
19. Aslam, S., Javaid, N., Khan, F.A., Alamri, A., Almogren, A., Abdul, W.: Towards
efficient energy management and power trading in a residential area via integrating
a grid-connected microgrid
20. Abhinav, S., Schizas, I.D., Ferrese, F., Davoudi, A.: Optimization-based AC micro-
grid synchronization. IEEE Trans. Ind. Inform. 13, 2339–2349 (2017)
21. Dehghanpour, K., Nehrir, H.: Real-time multiobjective microgrid power manage-
ment using distributed optimization in an agent-based bargaining framework. IEEE
Trans. Smart Grid 2017. 2708686
22. Liu, J., Chen, H., Zhang, W., Yurkovich, B., Rizzoni, G.: Energy management
problems under uncertainties for grid-connected microgrids: a chance constrained
programming approach. IEEE Trans. Smart Grid 8, 2585–2596 (2017)
23. Luo, Z., Wu, Z., Li, Z., Cai, H., Li, B., Gu, W.: A two-stage optimization and
control for CCHP microgrid energy management. Appl. Therm. Eng. 125, 513–
522 (2017)
24. Moga, D., Petreus, D., Muresan, V., Stroia, N., Cosovici, G.: Optimal generation
scheduling in Islanded microgrids. IFAC-PapersOnLine 49, 135–139 (2016)
25. Chen, J., Chen, J.: Stability analysis and parameters optimization of islanded
microgrid with both ideal and dynamic constant power loads. IEEE Trans. Ind.
Electron. 65, 3263–3274 (2017)
26. Su, W., Wang, J., Roh, J.: Stochastic energy scheduling in microgrids with inter-
mittent renewable energy resources. IEEE Trans. Smart Grid 5, 1876–1883 (2014)
27. Shuai, H., Fang, J., Ai, X., Tang, Y., Wen, J., He, H.: Stochastic optimization
of economic dispatch for microgrid based on approximate dynamic programming.
IEEE Trans. Smart Grid 2018 (2018). 2798039
28. Razmara, M., Bharati, G.R., Shahbakhti, M., Paudyal, S., Robinett III., R.D.:
Bilevel optimization framework for smart building-to-grid systems. IEEE Trans.
Smart Grid 9, 582–593 (2016)
29. Esfahani, M.M., Hariri, A., Mohammed, O.A.: A multiagent-based game-theoretic
and optimization approach for market operation of multi-microgrid systems. IEEE
Trans. Ind. Inform. (2018)
30. Li, D., Sun, H., Chiu, W.-Y., Vincent Poor, H.: Multiobjective Optimization for
Demand Side Management in Smart Grid
31. Macedo, M.N.Q., Galo, J.J.M., De Almeida, L.A.L., Lima, A.D.C.: Demand side
management using artificial neural networks in a Smart Grid environment. Renew.
Sustain. Energy Rev. 41, 128–133 (2015)
32. Logenthiran, T., Srinivasan, D., Shun, T.Z.: Demand side management in smart
grid using heuristic optimization. IEEE Trans. Smart Grid 3(3), 1244–1252 (2012)
33. Elham, S., Shahram, J.: Optimal residential appliance scheduling under dynamic
pricing scheme via HEMDAS. Energy Build. 93, 40–49 (2015)
52 H. M. Faisal et al.

34. Adika, C.O., Wang, L.: Smart charging and appliance scheduling approaches to
demand side management. Int. J. Elect. Power Energy Syst. 57, 232240 (2014)
35. Jovanovic, R., Bousselham, A., Bayram, I.S.: Residential demand response schedul-
ing with consideration of consumer preferences. Appl. Sci. 6(1), 1–16 (2016)
36. Shirazi, E., Zakariazadeh, A., Jadid, S.: Optimal joint scheduling of electrical and
thermal appliances in a smart home environment. Energy Convers. Manage. 106,
181–193 (2015)
 A New Memory Updation Heuristic
Scheme for Energy Management System
in Smart Grid

Waleed Ahmad1 , Nadeem Javaid1(B) , Sajjad Khan1 , Maria Zuraiz2 ,

Tayyab Awan3 , Muhammad Amir1 , and Raza Abid Abbasi1
1
COMSATS University, Islamabad 44000, Pakistan
[email protected]
2
COMSATS University Islamabad, Abbotabad Campus, Abbotabad, Pakistan
3
National University of Science and Technology, Islamabad, Pakistan
http://www.njavaid.com

Abstract. In the last decade, high energy demand is observed due to

increase in population. Due to high demand of energy, numerous chal-
lenges in the existing power systems are raised i.e., robustness, stability
and sustainability. This work is focused for the residential sector Energy
Management System (EMS), especially for the smart homes. An EMS
is proposed which shifts the electricity load from high price to low price
hours. To fulfill the high load demand of electricity consumers, we have
proposed a new Memory Updation Heuristic Scheme (MUHS), which
efficiently schedule the appliance from on peak to off peak hours. The
objective of our new scheme MUHS is to automate the EMS. The sig-
nificance of our new proposed MUHS scheme shown the efficiency by
reducing Cost, Peak to Average Ratio (PAR) and increase User Comfort
(UC) by balancing the load demand in peak times.

Keywords: Smart homes · Demand side management ·

Energy Management System · Appliance scheduling ·
Critical Peak Pricing

1 Introduction
In the last couple of years, high energy consumption in residential areas [1], has
been observed due to an increase in household electric appliances. The energy
demand has increased upto 27% due to the increase in high power consumption
appliances, for example: appliances used for cooling and heating systems. These
appliances are the main reason of energy shortage. The traditional utilities have
insufficient resources to fulfill the today’s load demand. Therefore, more power
generation is required to provide comfort to the users. The Home Energy Man-
agement System (HEMS) is adopted along with Demand Response (DR) to the
cope with peak load by scheduling the high-power electric appliances by shifting

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 53–66, 2020.
https://doi.org/10.1007/978-3-030-15032-7_5
54 W. Ahmad et al.

the On-peak hours power load to the Off-peak hours. In this regards, the inter-
mittent energy resources brought into consideration in [2]. Demand Side Energy
Management System (DSEM) in [3] is proposed with Direct Load Controlling
(DLC) integrated with Dynamic Pricing (DP) model.
The power supply companies started to manage the load centrally by schedul-
ing the smart appliances to reduce electricity bill and Peak to Average Ratio
(PAR) in congested time intervals. The conventional grid system is unstable
due to lack of control and unidirectional communications. To achieve these chal-
lenges and remove the flaws of the conventional system, Smart Grid (SG) in
[4] has emerged which is comprised of bidirectional communication, fault toler-
ance, and real time self support recovery. SG is facilitated with Advance Metering
Infrastructure (AMI) that helps in bidirectional communication between the end
users and service provider. SG has two important key components: DSM and DR.
With the help of DSM, the utility companies are providing consumers’ aware-
ness programs to use the electricity wisely. In DR schemes, the utilities motivate
the consumers to actively engage with the SG to minimize the load demand and
electricity bill in the pre-defined budget. The consumers have reduced the energy
cost through DR schemes efficiently [5,6].

2 Related Work

In the last few years, the researchers have made a tremendous work in SG to
decrease the energy cost for the consumers using different optimization tech-
niques. To balance the load and reduce the energy cost with the increasing
demand are fascinating and tough research problems, that have been addressed
by the experts along with their efforts for solving it. Several DSM methods have
been shown previously to minimize the users electricity bill and waiting time,
while maximizing individuals comfort. In [7], an Integer Linear based Program-
ming (ILP) scheme was proposed, which is used to control the energy supply in
the residential area. This proposed technique successfully achieved the desired
goal by shifting the high power appliances load to the OFF-peak hours. In [8],
Genetic Algorithm (GA) was proposed. The aim of the proposed algorithm was
to minimize the price and load demand equilibrium; both in the demand and
supply side. In [9], the authors presented heuristic techniques, Particle Swarm
Optimization (PSO) with GA. This work shows the reduction in peak power
demand and decreased the energy cost. This technique is employed three price
tariff plans: Time Of Use (ToU), Real Time Pricing (RTP) and Critical Peak
Pricing (CPP) by using Multi Knapsack Problem (MKP). Their proposed opti-
mization scheme efficiently achieved the goal by lesson the energy cost and PAR.
Simulations also revealed that GA is more efficient than PSO. In [10], Multiple
Integer Linear Programming (MILP) is used with HEMS to tackle the load
demand using Renewable Energy Resources (RES), i.e., electric mobiles, and
energy storage systems. In [11], the authors present a model for cost reduction
in residential sector by using Mixed Integer Non-Linear Programming (MINLP)
by including ToU pricing model. This proposed method minimizes the cost of
 A New Memory Updation Heuristic Scheme for EMS in Smart Grid 55

Table 1. Related work schemes comparison

Technique(s) Achievement(s) Limitation(s)

ILP The proposed scheme achieved cost UC is neglected
reduction
PSO GA and This model reduces cost and PAR UC is ignored
MKP
GA Simulation results show decrements UC is ignored and simulation
of cost and PAR performed on residential area
only
DP Minimization of cost Authors did not mention
equipments cost and
maintenance
MILP and Proposed scheme successfully Neglected the cost factor
heuristic manages the load equilibrium
algorithm
MILP This technique efficiently Par while UC is ignored
minimized the cost
GA-DSM Model shows reduction in Authors did not mention
electricity consumption PAR and UC
MINLP The proposed scheme accomplished PAR is not considered in
desired results by reducing the cost proposed work
GA and BPSO This approach shows minimization UC is neglected and focused
in the cost and PAR only on smart home
residential areas
CSA The proposed scheme efficiently PAR and UC did not
schedules the shift able appliances presented in their research
while considering users load work
demand

energy by the total of 25%. However, PAR is not evaluated in their research
work. GA along with Binary PSO algorithm is proposed in [12].
This optimization scheme reduce the user’s electricity bill and PAR. Simu-
lations show effectiveness of the described model. In [13], the authors propose
knapsack bounds scheme to lower the peak load and minimize the rebound peaks
of the different ON-peak time intervals. In [14], HEMS is used with DR which
offers strategies to optimize the home appliances operations. The scheduling of
appliances is a challenging task and needs the optimal solutions. HEMS focuses
on UC as well as cost minimization; however, these two factors have trade off,
i.e., in order to reduce cost, User Comfort (UC) has to be sacrificed. In lit-
erature, it is often observed that HEMS works as a single objective or multi-
objective models. In this paper, the single objective model is used to decrease
the cost. This problem is solved by Gradient Base Repair Optimization (GBRO)
technique. A new model, an Evolutionary Accretive Comfort Algorithm (EACA)
56 W. Ahmad et al.

[13], has been introduced which depends on postulations that permits the time
and device based priorities. Evaluation is based on input data which consist
of devices power rating, ToU, and absolute UC. The EACA has the capability
to develop the electricity usage pattern on pre-determined consumers budget.
The results from the experiment reveal that the EACA optimization technique
efficiently reduces the cost and increases UC. In [15], a Jaya-based algorithm
is proposed, which evaluates the power levels of distributed electricity sources,
minimize the actual power drop and reduces the production expenses. Simula-
tion results are compared with other optimization techniques like: Strawberry
Algorithm (SA) and Enhanced Differential Evolution (EDE).

3 Problem Statement
In residential sector, DSM has many challenges like devices scheduling, irregular
usage of energy, consumers negligence for the defined schedules and infeasible
solutions for increasing the demands. To stabilize the consumers demands, many
DSM techniques are brought into consideration from the last few years. The
irregular nature of residential side demand causes many problems on supply
side management. Heuristic schemes are considered as one of the cost-effective
solutions. More generation of electricity is required which in turn raises the
production cost. For this purpose, an optimized system is required to schedule
smart devices in the residential side to schedule from ON-peak hours to OFF-
peak hours with less waiting time. In [16], the purpose of appliance scheduling is:
load balancing, and cost and PAR reduction. The above discussion clearly reveals
that price and PAR can be reduced by applying some optimization techniques
as presented in [17]. In this scheme, an EMS is designed for the efficient energy
management scheduling in peaks. Energy optimization of the heating and cooling
systems is also considered in one of the most important objectives of this work
because they consume a large portion of energy (for example, 64% of the total
residential energy in Canada).

4 System Model
Many researchers struggled on DSM and Supply Side Management (SSM). The
main focus of our research is on DSM in a domestic area. Every smart home
in the domestic area has Smart electrical appliances and Smart Meters (SMs)
which are connected to the EMC. SMs performs bidirectional communication
between Smart homes and Utility. EMC receive all the information from smart
devices and sensors inside a smart house by using a home area network. EMC
evaluates the data received from smart devices and calculates the total average
of the load. This required load demand after evaluation transfer to the utility
through SMs. After that, the utility satisfies the consumer demand and transfer
pricing scheme of electricity via SMs to the consumer. SMs then assign all these
pricing information and energy to EMC. So, EMC has now load demand and
power rating of smart devices and pricing tariffs collected from the utility. Now,
 A New Memory Updation Heuristic Scheme for EMS in Smart Grid 57

Fig. 1. Proposed system model

the smart appliances are scheduled by EMC, based on load demand and pricing
schemes received from smart homes and utility. EMC performs the scheduling
function based on algorithms installed in it. In our proposed model, we have
created a new scheme MUHS based EMC. Alongside we also created Grey Wolf
Optimization (GWO) based EMC and Crow Search Algorithm (CSA) based
EMC. We have compared the efficiency of all three schemes and shown that
our proposed Scheme successfully achieved the desired goal i.e., minimize Peak
Average Ration (PAR) and Maximize user comfort. All the bidirectional traffic
between EMC to SMs and SMs to the utility is performed by Wi-Fi routers via
wide area network. Devices with their similar type, energy rating and Length of
Operational Time (LOT) are given in Table 2.
In our model, we have deemed 15 smart appliances from a smart home. With
the opinion associated with electricity payment, we used CPP value signal per-
taining to payment calculation. In this paper, we break 1-day time into 24-h
time intervals. The actual cause of this specific action could be that, the waiting
time of some smart appliances are usually equal to 1 h schedule. We have parti-
tioned smart home appliances into 24-h schedule. A lot of the home appliances
deemed within this paper are usually working continually for an hour. Due to
this, we avoid the actual wastage associated with time and acquire 24-h time
schedule. In all earlier mentioned information, the actual electricity repayment
and also delay time from the smart appliances are usually decreased. With that,
our model is more cost effective and robust. Our goal of this activity is to lower
the power consumption throughout peak time slots to diminish total power cost
and to reduce waiting time (Fig. 1).
58 W. Ahmad et al.

Now, mathematically formulation of afore named goals, we have mathemat-

ical equations here. Total power utilization is expressed applying Eq. 1.


24
Load = p ∗ S(t), S(t) = [1/0] (1)
t=1

PAR is defined in Eq. 2 and total expense for 24-h is figured in Eq. 3.

P AR = (M ax(Load)/Avg(Load)) (2)


24
Cost = ()EP ∗ p ∗ S(t)), S(t) = [1/0] (3)
t=1

The Eq. 1 shows power rating of devices, and S(t) represent the ON and OFF
status of the devices, where 1 indicates that appliance is ON at a particular
time interval and 0 indicates a device is OFF. In Eq. 2, PAR is measured as the
maximum load from 24-h power utilization form and then split it by the average
of the same 24 h energy utilization pattern. The EP in Eq. 3 shows the energy
price for a 24-h time slot.
During this paper, the actual smart appliances presumed are usually divided
throughout a few distinct different types like (Delay-Tolerant, Delay-Intolerant
and Delay-Intolerant with Flexible Load.

4.1 Appliances Classification

4.1.1 Delay-Tolerant Appliances

Appliances that can be interrupt any time in 24-h time slot are delay-tolerant
shown in Table 1. Shifting the appliance time slot from On-peak hour to Off-peak
hour can decrease cost, PAR while UC may be sacrificed.

Table 2. Delay-tolerant appliances

Type Appliances
AC 1.5
Refrigerator 1.66
Iron 1
Vacuum cleaner 0.7
Water Heater 5
Dishwasher 1.32
Water Pump 1
 A New Memory Updation Heuristic Scheme for EMS in Smart Grid 59

4.1.2 Delay-Intolerant Appliances

Delay-Intolerant is those electrical smart devices that cannot be suspended while
they are in running condition. These kind of appliances can be shift from one
time interval to another time interval before their working time.

4.1.3 Delay-Intolerant Base Appliances

Base appliances are those electrical devices which cannot be stopped while they
start working. These kind of appliances depends upon consumer demand.

4.2 Price Tariff

Several types of pricing schemes have been implemented around the globe,
Through the help of these pricing tariffs, the consumer can schedule their appli-
ances from On-Peak to Off-peak time intervals. In our proposed scheme, We
have implemented CPP pricing scheme and achieved our objective by reducing
PAR, and maximize the UC by reducing in their delay time. The main focus is
to balance the energy consumption during peak price times to decrease the cost.
However, cost and waiting time always have a trade-off between them (Table 3).

Table 3. Appliances used in simulations

Appliances Power rating (kwh) LOT (h)

Refrigerator 1.666 24
Vacuum Cleaner 0.7 0.7
Waterpump 1 8
Washing Machine 1.4 3
Cloth dryer 5 4
Dishwasher 1.32 3
Water Heater 5 8
Iron 2.4 3
AC 1.5 8
Cooker 0.225 4
Toaster 0.8 1
Printer 0.011 2
Light 0.18 12
Blender 0.3 2
Oven 2.4 4
60 W. Ahmad et al.

5 Meta-heuristic Algorithms
Main-stream optimization methods are mostly numerical algorithms like, LP,
Convex Programming (CP) and ILP, etc. are not doing properly for large quan-
tity of devices due to their slow computation. Therefore, we go for meta-heuristic
methods for wise home appliances scheduling to accomplish our major aim that
will be energy cost minimization and minimize delay time. We choose meta-
heuristic rather than heuristic and mathematical practices since these are rela-
tively efficient than mathematical practices while working with a large number of
devices and these are the problem independent, whereas heuristic practices are
issue dependent, this means heuristic practices are made to resolve a particular
problem. The selected meta-heuristic calculations are mentioned in more detail
in the next sections.

5.1 CSA

A new meta-heuristic optimizer in CSA [18] is proposed by Alireza Askarzadeh

in 2016. It is inspired by the smart behavior of storing excess food by crows.
CSA is a meta-heuristic population centered algorithm which operates on the
basis of the indisputable fact that crows collect their additional food in hiding
locations for later use. Taking a food stock which is concealed by a crow is
a difficult task since if crow note that another crow is following it, the crow
fools another crow by setting its path and planning to a different location as
opposed to the actual food resource concealed by it. From relating them with
the optimization method, the crows would be the searchers, the surroundings
could be the research space, the location of the surroundings can relate with the
possible answer, fitness function is evaluated based on food source quality. In
whole search space the best food origin is considered as Best solution.

5.2 GWO

GWO [19], Meta-Heuristic algorithm is a new hunting technique of grey wolves

inspired by the leadership chain. There are four types of wolves incorporated in
method used by the leadership chain of grey wolves. Furthermore we find there
are three main hunting phases which are, searching for the prey, attacking over
the prey and encircling the prey. In compliance with the social hierarchy of grey
wolves, fittest solution to be practiced is the alpha, afterwards the beta, delta
and omega comes accordingly.

5.3 MUHS

In this section, we define our proposed Meta-heuristic algorithm MUHS. In the

MUHS, we have used initialization parameters and generate random population.
Now, in MUHS, population updation is completely based on fittest solutions. On
the basis of these best solutions, we have updated and modify the search agents
 A New Memory Updation Heuristic Scheme for EMS in Smart Grid 61

Algorithm 1. Algorithm for MUHS

1: Start
2: Initialize all parameters Max iter: (N P, D, A)
3: Randomly generate initial population Xi(i = 1, 2, ..., n)
4: X(i, j) = rand(N P, D);
5: ittr = 0, transf ormation probability
6: while ittr < M aximum Iteration do
7: for i = 24 to P opulation do
8: Let Y is a random position of search space
9: Evaluate the position of the A
10: Initialize the memory of each A
11: if x(i, :) ≥ l&x(i, :) ≤ u thenx(i, :) = x(i, :)
12: transf orm local candidate solution ;
13: else
14: transf orm global candidate solution ;
15: end if
16: end For
17: Identif y the best and worst candidate solutions
18: if Solution moves towards the best then
19: U pdate solution ;
20: else
21: M odif y solution via mutation ;
22: end if
23: if N ew solution moves towards worst solutions then
24: U pdate solution ;
25: else
26: Discard new solution and keep old solution ;
27: end if
28: end while

according to environments. The storage or population operation is performed by

utilizing the comparison of the fitness of every single option of the population
with the fitness of storage positions, and then, when the condition holds true,
the storage of the population is modified by the best option which is originated
in the last result of main positions in MUHS. Therefore, we choose the updation
with its position, review and update memory by finding the best fittest criterion.
Our goal is to find more optimized solution by having large number of random
data to achieve best results. Following are the 5 steps.

1. Initialization of the population and parameters

2. Probability checking of the new environment
3. Encompassing the solution
4. Review and update memory
5. Finding the best fitness criteria.
62 W. Ahmad et al.

Fig. 2. CPP pricing scheme

Fig. 3. Per hour load

Fig. 4. PAR
 A New Memory Updation Heuristic Scheme for EMS in Smart Grid 63

Fig. 5. Hour cost

6 Simulation and Reasoning

As this part of our discussion depict the results of our simulation in detail.
By incorporating the simulations, algorithms are ratify on the bases of PAR,
delay time, energy consumption and energy cost. We have incorporated 15 sin-
gle home appliances, for the simulation. For calculation of electricity bills we
have used CPP pricing method. We have shown CPP pricing signal in Fig. 2.
For each unscheduled slot Fig. 3 shows the consumption pattern as well as sched-
uled scenarios. The proposed MUHS technique results indicates that scheduling
of power in better way. The overall energy consumption of MUHS is less than
other techniques to avoid the peak creation at any slot of the day. It is proposed
that MUHS optimization technique is better in shifting the load from peak time
intervals to off peak interval time. User satisfaction is affected by the load shift-
ing while it gives benefits to the user as in cost reduction. DSM just not only
limits its usefulness to consumers but also beneficial for utility as well. Utility is
assisted by the decrease in PAR to keep is stability which leads to cost reduc-
tion. The performance of different algorithms optimization in the form of PAR
is shown in Fig. 4 which clearly shows the proposed MUHS outperformed the
other techniques in terms of PAR. In Fig. 5, CSA, GWO and MUHS per hour
cost is shown. Plots indicates the peak hour reduction in scheduling scheme than
un-scheduling by shifting the load from peak price intervals to low price intervals.
Total cost production ratio of scheduled and unscheduled is presented in
Fig. 6, which clearly indicate the efficiency of our optimized algorithms in total
cost. However CSA and GWO beats our proposed scheme in term of cost because
our main goal was to achieve PAR and waiting time. It is worth saying that our
proposed scheme successfully beats the unscheduled case very efficiently. The
consumer satisfaction and delay time is measured in waiting time shown in Fig. 7.
The UC is assumed as, the particular user waiting time in which user turn on the
appliances. Therefore we can say that UC is opposite to delay time. The plots in
figure clearly indicate that proposed scheme efficiently reduce the overall cost and
waiting time, so it is worth mentioning that our proposed scheme outperformed
other described schemes in term of waiting time. Figure 8 represents the total
64 W. Ahmad et al.

Fig. 6. Cost

Fig. 7. Waiting time

Fig. 8. Total load

power load of scheduling and unscheduled appliances which clearly indicates the
total load is balance in all cases.

7 Conclusion
In our propose work, we have deployed DSM with EMC to manage the load
demand by shifting the smart devices from On-peak to Off-peak hours. We have
considered two heuristic schemes i.e., GWO and CSA. We proposed a new heuris-
tic scheme MUHA, which efficiently lower the PAR with considerable amount of
 A New Memory Updation Heuristic Scheme for EMS in Smart Grid 65

cost and also increase UC as compared to GWO and CSA. Appliance classifica-
tion and power rating helps EMC to control and schedule the appliance according
to user satisfaction. However, from the simulation results, performance is evalu-
ated and shows that our new scheme MUHA outperformed than GWO and CSA
in term of PAR and delay-time.

References
1. Cena, G., Valenzano, A., Vitturi, S.: Hybrid wired/wireless networks for real-time
communications. IEEE Ind. Electron. Mag. 2(1), 8–20 (2008)
2. Ganji Tanha, M.: Security constrained unit commitment reserve determination in
joint energy and ancillary services auction (Doctoral dissertation)
3. Strbac, G.: Demand side management: benefits and challenges. Energy Policy
36(12), 4419–4426 (2008)
4. Gellings, C.W.: The concept of demand-side management for electric utilities. Proc.
IEEE 73(10), 1468–1470 (1985)
5. Chaabene, M., Ben Ammar, M., Elhajjaji, A.: Fuzzy approach for optimal energy
management of a domestic photovoltaic panel. Appl. Energy 84(10), 992–1001
(2007)
6. Pradhan, V., Balijepalli, V.M., Khaparde, S.A.: An effective model for demand
response management systems of residential electricity consumers. IEEE Syst. J.
10(2), 434–445 (2016)
7. Khan, M.A., Javaid, N., Mahmood, A., Khan, Z.A., Alrajeh, N.: A generic demand
side management model for smart grid. Int. J. Energy Res. 39(7), 954–964 (2015)
8. Samadi, P., Wong, V.W.S., Schober, R.: Load scheduling and power trading in
systems with high penetration of renewable energy resources. IEEE Trans. Smart
Grid 7(4), 1802–1812 (2016)
9. Agnetis, A., de Pascale, G., Detti, P., Vicino, A.: Load scheduling for household
energy consumption optimization. IEEE Trans. Smart Grid 4(4), 2364–2373 (2013)
10. Bradac, Z., Kaczmarczyk, V., Fiedler, P.: Optimal scheduling of domestic appli-
ances via MILP. Energies 8(1), 217–232 (2014)
11. Ullah, I., Javaid, N., Khan, Z.A., Qasim, U., Khan, Z.A., Mehmood, S.A.: An
incentive based optimal energy consumption scheduling algorithm for residential
users. Procedia Comput. Sci. 52, 851–857 (2015)
12. Yalcintas, M., Hagen, W.T., Kaya, A.: An analysis of load reduction and load
shifting techniques in commercial and industrial buildings under dynamic electric-
ity pricing schedules. Energy Build. 88, 15–24 (2015)
13. Khan, A., Javaid, N., Khan, M.I.: Time and device based priority induced comfort
management in smart home within the consumer budget limitation. Sustainable
cities and society (2018)
14. Khalid, A., et al.: Cuckoo search optimization technique for multi-objective home
energy management. In: International Conference on Innovative Mobile and Inter-
net Services in Ubiquitous Computing. Springer, Cham (2017)
15. Samuel, O., Javaid, N., Ashraf, M., Ishmanov, F., Afzal, M., Khan, Z.: Jaya-based
optimization method with high dispatchable distributed generation for residential
microgrid. Energies 11(6), 1513 (2018)
16. Khalid, A., Javaid, N., Guizani, M., Alhussein, M., Aurangzeb, K., Ilahi, M.:
Towards dynamic coordination among home appliances using multi-objective
energy optimization for demand side management in smart buildings. IEEE Access
6, 19509–19529 (2018)
66 W. Ahmad et al.

17. Marzband, M., et al.: Real time experimental implementation of optimum energy
management system in stand alone microgrid by using multi-layer and colony opti-
mization. Int. J. Electr. Power Energy Syst. 75, 265–274 (2016)
18. Mirjalili, S., Mirjalili, S.M., Lewis, A.: Grey wolf optimizer. Adv. Eng. Softw. 69,
46–61 (2014)
19. Askarzadeh, A.: A novel metaheuristic method for solving constrained engineering
optimization problems: crow search algorithm. Comput. Struct. 169, 1–12 (2016)
 Towards Efficient Scheduling of Smart
Appliances for Energy Management
by Candidate Solution Updation
Algorithm in Smart Grid

Sahibzada Muhammad Shuja1 , Nadeem Javaid1(B) ,

Muhammad Zeeshan Rafique2 , Umar Qasim3 , Raja Farhat Makhdoom Khan1 ,
Ayesha Anjum Butt1 , and Murtaza Hanif4
1
COMSATS University Islamabad, Islamabad 44000, Pakistan
[email protected]
2
University of Lahore (Islamabad Campus), Islamabad 44000, Pakistan
3
Cameron Library, University of Alberta, Edmonton, AB T6G 2J8, Canada
4
Central South University, Changsha 410083, China
http://www.njavaid.com

Abstract. The energy demand is increasing day by day due to the

huge amount of residential appliance’s energy consumption, which cre-
ates more shortage of electricity. Industrial and commercial areas are
also consuming large amount of energy, but residential energy demand
is more flexible as compared to other two. Nowadays, many of the tech-
niques are presented for scheduling of Smart Appliances to reduce the
peak to average ratio (PAR) and consumer delay time. However, they
didn’t consider the total electricity cost and consumer waiting time. In
this paper, we reduce the cost through load shifting techniques. In order
to consider above objective, we employed some feature of the Jaya algo-
rithm (JA) on a bat algorithm (BA) to develop a candidate solution
updation algorithm (CSUA). Simulation was conducted to compare the
result of existing BA and Jaya for single smart home with 15 smart appli-
ances. We used time of use (ToU) and critical peak price (CPP). The
result depicts that successful achievement of load shifting from higher
price time slot to lower price time slot, which basically bring out the
reduction in electricity bills.

Keywords: BA · JA · CSUA · Metaheuristic techniques ·

Appliances scheduling · Home Energy Management ·
Demand Side Management · Smart grid

1 Introduction
The Conventional Grid (CG) has lack of some capabilities like self-effacing, real-
time reinforcement, observation of transmission lines, bi-directional communica-
tion and fault tolerance [1]. The invention of Smart Grid (SG) has appeared to
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 67–81, 2020.
https://doi.org/10.1007/978-3-030-15032-7_6
68 S. M. Shuja et al.

handled the limitation of CG. SG provide feature of bi-directional transmission

consumer and source of energy (Utility) provided by underlying Advanced Meter-
ing Infrastructure (AMI) [2]. The smart grid comprises of two main categories
for objectives of electricity cost and Peak to Average Ratio (PAR) while consid-
ering user comfort, which are Supply Side Management (SSM) and Demand Side
Management (DSM). These categories deal with production and transmission of
energy at low cost to consumer [3]. There are various kinds of DSM techniques
implemented by sources of energy for consumers to efficiently use the electricity.
Moreover, different incentive are given by utility companies for consumers to
reduce the wastage of electricity. This makes the consumer to optimally sched-
ule the electricity consumption for reduction of cost. These methods induce the
balance between the demand and supply of energy for consumers. Many of the
solutions are provided by Demand Response Program (DRP) to consumers in
[4]. These solutions influence the consumers to change the living way of life i.e.,
change the energy consumption pattern upon dynamic electricity rate. The elec-
tricity bills along PAR are minimized by consumer side through efficient man-
aging of electricity consumption. All around the world, utility companies set
up various electricity tariffs: Real Time Peak Price (RTP), Critical Peak Price
(CPP), Inclined Block Rate (IBR) etc. The electricity can be maintained for
usage of residential appliances through cost and PAR reduction. Recently lots of
DSM mechanisms and heuristic methods are used to adopt the previously men-
tioned challenges. Moreover, mathematical models of integer programming are
also presented for cost of energy utilization and management of load for residen-
tial appliances. In previous work, lots of heuristic and non- heuristic method are
given to schedule the home appliances i.e. particle swarm optimization, adaptive
cuckoo search, elephant hardening optimization, Mixed Integer Linear Program-
ming (MILP) and Mixed Integer Non-Linear Programming (MINLP).
The complexity is produced due to integrating Renewable Energy Source
(RES) along supply of energy from Grid, in which Photovoltaic (PV), Wind
Turbine (WT) and fuel generators are some sources of RES. The control level
of energy sources is risen in [5,6]. Now a day, researchers prefer the studies of
finding substitute of previous grid with more efficient and reliable grid system.
To make efficient SG, DRP is used through efficient utilization of energy along
interaction part of system between grid and consumer [7]. In DRP, utility with
RES of energy help consumer to control the use of electricity for minimizing cost
and balancing load between supply and demand sides of energy such that load on
generation and distribution could be minimize. Electricity consumers can satisfy
their comfort level in scheduling of residential appliances through efficient use of
DRP.
The consumption of electricity for residential electric appliances has been
broadly studied for being recent work. However, every consumer has different
kind of electric appliances depend on their operational time and consumption
criteria. As electric appliances of various consumption energy patterns give vari-
able electric price of consumption, we must present an efficient scheme of DRP
for different classes of appliances. Residential appliances of various categories are
 Candidate Solution Updation Algorithm in Smart Grid 69

in [8], which are mention by author in following five classes elastic-less memory,
elastic-full memory, elastic-partial memory, inelastic-interruptable and inelastic-
uninterruptable operation. Elastic and inelastic residential appliances are not
combine in use for studies in [9], because both of them have some limitation.
However, in most of the cases both of the appliance are not considered at a time.
Home Energy Management System (HEMS) for scheduling of smart home
appliances in architecture of smart grid is comprises of multiple parameters smart
meter, smart appliance, Electricity Storage System (ESS) and RES. HEMS is
an intelligent technology that can handle autonomous direction given by the
source of energy to optimize the load consumption for electric appliances in
peak hour. In the acknowledgment of last researches for HEMS, authors made
types for its energy management, which is categorizes as real-time and presuming
based management. The previous strategy for optimal solution is to predict the
consumption of energy on the basis of historical data. According to previous
research in [10], they were using presuming method for energy management,
while their results were not fine and accurate that’s why real-time optimization is
used to overcome presuming faults. However, their ideas were costly and complex,
that is only applicable for high scale areas. However, their presuming approach
of energy management was costly and complex, that is only applicable for high
scale areas [11].
The real time optimization of energy management is presented in [12], to min-
imize the energy consumption during peak hours. They show prominent results
for managing energy in residential areas, because they are easy to deploy upon
different appliances. However, user comfort is not satisfying to optimal schedule
the appliances. Residential appliances can manage their load through optimal
scheduling scheme, therefore smart grid distribution pattern of electricity shows
same behaviour of scheduling pattern. Due to which derived energy from other
production sources could be used with minimum cost. Furthermore, optimal
scheduling of residential appliances provide efficient and reliable operation of
renewable energy sources (Table 1).

2 Related Work

A notably large amount of work has been published on scheduling smart home for
smart appliances in home energy management system HEMS. Researchers have
the main concentration point of scheduling is residential areas, in which they
are trying to solve it in an efficient way. Huge number of scheduling method has
been evolved to design efficient energy consumption system, whose idea is to
optimally schedule energy consumption appliances in a smart home. They focus
on different condition to consume energy efficiently in reference of lesson the
cost, decreasing the PAR or increasing user comfort through individual way or
combined effect among for those parameters. At some extend for DSM technique
in HEMS are presented for both cost and PAR is decreased or one of them.
HEMS are not basically concern with cost or PAR reduction, also user comfort.
They are also integrated to create a trade-off between cost and user comfort as
70 S. M. Shuja et al.

Table 1. Summary of related work

Proposed Achievements of technique Limitations of technique

technique
MILP along Decrease in PAR and cost of User comfort is not considered
RES [13] electricity with RES for
home appliances
MILP on MG Cost reduction for home Micro turbines are not used
[14] appliances in Micro Grid
MILP along Reduce total energy cost Cost on implementation,
DERs [16] through RES and ESS in maintenance and fuel for generators
Micro grid is not preferred in their formulation
ILP [17] Perform balance management Cost of energy consumption is not
between demand and supply considered
GA [18] Optimal solution for cost User comfort is ignored
reduction and load balancing
GA and PSO Optimization for reduction Ignored possibility of user comfort
[19] cost and PAR on RTP, CPP
and TOU signals
GA with RES Control high demand and Cost for installation and
[20] high cost of energy maintenance of RES
GA on DMS Obtained optimal decrease in They neglect PAR and user
[21] energy consumption comfort
2PEM-PSO Optimization of operational They neglect to address the energy
[22] load cost and PAR
MINLP [23] Consumer to save 25% or They ignore PAR in their work
more energy cost on RTP
signal
DP-Dynamic Manage extra energy They neglected the cost of RES’s
Programing production and also shift deployment and maintenance
[24] operational time of
residential appliances in
different time-interval
GA and They perform optimal User comfort is ignored
BPSO [25] scheduling for minimization
of cost and PAR

energy management has multiple dimensions and parameters to schedule them

in optimal way. Few of the published work are presented below.
There are some pricing tariff mentioned for optimal scheduling of residential
appliances RTP, Time of Use (ToU), CPP etc. In [13], a MILP-model is presented
to decrease energy consumption, PAR and consumption cost along integration
of RES for scheduling residential consumers appliances. There results show that
given scheme achieved best target for cost and peak load reduction. To implement
 Candidate Solution Updation Algorithm in Smart Grid 71

MILP in [14], they present method of saving cost for consumer of energy produce
by Micro Grid (MG). The effects of temperature, heat and energy demand are
analyzed in depth. They integrated WT and PV for Distributed Energy Resource
(DER), while not using MT. In [15], a MILP-model is used to optimize the total
energy cost. They used different DERs for to make the production part of their
structure, which is MT, PV, WT, batteries and diesel generators. However, Cost
on implementation, maintenance for their system and fuel for generators is not
preferred in their formulation. On the other hand in [16], the basic scheme of
thought to implement ILP-model for residential areas, in which they performed
balance between energy demand and supply. Their proposed methodology pro-
ficiently moves optimal operation time and optimal load for time-shiftable and
load-shiftable residential appliances, individually. Test results present that their
proposed system pointedly archived the guaranteed objectives.
In [17], a Genetic Algorithm (GA) is used to present optimal solution for
electricity consumption cost reduction and load balancing between demand and
supply of electricity. The optimization scheduling method are followed by authors
in [18], which are Genetic Algorithm (GA) and Particle Swarm Optimization
(PSO) for the reduction of cost and PAR. They used three different cost tariff
signal RTP, CPP and ToU to determine energy cost through multiple knapsack
problem (MKP). There results give effectiveness to their method, while GA
perform best then other algorithm. Reduction of the electricity cost in smart
grid was grasped in [19], they integrate RES and storage devices like as batteries,
by using GA scheme. When energy demand and cost is high for defined interval,
they use storage devices to resolve this issue. Author mentioned limits of the
charging and discharging of batteries, they can also install a controller to safe
battery from any damage. However, cost for installation and maintenance of
RES is ignored by author.
The author used Demand Side Management (DSM) with GA scheme in [20],
in which they perform DSM with GA and DSM without GA on energy consump-
tion and compare results of their data-sets. There results simulation show that
the GA-DSM obtained optimal decrease in energy consumption, which analyzes
to get 21.91% reduction in peak hour. However, they neglect PAR and user com-
fort. In [21], they used DRP in DSM for HEMS to optimally schedule the oper-
ation of residential appliances. They proposed PSO technique with embedded
2PEM-two point estimation method for optimization of operational load. They
compare result of gradient based 2PEM-PSO with Latin Hypercube sampling
(LHS) GA with PSO (GAPSO). However, their results clarify this that given
scheme is much better and more effective then LHS-GAPSO. They neglect to
address the energy cost and PAR, which not reliable for electricity consumers.
In [22], a MINLP with Electricity Tariff (ET) ToU is modeled to use for home
appliance’s consumption cost minimization. However, they ignore PAR in their
work. Another scheme for the reduction of cost is presented in [23], they make
possible use of game theory through which they can interact with consumers for
extra energy production and also shift operational time of residential appliances
in different time-interval by using Dynamic Programming (DP). Author discuss
72 S. M. Shuja et al.

the scenario of resident, in which consumers generate electricity for their appli-
ances through RES and also sold extra energy to other consumer in neighbor or
to commercial utility sources. They neglected the cost of RES’s deployment and
maintenance.
Heuristic methods of GA and Binary-PSO are presented in [24], they perform
optimal scheduling for minimization of cost and PAR. Their result are analyzes
that the given proposition is an effective to reduce the describe parameters. An
optimal scheduling for residential appliances in [25], for reduction of electricity
cost and loads PAR by using GA scheme of scheduling on consumer appliances.
They perform analysis of cost by electricity tariff signal of RTP and IBR. The
results graphs of proposed method get better response for single as well as mul-
tiple consumers.

3 Problem Statement
The main objectives of smart grid are decreasing cost, PAR and increasing con-
sumer comfort. However, a trade-off exists between those parameters. The user
comfort is generally sacrificed by reducing PAR and energy consummation cost.
Recently HEM-model is proposed to schedule the energy load during on peak
and off peak hours along RTP signal [31]. They also make consumer to change
the scheduler at run-time through hybrid optimization models, while using elec-
tricity tariffs: RTP, CPP and ToU. Their results show the significant reduction
in electricity cost and the trade-off exists between cost and user comfort. How-
ever, the PAR is not considered. A general model of Energy Management System
(EMS) is presented with Home Area Networks (HAN) in [32]. Which proposes
an heuristic algorithm GA for efficient optimization of home energy, while using
RTP and IBR signals to efficiently manage the home appliances. Results show
the significant reduction in electricity cost and PAR. However, the user comfort
is ignored. In our work, we performed BA, Jaya and proposed a new CSUA
algorithm for scheduling of home appliances.

4 Proposed System Model

The proposed system model is shown in Fig. 1. Our system model is working
for scheduling smart home appliances. Cost of electricity and PAR is increas-
ing periodically so there is need of well adaptable system, which can minimize
average electricity price and PAR. A HEMS is used for optimization of res-
idential appliances to regulate the schedule of their operational time. HEMS
ease the consumer with respect of reduction in electricity dissipation and elec-
tricity cost. For optimization the appliances are classified into three different
categories; Shiftable appliances, Non-Shiftable appliances and fixed appliances.
Shiftable are those appliances whose operational time can be shifted from peak
hour time slots to non-peak hour time slots, while Non-Shiftable operational
time cannot be shifted or interrupted during their operational time and fixed
appliances operate on the basis of consumer demand which cannot be fixed for
 Candidate Solution Updation Algorithm in Smart Grid 73

specific time slots. Appliances of different classification with their power rating
are given in Table 2. In SG system consumer can bi-directionally communicate
with utility through SM, this communication can give information to the con-
sumer for recently consumed electricity and estimated cost of electricity. In this
paper we used CPP and RTP pricing scheme through which load is shifted from
peak hours to non-peak hours, which enable us to know the energy consumed
and cost charged against per hour. We can reduce our electricity cost by know-
ing the consumption of energy. In our work, we used our proposed algorithm in
EMC of SM, compared the result with SA and Firefly algorithm. The results
shows that proposed algorithm perform in a better way from other algorithm
for scheduling of residential appliances.

Fig. 1. Proposed system model

SM behaves like a bidirectional communication for information exchange

between consumer house and energy providers to schedule the appliances. More-
over, every algorithm is applied in EMC for appliances scheduling. In our case,
we used CSUA based EMC along with BA and Jaya algorithm based EMCs,
then we compared their results. In our simulation, we derived better result for
our proposed scheme of CSUA. We performed simulation on 15 appliances of
a smart house for calculation of electricity cost, PAR and waiting time. In this
paper, we break down a day of 24 h into 120 time slot. In other word, we can
say that we gave 5 slots for one hour. Because, waiting time of few appliances is
less than one hour as they are not working for one complete hour.
The main class of appliances is divided into two classes; one is scheduled
and other one non-scheduled appliances. The scheduled appliances are further
divided into interruptible, non-interruptible and base appliances class. Powers
rating of different appliances are given in Table 2.
74 S. M. Shuja et al.

Table 2. Appliances for simulations

Appliances class Appliances Power rating (kWh)

Interruptible Vacuum Cleaner 1.2
Electric Water Heater 2.6
Water Pump 1
Dish Washer 2.5
Steam Iron 1.2
Refrigerator 0.225
Air Conditioner 1.5
Non-Interruptible Washing Machine 3
Tumble Dryer 3.3
Base Oven 1.23
Blender 0.3
Ceiling Fan 0.1
Desktop PC 0.3
TV 0.3
Laptop 0.1

The mathematical model of our objective for estimation of total energy con-
sumption is given in Eq. 1.

24
Load = (P R ∗ S(t)), S(t) = [1/0] (1)
t=1

In Eq. 1, PR shows Power Rating and S(t) values shows the ON/OFF status
of appliances.
Mathematical formulation for PAR calculation by taking total energy con-
sumption in 24 h in Eq. 2.

P AR = (M ax(Load)/(Load)) (2)

Total cost estimation for 24 h electricity consumption is given in Eq. 3.


24
Cost = (P P ∗ P R ∗ S(t)), S(t) = [1/0] (3)
t=1

In Eq. 3, PP shows the electric price of 24 h.

5 Scheduling Techniques
In traditional way many kind of mathematical technique are used for schedul-
ing of home appliances; ILP, MILP, CP, DLP and MINLP. We selected Meta-
Heuristic technique as it can achieve optimization on any problem. We selected
two Meta-Heuristic techniques BA, Jaya and our proposed CSUA, which are
described in further subsection.
 Candidate Solution Updation Algorithm in Smart Grid 75

5.1 Bat Algorithm (BA)

We used BA [33] in our paper for solving problem of scheduling. In BA algo-

rithm mainly bats have an acoustic attribute of echolocation. The echolocation
is broadly used by well-known example of Microbats. Microbats mostly prey for
insects to eat. Microbats used sonar for echolocation; due to this they can iden-
tify prey, avoid hurdles and target hurdles in the dark. Bats produce high pulse
sound, to hear the reflected echo from their surrounded prey. The sound pulse is
different at various hunting strategy. For need of optimal solution for our appli-
ances, we can use echolocation attribute of Microbats to many bat algorithms.
However, easier rule for BA is mention below:

• To detect the distance and differentiate between food and hurdle, bats use
echolocation attribute.
• To search for food, bats randomly fly with velocity Vi , at location Xi , sound
loudness A0 , variable wavelength λ and constant frequency Fmin .
• Bats adapt their frequency production rate r  [0, 1] of pulse for proximity of
food.
• As loudness of sound vary in different situation, we assume it from A0 to
Amin . Amin is a fixed minimum value of loudness.

5.2 Jaya Algorithm

The Jaya (a word meaning victory) is the new approach of population-based

optimization introduced by Rao [34] for optimal solution of problem. Jaya con-
trolling parameters are not like other heuristic technique’s parameter; it has
only two simple control parameters of population size (m) (number of candidate
solutions) and the number of generation (Gn ) (number of iteration). In this tech-
nique optimization is extracted from an idea of solution for a particular problem
to move it toward best solution and away from worst solution. Let, f (x) is an
objective function for minimum and maximum values. For any ith value of itera-
tion, candidate with best and worst values of f (x) are represented as (f (x)best )
and (f (x)worst ). In a simple words algorithm is describe below:

• Initialize the population, control variable and check criteria.

• Classify best and worst values among candidate solutions.
• Modify the candidate solution on the basis of best and worst values.

Cj,l,t = Cj,l,t + r1,j,l (Cj,best,t − |Cj,l,t |) − r2,j,l (Cj,worst,t − |Cj,l,t |). (4)

• Check modified solution is best than previous, if it is, then lubricate the
previous one with modified either leave it.
• Is criteria good enough? then set the best solution to use otherwise repeat
the steps.
76 S. M. Shuja et al.

5.3 Candidate Solution Updation Algorithm (CSUA)

It is our proposed meta-heuristic technique for optimization. We put some deci-

sion based features of Jaya in BA to buildup CSUA algorithm. In BA, the
comparison of some random numbers with fixed value 0.5 occurs to update the
population, which is assigned to the parameter of loudness of sound A. By com-
paring random number, we can’t get refined updated population. If the condition
is satisfied, the random flying value of the bat updates the population which is
also not a good approach. Every time population gets updated by random num-
bers that makes the output of unrefined updated population. Because of putting
random solutions in comparison step and population updation step, we may not
achieve the best population.
Whereas, in Jaya algorithm objective function or population updation is per-
formed by using comparison of fitness through minimum and maximum value of
solution of the population with fitness of best and worst position of that number
of solution. When comparison condition is satisfied, then the objective function
or population is updated by the well refined solution which is extracted from
two main state [34] of Jaya. The population in Jaya is not updated by random
number of solution (which is in BA) in each iterations of objective function or
population updation. Therefore, we used Jaya along with their condition instead
of comparison between random solutions condition with A = 0.5 of BA for pop-
ulation updation. We also replaced the of parameter of BA: adapting frequency,
updating velocity and position in [33] from Eqs. 2 to 4 with fittest solution from
best and worst condition of Jaya in each iteration of population.
Algorithm 1. Candidate Solution Updation Algorithm.
Require: Input: [Initialize random population, MaxIt, Electricity Price;]
1: Search for minimum and maximum value for fitness of solution;
2: i = mini, j = maxi, iter = 0;
3: for t = 1 : 24 do
4: Let mini and maxi are values in random population of search space;
5: Evaluate the value of mini and maxi;
6: Initialize the value for best and worst solution CSUA;
7: while iter ≤ itermax do
8: iter = iter + 1;
9: Determine the Population Size;
10: Evaluate objective value of each one of the population;
11: Obtain new solution;
12: Accept new solution If it gives a better objective value;
13: Conduct the local search;
14: for j = 1 : C do
15: Randomly get a CSUA j to follow i;
16: Candidate Solution Updation by equation 1.4;
17: end for
18: Start Mutation Strategy;
19: End Mutation Strategy;
20: end while
21: Evaluate the new Population of the CSUA;
22: end for
 Candidate Solution Updation Algorithm in Smart Grid 77

6 Simulation Results and Reasoning

Simulation and their detail are described in this section. We create simulations
in MATLAB for evaluation of our proposed technique CSUA. In our proposed
algorithm, we compared our technique with present meta-heuristic optimization
algorithm; BA and Jaya. This evaluation result is analyzed in term of user elec-
tricity cost, PAR, hourly energy utilization and waiting time. In our simulation,
we derived best solution for 15 appliances with general existence in single home.
Appliances are divided into three main classes interruptible, non-interruptible
and base appliances.

Fig. 2. ToU per hour load Fig. 3. CPP per hour load

The hourly based energy consumption is given for ToU in Fig. 2 and CPP in
Fig. 3 with respect of scheduled and unscheduled patterns. The proposed CSUA
shows that its peaks except for ToU signal at mid four hours from 14 to 18
and for CPP signal at last three hours from 22 to 24 other than other hours
are best than BA and Jaya algorithm. It is admitted that pattern of scheduled
per hour load is far better than unscheduled. At last, our simulation shows that
proposed algorithm CSUA optimize hourly load consumption in a better way by
transferring load from peak hour to non-peak hour.

Fig. 4. ToU PAR Fig. 5. CPP PAR

The results of BA, Jaya and CSUA algorithms of PAR are shown for ToU
in Fig. 4 and CPP in Fig. 5 signals. In ToU signal, figure shows the proposed
CSUA outperformed than other algorithm as PAR is minimized 70% by CSUA,
65.71% by BA and 62.86% by Jaya as compared with unscheduled case. While
in CPP signal, results depict that BA is performed better as compared with our
algorithm and other one, because PAR is reduced 70% by BA, 68.57% by Jaya
78 S. M. Shuja et al.

and 67.14% by CSUA with respect of unscheduled case. Simulation results show
that proposed CSUA in ToU and BA in CPP signals performed best among other
respected techniques. However, there is always trade-off exist between total cost
of electricity between cost and PAR.

Fig. 6. ToU hourly cost Fig. 7. CPP hourly cost

Hourly cost of BA, Jaya and CSUA are shown in Figs. 6 and 7. Our plots
depict lower cost peaks in higher price time slots through scheduled algorithm
as compared unscheduled case. Total electricity cost in Figs. 8 and 9 show the
performance of our proposed algorithm CSUA in comparison with other existing
algorithm BA and Jaya. Plots show that our algorithm outperform than BA and
Jaya for both ToU and CPP pricing scheme. Simulation results are for both ToU
and CPP signals are shown in Figs. 10 and 11. Results depict that our proposed
scheme sacrificed the waiting time as compared with existing schemes, due to
which consumer comfort for user is compromised.

Fig. 8. ToU total cost Fig. 9. CPP total cost

Fig. 10. ToU waiting time Fig. 11. CPP waiting time
 Candidate Solution Updation Algorithm in Smart Grid 79

7 Conclusion
In this paper, we have perform the load shifting scheme from peak hours to
non-peak hours of DSM along with EMC based on BA, Jaya and CSUA. CSUA
reduce the PAR, total electricity cost and consumer waiting time to increase the
comfort level. We employed different kind of home appliances and performed
scheduling according to their energy consumption pattern. We evaluate the per-
formance of various algorithm used in our simulation, it is admitted that the
CSUA performed better than BA and Jaya for reduction of electricity cost and
consumer waiting time. In results simulation, we can say that their is trade-off
of electricity cost with both PAR and waiting time.

References
1. Gungor, V.C., Sahin, D., Kocak, T., Ergut, S., Buccella, C., Cecati, C., et al.:
Smart grid technologies: communication technologies and standards. IEEE Trans.
Industr. Inf. 7(4), 529–539 (2011)
2. Esther, B.P., Kumar, K.S.: A survey on residential demand side management archi-
tecture, approaches, optimization models and methods. Renew. Sustain. Energy
Rev. 59, 342–351 (2016)
3. Wu, Z., Tazvinga, H., Xia, X.: Demand side management of photovoltaic-battery
hybrid system. Appl. Energy 148, 294–304 (2015)
4. Strbac, G.: Demand side management: benefits and challenges. Energy Policy 36,
4419–4426 (2008)
5. Zhao, J., Guo, Z., Su, Z., Zhao, Z., Xiao, X., Liu, F.: An improved multi-step
forecasting model based on WRF ensembles and creative fuzzy systems for wind
speed. Appl. Energy 162, 808–826 (2016)
6. Heng, J., Wang, J., Xiao, L., Lu, H.: Research and application of a combined model
based on frequent pattern growth algorithm and multi-objective optimization for
solar radiation forecasting. Appl. Energy 208, 845–866 (2017)
7. Medina, J., Muller, N., Roytelman, I.: Demand response and distribution grid
operations: opportunities and challenges. IEEE Trans. Smart Grid 1(2), 193–198
(2010)
8. Roh, H.-T., Lee, J.-W.: Residential demand response scheduling with multiclass
appliances in the smart grid. IEEE Trans. Smart Grid 7(1), 94–104 (2016)
9. Baharlouei, Z., Hashemi, M., Narimani, H., Mohsenian-Rad, H.: Achieving opti-
mality and fairness in autonomous demand response: benchmarks and billing mech-
anisms. IEEE Trans. Smart Grid 4(2), 968–975 (2013)
10. Agnetis, A., de Pascale, G., Detti, P., Vicino, A.: Load scheduling for household
energy consumption optimization. IEEE Trans. Smart Grid 4, 2364–2373 (2013)
11. Colmenar-Santos, A., de Lober, L.N.T., Borge-Diez, D., Castro-Gil, M.: Solutions
to reduce energy consumption in the management of large buildings. Energy Build.
56, 66–77 (2013)
12. Marzband, M., Ghazimirsaeid, S.S., Uppal, H., Fernando, T.: A real-time evalua-
tion of energy management systems for smart hybrid home Microgrids. Elec. Power
Syst. Res. 143, 624–633 (2017)
13. Zhang, D., Evangelisti, S., Lettieri, P., Papageorgiou, L.G.: Economic and environ-
mental scheduling of smart homes with microgrid: DER operation and electrical
tasks. Energy Convers. Manag. 110, 113–124 (2016)
80 S. M. Shuja et al.

14. Zhang, Y., Zhang, T., Wang, R., Liu, Y., Guo, B.: Optimal operation of a smart
residential microgrid based on model predictive control by considering uncertainties
and storage impacts. Sol. Energy 122, 1052–1065 (2015)
15. Liu, G., Xu, Y., Tomsovic, K.: Bidding strategy for microgrid in day-ahead market
based on hybrid stochastic/robust optimization. IEEE Trans. Smart Grid 7, 227–
237 (2016)
16. Zhu, Z., Tang, J., Lambotharan, S., Chin, W.H., Fan, Z.: An integer linear pro-
gramming based optimization for home demand-side management in smart grid.
In: Proceedings of the 2012 IEEE PES Innovative Smart Grid Technologies (ISGT),
Washington, DC, USA, 16–20 January 2012, pp. 1-5 (2012)
17. Mohamed, F.A., Koivo, H.N.: Online management genetic algorithms of microgrid
for residential application. Energy Convers. Manag. 64, 562–568 (2012)
18. Mahmood, A., Javaid, N., Khan, N.A., Razzaq, S.: An optimized approach for home
appliances scheduling in smart grid. In: Proceedings of the 2016 19th International
Multi-Topic Conference (INMIC), Islamabad, Pakistan, 5–6 December 2016, pp.
1–5 (2016)
19. Mary, G.A., Rajarajeswari, R.: Smart grid cost optimization using genetic algo-
rithm. Int. J. Res. Eng. Technol. 3, 282–287 (2014)
20. Bharathi, C., Rekha, D., Vijayakumar, V.: Genetic algorithm based demand side
management for smart grid. Wirel. Pers. Commun. 93, 481–502 (2017)
21. Huang, Y., Wang, L., Guo, W., Kang, Q., Wu, Q.: Chance constrained optimization
in a home energy management system. IEEE Trans. Smart Grid (2016). https://
doi.org/10.1109/TSG.2016.2550031
22. Setlhaolo, D., Xia, X., Zhang, J.: Optimal scheduling of household appliances for
demand response. Electr. Power Syst. Res. 116, 24–28 (2014)
23. Samadi, P., Wong, V.W., Schober, R.: Load scheduling and power trading in sys-
tems with high penetration of renewable energy resources. IEEE Trans. Smart Grid
7, 1802–1812 (2016)
24. Ullah, I., Javaid, N., Khan, Z.A., Qasim, U., Khan, Z.A., Mehmood, S.A.: An
incentive-based optimal energy consumption scheduling algorithm for residential
users. Procedia Comput. Sci. 52, 851–857 (2015)
25. Shirazi, E., Jadid, S.: Optimal residential appliance scheduling under dynamic
pricing scheme via HEMDAS. Energy Build. 93, 40–49 (2015)
26. Erdinc, O.: Economic impacts of small-scale own generating and storage units, and
electric vehicles under different demand response strategies for smart households.
Appl. Energy 126, 142–150 (2014)
27. Javaid, N., Ullah, I., Akbar, M., Iqbal, Z., Khan, F.A., Alrajeh, N., Alabed, M.S.:
An intelligent load management system with renewable energy integration for
smart homes. IEEE Access 5, 13587–13600 (2017)
28. Cakmak, R., Altas, I.H.: Scheduling of domestic shiftable loads via Cuckoo Search
optimization algorithm. In: 2016 4th International Istanbul Smart Grid Congress
and Fair (ICSG), pp. 1–4. IEEE, April 2016
29. Li, C., Yu, X., Yu, W., Chen, G., Wang, J.: Efficient computation for sparse load
shifting in demand side management. IEEE Trans. Smart Grid 8(1), 250–261 (2017)
30. Bharathi, C., Rekha, D., Vijayakumar, V.: Genetic algorithm based demand side
management for smart grid. Wireless Pers. Commun. 93(2), 481–502 (2017)
31. Khalid, A., Javaid, N., Guizani, M., Alhussein, M., Aurangzeb, K., Ilahi, M.:
Towards dynamic coordination among home appliances using multi-objective
energy optimization for demand side management in smart buildings. IEEE Access
6, 19509–19529 (2018). https://doi.org/10.1109/ACCESS.2018.2791546
 Candidate Solution Updation Algorithm in Smart Grid 81

32. Marzband, M., Yousefnejad, E., Sumper, A., Domı́nguez-Garcı́a, J.L.: Real time
experimental implementation of optimum energy management system in stan-
dalone microgrid by using multi-layer ant colony optimization. Int. J. Electr. Power
Energy Syst. 75, 265–274 (2016)
33. Yang, X.-S.: A new metaheuristic bat-inspired algorithm. In: Nature inspired coop-
erative strategies for optimization (NICSO 2010), pp. 65–74. Springer, Heidelberg
(2010)
34. Radosavljevi’c, J., Klimenta, D., Jevti’c, M., Arsi’c, N.: Optimal power flow using
a hybrid optimization algorithm of particle swarm optimization and gravitational
search algorithm. Electr. Power Compon. Syst. 43, 1958–1970 (2015)
 Minimizing Daily Electricity Cost Using
Bird Chase Scheme with Electricity
Management Controller in a Smart Home

Raza Abid Abbasi1 , Nadeem Javaid1(B) , Shujat ur Rehman2 , Amanulla2 ,

Sajjad Khan1 , Hafiz Muhammad Faisal1 , and Sajawal Ur Rehman Khan1
1
COMSATS Institute of Information Technology, Islamabad 44000, Pakistan
[email protected]
2
Quaid-i-Azam University Islamabad, Islamabad 44000, Pakistan
http://www.njavaid.com

Abstract. Integration of Demand Side Management (DSM) strategies

within Smart Grid (SG) helps the utilities to mange and control the
power consumer load to meet the power demand. Schemes adapted by
DSM are used for reducing the load on utilities at peak time, which
is achieved by managing the user appliances according to the changes
in load on utility and individual smart home. This work is focused on
hourly scheduling of the appliances being used in a smart home targeting
the daily electricity cost minimization. A new heuristic scheme is intro-
duced for hourly appliances scheduling on user side in this paper. The
proposed scheme works at the electricity management controller level,
installed in a smart home, within a SG infrastructure. The proposed
scheme results are compared with other heuristic schemes as well. From
extensive simulations it is depicted that proposed scheme performs best
and outperforms other schemes in term of electricity cost minimization.

Keywords: STLF · Smart grid · Xgboost · Machine learning

1 Introduction
Energy consuming devices are growing exponentially, whereas energy produc-
ing sources are decreasing, i.e., oil, coal and gas. Rapidly growing at a pace of
0.3% annually, residential consumers has large share of 27% among others in US
electricity consumers [1]. Development in every field of life is cause of an evo-
lution in energy management system as well. Different type of energy resources
are explored to compete the energy demand of the community [2]. The use of
Smart Grid (SG) and its features like Demand Side Management (DSM), load
forecasting and cost forecasting is coming out to be the best substitute for the
energy management so far. SG is built as a communications network for the
energy industry. Main element of the SG is AMI which supports communication
between consumer and utility. Smart meters are used to figure out the energy
usage pattern of the consumer.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 82–94, 2020.
https://doi.org/10.1007/978-3-030-15032-7_7
 Minimizing Cost Using BCS 83

SG is the intelligent power grid which combines the information and commu-
nication technology in traditional grid to enable communication between pro-
ducer and consumer. SG is capable of efficient use of energy being produced.
Smart Meters (SMs) data of big volume, big velocity and big value can be used
for forecasting energy consumption and pricing. It helps the supplier to forecast
energy consumption demand and energy prices appropriately.
Power consumption prediction has importance to many applications, i.e.,
voltage and frequency adjustments, Demand Response (DR) and essential
administration. long term load forecasting (e.g., 1 year to 10 years) is essential
for outlining distribution and transmission network. Short Term Load Forecast-
ing (STLF) (e.g., from hours to days) in future is useful for DR, security affairs
and online setup of the power management system. STLF want to assure the
authenticity of the system accessories and handle power failure losses. It include
peak energy usage, daily energy load and Very STLF which forecasts using units
of less than an individual day.
In SG energy provider can handle the DR of attached devices to reduce load
on SG. DR are the modifications brought in the energy demand of the con-
sumer for the rules defined by the utility. Objective of DSM is to apply different
schemes for encouraging the consumer to transfer the energy consumption time
from on peak hours to off peak hours which increases efficiency of SG. The goal
of DSM is to minimize load on the grid by educating the users to change their
energy consumption pattern. DSM helps in decreasing peak demand, alleviat-
ing price fluctuation, and delay in system upgrade. DSM may involve the cus-
tomer to contribute in the system, where user can minimize the price of energy
through energy preservation, peak clipping and load transfer. In-fact, load shift-
ing can be considered as the mostly used technique where energy load is handled
using a DR.

2 Related Work

In SG utility is capable of managing the DR of attached loads i.e., Smart Homes

(SHs) to curtail the total pressure at the source [3].
Huangs et al. [4] suggested plan for smart buildings based on multi agent
minority game to curtail the peak load from source. It focused on reducing
power input from grid and increasing solar power usage to manage load demand.
However, it can not work if a SH does not have solar panel. In [5], Ruelens
proposed the use of batch Reinforcement Learning (RL) to DR which proved
to be more reliable for large-scale implementation as compared to traditional
model-based approaches. They proposed a model-free Monte Carlo scheme that
implements a metric depending on the state-action value function or Q-function
and implement it in finding the next day electricity required for house-hold heat
pumps by mining the historic weather information. It was modeled on a limited
set of appliances. In residential sector, energy needed for electric water heaters
was handled in [6]. It suggested conversion of electric energy into heat energy
during the absence of peak hours which will help in reducing load during peak
84 R. A. Abbasi et al.

hours. It measures temperature state of water heaters for load transferring, and
provides adjustments provision to the utility. DR management was done for a
limited set of appliances; however during conversion of energies, i.e., electric to
heat energy; storage cost of heat energy was not taken into consideration. To
provide a capable DR administration data analytic can be used wisely. In [7],
Sakurama and Miura proposed a decentralized DR management algorithm which
determines control signal by using communication network of SMs. They revealed
that the DR scheme can be more efficient when the behaviour of consumer using
energy is available.
Jindal et al. [8] proposes a comprehensive scheme based on combination of
Decision Tree (DT) and Support Vector Machine (SVM) classifier for analyzing
the data gathered from SH for theft detection in electric grid networks. It is capa-
ble of identifying theft at any level in transmission and distribution. Proposed
scheme can be considered a two-level approach for data processing and data
analysis because output produced by DR scheme is used as input to the SVM
classifier. In [9], authors used data analytics techniques on a rich, multi-year,
and high-frequency annotated data for evaluating STLF on SMs in an average
size city. They used different state-of-the-art forecasting algorithms, i.e., autore-
gressive model, kernel methods, exponential smoothing and state-space mod-
els. They found that STLF accuracy improves at larger meter aggregation, i.e.,
Feeder, substation, and system-wide level. Cumulative Sum (CUSUM) is used
in combination with SheWart algorithm to identify SMs with unusual electricity
usage pattern in [10]. Preprocessing is performed on data to find uncommon
meter readings by analyzing long missing or zero data and outliers. In first step,
temperature effect is normalized and then CUSUM and SheWart is used. Afore-
mentioned studies focused on data analytics; however, did not focus on managing
the DR of SHs for peak load curtailment. For this area, Jindal et al. [11] outline
SHs with extra electricity consumption for DR of those SHs. They SVM for clas-
sification purpose but did not consider real time modification in energy demand
of user and was applicable only to SHs exceeding specific consumption limit.
Cluster-Based Aggregate Forecasting (CBAF) algorithm was used by Misiti
et al. [12] and Alzate and Sinn [13] for load forecasting in a grid. Where Mis-
iti et al. evaluated just for industrial users where Alzate and Sinn worked for
house-hold consumers and different enterprises. They utilized wavelet included
plan and kernel spectral clustering for discovering likeness between time series.
However, they did not research the impact of various user sizes to the change
given by CBAF. In [14], Tri Kurniawan Wijaya et al., used different Machine
Learning (ML) algorithms for individual house-hold STLF and total anticipat-
ing (e.g., 1 h and 1 day in future) in private power utilization. They used scale
independent and robust accuracy metrics, i.e., normalized mean absolute error
and normalized root mean square error for accuracy measurement at house hold
level. Anish Jindal et al. [15], proposed scheme for DSM which aims at load
curtailment from grid at peak load time. They introduced different factors for
analysis, i.e., appliance priority index, appliance adjustment factor and appli-
ance curtailment priority along with an incentive scheme to increase the user
 Minimizing Cost Using BCS 85

participation. Different algorithms were introduced depending on these factors,

with respect to customer perspective and utility perspective.
A STLF scheme for electricity has been proposed by Jihoon Moon et al. [16],
for higher education institutes such as universities, etc. They used 2-stage pre-
dictive analytics for improving results. In proposed scheme, electric load pattern
is found using moving average method over day of the week, where daily electric
load is predicted using random forest method. Dogan Keles et al. [17], worked
on forecasting STLF at the European power exchange next day market using
a model developed on the basis of Artificial Neural Network (ANN). Different
cluster based algorithms are used for proper selection and organization of input
data. Then best fitting ANN configurations are found and used for in-sample
and out-of-sample analysis.
SG is of great importance with respect to the pricing point of view as well.
In [18], M. Erol-Kantarci et al. suggested that due to the flexibility of DSM,
it allows consumers to take part in the process of power management system.
They can minimize their electricity cost by load shifting and conservation. It will
encourage consumer to transfer energy consumption on peak or off peak hours
depending on the variable electricity pricing. Customer wants to know the exact
price of electricity, therefore, they need price forecasting to be exact point [18].
In this exact point forecasting, they can decide to switch on or switch off a device
depending on threshold. Data analytics need to work here on big historical price
data for price classification. We can classify the price by categorizing the prices
according to different thresholds on the basis of exact point prediction scheme
values. While relating forecasting, exact price is difficult due to different variable
factors. Therefore, price taxonomy for electricity using big price data is in focus
for researchers.
For managing price classification challenge, researches used conventional clas-
sifiers, i.e., SVM, Naive Bayes, neural network and DT mostly [19]. In [20], it is
stated that SVM has better accuracy, however higher complexity. Data redun-
dancy and high dimensionality of the features are the serious threat to ML
classifiers.

3 Problem Statement

Replacement of traditional hierarchical grids with SG brought new challenges for

energy conservation and efficient usage. Since the evolution of DSM, it came up
with a number of challenges like load shifting, communication fairness, security
and privacy. DSM aims at handling the irregular consumption of energy on
customer side because it increases load on the utility. To fulfill the demand
during peak hours, extra electric power is generated which result in an increase
in the cost. The existing studies have already done a lot of work to reduce the
cost, Peak to Average Ratio (PAR) and discomfort by efficiently scheduling the
household appliances.
In [21], authors proposed an Integer Linear Programming (ILP) based opti-
mization mechanism for home DSM in SG. However, It works well only for large
86 R. A. Abbasi et al.

number of appliances. In [22], authors worked on DSM and introduced schedul-

ing by formulating Mixed Integer Non Linear Programming (MINLP) scheme for
mixed appliances. It helps to reduce the total electricity cost. Home Area Net-
work (HAN) controller was proposed in [23], for reducing peak loads. They used
a load reduction algorithm which is modelled using a unified modeling language
state machine model. However, cost of HAN is high. Authors in [24,25] proposed
classification model by classifying the user devices in different classes and then
scheduled them according to prioritization at real time. It has two classes where
low i.e., delay tolerant and high i.e., delay intolerant, priority classes exist.
Flexible rescheduling of appliances is required for dynamic scheduling accord-
ing to the user need. However, there is a trade-off in PAR vs cost and cost vs
comfort. Cost and PAR can be reduced by load shifting and integrating renew-
able energy resources. In this work we will use our proposed scheme, i.e., Bird
Chase Scheme (BCS); for daily electricity cost minimization.

4 Proposed System Model

Home Energy Management Systems (HEMS) are getting popular. HEMS is

based on DSM which enables efficient use of energy. In a SG infrastructure,
houses are equipped with SMs. SM enables two way communication between
utility and consumer. Devices inside a home are connected to each other and
Energy Management Controller (EMC) through HAN. Hence, smart appliances
in a house share information with EMC using HAN. EMC receives the informa-
tion sent from smart appliances and then perform required processing on that
information for taking important decisions. EMC receives the total load and
other requirement and shares that with the utility using SM. Utility receives the
information sent through SM and perform necessary actions accordingly. Utility
sends the pricing signal along with the user demanded energy to the consumer
using SM. SM shares the information received from utility with the EMC. Here,
EMC has both the pricing information received from utility and information
received from smart appliances, i.e., appliance power rating information, Length
of Operation Time (LoT) information, operational time interval information.
EMC uses the information received from utility and appliances and perform opti-
mal scheduling of appliances. It uses the Jaya, Crow Search Algorithm (CSA)
and BCS algorithms for scheduling. The objective of the scheduling is to min-
imize PAR, maximize consumer satisfaction and cost reduction. The discussed
system model is displayed in Fig. 1.
In our proposed scheme, we consider a single home, using 11 smart appli-
ances of different categories. We categorize appliances in three main categories.
Fixed appliances, shiftable appliances and interruptible appliances. Fixed appli-
ances are those that needs to be switch on according to the specified schedule.
Shiftable appliances are those, that can be scheduled but can not be interrupted
once their status is set on. Interruptible appliances can be scheduled as well as
interrupted at any time. Moreover, we have a condition that washing dryer will
always start working after washing machine has finished its operation. Table 1,
 Minimizing Cost Using BCS 87

shows the categorized appliances with their corresponding LoT and power rat-
ing. The proposed schemes objective is to: minize PAR, maximize user confort
by reducing waiting time and reducing total cost. Equation 1 is used for total
energy consumption for a single day.

24
Load = (Prate ∗ St ), St = [1/0] (1)
t=1

Total cost for a single day is calculated using the mathematical Eq. 2.
P AR = (M ax(Load)/Avg(Load)) (2)
Equation 3 is used for PAR calculation.

24
Cost = (Pprice ∗ Prate ∗ St ), St = [1/0] (3)
t=1

Here, in Eq. 1, Prate is the device power rating, and St is defining the status
i.e., on or off of a device. If status is 1 it means that device is on at that time
and if staus is 0 it means device is off at that time. In Eq. 2, P AR is peak
to average ration. P AR is computed by selecting highest load from single day
energy consumption then dividing that by the average of that day. In Eq. 3,
Pp rice represents the power prices for single day.

Table 1. Control parameters

Classes Appliances LoT (h) PR (kWh)

Fixed appliances Light 12 0.1
Fan 16 0.1
Oven 9 3
Blender 4 1.2
Shiftable appliances Washing machine 5 0.5
Cloth dryer 4 4
Dish washer 4 1.5
Interruptable appliances AC 12 1.1
Refrigerator 12 1.2
Iron 6 1.1
Vacuum cleaner 5 0.5

5 Metaheuristic Optimization Algorithms

Conventional techniques like, LP, Convex Programming (CP), ILP, Mixed ILP
(MILP) and MI Non LP (MINLP) doesn’t perform well over large number of
88 R. A. Abbasi et al.

Fig. 1. System model

computations. Due to their deterministic nature, they are not sufficient for real
time optimization. Where, heuristic techniques are problem dependent. They
are designed to solve a particular problem. meta-heuristic algorithms can be
designed to solve almost any problem. Therefore, we selected heuristic program-
ming algorithms to solve the home appliances scheduling problem. Our main
objective is total energy consumption cost minimization. The selected meta-
heuristic algorithms are discussed now.

5.1 CSA

CSA is derived from the intelligent behavior of crows. Alireza Askarzadeh intro-
duced CSA in 2016. It is a novel meta-heuristic algorithm used for optimization.
CSA has been extensively used to solve numerous complex engineering problems.
Crows are intelligent enough to store their food for future use. While saving food
they ensure it that they are not followed by any other crow, who can steal their
food. If they find someone following them or keeping an eye on them, they change
their direction and find some other place for that. They continue this process
untill they find best possible location for hiding food. By considering locations
as possible solutions, crows as searchers, environment as the search space we
can map with the optimization process. The food source quality is considered as
the fitness function, and the best food source in the whole environment can be
mapped with the global best solution. CSA steps for scheduling appliances in a
SH for cost optimization as are listed below: 1. Initialize the problem and the
parameters 2. Initialize the memory and crow positions 3. Evaluate the positions
using objective function 4. Creating new positions for crows 5. Checking feasi-
bility of the new positions 6. Evaluate new positions using objective function 7.
Update the memory of the crows 8. comparing termination criteria.
 Minimizing Cost Using BCS 89

5.2 Jaya

Jaya is a Sanskrit word, which means victory. Jaya algorithm aims to become
winner by approaching best solution. This algorithm always try to get closer to
the best solution and get away from the worst solution. In first step we initial-
ize population size which is number of possible schedulings, number of design
variables which is number of appliances in our case and termination criterion. In
second step it identifies the best and worst solutions in the population. In third
step it updates population solutions based on previously selected best and worst
solutions. In fourth step we compare the updated solution with the best solution
corresponding to the design variable. If new solution is better we replace the
previous one, otherwise keep the same. If termination criterion is met, we report
the optimum solution otherwise repeat steps one to five.

5.3 BCS

BCS is based on the birds intelligence behavior to secure their food from other
birds in the flock. We can consider locations as possible solutions, birds as
searchers, environment as the search space. We can map them with the opti-
mization process. The food source quality is considered as the fitness function,
and the best food source in the whole environment can be mapped with the
global best solution. The steps followed in BICS are listed below.

Algorithm 1. Proposed: Bird Chase Scheme

Require: Minimum cost Scheduling
Initialize the bird awareness probability, memory and population
while Termination criteria is not met do
for i=1 to size(p) do
Apply Rosenbrock fitness function on p(i)
end for
generate random number of birds
for j=1 to size(p) do
update bird position using awareness probability
end for
Apply mutate on updated population
for i=1 to size(p) do
Apply Rosenbrock fitness function on updated p(i)
end for
Apply mutate on updated population
Find local fit solution
Apply scheduling conditions
end while
Find global fit solution
scheduling Appliances
90 R. A. Abbasi et al.

6 Simulation Results and Discussion

This section highlights the proposed system performance, which is validated
through extensive simulations in MATLAB software. Then, the results achieved
are discussed hereafter. The results are produced for Real Time Pricing (RTP)
tariff using our proposed algorithm BCS. We compare the results of BCS with
other implemented techniques such as Jaya and CSA. We study a single home for
the purpose of simulation with eleven appliances, i.e., D = 11 those needs to be
scheduled. For the purpose of simulation, these appliances are categorized into
three different groups: Fixed Appliances (FA), Shift-able Appliances (SA), Inter-
ruptible Appliances (IA). We selected these appliances because of their frequent
use in summer. Group FA includes light, fan, oven and blender. These appliances
will can neither be interrupted nor shifted. Group SA includes washing machine,
cloth dryer and dish washer. These appliances can be shifted, however can not
be interrupted once they are turned on. Group IA includes AC, refrigerator, iron
and vaccume cleaner. These appliances can be shifted as well as interrupted.
Real Time Pricing (RTP), pricing scheme is used for load, cost and PAR
calculation. Figure 2 is displaying the RTP pricing signal for 24 h. Where, x-axis
represents the 24 h and y-axis represents price in cents per kilowatt-hour.

6.1 RTP Tariff

As mentioned earlier, proposed scheme will be evaluated using RTP. Here, we
will discuss the RTP tariff. The RTP is based on the per hour use of electricity
and is known as dynamic price rate as well. RTP provides information about
real cost of energy at a specific time to the consumer. Electric prices vary with
time. RTP enables the user to adjust their working hours from on peak hours
to off peak hours, resulting in saving. RTP is enabled using SM that allows the
exchange of information between utility and consumer.

30
Price (cent/kWh)

RTP
20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Time (hour)

Fig. 2. Pricing signal

Figure 3 is displaying the power consumption pattern for unscheduled scheme

and scheduled schemes i.e., CSA, Jaya and BCS. X-axis is showing the time in
hour and y-axis is showing load per hour in kWh. It is note-able that all three
scheduling schemes are performing better load distribution as compared to non
scheduled. It is evident from the figure that proposed BCS is scheduling the
 Minimizing Cost Using BCS 91

10
Unscheduled

Per hour load (kWh)

Jaya
8 CSA
BCS
6

0
1 2 3 4 5 6 7 8 9 101112131415161718192021222324
Time (hour)

Fig. 3. Per hour load

300
Unscheduled
Per hour cost (cent)

Jaya
CSA
200 BCS

100

0
1 2 3 4 5 6 7 8 9 101112131415161718192021222324
Time (hour)

Fig. 4. Hourly cost

energy consumption in an proficient way. While using BCS load during the last
four hours is maximum as a result of reducing load during peak hours.
Figure 4 shows the hourly cost for Jaya, CSA and our proposed scheme BCS.
It is visible from plot that BCS is performing better than competitors while
shifting the load from on peak hours to off peak hours. Moreover, we found that
even Jaya and CSA are performing better than unscheduled.
The PAR performance of Jaya, CSA and BCS is displayed in Fig. 5. X-axis
is showing bars for Unscheduled, Jaya, CSA and BCS where y-axis is showing
the value of PAR. The graph is depicting that the performance of our proposed
scheme BCS is not optimal in case of Peak to Average Ratio. Here, CSA outper-
formed all other schemes. The PAR is reduced to 53.64%, 52.51% and 39.64%
by CSA, Jaya and BCS respectively. Here, CSA outperformed all other schemes.
It is worth mentioning that all schemes are performing better than unscheduled.
Waiting time of a consumer is displayed in Fig. 6. The consumer waiting time
is actually depicting the user satisfaction level. In current scheme, user comfort
is measured in terms of time a user will wait for a particular appliance to turn
on. The wait is discomfort in real. If waiting time is high, it means discomfort is
high. So, the waiting time and user comfort are indirectly proportional to each
other. The calculated waiting time is 5.267235 h, 5.264078 h and 5.953346 h for
Jaya, CSA and BCS respectively. It is evident that CSA outperformed the other
techniques in the case of user comfort.
92 R. A. Abbasi et al.

PAR
3

0
Unscheduled Jaya CSA BCS

Fig. 5. Peak to average ratio

5
Waiting time (hour)

0
Jaya CSA BCS

Fig. 6. Waiting time

1500
Total cost (cent)

1000

500

0
Unscheduled Jaya CSA BCS

Fig. 7. Total cost

In Fig. 7, total electricity cost for 24 h is displayed. The plot is displaying the
cost calculated by the use of different schemes. The unscheduled cost is 1410.98
cents, where, Jaya, CSA and BCS costs 1137.7832 cents, 1157.3978 cents and
1108.6057 cents respectively. The cost is reduced by all schemes as compared
to the unscheduled. However, The calculated values are depicting that proposed
scheme BCS outperformed all other techniques. Electricity cost is reduced by
19.94%, 18.95% and 24.58% from Jaya, CSA and JSCA techniques respectively
as compared to the unscheduled. From graphs we can see that their is a trade-off
between PAR, user comfort, i.e., waiting time and cost. The proposed scheme
minimizes the cost but is compromising PAR and user comfort.
 Minimizing Cost Using BCS 93

7 Conclusion
In this proposed solution we focused on DSM for minimizing the total electricity
cost. A new load shifting strategy for appliance scheduling is proposed, that
works at EMC level in a SG. It works on scheduling different appliances used
in houses, considering their usage patterns. We scheduled appliances for a single
day. We used CSA, Jaya and our proposed BCS for scheduling appliances. By
comparing the results of different schemes used for simulations, it is evident, that
BCS performed best as compared to CSA and Jaya in terms of total electricity
cost. Electricity cost is reduced by 24.58% using BCS. However, there is a trade-
off between different parameters, like PAR, consumer waiting time and cost.

References
1. U.S. Energy Information Administration, Annual Energy Outlook 2018, pp. 121,
122, Residential and commercial energy consumption grows gradually from 2017 to
2050. https://www.eia.gov/outlooks/aeo/pdf/AEO2018.pdf. Accessed Oct 2018
2. Luo, F., Ranzi, G., Wan, C., Xu, Z., Dong, Z.Y.: A multi-stage home energy
management system with residential photovoltaic penetration. IEEE Trans. Ind.
Inform. 15(1), 116–126 (2018)
3. Gungor, V.C., Sahin, D., Kocak, T., Ergut, S., Buccella, C., Cecati, C., Hancke,
G.P.: Smart grid and smart homes: key players and pilot projects. IEEE Ind.
Electron. Mag. 6(4), 18–34 (2012)
4. Huang, H., Cai, Y., Xu, H., Yu, H.: A multi-agent minority-game- based demand-
response management of smart buildings towards peak load reduction. IEEE Trans.
Comput.-Aided Des. Integr. Circuits Syst. 36(4), 573–585 (2017)
5. Ruelens, F., Claessens, B.J., Vandael, S., De Schutter, B., Babuška, R., Belmans,
R.: Residential demand response of thermostatically controlled loads using batch
reinforcement learning. IEEE Trans. Smart Grid 8(5), 2149–2159 (2017)
6. Pourmousavi, S.A., Patrick, S.N., Nehrir, M.H.: Real-time demand response
through aggregate electric water heaters for load shifting and balancing wind gen-
eration. IEEE Trans. Smart Grid 5(2), 769–778 (2014)
7. Sakurama, K., Miura, M.: Communication-based decentralized demand response
for smart microgrids. IEEE Trans. Ind. Electron. 64(6), 5192–5202 (2017)
8. Jindal, A., Dua, A., Kaur, K., Singh, M., Kumar, N., Mishra, S.: Decision tree
and SVM-based data analytics for theft detection in smart grid. IEEE Trans. Ind.
Informat. 12(3), 1005–1016 (2016)
9. Mirowski, P., Chen, S., Kam Ho, T., Yu, C.N.: Demand forecasting in smart grids.
Bell Labs Tech. J. 18(4), 135–158 (2014)
10. Wu, Z., Zhao, T., He, L., Shen, X.: Smart grid meter analytics for revenue pro-
tection. In: Proceedings of the IEEE International Conference on Power System
Technology, pp. 782–787 (2014)
11. Jindal, A., Kumar, N., Singh, M.: A data analytical approach using support vector
machine for demand response management in smart grid. In: 2016 IEEE Power
and Energy Society General Meeting (PESGM) 17–21 July, pp. 1–5 (2016)
12. Misiti, M., Misiti, Y., Oppenheim, G., Poggi, J.-M.: Optimized clusters for disag-
gregated electricity load forecasting. REVSTAT 8, 105–124 (2010)
94 R. A. Abbasi et al.

13. Alzate, C., Sinn, M.: Improved electricity load forecasting via kernel spectral clus-
tering of smart meters. In IEEE 13th International Conference on Data Mining
(ICDM), pp. 943–948 (2013)
14. Wijaya, T.K., Vasirani, M., Humeau, S., Aberer, K.: Cluster-based aggregate fore-
casting for residential electricity demand using smart meter data. In: IEEE Inter-
national Conference on Big Data (Big Data)
15. Jindal, A., Singh, M., Kumar, N.: Consumption-aware data analytical demand
response scheme for peak load reduction in smart grid. IEEE Trans. Ind. Electron.
65(11), 8993–9004 (2018)
16. Moon, J., Kim, K.-H., Kim, Y., Hwang, E.: A short-term electric load forecasting
scheme using 2-stage predictive analytics. In: 2018 IEEE International Conference
on Big Data and Smart Computing (2018)
17. Keles, D., Scelle, J., Paraschiv, F., Fichtner, W.: Extended forecast methods for
day-ahead electricity spot prices applying artificial neural networks. Appl. Energy
162, 218–230 (2016)
18. Erol-Kantarci, M., Mouftah, H.T.: Energy-efficient information and communica-
tion infrastructures in the smart grid: a survey on interactions and open issues.
IEEE Commun. Surv. Tutor. 17(1), 179–197 (2015)
19. Huang, D., Zareipour, H., Rosehart, W.D., Amjady, N.: Data mining for electricity
price classification and the application to demand-side management. IEEE Trans.
Smart Grid 3(2), 808–817 (2012)
20. Fleury, A., Vacher, M., Noury, N.: SVM-based multimodal classification of activ-
ities of daily living in health smart homes: sensors, algorithms, and first experi-
mental results. IEEE Trans. Inform. Technol. Biomed. 14(2), 274–283 (2010)
21. Zhu, Z., Tang, J., Lambotharan, S., Chin, W.H., Fan, Z.: An integer linear pro-
gramming based optimization for home demand-side management in smart grid.
In: 2012 IEEE PES Innovative Smart Grid Technologies (ISGT) (2012)
22. Maa, K., Hua, S., Yanga, J., Xua, X., Guanb, X.: Appliances scheduling via cooper-
ative multi-swarm PSO underday-ahead prices and photovoltaic generation. Appl.
Soft Comput. 62, 504–513 (2018)
23. Bazydlo, G., Wermiński, S.: Demand side management through home area network
systems. Electr. Power Eng. Syst. 97, 174–185 (2018)
24. Hong, S.H., Yu, M., Huang, X.: A real-time demand response algorithm for het-
erogeneous devices in buildings and homes. Energy 80, 123–132 (2015)
25. Logenthiran, T., Srinivasan, D., Shun, T.Z.: Demand side management in smart
grid using heuristic optimization. IEEE Trans. Smart Grid 3, 1244–1252 (2012)
 Optimal Power Flow with Uncertain
Renewable Energy Sources Using Flower
Pollination Algorithm

Muhammad Abdullah1 , Nadeem Javaid1(B) , Inam Ullah Khan2 ,

Zahoor Ali Khan3 , Annas Chand4 , and Noman Ahmad1
1
COMSATS University Islamabad, Islamabad 44000, Pakistan
[email protected]
2
COMSATS University Islamabad, Lahore 14000, Pakistan
3
Computer Information Science, Higher Colleges of Technology, Fujairah 4114, UAE
4
COMSATS University Islamabad, Abbottabad 22010, Pakistan
http://www.njavaid.com

Abstract. Optimal power flow (OPF) problem has become more signif-
icant for operation and planning of electrical power systems because of
the increasing energy demand. OPF is very important for system opera-
tors to fulfill the electricity demand of the consumers efficiently and for
the reliable operation of the power system. The key objective in OPF is to
reduce the total generating cost while assuring the system limitations.
Due to environmental emission, depletion of fossil fuels and its higher
prices, integration of renewable energy sources into the grid is essential.
Classical OPF, which consider only thermal generators is a non-convex,
non-linear optimization problem. However, incorporating the uncertain
renewable sources adds complexity to the problem. A metaheuristic algo-
rithm which solves the OPF problem with renewable energy sources is
to be implemented on a modified IEEE 30-bus system.

1 Introduction
The optimal power flow (OPF) problem formulation is important for planning
and operation of electrical power system. In literature, three types of problems
are generally referred to the power system; load flow, economic dispatch (ED),
and OPF. Load flow or power flow represents the generation, transmission net-
work, and load equations. In Load flow technique the reactive power limits,
security and contingency constraints are not considered therefore the solution
obtained from load flow analysis may not be optimal or physically feasible. In
ED formulation, the key objective is to minimize the generation cost by fulfilling
the power demand. However, ED formulation ignores or simplify the power flow
constraints [1]. The solution of OPF starts with load flow equation. The differ-
ence between OPF and load flow is that in OPF, power flow calculation is made
while system limitation and different objectives like cost, voltage profile improve-
ment etc. are also considered. Therefore, it is very important for the safe and
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 95–107, 2020.
https://doi.org/10.1007/978-3-030-15032-7_8
96 M. Abdullah et al.

pecuniary operation of the electrical power system by identifying the appropiate

settings of control parameters like voltages, the active power of generator buses,
transformer taps setting etc. The key purpose of the OPF is to optimize (min-
imize or maximize) a selected objective (cost, power loss, voltage stability etc.)
while fulfilling the equality, inequality and security restraints. The equality con-
straint consists of power flow equations. The inequality constraints comprise of
voltages, real and reactive power limitsFeb@2019 and transmission line capac-
ity [2,3]. Limited reserves of fossil fuels and environmental protection issues
have raised the concern to integrate renewable energy resources (RESs) into the
grid. Wind power plants and solar photovoltaic are amongst the most common
RESs. A lot of research work has been done on classical OPF, which consider
only thermal generators. Classical OPF which is by itself highly non-linear and
non-convex optimization problem because of different inequalities constraints,
however by incorporating uncertain RES further increase the complexity of the
problem [4]. Various optimization techniques and algorithms have been evolved
in a couple of decades to solve OPF problems. These techniques can be divided
into classical numerical analysis and modern metaheuristics algorithms. The
classical method includes quadratic programming, linear programming, interior
point, and dynamic programming etc. These techniques are inadequate for large-
scale problems because they comprise complex, lengthy calculation and they are
excessively dependent on initial guess values. In the past few years, different
metaheuristic algorithms have been developed. To overcome the deficiencies of
the classical methods, some of these algorithms have been successfully imple-
mented for solution OPF problems. Examples of some of the algorithms are;
moth-swarm algorithm (MSA), whale optimization algorithm (WOA), elephant
herding optimization (EHO) technique etc. In any case, due to the inconsistency
of the OPF solution for different objectives, no heuristic algorithm can be best
for solving all these objectives, Hence, there is always a room for new techniques
and algorithms that can effectively solve many of the objectives of OPF [4,5].
Increasing the participation of consumers in the electrical network operations
and accommodation of RESs are the most important features of smart grids. The
role of OPF in smart grids is very vital in order to decrease the transmission,
distribution losses, emission of fossil fuels, and the energy generation cost. Sub-
sequently, decreasing the electrical power consumption and prices, concurrently,
enabling consumers and electrical utilities to regulate the demand [6–10].

2 Problem Formulation

The OPF problem is formulated to maximize or minimize a specified objec-

tive function by attaining the settings of control variable while satisfying the
constraints of the electrical network. The OPF problem can be mathematically
described as:
Minimize:
fobj (x́, v́), (1)
 OPF with Renewable Energy Sources 97

subject to:
g(x́, v́) = 0, (2)

h(x́, v́) ≤ 0, (3)

where, v́ represents the control variables or independent variable and x́ denotes
the state (dependent) variables. The independent variables include the real power
of all the generator buses except the slack bus and all the generator bus voltages.
The control variables can be adjusted while satisfying the load flow equations.
The state or dependent variable describe the state of the system [11].

Control variables
PGb Real power at the generator buses excluding the slack bus.
VGb Generator buses voltage magnitude.
State variables
P G1 The Slack bus real power.
VL The load buses voltage.
QG Reactive power at the generator bus.
SI Line loading

3 RESs Uncertainty Model

3.1 Wind Generator Power Model
The output power of the wind generating units is determined by the wind speed.
The probability of the wind speed follows Weibull probability distribution func-
tion (PDF) [12–16]. The PDF of wind speed is expressed as:
 α   υ (α−1)
fwv (υ) = e−(υ/λ) f or 0 < υ < ∞, (4)
λ λ
where, α and λ represent the shape and scale parameter of Weibull distribution
respectively. The mean of the Weibull PDF is expressed as:

Mw = λ ∗ Γ (1 + α−1 ), (5)

where, Γ function is described as:

∞
Γ (x́) = e−t tx́−1 dt. (6)
0

The frequency distribution of wind speed and Weibull fitting are attained
by simulating 9000 monte-carlo scenarios, which is shown in Fig. 1. The wind
98 M. Abdullah et al.

Fig. 1. Wind speed probability distribution at bus 5.

farms are connected at bus 5 and 13 of IEEE 30-bus system. The output of wind
generators connected at bus 5 has cumulative rated power of 75 MW, consisting
of 25 wind generators, each having the rated power of 3 MW. Similarly, 15 wind
generators are connected in wind farm at bus number 13, of each having 3 MW
rated power. The output power of wind generators as a function of wind speed
can be formulated as:
⎧
⎪
⎨ 0  f or v ≤ vin and v > vout
v−vin
PW G = PW r v−vout f or vin ≤ v ≤ vr (7)
⎪
⎩
PW f or vr ≤ v ≤ vout ,

where, vin is cut-in wind speed, vr is rated wind speed, and vout is cut-out wind
speed. In this study, the cut-in, cut-out and rated wind speed are taken as 3
m/s, 25 m/s, and 16 m/s, respectively. From Eq. 7, it can be observed that the
output power from a wind generator is discrete in some region. The out power
of a wind farm is zero for wind speed less than cut-in speed or exceeding cut-
out speed. The rated power is produced by wind generators between cut-in and
cut-out wind speed. The probability of these discrete zones can be expressed as:
vin k vout k
f(PW G ) = 1 − exp[−( ) ] + exp[−( ) ] f or (PW G = 0), (8)
c c
vr k vout k
f(PW G ) = exp[−( ) ] − exp[−( ) ] f or (PW G = PW R ). (9)
c c
 OPF with Renewable Energy Sources 99

The wind power probability for the continuous region is given by:

k(vr − vin ) PW G k−1

f(PW G ) = [vin + (vr − vin )]
c ∗ Pwr
k Pwr
k (10)
vin + PW G
Pwr (vr − vin )
exp −( ) .
c

Fig. 2. Solar irradiance probability at bus 11.

3.2 Solar Photovoltaic Power Model

The solar photovoltaic (PV) output power depends on solar irradiance (G),
which obeys lognormal PDF [17–19]. The thermal generator at bus number 13
in IEEE 30-bus system is substituted with solar PV. The probability of solar
irradiance with mean and standard deviation can be described as:
1 −(ln x − μ2 )
fP V (G) = √ exp[ ] f or G > 0. (11)
Gσ 2π 2σ2

Lognormal distribution mean (MLgn ) is given by:

σ2
MLgn = exp(μ + ). (12)
2
100 M. Abdullah et al.

Frequency distribution of solar irradiance and lognormal fitting is shown in Fig. 2,

which is attained by simulating monte-carlo scenarios with 9000 samples size.
The output power of solar PV as a function of solar irradiance is formulated as:
⎧  
⎨PP V r G2 f or 0 ≤ G ≤ RC
PP V (G) =  std 
G
(13)
⎩PP V r G f or G ≥ RC ,
Gstd

where, Gstd represents the solar irradiance in the standard environment and Rc
represents the certain irradiance. The value of Gstd is assumed 800 W/m2 and
that of Rc is 120 W/m2 . The rated output power PP V r is 60 MW for solar PV
module at bus 11.

3.3 Hydro Generation Power Model

The hydro generators output power depends on the water flow rate (Qh ) and
pressure head (Hh ) [20–23]. The water flow rate follows Gumbel probability
distribution. The Gumbel PDF for water flow rate with scale parameter γ and
location parameter λ can be mathematically formulated as:

1 Qh − λ Qh − λ
fH (Qh ) = exp exp[− exp( ). (14)
γ λ λ
In the modified IEEE 30-bus system the thermal generator at bus number 13 is
replaced with 45 MW wind and 5 MW small hydropower unit. The Weibull fitting
and wind speed frequency distribution for wind power units and the frequency
distribution of water flow rate and Gumbel fitting is shown in Figs. 3 and 4. Both
these figures are obtained by simulating 9000 monte-carlo scenarios. The values
of all these parameters are realistically chosen and many of them are almost
same as provided in Ref. [12]. The output hydropower as a function of water
flow rate and pressure head can be described as:
PH (Qh ) = ηρgQh Hh , (15)
where, η is the efficiency of the generating unit and assumed as 0.85, ρ is the
density of water volume and taken as 1000 kg/m2 , g represents the value of
gravitational acceleration, Qh is the water flow rate, and Hh is the pressure
head of water across the turbine.

4 Objective Function
Objective function considered in this study is total generation cost minimization,
which consist of thermal generators and RESs cost functions.
Thermal generators cost model considering valve loading effect is expressed
as [12]:

N TG

CT G (PT G ) = ái + b́i + ći PT Gi

i=1 (16)
 
´ 
+ di ∗ sin(éi ∗ (P min
T Gi − PT G ) ,
i
 OPF with Renewable Energy Sources 101

where, NT G is the number of thermal generators. ái , b́i , and ći are the cost
coefficient of ith generator, producing output power PT Gi , d´i and éi are the
coefficients of valve point loading effect.
Generation from RESs does not require any fuel source for its operation.
However, if RESs are operated by private bodies, the independent system oper-
ator (ISO) must pay them according to the contractually agreed scheduled power
[23].
The direct cost function of the wind farm, solar PV, and combined wind and
hydro, in terms of scheduled power is described as:

CW (PW ) = gw PW , (17)

CP V (PP V ) = hpv PP V , (18)

CW H (PW H ) = gwh PW H = gw PW H,w + gh PW H,h , (19)
where, PW , PP V , and PW H are the scheduled output power from wind farm, solar
PV, and combined wind and hydro units. PW H,w and PW H,h are the electrical
energy contribution from the wind and hydro units. gw , hpv , and gwh are the
direct cost coefficient of these energy sources.
Due to stochastic nature of these RESs, the actual power produced may be
less or more than scheduled power. Therefore, the ISO must have reserve gener-
ating capacity to fulfill the demand. Reserve cost for the wind unit is expressed
as [18]:
CWR,i (PW sh,i − PW ac,i ) = krw,i (PW sh,i − PW ac,i )
 PW sh,i (20)
= krw,i (PW sh,i − pw,i )fw (pw,i )dpw,i .
0
In the case when the output power of the wind generators is greater then the
scheduled power, the ISOs pay penalty cost if the surplus power is not utilized
by them by reducing the power of thermal generators.

CWR,i (PW ac,i − PW sh,i ) = kpw,i (PW ac,i − PW sh,i )

 PW r,i
(21)
= kpw,i (pw,i − PW sh,i ) fw (pw,i )dpw,i ,
PW sh,i

where, PW sh,i is the scheduled power, PW ac,i is the available power from wind
plant, PW r,i is the rated power, and fw (pw,i ) is the probability density function
of wind power. Similarly, the reserve and penalty cost of solar (CP VR,i , CP VP,i ),
and combined wind and hydro (CW HR,i , CW HP,i ) can be calculated. The values
of probability distribution parameters and cost coefficients for thermal genera-
tors and RESs are almost the same as provided in [12]. The commulative cost
function can be mathematically expressed as:

CT otal = CT G (PT G ) + [CW (PW ) + CWR (PW sh,i − PW ac,i )

+CWP (PW ac,i − PW sh,i )] + [CP V (PP V ) + CP VR (PP V sh,i (22)
−PP V ac,i ) + CP VP (PP V ac,i − PP V sh,i )]
102 M. Abdullah et al.

5 Algorithms
The flower pollination algorithm (FPA) is one of the new meta-heuristic algo-
rithm that is inspired by the natural phenomena of pollination in flower. FPA is
used for the solution of diverse optimization problems. Since its inception, more
than 500 papers have been published on the use of FPA in different domain [24–
30]. In Ref. [30] the authors have mathematically proved the global convergence
capabilities of FPA using Markov chain theory. In this study, the FPA is used for
OPF problem incorporated with renewable energy sources. The FPA performed
better then SHADE-SF which is recently proposed in Ref. [12], in terms of the
total generation cost minimization.

Fig. 3. Wind speed probability distribution at bus 13.

6 Application and Results

It is essential to integrate clean and economical renewable energy sources into

the electrical grids to reduce environmental emission. A model is proposed which
consists of thermal generators along with RESs, to meet the energy demand
efficiently by OPF formulation. The proposed model is implemented on the IEEE
standard 30-bus test system. In IEEE 30-bus system there are total 30 buses and
41 branches. In these 30 buses, 6 are generator buses and 24 are load buses. The
 OPF with Renewable Energy Sources 103

Fig. 4. Distribution of river flow rate at bus 13 for small hydro.

Fig. 5. Convergence graph for total generation cost minimization.

104 M. Abdullah et al.

Table 1. IEEE 30-bus test system main charecteristics

Item Quantity Details

Buses 30
Branches 41
Generators buses 6 Slack bus at bus number 1 and other generator
buses at bus No. 2, 3, 5, 8, 11, 13
Load buses 24
Control variables 11 Real Power of generator buses except for slack
bus and voltages of all generator buses

Table 2. Values of control parameters for total generation cost minimization

Parameters Min Max SHADE-SF FPA

PT G1 (M W ) 50 140 134.90791 134.90791
PT G2 (M W ) 20 80 25.7995 25.7995
PW5 (M W ) 0 75 42.079 42.079
PT G8 (M W ) 10 35 10 10
PP V11 (M W ) 0 60 35.6326 35.6326
P(W +H)13 (M W ) 0 50 40.506 40.506
QT G1 (M V Ar) −20 150 −11.1306 −11.1306
QT G2 (M V Ar) −20 60 17.61549 17.61549
QW5 (M V Ar) −30 35 24.97633 24.97633
QT G8 (M V Ar) −15 40 40 40
QP V11 (M V Ar) −20 25 18.94608 18.94608
Q(W +H)13 (M V Ar) −25 30 22.28046 22.28046
VT G1 (p.u) 0.95 1.1 1.1 1.1
VT G2 (p.u) 0.95 1.1 1.0888 1.0888
VW5 (p.u) 0.95 1.1 1.0693 1.0693
VT G8 (p.u) 0.95 1.1 1.0872 1.0872
VP V11 (p.u) 0.95 1.1 1.1 1.1
V(W +H)13 (p.u) 0.95 1.1 1.0965 1.0965
Ploss (M W ) – – 5.8219 MW 5.525
Total cost ($/hr) – – 778.3248 777.3298

generator buses generally consist of thermal generators, these are connected at

bus number 1, 2, 5, 8, 11, 13. In the proposed scheme the 30-bus system is
modified as; at bus number 1, 2, 8 thermal generators, at bus number 5 wind
generator, at bus-11 solar photovoltaic, and at bus number 13 wind and run of
river small hydro generator are connected. Summary of IEEE 30-bus system is
presented in Table 1.
 OPF with Renewable Energy Sources 105

The total generating cost is minimized, considering all thermal generators

and RESs. The valve point effect is also considered for thermal generators. The
convergence performance of FPA and SHADE-SF is illustrated in Fig. 5. IEEE
30-bus system with three thermal power units at bus number 1, 2, and 8, two
wind generating units at bus number 5 and 11, and one solar photovoltaic at
bus number 13 is used as a test system. The real power of thermal, solar and
wind generators, and voltages of generator buses are the control variables. The
values of the voltage limits of PV buses, the real power of generating buses,
the objective function value, power loss, emission, and other parameters are
shown in Table 2. Five thousand iterations (generation) were run to explore the
performance of FPA and SHADE-SF. The convergence graph becomes nearly
straight after thousands iteration. The total cost in case of FPA was 781.2109
($/hr) and that of SHADE-SF was 782.4803 ($/hr). The FPA performs better
in terms of total generation cost minimization then SHADE-SF algorithm.

7 Conclusion

In this paper, the usefulness and application of FPA to solve OPF problem
incorporated with stochastic nature wind power, solar PV, and small hydro
generation in the power system is presented. Uncertain wind, solar PV, and
hydropower are modeled with Weibull PDF, lognormal PDF, and Gumbel PDF,
respectively. IEEE 30-bus system is adapted to compare the effectiveness of
SHADE-SF and FPA by considering the total generation cost minimization. It is
observed from simulation results that FPA effectively minimized total generation
cost in the system. Therefore it can be concluded that FPA outperforms The
SHADE-SF in terms of total generation cost minimization.

References
1. Mohamed, A.-A.A., et al.: Optimal power flow using moth swarm algorithm. Electr.
Power Syst. Res. 142, 190–206 (2017)
2. Duman, S.: A modified moth swarm algorithm based on an arithmetic crossover
for constrained optimization and optimal power flow problems. IEEE Access 6,
45394 (2018)
3. Reddy, S.S.: Optimal power flow using hybrid differential evolution and harmony
search algorithm. Int. J. Mach. Learn. Cybern., 1–15 (2018)
4. Attia, A.-F., El Sehiemy, R.A., Hasanien, H.M.: Optimal power flow solution in
power systems using a novel Sine-Cosine algorithm. Int. J. Electr. Power Energy
Syst. 99, 331–343 (2018)
5. Bai, W., Eke, I., Lee, K.Y.: An improved artificial bee colony optimization algo-
rithm based on orthogonal learning for optimal power flow problem. Control Eng.
Pract. 61, 163–172 (2017)
6. Javaid, N., et al.: Energy efficient integration of renewable energy sources in the
smart grid for demand side management. IEEE Access 6, 77077–77096 (2018)
7. Javaid, N., et al.: An intelligent load management system with renewable energy
integration for smart homes. IEEE Access 5, 13587–13600 (2017)
106 M. Abdullah et al.

8. Khan, M., et al.: Game theoretical demand response management and short-term
load forecasting by knowledge based systems on the basis of priority index. Elec-
tronics 7(12), 431 (2018)
9. Awais, M., et al.: Towards effective and efficient energy management of single home
and a smart community exploiting heuristic optimization algorithms with critical
peak and real-time pricing tariffs in smart grids. Energies 11(11), 3125 (2018)
10. Ahmad, A., et al.: An optimized home energy management system with integrated
renewable energy and storage resources. Energies 10(4), 549 (2017)
11. Bouchekara, H.R., Abido, M.A., Chaib, A.E., et al.: Optimal power flow using the
league championship algorithm: a case study of the Algerian power system. Energy
Convers. Manage. 87, 58–70 (2014)
12. Biswas, P.P., Suganthan, P.N., Amaratunga, G.A.: Optimal power flow solutions
incorporating stochastic wind and solar power. Energy Convers. Manage. 148,
1194–1207 (2017)
13. Roy, R., Jadhav, H.T.: Optimal power flow solution of power system incorporating
stochastic wind power using Gbest guided artificial bee colony algorithm. Int. J.
Electr. Power Energy Syst. 64, 562–578 (2015)
14. Panda, A., Tripathy, M.: Optimal power flow solution of wind integrated power
system using modified bacteria foraging algorithm. Int. J. Electr. Power Energy
Syst. 54, 306–314 (2014)
15. Panda, A., Tripathy, M.: Security constrained optimal power flow solution of wind-
thermal generation system using modified bacteria foraging algorithm. Energy 93,
816–827 (2015)
16. Shi, L., et al.: Optimal power flow solution incorporating wind power. IEEE Syst.
J. 6(2), 233–241 (2012)
17. Chang, T.P.: Investigation on frequency distribution of global radiation using dif-
ferent probability density functions. Int. J. Appl. Sci. Eng. 8(2), 99–107 (2010)
18. Reddy, S.S., Bijwe, P.R., Abhyankar, A.R.: Real-time economic dispatch consider-
ing renewable power generation variability and uncertainty over scheduling period.
IEEE Syst. J. 9(4), 1440–1451 (2015)
19. Reddy, S.S.: Optimal scheduling of thermal-wind-solar power system with storage.
Renewable Energy 101, 1357–1368 (2017)
20. Sharma, H., Singh, J.: Run off river plant: status and prospects. Int. J. Innov.
Technol. Exploring Eng. (IJITEE) 3(2) (2013)
21. Pandey, H.K., Dwivedi, S., Kumar, K.: Flood frequency analysis of Betwa river,
Madhya Pradesh India. J. Geol. Soc. India 92(3), 286–290 (2018)
22. Cabus, P.: River flow prediction through rainfall runoff modelling with a
probability-distributed model (PDM) in Flanders, Belgium. Agric. Water Man-
age. 95(7), 859–868 (2008)
23. Wijesinghe, A., Lai, L.L.: Small hydro power plant analysis and development. In:
2011 4th International Conference on Electric Utility Deregulation and Restruc-
turing and Power Technologies (DRPT). IEEE (2011)
24. Yang, X.-S.: Flower pollination algorithm for global optimization. In: International
Conference on Unconventional Computing and Natural Computation. Springer,
Heidelberg (2012)
25. Yang, X.-S., Karamanoglu, M., He, X.: Flower pollination algorithm: a novel app-
roach for multiobjective optimization. Eng. Optim. 46(9), 1222–1237 (2014)
26. Abdelaziz, A.Y., Ali, E.S., Abd Elazim, S.M.: Implementation of flower pollination
algorithm for solving economic load dispatch and combined economic emission
dispatch problems in power systems. Energy 101, 506–518 (2016)
 OPF with Renewable Energy Sources 107

27. Velamuri, S., Sreejith, S., Ponnambalam, P.: Static economic dispatch incorporat-
ing wind farm using flower pollination algorithm. Perspect. Sci. 8, 260–262 (2016)
28. Huang, S.-J., et al.: Application of flower pollination algorithm for placement of
distribution transformers in a low-voltage grid. In: 2015 IEEE International Con-
ference on Industrial Technology (ICIT). IEEE (2015)
29. Abdelaziz, A.Y., Ali, E.S., Abd Elazim, S.M.: Flower pollination algorithm and
loss sensitivity factors for optimal sizing and placement of capacitors in radial
distribution systems. Int. J. Electr. Power Energy Syst. 78, 207–214 (2016)
30. He, X., et al.: Global convergence analysis of the flower pollination algorithm:
a discrete-time Markov chain approach. Procedia Comput. Sci. 108, 1354–1363
(2017)
 Efficient Information Flow Control
by Reducing Meaningless Messages
in P2PPSO Systems

Shigenari Nakamura1(B) , Tomoya Enokido2 , Leonard Barolli3 ,

and Makoto Takizawa1
1
Hosei University, Tokyo, Japan
[email protected], [email protected]
2
Rissho University, Tokyo, Japan
[email protected]
3
Fukuoka Institute of Technology, Fukuoka, Japan
[email protected]

Abstract. In topic-based publish/subscribe (PS) model, each peer pro-

cess (peer) only receives an event message whose publication topics
include some common topic with the subscription topics. In a P2PPSO
(P2P (peer-to-peer) type of topic-based PS with Object concept) model,
each peer exchanges objects by publishing and receiving event messages
with no centralized coordinator. Here, objects are denoted by topics and
event messages are characterized by topics of the objects carried by the
event messages. Suppose an event message ej published by a peer pj car-
ries objects on some topics into a target peer pi . Here, objects in the peer
pj illegally flow to the peer pi if the peer pi is not allowed to subscribe
the topics. An object of a peer pi is illegal, whose topics the peer pi is
not allowed to subscribe. In this paper, we newly propose an ETOBSCO
(Efficient TOBSCO) protocol where meaningless event messages are not
published to reduce the number of event messages exchanged among
peers. In the evaluation, we show the fewer number of event messages
are delivered but longer update delay time to update replicas is taken in
the ETOBSCO protocol.

Keywords: Causally ordering delivery · Information flow control ·

P2P (peer-to-peer) · PS (publish/subscribe) model ·
TBAC (topic-based access control) model

1 Introduction
In distributed systems, peer processes (peers) are cooperating with one another
by manipulating and exchanging objects. An object is a unit of computation
resource, which is an encapsulation of data and operations to manipulate the
data. In secure information systems, every object has to be manipulated by only
users granted access rights. In the access control models [4], even if a peer is
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 108–119, 2020.
https://doi.org/10.1007/978-3-030-15032-7_9
 Efficient Information Flow Control by Reducing Meaningless Messages 109

not allowed to read data in an object oi , the peer can read the data by reading
another object oj if the data is written in the object oj [2]. Here, information
illegally flow from the object oi via the object oj to the peer. We have to prevent
illegal information flow among peers and objects in the access control models. In
order to keep information systems secure by preventing illegal information flow
among objects, types of protocols are proposed [8] based on the RBAC (role-
based access control) model [10]. On the other hand, content-based systems
like PS (publish/subscribe) systems [1,3] are getting more important in various
applications.
The TBAC (topic-based access control) model [9] is proposed to control sub-
scription and publication of topics by peers in topic-based PS systems [1,3].
Here, only a peer granted publication and subscription rights on a topic t is
allowed to publish and subscribe an event message with the topic t, respectively.
The topic sets pi .P and pi .S are sets of publication and subscription topics of
a peer pi , respectively. An event message ei published by a peer pi is received
by a target peer pj if the subscription pj .S and the publication ei .P include at
least one common topic. Here, topics in pi .S but not in pj .S are hidden in the
event message ei [9]. This means, the peer pj can receive objects related with
the hidden topics and include the data in the objects of the event message ei .
Thus, the event message ei may carry objects related on hidden topics.
In this paper, we consider the P2PPSO (P2P (peer-to-peer) [12] of topic-
based PS [11] with Object concept) model [7] based on the TBAC model. Here,
each peer can play both publisher and subscriber roles with no centralized coor-
dinator. Objects and event messages are characterized by topics. Event messages
published by a source peer carry objects to a target peer. By exchanging event
messages, objects are brought to the target peers. On receipt of an event mes-
sage with objects, a target peer holds replicas of the objects. In this paper, we
assume that only a peer which creates an object can update data in the object.
If data in an object is updated by a creator peer, every replica of the object
in another peer is also required to be updated to keep every replica mutually
consistent with the object.
Suppose a peer pi publishes an event message ei carrying an object o whose
data is related with a topic t in the P2PPSO system. An event message ei is
delivered to a target peer pj if the publication ei .P and the subscription pj .S
include at least one common topic. Suppose the event message ei is delivered
to the peer pj but the subscription pj .S does not include the topic t. Here, the
data of the object o on the topic t is delivered to the peer pj although the peer
pj is not allowed to subscribe the topic t. An object for a peer pj is illegal,
whose topics are not allowed to be subscribed by the peer pj . If the object o is
delivered to the peer pj by the event message ei , illegal objects in the peer pi
are carried to the target peer pj . In order to prevent illegal information flow,
the TOBSCO (Topics-of-Objects Based Synchronization with Causally Ordering
delivery) protocol is proposed. Here, illegal objects are not delivered to the target
peer. In addition, every event message is assumed to be causally delivered.
110 S. Nakamura et al.

In the TOBSCO protocol, if a peer updates data in its object, the peer
publishes an update event message to make other peers synchronize their replicas
with the object. Here, even if the peer updates the object, the topics of the object
may not change. Since every illegal information flow is prevented based on the
topics of each object and subscription of each peer, the update event message is
meaningless to prevent illegal information flow. In this paper, we newly propose
an ETOBSCO (Efficient TOBSCO) protocol to reduce the number of event
messages exchanged among peers by forcing peers avoid to publish meaningless
update event messages. We evaluate the TOBSCO protocol in terms of the
number of event messages delivered and the average update delay time of objects.
In the evaluation, we show the fewer number of event messages are delivered but
longer update delay time is taken in the ETOBSCO protocol than the TOBSCO
protocol.
In Sect. 2, we present the P2PPSO model, information flow relations based
on the TBAC model, and causal order. In Sect. 3, we propose the ETOBSCO
protocol. In Sect. 4, we evaluate the ETOBSCO protocol.

2 System Model

2.1 P2PPSO Model

The P2PPS (P2P [12] of PS [1,3]) model [9] is composed of a set P of peer
processes (peers) p1 , . . . , ppn (pn ≥ 1) which are cooperating with one another
by publishing and receiving event messages. Here, each peer pi can play both
publisher and subscriber roles differently from the conventional PS systems. Let
T be a set {t1 , . . . , ttn } (tn ≥ 1) of all topics in a system. In a topic-based PS
system [11], a peer pi publishes an event message ei with publication ei .P (⊆
T ). A peer pj specifies its subscription pj .S (⊆ T ) as a subset of the topics
T . An event message ei is received by a peer pj if ei .P ∩ pj .S = φ. Here, the
peers pi and pj are source and target peers of the event message ei , respectively.
Otherwise, the peer pj does not receive the event message ei .
A PS system is a content-based system [3]. In this paper, the contents of an
event message are composed of objects. A peer pi includes objects in an event
message e and then publishes the event message e. If another peer pj receives
the event message e, the objects carried by the event message e are stored in the
storage pj .D of the peer pj . Thus, objects are distributed to peers by publishing
and receiving event messages. We have to discuss whether or not a peer can
receive objects in addition to receiving each event message.
In this paper, we consider a P2PPSO (P2PPS with object concept) model.
Each peer pi is equipped with the storage pi .D where objects carried by event
messages are stored. Each event message ei carries some object in a storage pi .D.
Let e.O be a set of objects carried by an event message e. Let oii be an object
created by a peer pi . Let oji be a replica of an object oii which is held by a peer
pj , i.e. oji ∈ pj .D. We assume only a creator peer pi of an object oi can update
data in the object oi . A set oji .T of each object oji indicates topics of the data
 Efficient Information Flow Control by Reducing Meaningless Messages 111

included in the object oji . Here, e.T shows a set of topics of the objects which
the event message e carries, i.e. e.T = ∪o∈e.O o.T .

2.2 Information Flow

In the P2PPSO system, event messages carry objects to target peers. In order
to check if each peer can take objects carried by event messages, the peer has
to keep in record the objects and compare the objects with objects kept in
the peer. However, it is not easy for each peer to hold every object carried by
event messages due to computation and storage overheads. Each object oji is
characterized in terms of topics and has a variable oji .T . Topics in the variable
oji .T indicate topics of the object oji . Each event message ej has a variable ej .T
which shows topics of objects of ej . A set pi .S indicates topics which the peer
pi is allowed to subscribe.
We define information flow relations on objects and topics based on the
TBAC model [9].

Definition 1. Let ei be an event message published by a peer pi and pj be a

peer:

1. ei flows to pj (ei → pj ) iff (if and only if) ei .O = φ and ei .P ∩ pj .S = φ.

2. ei legally flows to pj (ei ⇒ pj ) iff ei → pj and ei .T ⊆ pj .S.
3. ei illegally flows to pj (ei
→ pj ) iff ei → pj and ei .T ⊆pj .S.

If an event message ei published by the peer pi flows to a peer pj (ei → pj ),

the event message ei can be received by the peer pj . Here, some information, i.e.
objects obtained by the peer pi flow into the peer pj . Otherwise, no information
from the peer pi flows into the peer pj because the event message ei is not
received by the peer pj .
The condition “ei .T ⊆ pj .S” shows that every topic in the variable ei .T
is also in the subscription pj .S. This means, the event message ei carries no
object on topics which the target peer pj is not allowed to subscribe. Hence, no
information illegally flows into the peer pj by receiving the event message ei .
The condition “ei .T ⊆pj .S” means that the event message ei carries objects
on topics into the target peer pj , which the target peer pj is not allowed to
subscribe.
An event message carries objects in a source peer pi to a target peer pj . The
target peer pj has to decide if the objects can be delivered to the peer pj . We
define legal and illegal objects in terms of topics.

Definition 2. Let oi be an object in an event message ei published by a peer

pi . Let pj be a target peer of ei :
o
1. oi is legal at pj (oi ⇒ pj ) iff oi ∈ ei .O, ei → pj , and oi .T ⊆ pj .S.
o
2. oi is illegal at pj (oi
→ pj ) iff oi ∈ ei .O, ei → pj , and oi .T ⊆pj .S.
112 S. Nakamura et al.

3 Protocols to Prevent Illegal Information Flow

3.1 TOBSCO Protocol

The illegal information flow relation (

→) on peers and objects is defined based on
the TBAC model. Information illegally flows to a target peer if an event message
carries at least one illegal object into the target peer. A TOBS (Topics-of-Objects
Based Synchronization) protocol is proposed to prevent illegal information flow
in the P2PPSO system [7].
In the P2PPSO system, a peer pi is allowed to publish and subscribe a topic
t in the publication pi .P and subscription pi .S, respectively. Let e.T show a set
of topics carried by the event message e. In the TOBS protocol, on receipt of
an event message ei , the target peer pj checks whether or not the event message
ei includes illegal objects. Illegal objects are not delivered to the target peer pj
while legal objects are delivered to the target peer pj . If a peer pi updates data
in an object oii , the peer pi publishes an update event message uei to make every
replica oji of the object oii mutually consistent with the object oii . In this paper,
we assume only the creator peer pi of the object oii can update the object oii . If
an object oii is updated by a creator peer pi , every replica oji has to be updated.
[TOBS protocol]

1. A peer pi publishes an event message ei :

a. ei .O = objects or replicas included in the event message ei from the
storage pi .D, i.e. ei .O ⊆ pi .D;
ei .T = {t | t ∈ oi .T and oi ∈ ei .O};
ei .P = publication topics of ei , i.e. ei .P ⊆ pi .P ;
b. pi publishes the event message ei ;
2. A peer pi receives an event message ej from a peer pj , i.e. “ej → pi ” holds:
a. if ej ⇒ pi , the event message ej is delivered to the peer pi and
i. if ej is an update event message uej , every replica oij in uej .O is
updated, i.e. oij = ojj ;
ii. otherwise,
A. if ojk ∈ ej .O, every replica oik (k = i) is updated, i.e. oik = ojk ;
B. if ojm ∈ ej .O and oim ∈pi .D, every replica ojm (m = i) is added to
the storage pi .D;
b. else
i. if ej = uej , /* update event message */
o
A. if ojj ⇒ pi , every replica oij is updated, i.e. oij = ojj ;
j o
B. if oj
→ pi , every replica oij is deleted from pi .D;
ii. otherwise,
o
A. if ojk ⇒ pi , every replica oik (k = i) is updated, i.e. oik = ojk ;
o
B. if ojk
→ pi , every replica oik (k = i) is deleted from pi .D;
o
C. every replica ojm (m = i) is added to pi .D such that ojm ⇒ pi and
oim ∈pi .D;
 Efficient Information Flow Control by Reducing Meaningless Messages 113

3. A peer pi updates data in an object oii in the storage pi .D:

a. pi makes an update event message uei where uei .O = {oii | oii is being
updated by pi }, uei .T = {t | t ∈ oii .T and oii ∈ uei .O}, and uei .P = uei .T ;
b. pi updates data in an object oii where oii .T ⊆ pi .S;
c. uei .O = {oii | oii is an updated object};
uei .T = {t | t ∈ oii .T and oii ∈ uei .O};
d. pi publishes an update event message uei ;
In the TOBS protocol, every event message is assumed to be causally deliv-
ered to every common target peer in the underlying network. However, in the
P2PPSO system where there is no centralized coordinator, some pair of event
messages may be received by different common target peers not in the same
order. According to the causality theory [5], an event message e1 causally pre-
cedes another event message e2 (e1 →c e2 ) iff the publication event of e1 happens
before e2 . Hence, if e1 →c e2 , e1 has to be delivered before e2 in the peer pj even
if the peer pj receives the event message e1 after e2 .
In the TOBSCO (TOBS with causally ordering delivery) protocol, each event
message is causally delivered to every target peer. Here, we assume the under-
lying network provides every pair of peers with reliable communication service,
i.e. every peer receives event messages in the sending order without message
loss and duplication. However, event messages published by different peers may
be received by different target peers in different orders. Hence, event messages
have to be causally delivered to target peers. The causally ordering broadcast
(CO) protocol [6] is proposed where vectors of sequence numbers of messages
are used. We also assume each event message is broadcast to every peer. In this
paper, ei .SEQ denotes the sequence number of an event message ei published
by a peer pi . Let ei .ACKj be a sequence number of an event message ei which
the peer pi expects to receive next from a peer pj (j = 1, . . ., pn). Each peer
pi also obtains a sequence number pi .SEQ which the peer pi expects to publish
next. Let pi .REQj be a sequence number of an event message which the peer pi
expects to receive next from a peer pj (j = 1, . . ., pn). A peer pi manipulates a
pn × pn matrix pi .AL. Each element pi .ALk,j shows a sequence number of an
event message which the peer pi knows that a peer pj expects to receive next
from a peer pk (j, k = 1, . . . , pn). min(pi .ALk ) is a minimum one of pi .ALk,1 ,
. . ., pi .ALk,pn . The sequence numbers are manipulated as follows:
[CO protocol]
1. A peer pi publishes an event message ei :
ei .SEQ = pi .SEQ;
pi .SEQ = pi .SEQ + 1;
ei .ACKj = pi .REQj (j = 1, . . ., pn);
2. A peer pi receives an event message ej from a peer pj :
pi .REQj = ej .SEQ + 1;
pi .ALk,j = ej .ACKk (k = 1, . . ., pn);
A peer receives only an event message ej which holds the condition ej .SEQ =
pi .REQj in order to receive event messages published by a peer pj in the pub-
lication order. Every event message arriving at the peer pi in the underlying
114 S. Nakamura et al.

network are kept in the buffer RBFi of the peer pi . Here, it is guaranteed that
every pair of event messages published by each peer are stored in the publish-
ing order. Then, it is checked whether or not the event message ej satisfies the
condition “ej .SEQ < min(pi .ALj ) (= min{pi .ALj,1 , . . . , pi .ALj,pn })”. If the
condition holds, the peer pi correctly receives the event message ej . After that,
only the event message ej whose publication ej .P includes some common topic
with the subscription pi .S is moved to a second buffer SBFi of the peer pi . Every
event message in the second buffer SBFi is reordered in the causal order. The
peer pi dequeues a top event message e from the second buffer SBFi and the
event message e is delivered to the peer pi in the causal order.

3.2 ETOBSCO (Efficient TOBSCO) Protocol

In Example 1, the peer p1 updates the object o1 and publishes an update event
message ue2 to make the other peers p2 and p3 synchronize the replicas with
the object o1 . Here, the topic set o1 .T does not change although the peer p1
updates the object o1 . In the TOBSCO protocol, every illegal information flow
is prevented based on the topics of each object and the subscription topics of
each peer. Hence, if a peer pi updates its own object oi but the topics of the
object oi does not change, it is meaningless to inform the update to each peer pj
which obtains a replica oji from the viewpoint of preventing illegal information
flow.

Definition 3. A peer pi alters an object oi iff the peer pi updates data of the

object oi but the topics in oi .T do not change.

Definition 4. The update event message uei is meaningless iff a peer pi pub-

lishes the update event message uei to make every other peer synchronize the
replica of the object oi which the peer pi alters.

In this paper, we newly propose an ETOBSCO (Efficient TOBSCO) protocol

to reduce the number of event messages by forcing every peer avoid publishing
the meaningless update event messages. Here, peers cannot know that an object
is altered by a peer pi unless the peer pi publishes an event message carrying
the altered object. In order to prevent this problem, if an object o which has at
least one common topic with the object oi altered by the peer pi , i.e. o.T ∩ oi .T
= φ, is delivered to the peer pi , the peer pi publishes an update event message
carrying the altered object oi .

Example 1. Suppose there are three peers p1 , p2 , and p3 as shown in Fig. 1. The
peers p1 , p2 , and p3 have subscriptions and publications as p1 .S = p1 .P {w,
x}, p2 .S = p2 .P {w, x, y, z}, and p3 .S = p3 .P {w, x, y}. The peers p1 , p2 ,
and p3 create objects o1 , o2 , and o3 where o1 .T = {w}, o2 .T = {w, x}, and
o3 .T = {x, y}, respectively. Figure 1 shows how the peers communicate in the
TOBSCO protocol. Here, the peer p1 publishes an update event message ue2
because the peer p1 updates data in the object o1 . In the peer p2 , since e1 ⇒ p2
and ue2 ⇒ p2 , the event messages e1 and ue2 are delivered at the first delivery
 Efficient Information Flow Control by Reducing Meaningless Messages 115

event of the peer p2 . First, the object o1 carried by the event message e1 is
delivered to the peer p2 and is stored in the storage p2 .D of the peer p2 . Then,
the replica o21 is synchronized with the object o1 carried by the update event
message ue2 . In the peer p1 , since e4 ⇒ p1 , the event message e4 is delivered at
the second delivery event of the peer p1 in Fig. 1. The object o2 is delivered to
the peer p1 and is stored in the storage p1 .D. On the other hand, e5
→ p1 holds
o
because the object o3 where o3
→ p1 is carried by the event message e5 . Here,
the illegal object o3 is not delivered to the peer p1 to prevent illegal information
flow.
In Fig. 2, the ETOBSCO protocol is performed in the same scenario of the
Fig. 1. The update event message ue2 published by the peer p1 is meaningless
because the peer pi alters the object o1 . Here, the peer p1 does not publish the
update event message ue2 in the ETOBSCO protocol. Hence, fewer number of
event messages are exchanged among peers in the ETOBSCO protocol than the
TOBSCO protocol. On the other hand, it takes longer time to make peers p2
and p3 synchronize their replicas of the altered object o1 with the object o1 in
the ETOBSCO protocol than the TOBSCO protocol. For example, the peer p2
synchronizes its replica o21 with the altered object o11 at the second delivery event
of the peer p2 in the ETOBSCO protocol while the peer p2 is synchronized at
the first delivery event of in the TOBSCO protocol.

Fig. 1. TOBSCO protocol Fig. 2. ETOBSCO protocol

4 Evaluation

We evaluate the ETOBSCO protocol on a topic set T = {t1 , . . . , ttn } (tn ≥ 1)

and a peer set P = {p1 , . . . , ppn } (pn ≥ 1) in terms of the number of event
messages delivered and the average update delay time of every object altered. In
the ETOBSCO protocol, every illegal object is not delivered to peers to prevent
illegal information flow. Every event message is causally delivered to peers. The
meaningless update event messages are not published to reduce the number of
event messages exchanged among peers.
116 S. Nakamura et al.

In the evaluation, topics in the publication pi .P and the subscription pi .S of

each peer pi are randomly taken in the topic set T . Let mstn be the maximum
number of topics which can be included in the subscription pi .S of each peer
pi . The number of topics in the subscription pi .S is randomly selected out of
numbers 1, . . ., mstn. Topics in the publication pi .P are randomly selected so
that 1 ≤ |pi .P | and pi .P ⊆ pi .S. After publication and subscription of each peer
pi are decided, the peer pi creates one object oi where 1 ≤ |oi .T | and oi .T ⊆
pi .P .
We consider a pair of update operations on an object, full and partial update
operations. If a peer pi issues a full update operation to an object oii , the whole
data of the object oii is fully overwritten. Since every data is changed in the object
oii , i.e. oii is deleted and created, topics on the object oii are totally changed with
new ones. Hence, the variable oii .T gets empty, i.e. oii .T = φ. Then, the variable
oii .T randomly includes topics so that oii .T is a subset of the publication pi .P , i.e.
oii .T ⊆ pi .P . On the other hand, if the peer pi issues a partial update operation
to an object oii , only some data of the object oii is overwritten. Some topic
on the object oii is considered to be deleted or some new topic is given to the
object oii .
In the evaluation, we consider each peer pi alters an object oi by full and
partial update operations. Let ap be a alteration probability. In this paper, we
decided ap is 0.5. This means, a peer pi alters an object oi with probability 0.5
in full or partial update operation, respectively.
In the evaluation, we consider a pair of sets T and P which are composed of
fifty topics t1 , . . . , t50 (tn = 50) and ten peers p1 , . . . , p10 (pn = 10), respectively.
For each peer pi , subscription topics in pi .S and publication topics in pi .P are
randomly taken in the fifty topics T . Here, mstn = 20, i.e. the number of topics
in the subscription of each peer pi is randomly selected out of numbers 1, . . . , 20.

1. One peer pi is randomly selected in the peer set P and the peer pi randomly
includes some object or replica oi such that oi .T ⊆ pi .P in an event message
ei . Publication ei .P is decided so that ei .P is same as the topic set ei .T of
the event message ei . The peer pi publishes the event message ei .
2. Each peer pi creates an object o where o.T is a subset of pi .P with probability
0.01.
3. Each peer pi fully or partially updates data in the object oii where pi is the
creator peer with probability 0.02. The peer pi alters the object oii in full or
partial update operation with probability 0.5. Only if the peer pi alters the
object oii , the peer pi does not publish an update event message uei .
4. A target peer pi receives an event message ej if pi .S ∩ ej .P = φ, i.e. ej → pi .
Only the object oj such that oj .T ⊆ pi .S carried by the event message ej is
delivered to the peer pi . If the object oj has at least one common topic with
the object oii altered by the peer pi , i.e. oj .T ∩ oii .T = φ, the peer pi publishes
an update event message carrying altered object oii .

Let n be the number of publication events to occur in the simulation (0 ≤

n ≤ 500). For each n, one hundred different peer sets P1 , . . . , P100 are randomly
 Efficient Information Flow Control by Reducing Meaningless Messages 117

generated. For each set Pk , n publication events iteratively occur one hundred
times. After that, we calculate the average number of event messages delivered
and update delay time of altered objects in the ETOBSCO protocol.
In the simulation, we make the following assumptions:
• No event message is lost in a system.
• If a peer knows at least one event message exists which the peer has not
received yet, the peer does not publish any event message until the peer
receives the event message.
• The simulation steps 1 to 4 are in one time unit [tu]. Let mdt be the maximum
delay time. In this paper, we decided mdt is 10. This means, delay time of
each event message for each peer is randomly decided out of 1 to 10 [tu].
However, the delay time of the event message ei for the peer pi is 0 [tu].

8,000
TOBSCO
ETOBSCO
em
Number of event messages delivered

7,000

6,000

5,000

4,000

3,000

2,000

1,000

0 100 200 300 400 500

Number n of publication events

Fig. 3. Number of event messages delivered.

Figure 3 shows the numbers of event messages delivered in the TOBSCO

and ETOBSCO protocols. The dotted line with the label “em” shows the total
number of event messages except for update event messages delivered. Here, the
difference between em-line and ETOBSCO-line shows how many update event
messages are published in the ETOBSCO protocol. The number of event mes-
sages delivered in the ETOBSCO protocol is fewer than the TOBSCO protocol.
For example, about 7,770 event messages are delivered in the TOBSCO protocol
but about 5,320 event messages are in the ETOBSCO protocol for five hundred
publication events (n = 500).
Figure 4 shows the average update delay time of altered objects in the TOB-
SCO and ETOBSCO protocols. Update delay time means how many time units
it takes until an event message carrying the altered object is delivered to a peer
from time the object is altered. The update delay time of altered objects in the
ETOBSCO protocol is longer than the TOBSCO protocol. For example, update
delay time is about 29 [tu] in the TOBSCO protocol but about 48 [tu] in the
ETOBSCO protocol for five hundred publication events (n = 500).
118 S. Nakamura et al.

60

50

Update delay time [tu]

40

30

20

10

TOBSCO
ETOBSCO
0
0 100 200 300 400 500
Number n of publication events

Fig. 4. Update delay time.

5 Concluding Remarks
In the P2PPSO (P2P publish/subscribe with object concept) system, each peer
exchanges objects with other peers by publishing and subscribing event mes-
sages with no centralized coordinator. Here, illegal information flow occurs such
that the objects related with topics which the target peers are not allowed to
subscribe are delivered to the target peers by receiving event messages with
the objects. In order to prevent illegal information flow from occurring in the
P2PPSO system, the TOBSCO (topics-of-objects-based synchronization with
causally ordering delivery) protocol is proposed. In the TOBSCO protocol, ille-
gal objects are not delivered to the target peers and every event message is
causally delivered. In this paper, we newly proposed the ETOBSCO (Efficient
TOBSCO) protocol to reduce the number of event messages exchanged among
peers. In the ETOBSCO protocol, the peer which alters its object does not pub-
lish an update event message. In the evaluation, we showed the fewer number
of event messages are delivered in the ETOBSCO protocol than the TOBSCO
protocol. However, it takes longer time to synchronize the replicas with an object
altered in the ETOBSCO protocol than the TOBSCO protocol.

Acknowledgements. This work was supported by Japan Society for the Promotion
of Science (JSPS) KAKENHI Grant Numbers JP15H0295, JP17J00106.

References
1. Google alert. http://www.google.com/alerts. Accessed 1 Aug 2018
2. Denning, D.E.R.: Cryptography and Data Security. Addison Wesley, Boston (1982)
3. Eugster, P.T., Felber, P.A., Guerraoui, R., Kermarrec, A.M.: The many faces of
publish/subscribe. ACM Comput. Surv. 35(2), 114–131 (2003)
4. Fernandez, E.B., Summers, R.C., Wood, C.: Database Security and Integrity. Adi-
son Wesley, Boston (1980)
 Efficient Information Flow Control by Reducing Meaningless Messages 119

5. Lamport, L.: Time, clocks, and the ordering of event in a distributed systems.
Commun. ACM 21(7), 558–565 (1978)
6. Nakamura, A., Takizawa, M.: Causally ordering broadcast protocol. In: Proceed-
ings of IEEE the 14th International Conference on Distributed Computing Sys-
tems, pp. 48–55 (1994)
7. Nakamura, S., Enokido, T., Takizawa, M.: Information flow control in object-based
peer-to-peer publish/subscribe systems. Concurrency and Computation: Practice
and Experience (accepted)
8. Nakamura, S., Enokido, T., Takizawa, M.: A flexible read-write abortion protocol
with role safety concept to prevent illegal information flow. J. Ambient Intell.
Humaniz Comput. 9(5), 1415–1425 (2018)
9. Nakamura, S., Ogiela, L., Enokido, T., Takizawa, M.: An information flow control
model in a topic-based publish/subscribe system. J. High Speed Netw. 24(3), 243–
257 (2018)
10. Sandhu, R.S., Coyne, E.J., Feinstein, H.L., Youman, C.E.: Role-based access con-
trol models. IEEE Comput. 29(2), 38–47 (1996)
11. Setty, V., Steen, M.V., Vitenberg, R., Voulgaris, S.: Poldercast: fast, robust,
and scalable architecture for P2P topic-based pub/sub. In: Proceedings of
ACM/IFIP/USENIX 13th International Conference on Middleware, pp. 271–291
(2012)
12. Waluyo, A.B., Taniar, D., Rahayu, W., Aikebaier, A., Takizawa, M., Srinivasan, B.:
Trustworthy-based efficient data broadcast model for P2P interaction in resource-
constrained wireless environments. J. Comput. Syst. Sci. 78(6), 1716–1736 (2012)
 Comparison of Intelligent Algorithms
with FACTS Devices for Minimization of Total
Power Losses

Shohreh Monshizadeh(&), Gunne John Hegglid,

and Svein Thore Hagen

Department of Electrical Engineering, Information Technology, and Cybernetics,

University College of Southeast Norway, Porsgrunn, Norway
{Shohreh.Monshizadeh,gunne.j.hegglid,
svein.t.hagen}@usn.no

Abstract. Optimal power flow plays a vital role in providing secure, economic,
and reliable operation of electrical power system. This paper presents a com-
parison of two types of intelligent algorithms for solving optimal power flow
problems, incorporating Flexible AC transmission systems (FACTS) devices.
Particle Swarm optimization (PSO) and imperialist competitive algorithm
(ICA) have been analyzed and simulated with MATLAB software in order to
reduce the total power losses of the network with three types of FACTS devices:
unified power flow controller (UPFC), Static Var Compensators (SVC), and
Thyristor Controlled Series Capacitor (TCSC). The proposed approach is sim-
ulated in IEEE-30 buses in MATLAB software and the results are compared
with several tables and diagrams. It is observed that PSO algorithm with UPFC
as the FACTS devices can reduce more active and reactive power losses than the
other types of the FACTS devices and intelligent algorithms. IEEE 30 buses is
used to illustrate the effectiveness of proposed FACTS devices and intelligent
algorithm for solving optimal power flow problem. The proposed method is
simulated with MATLAB programming and the results are compared with
several tables and figures.

Keywords: Optimal power flow
Particle Swarm Optimization (PSO)

Imperialist Competitive Algorithm (ICA)

FACTS controllers (TCSC, UPFC, SVC)
Active and reactive power losses

1 Introduction

Optimization is a mathematical tool to enhance the performance of a system. In recent

years, the term “optimal power flow” has been the center of attention for solving many
problems of electrical power systems, including the high cost of fuels and power losses
[1]. Carpentier was the first one who introduced optimal power flow in 1962 after that
OPF had been developed by many researchers for the aim of enhancing the perfor-
mance of power systems [1]. Optimization is described as a mathematical formulation
with a large number of variables, including equality and inequality of constraints [2].
Optimization methods are classified in two main categories that consist of conventional

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 120–131, 2020.
https://doi.org/10.1007/978-3-030-15032-7_10
 Comparison of Intelligent Algorithms with FACTS Devices 121

and intelligent methods [2]. Classical techniques, which are used in many scientific
papers, include nonlinear programming, linear programming, and quadratic program-
ming [2]. There are several disadvantages to using classical techniques for large
dimension of power system with increasing the number of buses and generators, such
as large mathematical formulations, poor convergence [3]. The most prominent arti-
ficial intelligence techniques that are mentioned in many research papers are Genetic
Algorithm (GA), Evolutionary Programming (EP), and Particle Swarm Optimization
(PSO) [3]. These techniques have several advantages for a large number of nonlinear
constraints, including large choice of methodologies, suitability for solving multi-
objective optimization problems, multiple optimization solutions for a single simula-
tion run [3]. Among these algorithms, PSO and ICA algorithms known as the most
useful methods for solving optimization problems and for minimizing objective
functions like power loss and cost of fuel.
The main objective of this paper is to employ two types of prominent algorithms
(PSO and ICA) to reduce the total active and reactive power losses in presence of
FACT devices. The purpose of using the FACTS devices in electrical power systems is
enhancement of flexibility, security, and economical operation of systems (these goals
were first introduced by Hingorani in 1988) [4]. The most prominent FACTS devices in
recent scientific papers are TCSC (Thyristor controlled series compensation), UPFC
(unified power flow controllers), and SVC (static var compensators) that simulated in
this paper. By incorporating the use of FACTS devices with optimal power flow
analysis, can increase security, stability of voltage levels, and flexibility for the elec-
trical power system [5].
Several techniques have been proposed in the past, including: Nachimuthu suggests
an improvement of the voltage profile and enhancing the stability by using Imperialist
competitive algorithm for solving OPF problems encountered when incorporating
FACTS devices. The FACTS device which is used in this paper is UPFC. The
advantages of using this algorithm are reduction of the total loss and installation cost of
UPFC devices, using a lower number of iterations for convergence response, and
improving the voltage profile [6]. Gopinath introduced several algorithms, including
PSO, Cuckoo search, Bat, and Firefly optimization in order to solve transmission line
problems, and select the optimize algorithm with IPFC device as FACTS controller in
order to solve objective function. The aim of this paper is to reduce the losses and
balance power flow of electrical energy in a multi-transmission line [7]. A. Immanuel
introduced a new method based on fuzzy logic for placing the optimal location of a
UPFC device based on a PSO algorithm. UPFC is the new facts device that can control
active and reactive power and has high flexibility and voltage magnitude for a power
system. In this papers use of fuzzy method based on the weakest node selected for
installing UPFC device in order to decrease the losses and the cost of the network [8].
This paper is divided into six sections. The first part is organized as introduction.
The next stage, Sect. 2, will illustrate the description of OPF with its formulation. In
Sect. 3, a brief description will be given of two intelligent algorithms, including ICA
and PSO algorithms, for comparative purposes. Section 4 will give a general
description of FACTS devices and three specific types of FACTS devices, with their
structures and models are discussed. In Sect. 5, the simulation results and programming
with MATLAB software are discussed and presented with several tables and figures,
and in Sect. 6 the final results are shown.
122 S. Monshizadeh et al.

2 Optimal Power Flow

Optimal power flow problems are the main challenges for many researchers for the aim
of operating and planning of power systems. Optimal power flow is described as a
nonlinear optimization problem with nonlinear objective functions and non-linear
constraints with both format equality and inequality constraints for solving the non-
linear equations [7]. The general optimal power flow problems can be expressed as
following:

Min f ðxÞ:
Constraints: gð xÞ ¼ 0 Equality constraints ð1Þ
H ð xÞ  0 Inequality constraints

In order to employ the limitation of control variables such as active and reactive
power and voltage magnitude, the penalty function is suggested to normalize the output
of control variables in their limitation area. The goal of penalty functions is converting
the constrained problems into unconstrained problems by introducing one penalty.
Penalty function can be express as Eq. 2 [9]:

ðxÞ ¼ f ðxÞ þ XðxÞ:

Pn o
ð2Þ
XðxÞ ¼ q gðxÞ2 þ ½maxð0; hðxÞÞ2 :

Where:
P(x) is the penalty function.
Ω(x) is the penalty.
P is penalty factor.
There are several types of objective functions which are described in many research
papers, including minimization of production cost, minimization of pollution emission,
and minimization of active and reactive losses [9]. In this paper, the objective function
which is considered is decreasing the transmission power losses. The Mathematical
formulation of transmission losses can be expressed as formula 3 [14]:
XNL XNL   
Floss ¼ i¼1 j¼1
g ij V 2
i þ V2
j  2  V i  Vj  cos d i  d j : ð3Þ

Where,
Vi is defined as the voltage magnitude at bus i.
gi,j is defined as the conductance of line i − j.
di is defined as the voltage angle at bus i.
N is defined as the total number of transmission lines.
In order to solve optimization problems, the constraints of the system divided in
two main categories, including the equality and inequality constraints that are generator
MW, controller voltage, transformer tapping, and reactive power injection [9]. Equality
constraints are the set of nonlinear formulation of power flow that are active and
 Comparison of Intelligent Algorithms with FACTS Devices 123

reactive power transmitted through each buses of the power system and inequality
constraints are the lower bound and upper bound of each control variables including the
voltages of generators, tap position of transformers, upper and lower bounds of active
and reactive power of generators. Variable limitation is expressed as formulation 4 [9].

ViðminÞ  Vi  ViðmaxÞ :
TiðminÞ  Ti  TiðmaxÞ :
ð4Þ
QiðminÞ  Qi  QiðmaxÞ :
PiðminÞ  Pi  PiðmaxÞ :

Where,
ViðminÞ , ViðmaxÞ are upper and lower limits of voltage magnitude at bus i.
TiðminÞ , TiðmaxÞ are upper and lower limits of tap position of transformer i.
QiðminÞ , QiðmaxÞ are upper and lower limits of reactive power source i.
PiðminÞ , PiðmaxÞ are upper and lower limits of power generated by generator i.

3 Intelligent Algorithms

In overall, two types of the algorithms are introduced to solve the optimal power flow
problems, including the classical and intelligent algorithms. In this paper two types of
the intelligent algorithms, PSO and ICA methods are introduced and the results of
using these algorithms for minimizing OPF problems tested in IEEE-30 buses system.

3.1 Particle Swarm Optimization (PSO) Algorithm

Kennedy and Eberhart were the first to introduce Particle Swarm Optimization as the
random search space method of optimization based on the sociable behavior of a group
of animals, including bird flocks or fish groups in order to find a food. Groups of fishes
or birds located in the random search space and there was just one piece of the food. No
bird has any knowledge about the place of the food and they only know about their
distance to the food. One of the best strategies is selecting one bird, which is near to the
food. Locations of each particles in the search space are influence on the experience
and knowledge of themselves and their neighbors. Thus, the location of the other
particles in a swarm are influenced on the method of searching particles [9].
In PSO, each random response is called one particle and the collection of particles
is introduced as a swarm. Therefore, each particle has fitness value according to the
objective function, which is classified as following: first agent, which has the best value
of group experience, considered Gьest and the other item, which has the best inter-
personal experience, mentioned Pьest [1].
In a physical search space, the direction of movement of each particle is called
velocity and the velocity of particles is called velocity vector [6]. The mathematical
algorithm about the relationship of velocity and the distance of each particle for
124 S. Monshizadeh et al.

reaching to the new position according to the best personal experience and global
experience until next location is shown (see Fig. 1).

Swarm Influence

Particle memory Influence

Current Motion Influence

Fig. 1. The movement of each particle to the imperialist.

After first iteration, according to the velocity and position vector, the new popu-
lation can be calculated. The velocity and situation of each particle can be calculated
according the Eqs. 5 and 6 [2]:

ð5Þ

Xkj þ 1 ¼ Xkj þ Vkj þ 1 : ð6Þ

Where,
Vkj The velocity of individual I at iteration K.
Rand1 and rand2 Random numbers between (0, 1).
Xkj The location of individual I at iteration k.
C1, C2 Velocity coefficients.
W Inertial coefficient.

Process of OPF with PSO Algorithm

In this paper, the objective function of optimal power flow is the minimization of active
and reactive power losses in the transmission lines. According to the particle swarm
optimization, various stages are included for solving OPF problem that is described in
the following stages [15]:
Step (1) Identification of the bunch of initial variables as the control variables (Tap,
PG, VG) between Lower bound and Upper bound.
Step (2) Identification of slack generator with largest capacity among N generators,
then optimization applied for N − 1 generators.
Step (3) Particles of population should be selected randomly then identifies max-
min of each particles. (Randomly selection of 24 variables of OPF).
 Comparison of Intelligent Algorithms with FACTS Devices 125

Step (4) Value of the objective function (target function) for this category of
solutions is calculated (Cost function or Loss function).
Step (5) Gbest is the best solution until one especial repetition (for example 100
repetition), and Pbest is the best solution in the each repetition.
Step (6) In the next section, using the two formulas 5 and 6, the new replies are set
for the next iteration (the new population can be calculated).
Step (7) Check constraints.
Step (8) Producing the new population, the position and velocity of each particle
should be modified according to Eq 5.
Step (9) Check constraints.
Step (10) Evaluating the step criteria (such max iteration).
Step (11) Global best (Gbest) is considered as optimal value that is lastly generated
by particles [15].

3.2 Imperialist Competitive Algorithm (ICA)

Imperialist competitive algorithm is a modern optimization algorithm, which is inspired
from the real world competition between imperialists and colonies [10]. The algorithm
starts the process of its optimization based on initial random population of several
countries. Firstly, the algorithm starts with a random number of countries (N_country)
and the cost of the initial countries is calculated according to the Eq. 7 [6]:

Costn ¼ f ðcountryn Þ: ð7Þ

Among N random countries, Nimp country from the powerful country that has the
lowest price, the lowest objective function is selected as imperialist. The other countries
are selected as the colonies. Colonies are divided among imperialists and the number of
colonies for each imperialist shows the strength of each imperialist. According to the
Fig. 2, imperialist countries attract the colonies to themselves based on assimilation
policy [15].

Fig. 2. The assimilation process.

126 S. Monshizadeh et al.

The new location of the colony after attracting by imperialist can be calculated as
formulation 8, which d is defined as the distance between colony and imperialist and b
is the parameter that modify the area that colonies randomly search around the
imperialist.

Xnew ¼ Xold þ ð0; b  dÞ: ð8Þ

After that, the Revolution process will be applied. In the revolution stage, if the
power of one colony is more than imperialist, the location of colony and imperialist
will have changed. Revolution occurs when power of each colony is more than the
power of imperialist. The other stage is called Colonialism, which is the fight between
imperialists for power. In colonialism, the weakest imperialist will be removed that the
weakest imperialist is defined as imperialist with just one colony [6].
In each iteration, the competition will be applied between imperialists and colonies.
During this competition, according to the fitness function, the weaker colony among
weakest imperialist will be selected then the weaker colony transfers to the more
powerful imperialist (the more powerful imperialist has a higher chance of attracting
the weakest colony from weakest imperialist). Figure 3 shows the process of colonial
competition [15]:

Fig. 3. Process of colonial competition.

The overall process of the ICA algorithm is as explained below [10]:

1. Definition of the objective function.
2. Select several random points in the search space, and then create the initial
imperialist.
3. Assimilation: Imperialists absorbing colonies.
4. Evaluation: Changing weaker imperialists with powerful colonies randomly.
5. Calculation of total cost of imperialist.
6. Imperialist competition: Select one colony among the weakest imperialists and then
give it to the most powerful imperialist.
 Comparison of Intelligent Algorithms with FACTS Devices 127

7. Remove the weaker imperialist.

8. If just one imperialist remains then stop; otherwise go to the step 2.

Implementing ICA in OPF

As mentioned before, ICA is introduced one efficient intelligent algorithm for solving
the optimal power flow studies. Several new algorithms have been proposed to solve
optimal power flow problems, including the limitation of product cost and transmission
losses. The following steps are proposed for implementing ICA in OPF programming
[16]:
Step (1) Selecting the input data of system (data of transmission lines, generators
and voltages of system).
Step (2) initialize the countries for optimization purposes.
Step (3) Evaluate fitness function using power flow studies using various methods,
including Quadratic programming, nonlinear programming and interior point
method.
Step (4) exchange the location of imperialist with colonies if there is stronger
colony in one imperialist.
Step (5) move colonies toward the imperialist.
Step (6) Calculation of function cost according to the imperialist and colonies.
Step (7) selecting one weakest colony from weakest imperialist and transmit to the
stronger imperialist.
Step (8) remove the weakest imperialist.
Step (9) check the criteria sections.
Step (10) if there is one imperialist stop otherwise update data and go to step 3.

4 FACTS Devices

Nowadays, the security and flexibility of a power system has been become the new
challenge for power systems. Flexible AC transmission system (FACTS) was proposed
to be added in power systems in order to solve several problems, such as overloading,
losses, voltage stability, and power flow [11]. The objective function that is used to
solve optimal power flow problems are including to reduce the transmission losses and
cost of production with incorporating FACTS devices [12]. There are different types of
FACTS devices, but three types of them are most relevant in designing power systems,
including TCSC (Thyristor Controlled Series Compensation), SVC (Static Var Com-
pensator) and UPFC (Unified Power Flow Controllers) [13].

5 Simulation Results

In this paper, the proposed method is coded on IEEE-30 buses with 41 transmission
lines. Programming of IEEE-30 buses is written with MATLAB software from Math
power box. The characterized of system consists of 6 generators, 30 load buses, and 41
branches, with powerbase 100 MVA. This paper is tested in a MATLAB environment
128 S. Monshizadeh et al.

to find a better solution for reducing the total transmission losses of the network with
PSO and ICA algorithms. In this paper, three types of FACT devices (TCSC, SVC and
UPFC) are compared based on their ability to solve OPF problems incorporating
intelligent algorithms, and the results of each type of FACTS devices are shown in the
tables and figures. Tables 1 and 2 show a brief description of 5 bus data and branch data
of IEEE-30 buses.

Table 1. Bus data of IEEE-30 buses.

Bus Voltage Generation Load
Mag (Pu) Ang (Deg) P (MW) Q (MVar) P (MW) Q (MVar)
1 1.00 0.00 71.095 24.07 – –
2 1.00 9.669 163.00 14.46 – –
3 1.00 4.771 85 −3.65 – –
4 0.987 −2.407 – – – –
5 0.975 −4.017 – – 90.00 30.00

Table 2. Branch data of IEEE-30 buses.

Branch From To From bus P Injection Q To bus P Injection Loss P Q
bus bus (MW) (MVar) (MW) Q(Var) (MW) (Mvar)
1 1 4 71.95 24.70 −71.95 −20.75 0 3.32
2 4 5 30.73 −0.59 −30.55 −13.69 0.174 0.94
3 5 6 −59.45 −16.31 60.89 −12.43 1.44 6.31
4 3 6 85 −3.65 −85 7.89 0 4.2
5 6 7 24.11 4.54 −24.01 −24.40 0.095 0.81

The power losses after using intelligent algorithms for solving OPF problem (PSO
and ICA) are shown in the Table 3. As can be seen, the proposed algorithms can reduce
the power losses and improve the performance of the system. The total active and
reactive power losses using intelligent algorithms are calculated as 4.955 MW and
51.31 MVar respectively, whereas after using artificial algorithms, PSO and ICA, it has
significantly declined by 2.757 MW for PSO and 1.478 MW for the ICA algorithm.

Table 3. Comparison of power losses before and after employing intelligent algorithms on
IEEE-30 buses.
Methods Power active loss Power reactive loss
IEEE 30 buses 4.955 51.31
PSO 2.198 19.65
ICA 3.477 23.33
 Comparison of Intelligent Algorithms with FACTS Devices 129

In the next stage, the results of three types of FACTS devices, TCSC, SVC and
UPFC, for optimization purposes are compared. The results are shown in Table 4. It is
clearly shown that reducing the objective function after using FACTS devices produces
better results. By compering three types of FACTS devices, it is shown that UPFC has
better result for reducing the active and reactive power losses. Table 4 shows the results
of using intelligent algorithms, PSO and ICA with FACTS devices to reduce the
transmission losses in the electrical power network IEEE-30 buses. Figure 4 compares
the presence of the three types of FACTS devices (UPFC & TCSC & SVC) with two
types of intelligent algorithms (PSO & ICA) for minimizing the transmission losses
with MATLAB software. Table 5 illustrates the comparison of two types of intelligent
algorithms that used in this paper and Table 6 shows the brief summary of simulation
results.

Table 4. Comparison Effects of FACTS devices on reducing the power active and reactive
losses incorporating of PSO and ICA algorithms on IEEE-30 buses.
Algorithms TCSC SVC UPFC UPFC & TCSC
Losses
Power Power Power Power Power Power Power Power
active reactive active reactive active reactive active reactive
loss loss loss loss loss loss loss loss
PSO 2.133 17.87 2.091 22.05 1.892 18.12 2.209 20.10
ICA 2.017 19.54 9.504 33.60 3.727 20.99 2.27 20.54

Fig. 4. (A) Reducing the total power losses with ICA algorithm in FACTS devices.
(B) Reducing the total power losses with a PSO algorithm in FACTS devices
130 S. Monshizadeh et al.

Table 5. Advantages of employing PSO and ICA for solving OPF problems.
PSO advantages ICA advantages
Fast convergence Robust behavior
Few control parameters Reduced losses
Useful for optimization multi model space Less number of iteration compared the
other algorithms
Very easily adaptive for both integer and Reduced installation cost
discrete optimization

Table 6. Comparing results of simulation for reducing the transmission losses in presence of
FACTS device (UPFC) with intelligent algorithms in IEEE-30 buses network.
Conditions Power active loss Power reactive loss
IEEE 30 buses 4.955 51.31
PSO algorithm 2.198 19.65
PSO in presence of UPFC 1.892 18.12
ICA algorithm 3.477 23.33
ICA in presence of UPFC 3.727 20.99

6 Conclusions

In this paper, two categories of artificial algorithms were simulated and coded with
MATLAB software and the effects of FACTS devices (TCSC, UPFC and SVC) are
displayed for minimizing the losses of the network. Table 5 compares the advantages
of PSO and ICA algorithms for solving OPF problems that are collected from many
research papers.
The PSO algorithm is the best intelligent algorithm for solving OPF problems
because of several important advantages: lower number of computational formulations
and fast convergence for response, which put PSO in the center of attention for many
researchers. Therefore, in this paper, the PSO algorithm with UPFC as FACTS device
has shown the most significant advantages in solving optimal power flow problems
better than the other types of algorithms, such as ICA. Table 6 indicates the brief
summary of before and after applying the intelligent algorithms (PSO and ICA) with
UPFC as a FACTS device in IEEE 30 buses. According to the results of simulation,
total transmission loss after using an intelligent algorithm would be decreased com-
pared of before using artificial optimization techniques. It can be clearly shown by
comparing two types of algorithms, PSO gives the better result compared with the ICA
algorithm, and UPFC as a FACTS device can solve OPF problems that have better
effects on optimization than the other types of FACTS devices.
 Comparison of Intelligent Algorithms with FACTS Devices 131

References
1. Joseph, S.P., Dineshbalaji, C.: Transmission loss minimization using optimization technique
based on PSO. IOSR J. Electr. Electron. Eng. (IOSR-JEEE) 6(1), 1–5 (2013)
2. Jothi Swaroopan, N.M., Somasundarant, P.: Optimal power flow for security enhancement
using fuzzified particle swarm optimization. In: IEEE International Conference on
Sustainable Energy and Intelligent System, pp. 20–22 (2011). ISBN 978-9-38043-000-3
3. Amarnath, R.V., Ramana, N.V.: State of art in optimal power flow solution methodologies.
J. Theor. Appl. Inf. Technol. 30(2), 128–154 (2011)
4. Albatsh, F.M., Mekhilef, S., Ahmad, S., Mokhlis, H.: Enhancing power transfer capability
through flexible AC transmission system devices. Front. Inf. Technol. Electron. Eng. 16(8),
658–678 (2015). ISSN 2095-9184
5. Ma, R., Yan, H.: Multi objective optimal dispatch in power system with TCSC and load
voltage characteristic using fuzzy sets theory. In: 6th World Congress on Intelligent Control
and Automation, pp. 7562–7566 (2006)
6. Nachimuthu Deepa, S., Basha Rizwana, J.: Imperialist competitive algorithm for minimiza-
tion of losses by optimally location FACTS controllers in power system. Istanbul Univ.
J. Electr. Electron. Eng. (IU-JEEE) 15, 1889–1896 (2015)
7. Gopinath, B., Rubiya, S.: Comparative analysis of optimization algorithms for solving
optimal power flow using IPFC. Int. J. Sci. Technol. Eng. 2(11), 435–441 (2016)
8. Immanuel, A., Chengaiah, C.: Optimal power flow with UPFC using fuzzy-PSO with non-
smooth fuel cost function. Int. J. Eng. Res. Appl. 5(5), 01–06 (2015)
9. Leeton, U., Uthitsunthorn, D., Kwannetr, A., Sinsuphun, N., Kulworawanichpong, T.:
Power loss minimization using optimal power flow based on particle swarm optimization. In:
International Conference on Electrical Engineering/Electronics, Computer, Telecommuni-
cations and Information Technology (ECTI-CON 2010). IEEE (2010)
10. Nanadakumar, E., Dhanasekaran, R., Senapathi, N.K.: Optimal location of UPFC by BFO,
MSFL and ICA algorithms. IOSR J. Electr. Electron. Eng. (IOSR-JEEE) 10(2), 25–33
(2015)
11. Gaur, D., Mathew, L.: Optimal placement of FACTS devices using optimization techniques.
In: 3rd International Conference on communication Systems, pp. 1–15. IOP Publishing
(2018)
12. Ravi Kumar, G., Kameswara Rao, R., Tulasi Ram, S.S.: Power flow control and
transmission loss minimization model with TCSC and SVC for improving system stability
and security. In: IEEE Third International Conference on Industrial and Information Systems
(2008). ISSN 2164-7011
13. Tiwari, A., Swarnkar, K.K., Wadhwani, S., Wadhwani, A.K.: Optimal power flow with
FACTS devices using genetic algorithm. Int. J. Power Syst. Oper. Energy Manag. 1(2), 66–
72 (2011)
14. Alizadeh, M., Ghazizadeh Ehsaee, M., Shahbazi Ghareghavini, M.M.: Reducing the power
loss in presence of TCSC based on optimal power flow with intelligent algorithm. In: 4th
National Conference and 2nd International Conference Applied Research in Electrical,
Mechanical and Mechatronic Engineering, pp. 1–11 (2016)
15. Smita, P., Vaidya, B.N.: Optimal power flow by particle swarm optimization for reactive loss
minimization. Int. J. Sci. Technol. Res. 1(1), 1–6 (2012)
16. Saebi, J., Mashhadi, H.R.: Imperialist competitive algorithm for reactive power dispatch
problem in electricity market. International Trans. Electr. Energy Syst., 858–874 (2013)
 Energy-Efficient Recovery Algorithm
in the Fault-Tolerant Tree-Based Fog
Computing (FTBFC) Model

Ryuji Oma1(B) , Shigenari Nakamura1 , Dilawaer Duolikun1 ,

Tomoya Enokido2 , and Makoto Takizawa1
1
Hosei University, Tokyo, Japan
[email protected], [email protected],
[email protected], [email protected]
2
Rissho University, Tokyo, Japan
[email protected]

Abstract. In order to reduce the traffic of networks and servers, the

fog computing model is proposed to realize the IoT (Internet of Things).
In our previous studies, the fault-tolerant tree-based fog computing
(FTBFC) model is proposed to be tolerant of faults of fog nodes. If
a fog node fj is faulty, child fog nodes of fj are disconnected. Another
operational fog node fi at the same level as fj is a candidate parent node
of disconnected nodes. One candidate node is selected to be a parent of
every disconnected node. However, the new parent node fi has to addi-
tionally process data from the disconnected nodes. In order to reduce
the energy consumption of each new parent node, we propose a modified
FTBFC (MFTBFC) model where disconnected nodes are partitioned
into groups and fog nodes in each group are connected to a different can-
didate node. We also propose an SMPR (selecting multiple parents for
recovery) algorithm to select a candidate parent node for each discon-
nected node so that the electric energy consumption of each new parent
node can be reduced. In the evaluation, we show the energy consumption
and execution time of each new parent fog node can be reduced in the
SMPR algorithm.

Keywords: Energy-efficient fog computing ·

Fault-tolerant tree-based fog computing (FTBFC) model ·
Modified FTBFC (MFTBFC) model · SMPR algorithm

1 Introduction
In the Internet of Things (IoT) [5], not only computers like servers and clients
but also sensors and actuators are interconnected in networks. In the cloud com-
puting model [2], data collected by sensors are transmitted to servers in a cloud
through networks. Networks are congested to transmit huge volume of sensor
data and servers are also overloaded to process sensor data. In order to reduce
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 132–143, 2020.
https://doi.org/10.1007/978-3-030-15032-7_11
 Energy-Efficient Recovery Algorithm 133

the communication and processing traffic of networks and servers in the IoT, the
fog computing (FC) model [11] is proposed. A fog node processes sensor data and
sends the processed data to another fog node. On receipt of processed data from
fog nodes, a fog node further processes the data and sends the processed data
to fog nodes. Thus, data processed by fog nodes is finally delivered to servers.
Since some subprocesses done by servers in the cloud computing model are per-
formed by fog nodes on behalf of the servers, the processing load of servers can
be reduced in the FC model. Thus, subprocesses to handle the data are per-
formed on not only servers but also fog nodes. Furthermore, the IoT is more
scalable than the Internet because not only computers but also various things
like cars [1] are interconnected. In addition to increasing the performance, we
have to reduce the electric energy consumed by servers, fog nodes, and devices
in the IoT.
In the tree-based fog computing (TBFC) model [6,7,9,10], fog nodes are hier-
archically structured in a height-balanced tree. A root node of the tree denotes
a cluster of servers. Fog nodes at the bottom layer are edge fog nodes which
communicate with devices, i.e. sensors and actuators. Sensors send data to edge
nodes. Each edge node generates output data by processing input data and sends
output data to a parent node. An application process is assumed to be a sequence
of subprocesses. Every node at each level is equipped with the same subprocess.
Each node receives input data from child nodes and performs the subprocess on
the data. Then, the node sends the processed output data to a parent node. It
is shown the energy consumption and execution time of nodes are shown to be
reduced in the TBFC model compared with the cloud computing model [6,7,10].
The fault-tolerant TBFC (FTBFC) model [8] is proposed to be tolerant of
faults of fog nodes. Child nodes of a faulty node are disconnected in the TBFC
model. An operational node at the same level as a faulty node is a candidate
node. In the FTBFC model, one candidate node is selected as a new parent
node and data processed by disconnected nodes is sent to the candidate node of
the faulty node. In the evaluation, the total energy consumption and execution
time of all the nodes are shown to be reduced but the reduction ratio is small
in scalable systems. In this paper, we propose a modified FTBFC (MFTBFC)
model where there are multiple new parent nodes for disconnected nodes. We
also propose an SMPR (selecting multiple parents for recovery) algorithm to
select a candidate node as a new parent node of each disconnected node so that
the energy consumption of new parent nodes can be reduced. In the evaluation
we show the energy consumption and execution time of each new parent node
can be reduced in the MFTBFC model [3,4].
In Sect. 2, we present the FC model. In Sect. 3, we propose the MFTBFC
model and SMPR algorithm. In Sect. 4, we evaluate the MFTBFC model.
134 R. Oma et al.

2 System Model
2.1 Tree Structure of Fog Nodes

The fog computing (FC) model [11] to efficiently realize the IoT is composed of
devices, fog nodes, and clouds of servers. Sensor data is processed by an applica-
tion process. In the cloud computing (CC) model [2], application processes are
performed on servers. A sensor collects data obtained by sensing events occur-
ring in physical environment. Devices, i.e. sensors and actuators are connected
to edge fog nodes. Sensor data is sent to edge fog nodes and delivered to servers
by fog nodes. Some subprocesses of an application process are performed on
a fog node to process a collection of input data sent by sensors and other fog
nodes. Then, output data, i.e. processed data is generated by a fog node through
performing a subprocess on the input data. Output data processed by a fog node
is sent to another fog node. Servers finally receive data processed by fog nodes.
Actions decided by servers are sent to edge fog nodes and the edge fog nodes
issue the actions to actuator nodes.
In the tree-based fog computing (TBFC) model [6,7,10], fog nodes are struc-
tured in a height-balanced tree Fig. 1. Here, the root node f0 denotes a cloud of
servers. Each node fR has lR (≥ 1) child nodes fR0 , fR1 , ..., fR,lR −1 . Here, fRi is
the ith child node of the node fR . Let ch(fR ) show a set {fR0 , fR1 , ..., fR,lR −1 }
of child nodes of a node fR and pt(fRi ) denote a parent node fR of a child node
fRi . If a node fR is a root node of the tree, R = 0, i.e. fR is f0 . If a node fR is an
ith child node of a node fR , fR is fR i , i.e. R = R i. For example, the third child
node of a node f01 is f012 . Thus, the label R of a node fR shows a path from
the root node f0 to the node fR . Let as(fR ) and ds(fR ) be sets of ancestor and
descendant nodes of a node fR , respectively. A least upper bound lub(fR , fS )
nodes of fR and fS is a common ancestor node fT of whose any descendant node
is not an ancestor of the nodes fR and fS . A node at the bottom layer shows an
edge node which communicate with sensors and actuators.
A node fR takes input data dRi from each child node fRi (i = 0, 1, ..., lR -1).
DR shows a collection of input data dR0 , dR1 , ..., dR,lR −1 . A subprocess in
the node fR does the computation on the collection DR of input data
dR0 , dR1 , ..., dR,lR −1 and then generates output data dR . Let p(fR ) denote a
subprocess to be performed on a node fR . Then, the node fR sends the out-
put data dR to a parent node pt(fR ). The input data dR0 , dR1 , ..., dR,lR −1 and
output data dR are stored by the storage module SR in the storage DBR . A
notation |d| shows the size [Byte] of data d. The ratio |dR |/|DR | is the output
ratio ρR of a node fR .
Let p be an application process to handle sensor data. In this paper, we
assume a process p is realized as a sequence of subprocesses p0 , p1 , ..., pm
(m ≥ 0). Here, p0 and pm are root and edge subprocesses, respectively. The
subprocess pm takes sensor data dm+1 from sensors, generates output data by
doing computation on the data, and sends the output data dm to the subprocess
pm−1 . The process pm is an edge subprocess on an edge node. Thus, each subpro-
cess pi receives input data di+1 from a preceding subprocess pi+1 and outputs
 Energy-Efficient Recovery Algorithm 135

Fig. 1. TBFC model.

data di to a succeeding subprocess pi−1 , which is obtained by processing the

input data di+1 from the subprocess pi+1 . In the CC model, all the subprocesses
p0 , p1 , ..., pm are performed on a server.
The edge subprocess pm is performed on edge nodes of level h - 1. The
subprocess pm on each edge node fR receives sensor data di from each descendant
sensors sRi (i = 0, 1, ..., lR -1, lR ≥ 1). The edge subprocess pm on each edge
node receives, processes input data DR from sensors, and sends output data dR
to the succeeding subprocess pm−1 . The subprocess pm−1 is performed on nodes
of level h - 2. Thus, a same subprocess pm−h+l+1 is performed on each node
fR of each level l (l = 0, 1, ..., h-1). The subprocess pm−h+2 is performed on
nodes f00 , ..., f0,l0 −1 of level 1. A subsequence pm−h+1 , pm−h , ..., p0 (m ≥ h) of
subprocesses are performed on the root node f0 .

2.2 Energy Consumption of Fog Nodes

Each node fR consumes the energy T ER (iR ) for input data DR of size iR (=
|DR |) and output data dR of size oR (= |dR | = ρR · iR ) [6,7,10]:
T ER (iR ) = EIR (iR ) + ECR (iR ) + ESR (iR + oR ) + δR · EOR (oR ). (1)
= EIR (iR ) + ECR (iR ) + ESR ((1 + ρR ) · iR ) + δR · EOR (ρR · iR ).
Here, iR = oR0 + oR1 + ... + oR,lR −1 . If fR is a root, δR = 0, else δR =
1. EIR (x), ECR (x), ESR (x), and EOR (x) show the energy consumption of
input, computation, storage, and output modules [6] to process data of size x,
respectively.
The execution time ETR (iR ) of each node fR to receive and compute input
data DR , send output data dR , and store the data DR and dR are given as
follows:
ETR (iR ) = T IR (iR ) + T CR (iR ) + T SR (iR + oR ) + δR · T OR (oR ). (2)
= T IR (iR ) + T CR (iR ) + T SR ((1 + ρR ) · iR ) + δR · T OR (ρR · iR ).
136 R. Oma et al.

Here, T IR , T CR , T SR , and T OR are execution time of input, computation,

storage, and output modules for input data of size iR . The power consumption
P IR and P OR of the input IR and output OR modules are proportional to the
receiving and transmission rates of a node fR , respectively. Hence, the energy
consumption EIR (iR ) and EOR (iR ) to receive and send data for input data of
size iR are given as follows:

EIR (iR ) = P IR · T IR (iR ). (3)

EOR (ρR · iR ) = P OR · T OR (ρR · iR ). (4)

P SR [W] shows the power consumption of a node fR to store data in a

database DBR which depends on the access rate aR [bps]. Hence, the energy
consumption ESR (iR ) of a fog node fR to store input and output data of size
iR is given as follows:

ESR ((1 + ρR ) · iR ) = P SR · T SR ((1 + ρR ) · iR ). (5)

3 A Modified Fault-Tolerant Tree-Based Fog Computing

(MFTBFC) Model
If a fog node fR stops by fault, every child fog node fRi is disconnected in the
tree. A node at the same level as a node fR is a candidate parent node of each
child node fRi . Let cn(fR ) be a set of candidate nodes for a node fR . We assume
each node fRi knows every candidate parent node. In the FTBFC model [8], one
candidate node fU is selected as a new parent node of all the disconnected node
fR0 , fR1 , ..., fR,lR −1 of a faulty node fR . A new parent node fU has to process
input data DR of size iR from the nodes fR0 , fR1 , ..., fR,lR −1 in addition to
input data DU of size iU from all the child nodes fU 0 , fU 1 , ..., fU,lU −1 . Hence,
the energy consumption T EU and execution time ETU of the node fU increase
to N T EU and N ETU , respectively, as follows:

N T EU = T EU (iU + iR ). (6)
N ETU = ETU (iU + iR ). (7)

The ratio T EU (iR + iU )/T EU (iU ) stands for increasing energy ratio IERU
of the node fU . The size ρU · iU of output data dU of the node fU also increases
to ρU · (iR + iU ). The ratio ETU (iR + iU )/ETU (iU ) is the increasing execution
time ratio IT RU of the node fU , T EU (iR + iU )/T EU (iU ) ≥ 1 and ETU (iR +
iU )/ETU (iU ) ≥ 1.
On the other hand, in a parent node fT of the faulty node fR , the energy con-
sumption T ET and execution time ETT decrease to N T ET and N ETT , respec-
tively, as follows:

N T ET = T ET (iT − ρR · iR ). (8)
N ETT = ETT (iT − ρR · iR ). (9)
 Energy-Efficient Recovery Algorithm 137

The size ρT · iT of output data dT of the node fT also decreases to ρT · (iT −

ρR · iR ). The ratios T ET (iT - ρR · iR )/T ET (iT ) and ETT (iT - ρR · iR )/ETT (iT )
are smaller than one.

Algorithm 1. ME (minimum energy) algorithm

Input : fR = faulty node;
Output: fU = new parent node;
1 E = ∞;
2 for each candidate fog node fS in cn(fR ) do
3 N T ES = T ES (iS + iR );
/* fog node fU whose N T EU is minimum. */
4 if N T ES < E then
5 U = S;
6 E = N T ES ;

A candidate node of a faulty node fR in the set cn(fR ) is selected as a

new parent fog node for every disconnected node in the minimum energy (ME)
algorithm Algorithm 1. Here, a candidate node fU whose N T EU (= T EU (iU +
iR )) is minimum is selected to be a new parent node of the disconnected nodes
fR0 , fR1 , ..., fR,lR −1 .
We consider how much energy the new parent node fU consumes to process
input data DU and DR . We consider a subprocess p(fR ) whose computation
complexity is O(x) and O(x2 ) for size x of input data. We consider a four-ary
height balanced tree with height h. One non-root, non-edge node fR is randomly
selected as a faulty node. Then, a new parent node fU is selected for fR in
the ME algorithm. Figures 2 and 3 show the energy consumption N T EU of a
new parent node fU . Here, N M E stands for the energy N T EU consumed by
a new parent node fU to just process input data from its child nodes. M E
shows energy consumption N T EU of a new parent node fU selected in the ME
algorithm, which additionally processes input data from disconnected nodes.
N T EU is about double of T EU for complexity O(x) and about 4.5 for O(x2 ).
As shown here, the energy to be consumed by a new parent node has to be
reduced. In this paper, the energy consumption of a new parent node is tried
to be reduced by taking multiple new parent nodes in this paper as discussed
later. This means, fewer number of disconnected nodes are connected to each
new parent node.
In order to reduce the energy consumption of a new parent node, disconnected
nodes fR0 , fR1 , ..., fR,lR −1 are distributed to multiple nodes as shown in Fig. 4.
In this paper, we propose an SMPR (selecting multiple parents for recovery)
algorithm to select a candidate node fU for each disconnected node fRi . The
SMPR algorithm is shown in Algorithm 2. First, a disconnected node fD whose
size oR of output data dD is maximum is selected. Then, a candidate node fU
where energy consumption T E(iU +oD ) to process both input data DU from the
138 R. Oma et al.

Fig. 2. Energy consumption of a new parent fog node of computation complexity O(x).

Fig. 3. Energy consumption of a new parent fog node of computation complexity O(x2 ).

child nodes fU 0 , fU 1 , ..., fU,lU −1 and output data dD from the disconnected node
fD is minimum. Here, the disconnected node fD is connected to the candidate
node fU . The node fD is now not disconnected. The size iU of input data of
the node fU is incremented by oD . Then, another disconnected node fD whose
output data size oD is maximum is selected. A candidate node fU is found for the
disconnected node fD is connected to the node fU and the node fD in the same
way. This procedure presented here is iterated until there is no disconnected
node.
 Energy-Efficient Recovery Algorithm 139

Fig. 4. SMPR algorithm.

Algorithm 2. SMPR (selecting multiple parents for recovery) algorithm

Input: fR = faulty node;
1 F = ch(fR );
/* set of disconnected nodes. */
2 while F = φ do
3 select a disconnected node fD in F whose size oD of output data is
maximum;
4 C = cn(fR );
/* set of candidate nodes of fR . */
5 while C = φ do
6 select a candidate node fU where T EU (iU + oD ) is minimum in C;
7 connect fD to fU ;
8 ch(fU ) = ch(fU ) ∪ {fD };
9 pt(fD ) = {fU };
10 iU = iU + oD ;
11 C = C - {fU };
12 F = F - {fD };

4 Evaluation
We evaluate the SMPR algorithm to select a candidate node for each discon-
nected node in the MFTBFC model proposed in this paper. We consider a
height-balanced k-ary tree with height h (≥ 1) of the MFTBFC model, where
each non-edge node fR has k (≥ 1) child nodes fR0 , fR1 , ..., fR,k−1 (lR = k) and
every edge node is at level h − 1. There are totally k h−1 edge nodes. Let SD be
the total size [Byte] of sensor data sent by all the sensors. That is, the sensors
totally send sensor data SD [Byte] to k h−1 edge nodes. In this paper, we assume
SD is 1 [MB]. Each edge node receives some of sensor data of SD. The size of
sensor data which each edge node receives is randomly decided.
In this evaluation, an application process p is a sequence of subprocesses
p0 , p1 , ..., pm (m ≥ h - 1) where pm is an edge subprocess. As presented in this
paper, a subprocess pm−h+1+l is deployed on fog nodes of each level l (1 ≤ l <
h). Subprocesses pm−h+1 , pm−h , ..., p0 are performed on a root node, i.e. server
140 R. Oma et al.

f0 of layer 0. We consider a pair of types of subprocesses which receive input

data of size x, which are characterized by the computation complexity O(x) or
O(x2 ).
In the evaluation, one node fR is randomly selected to be faulty for each level
l (0 < l < h - 1). cn(fR ) is a set of candidate nodes of the faulty node fR . We
assume neither a root node nor every edge node is faulty for simplicity. That is,
every sensor data is received by one edge node. If a node fR is faulty, k child
nodes fR0 , fR1 , ..., fR,k−1 of the faulty node fR are disconnected.
We consider a random (RD), ME, and SMPR algorithms to select a candidate
node for each disconnected node in the set cn(fR ) of a faulty node. In the RD and
ME algorithms, one candidate node is selected for all the disconnected nodes.
In the RD algorithm, a new parent node is randomly selected in the candidate
nodes. In the ME algorithm, a new parent node is selected, whose expected
energy consumption is minimum in the candidate nodes. On the other hand, a
candidate node whose expected energy consumption is minimum is selected as a
new parent node for each disconnected node. That is, multiple candidate nodes
are selected as new parent nodes.
Figures 5 and 6 show the average energy consumed by a new parent node in
the tree where one node is randomly faulty at level l (0 < l < h) for the com-
putation complexity O(x) and O(x2 ), respectively. ST E, M T E, and RT E show
energy consumed by a new parent node in the SMPR, ME, and RD algorithms,
respectively, where a node is randomly selected to be faulty. Let N ST E, N M T E,
and N RT E be average energy consumption of a new parent node selected by
the SMPR, ME, and RD algorithms, respectively, where no fault occurs in the
tree. Figures 5 and 6 show the ratios ST E, M T E, RT E, N ST E, N M T E, and
N RT E to RT E of h = 3. RT E is maximum for a tree of h = 3. The energy
consumption of a new parent node selected by the SMPR algorithm is about
34% and 25% for computation complexity O(x) and 57% and 44% for O(x2 )
smaller than the RD and ME algorithms, respectively.

Fig. 5. Energy consumption of a new parent node of computation complexity O(x) for
height h.
 Energy-Efficient Recovery Algorithm 141

Next, we measure the execution time SET , M ET , and RET of a new par-
ent node selected in the SMPR, ME, and RD algorithms, respectively, where
a node is randomly selected to be faulty. Let N SET , N M ET , and N RT E be
a execution time of a new parent node selected by the SMPR, ME, and RD
algorithms, respectively, where no fault occurs in the tree. Figures 7 and 8 show
the execution time ratios SET , M ET , RET , N SET , N M ET , and N RT E of a
new parent node to RT E of h = 3 for computation complexity O(x) and O(x2 ),
respectively. The execution time of a new parent fog node selected in the SMPR
algorithm is about 34% and 25% for computation complexity O(x) and 57% and
44% for O(x2 ) shorter than the RD and ME algorithms, respectively.
As shown in Figs. 5, 6, 7 and 8, the energy consumption and execution time
of a new parent node for disconnected nodes can be reduced.

Fig. 6. Energy consumption of a new parent node of computation complexity O(x2 )

for height h.

Fig. 7. Execution time of a new parent node of computation complexity O(x) for
height h.
142 R. Oma et al.

Fig. 8. Execution time of a new parent node of computation complexity O(x2 ) for
height h.

5 Concluding Remarks
The IoT is scalable and includes sensors and actuators in addition to servers.
Subprocesses of an application process are performed on not only servers but also
fog nodes in the fog computing (FC) model in order to reduce the traffic of net-
works and servers. In this paper, we proposed the modified FTBFC (MFTBFC)
model where disconnected nodes of a faulty node are partitioned into groups
and nodes in each group are connected to a different candidate node. We newly
proposed the SMPR (selecting multiple parents for recovery) algorithm where a
candidate node is selected for each disconnected node as a new parent node so
that energy consumption of the new parent node can be reduced. Since discon-
nected nodes are connected to multiple new parent nodes, the traffic of each new
parent node can be reduced. We evaluated the SMPR algorithm of the MFTBFC
model in terms of the energy consumption and execution time of a new parent
node. In the evaluation, we showed the energy consumption and execution time
of a new parent node can be reduced in the SMPR algorithm compared with
other algorithms.

Acknowledgements. This work was supported by JSPS KAKENHI grant number

15H0295.

References
1. Iot for automotive - ibm watson iot. https://www.ibm.com/internet-of-things/
explore-iot/vehicles/automobiles
2. Creeger, M.: Cloud computing: an overview. Queue 7(5), 3–4 (2009)
3. Enokido, T., Ailixier, A., Takizawa, M.: Process allocation algorithms for saving
power consumption in peer-to-peer systems. IEEE Trans. Indus. Electron. 58(6),
2097–2105 (2011)
 Energy-Efficient Recovery Algorithm 143

4. Enokido, T., Ailixier, A., Takizawa, M.: An extended simple power consumption
model for selecting a server to perform computation type processes in digital ecosys-
tems. IEEE Trans. Indus. Inform. 10, 1627–1636 (2014)
5. Hanes, D., Salgueiro, G., Grossetete, P., Barton, R., Henry, J.: IoT Fundamentals:
Networking Technologies, Protocols, and Use Cases for the Internet of Things.
Cisco Press, Indianapolis (2018)
6. Oma, R., Nakamura, S., Duolikun, D., Enokido, T., Takizawa, M.: An energy-
efficient model for fog computing in the Internet of Things (IoT). Internet Things
1–2, 14–26 (2018). https://doi.org/10.1016/j.iot2018.08.003
7. Oma, R., Nakamura, S., Duolikun, D., Enokido, T., Takizawa, M.: Evaluation of
an energy-efficient tree-based model of fog computing. In: Proceedings of the 21st
International Conference on Network-Based Information Systems, NBiS 2018, pp.
99–109 (2018)
8. Oma, R., Nakamura, S., Duolikun, D., Enokido, T., Takizawa, M.: Fault-tolerant
fog computing models in the IoT. In: Proceedings of the 13th International Con-
ference on P2P, Parallel, Grid, Cloud and Internet Computing, 3PGCIC 2018, pp.
14–25 (2018)
9. Oma, R., Nakamura, S., Enokido, T., Takizawa, M.: An energy-efficient model
of fog and device nodes in IoT. In: Proceedings of IEEE the 32nd International
Conference on Advanced Information Networking and Applications, AINA 2018,
pp. 301–306 (2018)
10. Oma, R., Nakamura, S., Enokido, T., Takizawa, M.: A tree-based model of energy-
efficient fog computing systems in IoT. In: Proceedings of the 12th International
Conference on Complex, Intelligent, and Software Intensive Systems, CISIS 2018,
pp. 991–1001 (2018)
11. Rahmani, A.M., Liljeberg, P., Preden, J.-S., Jantsch, A.: Fog Computing in the
Internet of Things. Springer, Heidelberg (2018)
 Energy-Efficient Group Migration
of Virtual Machines in a Cluster

Dilawaer Duolikun1(B) , Tomoya Enokido2 , and Makoto Takizawa1

1
Hosei University, Tokyo, Japan
[email protected], [email protected]
2
Rissho University, Tokyo, Japan
[email protected]

Abstract. In order to realize green society, we take the process migra-

tion approach to reducing the total electric energy consumption of clus-
ters by migrating virtual machines. In our previous studies, the DM
(Dynamic Migration) algorithm is proposed where virtual machines are
dynamically resumed and suspended and a group of virtual machines on
a host server migrate to a guest server so that the total energy consump-
tion of the host and guest servers can be reduced. In this paper, we newly
propose a DGM (Dynamic Group Migration) algorithm. Here, there is
a seed virtual machine on each server. A virtual machine is created as
a clone of the seed virtual machine so that the number of processes on
each virtual machine is kept fewer. In the evaluation, we show the total
electric energy consumption of servers can be more reduced in the DGM
algorithm compared with other algorithms.

Keywords: Dynamic virtual machine migration ·

Group migration of virtual machines · DGM algorithm · Clone

1 Introduction
In order to reduce energy [J] consumed by servers like cloud computing systems
[18], power consumption and computation models [9–13,15] are proposed to show
the total power consumption [W] and execution time of a whole server to perform
application processes. Based on the models, algorithms [10–12,14] are proposed
to select an energy-efficient server to perform an application process issued by a
client.
A process migration approach [2,4–6] is also proposed where a process on
a host server migrates to a more energy-efficient server. Especially, processes
can easily migrate from a host server to a guest server by migrating a vir-
tual machine. In the static migration algorithm [4], a cluster supports the fixed
number of virtual machines. Dynamic migration algorithms [7,8] are proposed
where virtual machines are dynamically resumed and suspended as the number
of application processes increases and decreases, respectively, and one virtual
machine migrates. In a dynamic migration algorithm DM(*) [1,8], multiple vir-
tual machines migrate from a host server to a guest server. Here, it is shown
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 144–155, 2020.
https://doi.org/10.1007/978-3-030-15032-7_12
 Energy-Efficient Group Migration of Virtual Machines in a Cluster 145

the energy consumption of servers and the average execution time of applica-
tion processes can be reduced by migrating virtual machines. In the dynamic
migration algorithms, enough number of dormant virtual machines are assumed
to be supported by each server. However, it is not easy to a priori know how
many number of virtual machines are needed. In this paper, there is one seed
virtual machine on each server. If more number virtual machines are required
to perform application processes, a new virtual machine is created as a clone
of the seed virtual machine. We propose an algorithm to estimate the energy
to be consumed by servers when new virtual machine is created when a new
application process is performed by taking into consideration time to create a
virtual machine. In the evaluation, we show the energy consumption of servers
and the average execution time of processes are reduced in the DGM algorithm.
In Sect. 2, we present a system model. In Sect. 3, we present the power con-
sumption and computation models. In Sect. 4, we propose the DGM algorithm.
In Sect. 5, we evaluate the DGM algorithm.

2 System Model
A cluster is composed of servers s1 , . . . , sm (m ≥ 1). An application process
is performed on a server st . In this paper, a process stands for an application
process which uses CPU. A process performed on a server is active. A idle process
is not active. In each server st , npt (≥ 1) homogeneous CPUs are supported. Each
CPU includes cct (≥ 1) cores, each of which supports ctt (≥ 1) threads. A server st
supports the total number ntt (= npt ·cct ·ctt ) of threads on nct (= npt ·cct ) cores.
Processes on different threads are performed independently of one another. A
thread is active if and only if (iff) at least one process is active. CPt (τ ) is a set of
active processes on a server st at time τ . V CPh (τ ) shows a set of active processes
on vmh at time τ . The size |vmh | shows the number of active processes on a
virtual machine vmh . vmh is smaller than vmk (vmh < vmk ) iff |vmh | < |vmk |.
Each server st supports a seed virtual machine svmt . A virtual machine vmh
is created on a server st as a clone svmt . A server is engaged iff at least one
ready virtual machine resides. V Mt is a set of virtual machines on a server st .
A virtual machine vmh is active and idle iff some process and no process are
active, respectively. A virtual machine vmh on a host server is assumed to be
able to migrate to any server in the live manner.
The migration time mgttu to migrate v virtual machines from a host server
st to a guest server su is given by a function mgt(v) = mt + β · (v − 1). Let crt
be time [sec] to create a clone of a virtual machine. Figure 1 shows time crt to
create a clone of a virtual machine with memory of size ms.

3 Estimation of Energy Consumption

We assume processes are fairly allocated with CPUs, cores, and threads in each
server [17]. In the MLPCM (multi-level power consumption) model [14,16], the
power consumption N Et (n) is minEt +napt (n)·bEt +nact (n)·cEt +natt (n)·tEt
146 D. Duolikun et al.

Fig. 1. Time crt to create a clone of a virtual machine

[W]. Here, napt (n) = n if n ≤ npt , else npt , nact (n) = n if n ≤ nct , else nct ,
and natt (n) = n if n ≤ ntt , else ntt . A server st consumes the maximum power
maxEt = minEt +npt ·bEt +nct ·cEt +ntt ·tEt [W] if every thread is active. The
power Et (τ ) of a server st at time τ is N Et (|CPt(τ )|). The energy consumed by
et
a server st from time st [tu (time unit)] to et is τ =st N Et (|CPt (τ )|) [Wtu].
If only a process pi is performed on a server st without any other process, the
execution time Tti [tu] of pi is shortest, Tti = minTti . A server st where minTti is
the shortest is f astest. minTi shows minTf i for a fastest server sf . One virtual
computation step [vs] is performed on a thread of the fastest server sf for one
time unit [tu] [12,13], i,e. the thread computation rate T CRf = 1 [vs/tu]. For
every server st (= sf ), T CRt ≤ T CRf (= 1). The total number V Ci [vs] of virtual
computation steps of pi is minTi [tu] · T CRf [vs/tu] = minTi [vs]. On a fastest
server sf , maxP CRf i = T CRf = 1. For every pair of processes pi and pj on
st , maxP CRti = maxP CRtj = T CRt (≤ 1) [13]. The server computation rate
SCRt (τ ) of a server st at time τ is att (τ ) · T CRt where att (τ )(≤ ntt ) is the
number of active threads. The maximum server computation rate maxSCRt
is ntt · T CRt . Here, att (τ ) is assumed to be natt (n) for n(= |CPt (τ )|) active
processes on st at τ . In the MLCM model [13], n processes are performed on st
at the server computation rate N SRt (n) [vs/tu]:

n · T CRt if n ≤ ntt .
N SRt (n) = (1)
maxSCRt (= ntt · T CRt ) if n > ntt .
 Energy-Efficient Group Migration of Virtual Machines in a Cluster 147

Each process pi is performed at rate N P Rti (n) = N SRt (n)/n(≤ T CRt )

[vs/tu] on a server st for n active processes. SCRt (τ ) [vs/tu] of a server
st at time τ is N SRt (n) for n = |CPt (τ )|. The process computation rate
P CRti (τ ) [vs/tu] of each process pi on st at τ is SCRt (τ )/n = N SRt (n)/n =
N P Rti (n). Suppose a process pi on a server st starts at time st and ends at
et. Here,the total number V Ci of virtual computation steps of a process pi is
et
V Ci = τ =st N P Rti (|CPt (τ )|) = minTi [vs]. If pi starts, pi gets active and
the variable plci is V Ci [vs]. If pi is active, plci = plci + N P Rti (|CPt (τ )|)(=
N P Rt (|CPt (τ )|)). Then, pi terminates if plci ≤ 0.
The total number of virtual computation steps to be performed by nt pro-
cesses is assumed to be αt (nt ) · nt :
⎧
⎪ 0.5 for nt ≤ ntt .
⎪
⎨
0.6 for ntt < nt ≤ 2 · ntt .
αt (nt ) = (2)
⎪
⎪ 0.8 for 2 · ntt < nt ≤ 4 · ntt .
⎩
1 for nt > 4 · ntt .
Suppose k processes newly start on a server st where nt processes are active.
Here, the total number of virtual computation steps of the k new and nt active
processes are is αt (nt ) · nt + k. It takes (αt (nt ) · nt + k)/N SRt (nt + k) [tu]
to perform (αt (nt ) · nt + k) steps of (nt + k) processes. Hence, the expected
termination time SETt (nt , k) [tu] and energy consumption SEEt (nt , k) [W tu]
of each server st are:

1 SETt (nt , k) = (αt (nt ) · nt + k)/N SRt (nt + k) [tu].

2 SEEt (nt , k) = SETt (nt , k) · N Et (nt + k)
= (αt (nt ) · nt + k) · N Et (nt + k)/N SRt (nt + k) [Wtu].

The nt active processes are expected to terminate on a server st at time

ETt = SETt (nt , 0). The expected termination time N Tt = M ETt (nt , tnv u ) and
energy consumption N Et = M EEt (nt , tnv u ) of st to which a group GVu of
virtual machines with tnv u processes migrate from su to st are:

⎧
⎨ mtut + ((1 − mtut /ETt ) · αt (nt ) · nt +
M T Et (nt , tnv u ) = αu (nu ) · tnv u )/N SRt (nt + tnv u ) if ETt > mtut . (3)
⎩
mtut + (αu (nu ) · tnv u )/N SRt (tnv u ) otherwise.

⎧
⎪
⎪ mtut · N Et (nt ) +
⎪
⎪
⎨ (M T Et (nt , tnv u ) − mtut ) · N Et (nt + tnv u )
M EEt (nt , tnv u ) = if ETt > mtut . (4)
⎪
⎪
⎪
⎪ ET t · N E (n
t t ) + minE t · (mt ut − ET t ) +
⎩
(M T Et (nt , tnv u ) − ETt ) · N Et (tnv u ) otherwise.

On the other hand, the server su consumes energy N Eu = SEEu (nu −

tnv u , 0) [Wtu] and it takes N Tu = SETu (nu − tnv u , 0) to perform (nu − tnv u )
processes. We introduce the following function T EE [3]:
148 D. Duolikun et al.


ET + EU + (T T − T U ) M U if T T ≥ T U .
T EE(ET, T T, M T, EU, T U, M U ) = (5)
ET + EU + (T U − T T ) M T if T T < T U.

The servers st and su consume energy T EE (N Et , N Tt , minEt , N Eu , N Tu ,

minEu ).
Next, a new process pi is issued to a server st . There are two ways. i.e. pi is
performed on an exiting virtual machine and on a new virtual machine created.
Every current process is estimated to terminate by time ETt = SEEt (α(nt ) ·
nt , 0). In the first case, every current process and the new processpi terminate
by time N ETt = SETt (α(nt ) · nt , 1). The server st consumes energy N EEt =
N Et (nt + 1) · N ETt .
Secondly, crt stands for time to create a clone. If crt ≥ ETt , one virtual
machine vmh is selected to perform pi as shown in Fig. 2(1). Next, suppose
crt < ETt . The process pi starts on a new virtual machine vmh at time τ + crt
after vmh is created, Then, α(nt )·nt ·crt/ETt steps of nt processes are performed
until time τ + crt. Here, totally α(nt ) · nt · (1 − crt/ETt ) + 1 steps have to be
performed after time τ + crt for N N Tt = SET t (α(nt ) · nt · (1 − crt/ETt ), 1). The
new process pi and nt current processes and the new process pi are estimated to
terminate by τ + crt + N N Tt Fig. 2(2). The server st consumes energy N N E t (=
N Et (nt ) · crt + N Et (nt + 1) · N N Tt ). Hence, if crt < ETt , a new virtual machine
can be created. If N N E t < N EE t , a new virtual machine vmh is created where
the process pi is performed. Otherwise, one virtual machine vmh is selected. The
DVM (dynamic virtual machine) algorithm is shown in Algorithm 1.

Fig. 2. Electric energy consumption.

4 A Dynamic Migration Algorithm

Suppose a cluster S is composed of servers s1 , . . . , sm (m ≥ 1). A variable V M is
a set of virtual machines in the cluster S. A variable V Mt (⊆ V M ) shows a set
of ready virtual machines on each server st . There is one seed virtual machine
 Energy-Efficient Group Migration of Virtual Machines in a Cluster 149

svmt on each server st . That is, V Mt = φ for every server st and V M = φ.

A virtual machine vmh is created as a clone of the seed virtual machine svmt .
V Mt = V Mt ∪ {vmh }. If vmh is dropped, V Mt = V Mt − {vmh }. A pair of
variables nt and nvh denote the numbers of processes on st and vmh , respectively.

Algorithm 1. Dynamic virtual machine (DVM) selection

Input : pi = a new process pi ;
Output: st = server; vmh = virtual machine on st
1 select a host server st where SEEt (nt , 1) is minimum;
2 ETt = SETt (α(nt ) · nt , 0);
3 N ETt = SETt (α(nt ) · nt , 1); N EEt = N Et (nt + 1) · N ET t ;
4 N N Tt = SETt (α(nt ) · nt · (1- crt / ETt ), 1); N N Et = N Et (nt ) · crt +
N Et (nt + 1) · N ET t ;
5 if V Mt = ∅ then
6 if nt / |V Mt | ≤ maxN V Mt , then
7 select a smallest virtual machine vmh ;
8 else
9 if crt < ETt , then
10 select a smallest virtual machine vmh ;
11 else
12 if N N E t < N Et then
13 create a virtual machine vmh as a clone of svmt ;
14 else
15 select a smallest virtual machine vmh ;

16 else
17 create a virtual machine vmh ;

Each engaged server st is periodically checked. Here, in the set V Mt =

{vmt1 , . . . , vmtvt }(vt ≥ 0) are ordered in terms of size as vmt1 ≤ . . . ≤ vmtvt .
[DGM (Dynamic Group Migration) algorithm]

1 If a client issues a process pi to cluster S, a virtual machine vmh on a host

server st is selected to perform pi by using the DVM algorithm Algorithm 1.
The process pi is performed on vmh .
2 For each server st , the GVM algorithm Algorithm 2 is periodically performed,
where a group GVt of virtual machines and a guest server su are selected. The
group GVt migrate from st to su .

5 Evaluation

The DGM algorithm is evaluated in the simulation. The total energy consump-
tion T EE [Wtu] and total active time T AT [tu] of servers s1 , . . . , sm (m ≥ 1) and
150 D. Duolikun et al.

Algorithm 2. Group virtual machine migration GV M t on a server st

Input : st = hoset server;
V Mt = set of virtual machines in st where vmt1 ≤ . . . ≤ vmtvt ;
Output: s = guest server;
GVt = group of virtual machines to migrate from st to su ;
1 EEt = SEEt (nt , 0); ETt = SETt (nt , 0);
2 for each server su (= st ), do
3 EEu = SEEu (nu , 0); ETu = SETu (nu , 0);
4 CEEtu = T EE(EEt , ETt , minEt , EEu , ETu , minEu );
5 nv = 0; N E = ∞; s = NULL; GVt = φ; i = 1;
6 while nvti > 0 do
7 nv = nv + nvti ; /* nv processes migrate from st to su */
8 N Et = SEEt (nt − nv, 0) N Tt = SETt (nt − nv, 0);
9 s = NULL;
10 for each server su (= st ) /* vmt1 , . . ., vmti to migrate to su */
11 do
12 N Etu = M EEu (nu , nv); N Ttu = M T Eu (nu , nvh );
13 N EEtu = T EE(N Et , N Tt , minEt , N Etu , N Ttu , minEu );
14 if N EEtu < CEEtu and N EEtu < N E, then
15 N E = N EEtu ; s = su ;
16 GVt = GVt ∪ {vmti }; /* vmti is selected */
17 else

18 i = i + 1;
19 if s = NULL, /* guest server is found */
20 then
21 migrate virtual machines in GVt from st to s;
22 else

the average execution time AET [tu] of processes p1 , . . . , pn (n ≥ 1) are measured

compared with random (RD), round robin (RR), and SGEA [13] algorithms. We
consider four servers s1 , . . . , s4 (m = 4) in our laboratory, whose power consump-
tion and performance parameters of each server st are shown in Table 1. In the
RD, RR, and SGEA algorithms, a set V M of virtual machines includes sixteen
virtual machines vm1 , . . . , vm16 (nv = 16) and no virtual machine migrate. Each
server st initially hosts four virtual machines, i.e. |V Mt | = |V M |/4 = 4.
In the RD algorithm, one virtual machine vmh is randomly selected to per-
form each process pi . In the RR algorithm, vmh is selected after vmh−1 . In
the SGEA algorithm, a server st is first selected so as to minimize the total
energy consumption of all the servers. Then, pi is performed on a smallest vir-
tual machine of the server st .
In the DGM algorithm, a group GVt of virtual machines migrate from a host
server st to a guest server su . Here, the migration time mttu of v virtual machines
is mgt(v)(= 2+2.5v−0.5). The clone creation time crt is 5 [tu]. In addition, there
is initially no virtual machine on each server st , i.e. V Mt = φ. Clones of a seed
 Energy-Efficient Group Migration of Virtual Machines in a Cluster 151

virtual machine are created and idle virtual machines are dropped depending on
average number maxN V Mt of processes on each virtual machine. Each engaged
server st is checked every σ = 5 [tu] and a group GVt of virtual machines migrate
to a guest server su if the total energy consumption of st and su can be reduced.
Processes p1 , . . . , pn (n ≥ 1) are randomly issued to the cluster S. One time
unit [tu] is assumed to be 100 [msec]. In each process configuration P Fng , the
minimum execution time minTi of each process pi is randomly taken from 15
to 25 [tu] and V Ci [vs] is minTi . The half n/2 of the processes randomly start
at time xtime/4 − xtime/40 to xtime/4 + xtime/40. One quarter n/4 of the
processes randomly start at time xtime/4 − xtime/20 to xtime/4 − xtime/40
and at xtime/4 + xtime/40 to xtime/4 + xtime/20. The other n/4 processes
randomly start from time 0 to xtime/4−xtime/20 and from xtime/4+xtime/20
to xtime − 1. The starting time stimei of each process pi is thus decided. The
ending time etimei of each process pi is obtained in the simulation. The execution
time ETi of each process pi is etimei − stimei + 1. xtime is 1,000 [tu] (= 100
[sec]). We randomly generate eight process configurations P Fn1 , . . . , P Fn8 for
each number n of processes.
EEt and ATt denote the energy consumption [Wtu] and active time [tu]
of each server st , respectively. T EE shows the total energy consumed by the
servers. ETi denotes the execution time of each process pi and T ET is ET1 +
. . . + ETn . Initially, ETi = 0. In the simulation, EEt is incremented by the
power consumption N Et (n) − minEt [W] for number n = |CPt (τ )| of active
processes on the server st and ATt is incremented by one for n > 0 at each time
τ . Then, if pi terminates on st at time τ , ETi = τ − stimei + 1 and T ET is
incremented by ETt . When the simulation ends, EEt and ATt show the total
energy [Wtu] and total active time [tu] of each server st , respectively. The total
energy consumption T EE is EE1 + . . . + EE4 . AET is T ET /n.
Figure 3 shows the total energy consumption T EE [Wtu] of the servers for
number n of processes. T EE of the RD is almost the same as the RR. In the
DGM algorithm, multiple virtual machines migrate from a host server to a guest
server. T EE of the DGM algorithm is smallest in the algorithms. For example,
T EE of the DGM is about 50% of the RD and RR, and 5% of the SGEA for
n = 2, 000.
Figure 4 shows the total active time T AT [tu] of the servers for the number
n of processes. T AT of the RR is almost same as the RD. T AT of the DGM
algorithm is about 45% of the RD and RR and about 5 to 10% shorter than
SGEA.
Figure 5 shows the average execution time AET [tu] of n processes. AET of
the DGM algorithm is the shortest in the algorithms. In the DGM algorithm,
AET of processes is about 5 to 10% shorter than SGEA.
152 D. Duolikun et al.

Fig. 3. Total electric energy consumption T EE (m = 4, σ = 5, maxN V Mt = 10).

Fig. 4. Total active time T AT (m = 4, σ = 5, maxN V Mt = 10).

 Energy-Efficient Group Migration of Virtual Machines in a Cluster 153

Fig. 5. Average execution time AET of processes (m = 4, σ = 5, maxN V Mt = 10).

Table 1. Parameters of servers.

Parameters s1 s2 s3 s4
npt 2 1 1 1
nct 8 8 6 4
ntt 32 16 12 8
CRTt [vs/tu] 1.0 1.0 0.5 0.7
maxCRt [vs/tu] 32 16 6 5.6
minEt [W] 126.1 126.1 87.2 41.3
maxEt [W] 301.3 207.3 136.2 89.6
bEt [W] 30 30 16 15
cEt [W] 5.6 5.6 3.6 4.7
tEt [W] 0.8 0.8 0.9 1.1

6 Concluding Remarks
In this paper, we discussed the virtual machine migration approach to reduc-
ing the energy consumption of servers. In this paper, we proposed the DGM
algorithm where a group of virtual machines migrate from a host server to a
guest server. In addition, virtual machines are created as clones of a seed vir-
tual machine on each server. In this paper, we consider time to create a clone
154 D. Duolikun et al.

and migrate virtual machines. In the evaluation, we showed the total energy
consumption and total active time of servers and the average execution time of
processes can be reduced in the DGM algorithm compared with non-migration
algorithms.

Acknowledgement. This work was supported by Japan Society for the Promotion of
Science (JSPS) KAKENHI 15H0295 and Grant-in-Aid for JSPS Research Fellow grant
18J10022.

References
1. Duolikun, D., Enokido, T., Takizawa, M.: Static and dynamic group migration
algorithms of virtual machines to reduce energy consumption of a server cluster.
Int. J. Trans. Comput. Collective Intell. (2019, accepted)
2. Duolikun, D., Enokido, T., Takizawa, M.: An energy-aware algorithm to migrate
virtual machines in a server cluster. Int. J. Space-Based Situated Comput. 7(1),
32–42 (2017)
3. Duolikun, D., Nakamura, S., Enokido, T., Takizawa, M.: An energy-efficient
dynamic live migration of multiple virtual machines. In: Proceedings of the
21st International Conference on Network-Based Information Systems, NBiS-2018
(2018, accepted)
4. Duolikun, D., Nakamura, S., Watanabe, R., Enokido, T., Takizawa, M.: Energy-
aware migration of virtual machines in a cluster. In: Proceedings of the 11th Inter-
national Conference on Broadband and Wireless Computing, Communication and
Applications, BWCCA 2016, pp. 21–32 (2016)
5. Duolikun, D., Watanabe, R., Enokido, T., Takizawa, M.: An Eco algorithm for
dynamic migration of virtual machines in a server cluster. In: Proceedings of the
20th International Conference on Network-Based Information Systems, NBiS 2017,
pp. 42–54 (2017)
6. Duolikun, D., Watanabe, R., Enokido, T., Takizawa, M.: Energy-aware dynamic
migration of virtual machines in a server cluster. In: Proceedings of the 12th Inter-
national Conference on Broadband and Wireless Computing, Communication and
Applications, BWCCA 2017, pp. 161–172 (2017)
7. Duolikun, D., Watanabe, R., Enokido, T., Takizawa, M.: An Eco migration algo-
rithm of virtual machines in a server cluster. In: Proceedings of IEEE the 32nd
International Conference on Advanced Information Networking and Applications,
AINA 2018 (2018, accepted)
8. Duolikun, D., Watanabe, R., Enokido, T., Takizawa, M.: Energy-efficient replica-
tion and migration of processes in a cluster. In: Proceedings of the 12th Interna-
tional Conference on Complex, Intelligent and Software Intensive Systems, CISIS
2018 (2018, accepted)
9. Enokido, T., Aikebaier, A., Deen, M., Takizawa, M.: Power consumption-based
server selection algorithms for communication-based systems. In: Proceedings of
the 13th International Conference on Network-based Information Systems, NBiS
2010, pp. 201–208 (2010)
10. Enokido, T., Aikebaier, A., Takizawa, M.: A model for reducing power consumption
in peer-to-peer systems. IEEE Syst. J. 4(2), 221–229 (2010)
11. Enokido, T., Aikebaier, A., Takizawa, M.: Process allocation algorithms for saving
power consumption in peer-to-peer systems. IEEE Trans. Indus. Electron. 58(6),
2097–2105 (2011)
 Energy-Efficient Group Migration of Virtual Machines in a Cluster 155

12. Enokido, T., Aikebaier, A., Takizawa, M.: An extended simple power consump-
tion model for selecting a server to perform computation type processes in digital
ecosystems. IEEE Trans. Indus. Inform. 10(2), 1627–1636 (2014)
13. Kataoka, H., Duolikun, D., Enokido, T., Takizawa, M.: Multi-level computation
and power consumption models. In: Proceedings of the 18th International Confer-
ence on Network-Based Information Systems, NBiS 2015, pp. 40–47 (2015)
14. Kataoka, H., Duolikun, D., Enokido, T., Takizawa, M.: Energy-aware server selec-
tion algorithm in a scalable cluster. In: Proceedings of IEEE the 30th International
Conference on Advanced Information Networking and Applications, AINA 2016,
pp. 565–572 (2016)
15. Kataoka, H., Nakamura, S., Duolikun, D., Enokido, T., Takizawa, M.: Multi-level
power consumption model and energy-aware server selection algorithm. Int. J. Grid
Utility Comput. 8(3), 201–210 (2017)
16. Kataoka, H., Sawada, A., Duolikun, D., Enokido, T., Takizawa, M.: Simple energy-
efficient server selection algorithm in a scalable cluster. In: Proceedings of the 11th
International Conference on Broadband and Wireless Computing, Communication
and Applications, BWCCA 2016, pp. 45–46 (2016)
17. Mittal, N., Garg, K., Ameria, A.: A paper on modified round robin algorithm. Int.
J. Latest Technol. Eng. Manag. Appl. Sci. 5(11), 93–98 (2015)
18. Rafaels, R.J.: Cloud Computing: From Beginning to End. Create Space Indepen-
dent Publishing Platform (2015)
 A Secure and Trustworthy Intelligent
System for Clustering in VANETs Using
Fuzzy Logic

Kevin Bylykbashi1(B) , Yi Liu1 , Donald Elmazi2 , Keita Matsuo2 ,

Makoto Ikeda2 , and Leonard Barolli2
1
Graduate School of Engineering, Fukuoka Institute of Technology (FIT),
3-30-1 Wajiro-Higashi, Higashi-Ku, Fukuoka 811–0295, Japan
[email protected], [email protected]
2
Department of Information and Communication Engineering,
Fukuoka Institute of Technology (FIT),
3-30-1 Wajiro-Higashi, Higashi-Ku, Fukuoka 811-0295, Japan
[email protected], {kt-matsuo,barolli}@fit.ac.jp,
[email protected]

Abstract. Recently, smart cities and Internet of Things (IoT) applica-

tions, such as Vehicular Ad-hoc Networks (VANETs) and Opportunistic
networks have been deeply investigated. However, these kinds of wireless
networks have security problems. Also, the vehicles can be not trust-
worthy, which brings different communication problems. In this work,
we present a Fuzzy Cluster Management System (FCMS) for VANETs.
For FCMS, we use four input parameters: Vehicle Relative Speed with
Vehicle Cluster (VRSVC), Vehicle Degree of Centrality (VDC), Vehicle
Security (VS) and Vehicle Trustworthiness (VT). The output parameter
is Vehicle Remain or Leave Cluster (VRLC). We evaluate the proposed
system by computer simulations. The simulation results show that vehi-
cles with the same VRSVC and with high VDC, VS and VT values have
higher possibility to remain in the cluster.

1 Introduction

The traffic congestion is one of the most serious problems faced by big cities
and has made the people live in difficult situations. Smart cities would like to
plan how to minimize their transportation problems due to the increasing pop-
ulation that produces congested roads [1]. Smart city can use Smart Computing
technologies to make the critical infrastructure components and services of a
city, which include city administration, education, health-care, public safety,
real estate, transportation, and utilities more intelligent, interconnected, and
efficient [2].
The Vehicular Ad-hoc Networks (VANETs) can help to alleviate this issue
improving vehicles’ mobility, increasing road safety and also seeking to have more
sustainable cities. VANETs are a type of wireless networks that have emerged
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 156–165, 2020.
https://doi.org/10.1007/978-3-030-15032-7_13
 A Secure and Trustworthy Intelligent System for Clustering in VANETs 157

thanks to advances in wireless communication technologies and the automotive

industry. Nowadays, every car is likely to be equipped at least with some of
the following devices: an on-board computer, a huge capacity storage device, a
GPS device, a communication module, an antenna, a short-range rear collision
radar device, cameras and various sensors which can gather information about
the road and environment conditions to ensure the driving safety.
There are different works for VANETs. In [3], the authors proposed a Message
Suppression Controller (MSC) for V2V and V2I communications. They consid-
ered some parameters to control the message suppression dynamically. However,
a fixed parameter still is used to calculate the duration of message suppression.
To solve this problem, the authors proposed an Enhanced Message Suppression
Controller (EMSC) [4] for Vehicular-DTN (V-DTN). The EMSC is an expanded
version of MSC [3] and can be used for various network conditions. But, many
control packets were delivered in the network.
Security and trust in VANETs is essential in order to prevent malicious agents
sabotaging road safety systems built upon the VANET framework, potentially
causing serious disruption to traffic flows or safety hazards. Several authors have
proposed cluster head metrics which can assist in identifying malicious vehicles
and mitigating their impact by denying them access to cluster resources [5].
Security of the safety messages can be achieved by authentication [6]. To
make the process of authentication faster [7], vehicles in the communication
range of an Road Side Unit (RSU) can be grouped to be in one cluster and a
cluster head is elected to authenticate all the vehicles available in the cluster.
In [8] is computed a cluster head selection metric based on vehicle direction,
degree of connectivity, an entropy value calculated from the mobility of nodes
in the network, and a distrust level based on the reliability of a node’s packet
relaying. Vehicles are assigned verifiers, which are neighbors with a lower distrust
value. Verifiers monitor the network behavior of a vehicle, and confirm whether
it is routing packets and advertising mobility and traffic information that is
consistent with the verifier’s own view of the neighborhood. The distrust value
for nodes which behave abnormally is then automatically increased, while it is
decreased for nodes which perform reliably. In this way, the trustworthiness of a
node is accounted for the cluster head selection process.
Fuzzy Logic (FL) is the logic underlying modes of reasoning which are approx-
imate rather then exact. The importance of FL derives from the fact that most
modes of human reasoning and especially common sense reasoning are approx-
imate in nature [9]. FL uses linguistic variables to describe the control param-
eters. By using relatively simple linguistic expressions it is possible to describe
and grasp very complex problems. A very important property of the linguistic
variables is the capability of describing imprecise parameters.
In this paper, we present a Fuzzy Cluster Management System (FCMS) for
VANETs. In our system we consider the vehicle relative speed with vehicle clus-
tering, its degree of centrality, the security and its trustworthiness to decide
if the vehicle will remain or leave the cluster. We evaluate the performance of
proposed system by computer simulations.
158 K. Bylykbashi et al.

The structure of the paper is as follows. In Sect. 2, we present VANETs. In

Sect. 3, we present our proposed system. In Sect. 4, we show simulation results.
Finally, conclusions and future work are given in Sect. 5.

2 VANETs
VANETs are considered to have an enormous potential in enhancing road traffic
safety and traffic efficiency. Therefore various governments have launched pro-
grams dedicated to the development and consolidation of vehicular communica-
tions and networking and both industrial and academic researchers are address-
ing many related challenges, including socio-economic ones, which are among
the most important [10–12].
VANET technology uses moving vehicle as nodes to form a wireless mobile
network. It aims to provide fast and cost-efficient data transfer for the advan-
tage of passenger safety and comfort. To improve road safety and travel com-
fort of voyagers and drivers, Intelligent Transport Systems (ITS) are developed.
ITS manages vehicle traffic, support drivers with safety and other information,
and provide some services such as automated toll collection and driver assist
systems [13].
In essence, VANETs provide new prospects to improve advanced solutions
for making reliable communication between vehicles. VANETs can be defined
as a part of ITS which aims to make transportation systems faster and smarter
in which vehicles are equipped with some short-range and medium-range wire-
less communication [14]. In a VANET, wireless vehicles are able to communi-
cate directly with each other (i.e., emergency vehicle warning, stationary vehicle
warning) and also served various services (i.e., video streaming, internet) from
access points (i.e., 3G or 4G) through roadside units [13].

3 Proposed System
Fuzzy sets and fuzzy logic have been developed to manage vagueness and uncer-
tainty in a reasoning process of an intelligent system such as a knowledge based
system, an expert system or a logic control system [15–20].
In this work, we use fuzzy logic to implement the proposed system. The
proposed system is shown in Fig. 1. For the implementation of our system we
consider the following input parameters to decide the Vehicle Remain or Leave
Cluster (VRLC) possibility output parameter.
Vehicle Relative Speed with Vehicle Cluster (VRSVC): This is an important
parameter in VANETs when considering cluster techniques for network man-
agement. A vehicle should be close to the Cluster Head and in order to remain
in the cluster it should move with the same speed as other cluster vehicles.
Otherwise, it will leave the cluster.
Vehicle Degree of Centrality (VDC): The number of connections the vehicle has
to other vehicles in the cluster is an important parameter to decide if this vehicle
will remain or leave the cluster.
 A Secure and Trustworthy Intelligent System for Clustering in VANETs 159

Fig. 1. Proposed system model for FCMS.

Fig. 2. Triangular and trapezoidal membership functions.

Vehicle Security (VS): In VANETs, the security is a critical issue. A vehicle

is secure if it has the proper safeguard techniques or mechanisms to keep the
communication and involved devices protected from malicious intentions.
Vehicle Trustworthiness (VT): It is important for vehicles to be trustworthy.
Trust is defined as a value assigned to a vehicle based on the services it received
and actions performed by the other vehicle. A trustworthy vehicle is a vehicle
that have been given a high trust value by other vehicles and can deliver high
quality services to other vehicles in the network.
These four parameters are not correlated with each other, for this reason
we use fuzzy system. As shown in Fig. 2, we use triangular and trapezoidal
membership functions for FLC, because they are suitable for real-time operation.
The x0 in f (x) is the center of triangular function, x0 (x1 ) in g(x) is the left (right)
edge of trapezoidal function, and a0 (a1 ) is the left (right) width of the triangular
or trapezoidal function. The membership functions that we use in our system
are shown in Fig. 3. In Table 1, we show the Fuzzy Rule Base (FRB) of FCMS,
which consists of 81 rules. The control rules have the form: IF “conditions”
THEN “control action”. For example, for Rule 1: “IF VRSVC is Sl, VDC is LI,
VS is Lo and VT is Lw, THEN VRLC is Le” or for Rule 45: “IF VRSVC is Sa,
VDC is MI, VS is Hi and VT is Hg, THEN VRLC is Re”.
160 K. Bylykbashi et al.

Fig. 3. Membership functions.

We explain in details the design of FLC in following.

The term sets of VRSVC, VDC, VS and VT are defined respectively as:
T (V RSV C) = {Slower (Sl), Same (Sa), F aster (F a)};
T (V DC) = {Little Important (LI), M edium Important (M I), V ery Important (V I)};
T (V S) = {Low (Lo), M edium (M e), High (Hi)};
T (V T ) = {Low (Lw), M iddle (M d), High (Hg)}.

The membership functions for input parameters of FLC are defined as:

µSl (V RSV C) = g(V RSV C; Sl0 , Sl1 , Slw0 , Slw1 );

µSa (V RSV C) = f (V RSV C; Sa0 , Saw0 , Saw1 );
µF a (V RSV C) = g(V RSV C; F a0 , F a1 , F aw0 , F aw1 );
µLI (V DC) = g(V DC; LI0 , LI1 , LIw0 , LIw1 );
µV I (V DC) = f (V DC; M I0 , M Iw0 , M Iw1 );
µV I (V DC) = g(V DC; V I0 , V I1 , V Iw0 , V Iw1 );
µLo (V S) = f (V S; Lo0 , Lo1 , Low1 );
µM e (V S) = f (V S; M e0 , M ew0 , M ew1 );
µHi (V S) = f (V S; Hi0 , Hi1 , Hiw0 );
µLw (V T ) = g(V T ; Lw0 , Lw1 , Lww0 , Lww1 );
A Secure and Trustworthy Intelligent System for Clustering in VANETs 161

Table 1. FRB of proposed fuzzy-based system.

Rule VRSVC VDC VS VT VRLC Rule VRSVC VDC VS VT VRLC

1 Sl LI Lo Lw Le 41 Sa MI Me Md WR
2 Sl LI Lo Md Le 42 Sa MI Me Hg Re
3 Sl LI Lo Hg WL 43 Sa MI Hi Lw WR
4 Sl LI Me Lw Le 44 Sa MI Hi Md Re
5 Sl LI Me Md WL 45 Sa MI Hi Hg Re
6 Sl LI Me Hg NRNL 46 Sa VI Lo Lw NRNL
7 Sl LI Hi Lw WL 47 Sa VI Lo Md VWR
8 Sl LI Hi Md VWL 48 Sa VI Lo Hg Re
9 Sl LI Hi Hg VWR 49 Sa VI Me Lw WR
10 Sl MI Lo Lw Le 50 Sa VI Me Md Re
11 Sl MI Lo Md Le 51 Sa VI Me Hg Re
12 Sl MI Lo Hg VWL 52 Sa VI Hi Lw WR
13 Sl MI Me Lw WL 53 Sa VI Hi Md Re
14 Sl MI Me Md VWL 54 Sa VI Hi Hg Re
15 Sl MI Me Hg VWR 55 Fa LI Lo Lw Le
16 Sl MI Hi Lw VWL 56 Fa LI Lo Md Le
17 Sl MI Hi Md NRNL 57 Fa LI Lo Hg WL
18 Sl MI Hi Hg WR 58 Fa LI Me Lw Le
19 Sl VI Lo Lw Le 59 Fa LI Me Md WL
20 Sl VI Lo Md WL 60 Fa LI Me Hg NRNL
21 Sl VI Lo Hg NRNL 61 Fa LI Hi Lw WL
22 Sl VI Me Lw VWL 62 Fa LI Hi Md VWL
23 Sl VI Me Md NRNL 63 Fa LI Hi Hg VWR
24 Sl VI Me Hg WR 64 Fa MI Lo Lw Le
25 Sl VI Hi Lw NRNL 65 Fa MI Lo Md Le
26 Sl VI Hi Md VWR 66 Fa MI Lo Hg VWL
27 Sl VI Hi Hg WR 67 Fa MI Me Lw WL
28 Sa LI Lo Lw WL 68 Fa MI Me Md VWL
29 Sa LI Lo Md VWL 69 Fa MI Me Hg VWR
30 Sa LI Lo Hg VWR 70 Fa MI Hi Lw VWL
31 Sa LI Me Lw NRNL 71 Fa MI Hi Md NRNL
32 Sa LI Me Md VWR 72 Fa MI Hi Hg WR
33 Sa LI Me Hg WR 73 Fa VI Lo Lw Le
34 Sa LI Hi Lw VWR 74 Fa VI Lo Md WL
35 Sa LI Hi Md WR 75 Fa VI Lo Hg NRNL
36 Sa LI Hi Hg Re 76 Fa VI Me Lw VWL
37 Sa MI Lo Lw VWL 77 Fa VI Me Md NRNL
38 Sa MI Lo Md VWR 78 Fa VI Me Hg WR
39 Sa MI Lo Hg WR 79 Fa VI Hi Lw NRNL
40 Sa MI Me Lw VWR 80 Fa VI Hi Md VWR
81 Fa VI Hi Hg WR
162 K. Bylykbashi et al.

µM d (V T ) = f (V T ; M d0 , M dw0 , M dw1 );
µHg (V T ) = g(V T ; Hg0 , Hg1 , Hgw0 , Hgw1 ).

The small letters w0 and w1 mean left width and right width, respectively.
The term set for the output parameter VRLC is defined as:
⎛ ⎞ ⎛ ⎞
Leave Le
⎜ W eak Leave ⎟ ⎜ WL ⎟
⎜ ⎟ ⎜ ⎟
⎜ V ery W eak Leave ⎟ ⎜ V W L ⎟
⎜ ⎟ ⎜ ⎟
V RLC = ⎜ ⎟ ⎜ ⎟
⎜ N ot Remain N ot Leave ⎟ = ⎜ N RN L ⎟ .
⎜ V ery W eak Remain ⎟ ⎜ V W R ⎟
⎜ ⎟ ⎜ ⎟
⎝ W eak Remain ⎠ ⎝ WR ⎠
Remain Re
The membership functions for the output parameter VRLC are defined as:
µLe (V RLC) = g(V RLC; Le0 , Le1 , Lew0 , Lew1 );
µW L (V RLC) = f (V RLC; W L0 , W Lw0 , W Lw1 );
µV W L (V RLC) = f (V RLC; V W L0 , V W Lw0 , V W Lw1 );
µN RN L (V RLC) = f (V RLC; N RN L0 , N RN Lw0 , N RN Lw1 );
µV W R (V RLC) = f (V RLC; V W R0 , V W Rw0 , V W Rw1 );
µW R (V RLC) = f (V RLC; W R0 , W Rw0 , W Rw1 );
µRe (V RLC) = g(V RLC; Re0 , Re1 , Rew0 , Rew1 ).

4 Simulation Results
In our system, we decided the membership functions and term sets by carrying
out many simulations. We present the simulation results in Figs. 4, 5 and 6. We
consider the VT and VS as constant parameters. We show the relation between
VRLC and VRSVC for different VDC values. The maximal value of VRLC is
when VRSVC is 50 unit. At this value the vehicle is moving with the same
velocity as other vehicles in the cluster.
In Fig. 4(a) and (b), are shown the simulation results considering VT value 10.
If VS and VT are both 10, the vehicle will always leave the cluster. This means
that vehicles with low trustworthiness and security should leave the cluster even
if they are moving at the same speed as other vehicles. We change the VS value
from 10 to 90. The increase of VS increases VRLC, which means that this vehicle
has more chances to remain in cluster because it is more secure. With the increase
of VDC we can see that the possibility of vehicle to remain in cluster is increased.
In Fig. 5(a) and (b), we increase the VT value to 50. We can see that by
increasing VT value, the VRLC is increased. Then in Fig. 6(a) and (b), we
increase the VT value to 90. It can be seen that the VRLC value is increased
much more.
We conclude that vehicles with the same VRSVC and with high VDC, VS
and VT values have higher possibility to remain in the cluster. This is due to the
fact that these vehicles are moving with the same velocity as the other cluster
vehicles, have high connectivity, security and trustworthiness.
 A Secure and Trustworthy Intelligent System for Clustering in VANETs 163

Fig. 4. Simulation results for FCMS2 when VT is 10.

Fig. 5. Simulation results for FCMS2 when VT is 50.

Fig. 6. Simulation results for FCMS2 when VT is 90.

5 Conclusions
In this work, we considered the effect of security and trustworthiness for a fuzzy
cluster management system in VANETs. We evaluated the proposed system by
computer simulations. From the simulation results, we conclude as follows.
164 K. Bylykbashi et al.

• Vehicles with the same VRSVC have high possibility to remain in cluster.
• When VDC is increased, the VRLC parameter is increased, so the probability
that the vehicle remains in the cluster is high.
• Vehicles with low trustworthiness and security should leave the cluster.
• Best VRLC values are achieved by vehicles with the same VRSVC and with
high VDC, VS and VT values.

In the future work, we will consider other parameters for the simulation
system and carry out extensive simulations.

References
1. Barba, C.T., Mateos, M.A., Soto, P.R., Mezher, A.M., Igartua, M.A.: Smart city
for VANETs using warning messages, traffic statistics and intelligent traffic lights.
In: 2012 IEEE Intelligent Vehicles Symposium (IV), pp. 902–907. IEEE (2012)
2. Washburn, D., Sindhu, U., Balaouras, S., Dines, R.A., Hayes, N., Nelson, L.E.:
Helping CIOs understand “smart city” initiatives. Growth 17(2), 1–17 (2009)
3. Honda, T., Ikeda, M., Ishikawa, S., Barolli, L.: A message suppression controller for
vehicular delay tolerant networking. In: Proceedings of the 29th IEEE International
Conference on Advanced Information Networking and Applications, IEEE AINA
2015, pp. 754–760 (2015)
4. Ikeda, M., Ishikawa, S., Barolli, L.: An enhanced message suppression controller for
vehicular-delay tolerant networks. In: Proceedings of the 30th IEEE International
Conference on Advanced Information Networking and Applications (IEEE AINA
2016), pp. 573–579 (2016)
5. Cooper, C., Franklin, D., Ros, M., Safaei, F., Abolhasan, M.: A comparative survey
of VANET clustering techniques. IEEE Commun. Surv. Tutorials 19(1), 657–681
(2017)
6. Wen, H., Ho, P.-H., Gong, G.: A novel framework for message authentication in
vehicular communication networks. In: Global Telecommunications Conference,
GLOBECOM 2009, pp. 1–6. IEEE (2009)
7. Huang, J.-L., Yeh, L.-Y., Chien, H.-Y.: Abaka: an anonymous batch authenticated
and key agreement scheme for value-added services in vehicular ad hoc networks.
IEEE Trans. Veh. Technol. 60(1), 248–262 (2011)
8. Daeinabi, A., Rahbar, A.G.P., Khademzadeh, A.: VWCA: an efficient clustering
algorithm in vehicular ad hoc networks. J. Netw. Comput. Appl. 34(1), 207–222
(2011)
9. Inaba, T., Obukata, R., Sakamoto, S., Oda, T., Ikeda, M., Barolli, L.: Performance
evaluation of a qos-aware fuzzy-based CAC for LAN access. Int. J. Space-Based
Situated Comput. 6(4), 228–238 (2016)
10. Santi, P.: Mobility Models for Next Generation Wireless Networks: Ad Hoc, Vehic-
ular and Mesh Networks. Wiley, Hoboken (2012)
11. Hartenstein, H., Laberteaux, L.: A tutorial survey on vehicular ad hoc networks.
IEEE Commun. Mag. 46(6), 164–171 (2008)
12. Zhang, W., Jiang, S., Zhu, X., Wang, Y.: Cooperative downloading with privacy
preservation and access control for value-added services in VANETs. Int. J. Grid
Utility Comput. 7(1), 50–60 (2016)
 A Secure and Trustworthy Intelligent System for Clustering in VANETs 165

13. Karagiannis, G., Altintas, O., Ekici, E., Heijenk, G., Jarupan, B., Lin, K., Weil,
T.: Vehicular networking: a survey and tutorial on requirements, architectures,
challenges, standards and solutions. IEEE Commun. Surv. Tutorials 13(4), 584–
616 (2011)
14. Booysen, M.J., Zeadally, S., van Rooyen, G.J.: Performance comparison of media
access control protocols for vehicular ad hoc networks. IET Netw. 1(1), 10–19
(2012)
15. Kandel, A.: Fuzzy Expert Systems. CRC Press, Boca Raton (1991)
16. Zimmermann, H.-J.: Fuzzy Set Theory and Its Applications. Springer, Heidelberg
(1991)
17. McNeill, F.M., Thro, E.: Fuzzy Logic: A Practical Approach. Academic Press,
Cambridge (1994)
18. Zadeh, L.A., Kacprzyk, J.: Fuzzy Logic for the Management of Uncertainty. Wiley,
Hoboken (1992)
19. Klir, G.J., Folger, T.A.: Fuzzy Sets, Uncertainty, and Information. Prentice Hall,
Upper Saddle River (1988)
20. Munakata, T., Jani, Y.: Fuzzy systems: an overview. Commun. ACM 37(3), 68–76
(1994)
 Terminal Access Data Anomaly Detection
Based on Random Forest for Power User
Electric Energy Data Acquisition System

Xiaobing Liang1, Bing Zhao1, Qian Ma2,3(&), Bang Sun2,3,

and Baojiang Cui2,3
1
China Electric Power Research Institute, Beijing 100192, China
{liangxiaobing,zhaob}@epri.sgcc.com.cn
2
School of Cyberspace Security,
Beijing University of Posts and Telecommunications, Beijing, China
{maqian95,confuoco,cuibj}@bupt.edu.cn
3
National Engineering Laboratory for Mobile Network Security, Beijing, China

Abstract. There are several drawbacks in rule-based traditional terminal access

data anomaly detection methods for power user electric energy data acquisition
system. They are easy to be bypassed, and the false positive rate and false
negative rate are often very high. To address these problems, we propose a
terminal access data anomaly detection model based on random forest focusing
on the communication protocol, namely 376.1 master station communication
protocol. Firstly, through analyzing the characteristics of the 376.1 master sta-
tion communication protocol, we construct an expressive multidimensional
feature set. Then we choose random forest to detect abnormal access data. The
experimental result shows that the detection model outperforms its counterparts.
Our work also provides a new idea for terminal access data anomaly detection.

Keywords: Power user electric energy data acquisition system

376.1 master station communication protocol
Anomaly detection

Feature extraction
Random forest

1 Introduction

The power user electric energy data acquisition system is a key power infrastructure for
residential and enterprise users. Through concentrator parameter setting and data col-
lection by the master station, the system collects and analyzes the electricity con-
sumption data of the distribution transformer and end users. There are many security
risks in the acquisition system. In order to ensure the security of access data trans-
mission, it is necessary to adopt security protection methods such as encryption and
network isolation for collected data. Besides, it is also an effective protection method to
detect malicious behaviors in the access data from the aspect of communication pro-
tocol, such as: data tampering and implantation of malicious programs (trojans, viruses,
malicious code and so on). Anomaly detection plays an important role in ensuring
security of collection equipment. Through analysis and research on the access data of
the terminal, abnormal behaviors can be discovered effectively. It also provides
© Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 166–175, 2020.
https://doi.org/10.1007/978-3-030-15032-7_14
 Terminal Access Data Anomaly Detection Based on Random Forest 167

decision basis for security management of the terminal device, thereby guaranteeing the
safety and efficient operation of the collected data transmission.
Anomaly detection is one of the important research directions in data mining. It
aims at discriminating abnormal data in the data set, where abnormal data refers to the
data that deviates from the normal behavior pattern. In many practical scenarios, the
data is highly unbalanced. For example, more than 99% of terminal access data would
be not malicious. Therefore, abnormal data is often much lacking due to high acqui-
sition cost and various difficulties in collection. However, a small amount of abnormal
data often indicates important behavioral pattern information. Under the background of
big data, in addition to traditional anomaly detection methods based on rules and
statistical methods, artificial intelligence techniques such as machine learning and deep
learning have also been introduced into anomaly detection. This also provides a new
idea for terminal access data anomaly detection.
There are several drawbacks in rule-based traditional terminal access data anomaly
detection methods for power user electric energy data acquisition system, such as: they
are easy to be bypassed, and the false positive rate and false negative rate are often very
high. To this end, we propose a terminal access data anomaly detection model based on
random forest focusing on the communication protocol, namely 376.1 master station
communication protocol. We firstly analyze the characteristics of the 376.1 master
station communication protocol, and construct a multidimensional feature set. Then we
choose random forest as the abnormal terminal access data detection model. The
experimental result shows that the detection model outperforms its counterparts.

2 Related Work

The existing anomaly detection methods can be divided into two types: supervised and
unsupervised anomaly detection methods, according to whether the training dataset
contains abnormal data. In some practical tasks, the abnormal data is rare or even
doesn’t exist. Therefore, only normal samples can be used for model training and
testing. Corresponding methods belong to unsupervised anomaly detection methods.
One-class support vector machine (OC-SVM) and clustering are two common unsu-
pervised models in anomaly detection. OC-SVM obtains a classification plane through
solving a mathematical optimization problem. It also adopts techniques such as the
kernel trick and slack variables to deal with the scenarios where data is linearly
inseparable and contains noise. The introduction of these tricks further improves the
detection efficiency of OC-SVM. Ma et al. [1] transform the original data into a new
feature space by defining the difference between the dataset and target class set. Based
on this, an OC-SVM based on the degree of difference is proposed for anomaly
detection. In addition, this model is combined with kernel principal component analysis
method to reduce the dimensions of the input space. Manuele et al. [2] propose a new
soft clustering algorithm based on OC-SVM. In the proposed clustering algorithm, each
cluster is characterized by a weighted OC-SVM. Clustering method aims to discover
the intrinsic structure of data, and divides the dataset into a number of disjoint subsets.
Each subset is usually called a cluster. For a new data instance, which cluster it belongs
to depends on the minimal distance from it to the center of each cluster. K-means [3] is
168 X. Liang et al.

a classic clustering algorithm, which has been widely used in anomaly detection for its
high efficiency and simplicity. However, since the algorithm randomly initializes the
cluster center, it is easy to be limited to local optima. To address this problem, Han
et al. [4] select the top K points with the highest density as the initial cluster centers by
measuring the density around all data points. This method only considers the distri-
bution density to select initial center points, so it still cannot guarantee the uniform
distribution of the cluster centers. Zuo et al. [5] uniformly select the K initial clustering
centers in the region with large data distribution density. The proposed initialization
method effectively alleviates the problem that the original algorithm was easy to fall
into local optimum due to random initialization, and improves the clustering efficiency.
Supervised anomaly detection methods use normal and abnormal data simultane-
ously to in model training and testing. A variety of supervised models have been
developed for the purpose of anomaly detection. Ye et al. [6] propose an anomaly
detection method based on principal component analysis and tabu search (PCA-TS).
The method accomplishes feature selection and dimension reduction through PCA-TS,
thereby providing effective low-dimensional feature input for the decision tree classi-
fier. Zhou [7] analyzes different characteristics of traffic behavior in time and space.
Combined with data mining and signal processing methods, he propose a communi-
cation network anomaly detection method based on feature correlation analysis. Cheng
[8] et al. propose an improved distributed intrusion detection model based on artificial
immune theory, through combination of misuse detection and anomaly detection.
Moreover, machine learning algorithms such as Bayesian networks and neural net-
works are also used in supervised anomaly detection [9, 10].

3 A Terminal Access Data Anomaly Detection Model

Based on Random Forest for Power User Electric Energy
Data Acquisition System

3.1 The Framework of Access Data Anomaly Detection

As current abnormal behaviors in the terminal access data appear complex and diverse
trend, terminal access data anomaly detection models are required to have higher
scalability and better real-time performance. Machine learning has unique advantages
in these respects. In this paper, we propose an access data anomaly detection model for
power user electric energy data acquisition system based on random forest. The
schematic diagram of the system is shown in Fig. 1. The proposed method mainly
consists of two parts: data processing and machine learning. The key step of data
processing is feature extraction. Machine leaning module accomplishes model selection
and parameter tuning of machine models. In this paper, we choose random forest as the
final detection model as it outperforms its counterparts. Figure 2 shows the overall
framework of terminal access data anomaly detection. The whole detection process
consists of four modules: feature extraction, data deduplication and quantization,
machine learning model training, and anomaly detection. In addition, according to the
feedback results of the anomaly detection model, the feature set will be updated and
optimized.
 Terminal Access Data Anomaly Detection Based on Random Forest 169

Fig. 1. The schematic diagram of power user electric energy data acquisition system

Fig. 2. The access data anomaly detection model based on random forest

The detail of the four modules in Fig. 2. is described as follows.

(1) Feature extraction module.
Feature extraction plays an important role in terminal access data anomaly
detection. An expressive feature set is constructed through analysis of the attribute
fields of 376.1 master station communication protocol. Raw communication
packet of the protocol is mapped into fixed-length feature vectors based on the
obtained feature set, which are used as input to machine leaning models. Details
about the feature set is presented in Sect. 3.2.
170 X. Liang et al.

(2) Data deduplication and quantization module.

To improve the speed of data processing and reduce the size of training set, it is
necessary to perform de-duplication processing on the obtained feature vectors.
For data samples which are mapped into a same feature vector, only one feature
vector is used to train and test the subsequent machine learning models. More-
over, for some features with a large range of values, they are quantized using log2
in this module.
(3) Machine learning module.
After feature extraction and processing of above two modules, a training set with
low redundancy is obtained. As for the machine learning model for access data
anomaly detection, we choose logistic regression, Naïve Bayes, random forest and
support vector machine (SVM) as candidate classification models. Then we select
random forest as the final terminal access data anomaly detection model through
the hold-out method, for its significant two-class classification performance.
Details of random forest are described in Sect. 3.3.
(4) Anomaly detection module.
In this module, the model file generated in the machine learning module is loaded
to detect new data instances and then a detection result is given.

3.2 Feature Extraction

Feature extraction is the first key step in terminal access data anomaly detection.
Constructing a high-quality feature set plays an important role in improving the
accuracy and detection efficiency of the subsequent anomaly detection model. Figure 3
shows the frame format of 376.1 master station communication protocol with data
acquisition terminal. More details of the communication protocol can be found in [11].
Through analyzing the attribute fields of the 376.1 protocol message data and statistical
analysis of the terminal access data, we construct a multi-dimensional feature set with
12 expressive features. It mainly includes the following four types of features:

Fig. 3. The frame format of 376.1 master station communication protocol with data acquisition
terminal

(1) Abnormal message header format.

Such features are extracted from scenarios where the packet header does not
conform to the constraints of protocol. This type of features mainly includes the
following four features:
 Terminal Access Data Anomaly Detection Based on Random Forest 171

a. Whether the first starting character is 68H.

b. Whether the header length of the message is 6 bytes.
c. Whether the second starting character is 68H.
d. Whether the value of protocol identification is 01 and 10.
(2) Abnormal message end format.
This type of feature is obtained by judging the end character of a message. It
mainly includes one feature: whether the ending character is 16H.
(3) Abnormal data unit.
This type of features is extracted from the data unit segment of the packet. It
mainly includes the following five features:
a. Whether the length of the data unit identification is in the interval (0,4)
b. Whether the value range of BCD code1 is beyond 0–9.
c. In the uplink packet, whether AFN code2 is 01H, 04H, or 05H.
d. Whether the value range of Fn3 is beyond 0–248.
e. For a particular Fn, whether the length of the corresponding data unit is less
than lower bound.
f. Whether the length of the data field of Fn satisfies the communication
protocol.
(4) Abnormal event content.
This type of features derives from the event recording segment. It mainly includes
one feature that for an event record [11], whether the difference between the end
pointer and the start pointer (indicate the number of uploaded events) is negative.

3.3 Random Forest Basics

The basic idea of ensemble models is to combine several weak classifiers into a more
powerful one. Random forest [12] is an ensemble model based on decision tree. Different
form GBDT [13], which is constructed by gradient boosting, random forest applies
bootstrap aggregating strategy to ensemble several single decision trees. In the training of
random forest, a new decision tree is trained with different bootstrap samples from the
original data. When a new data sample is input into random forest, it is firstly classified by
each of the tree in the forest. Then a vote is given by each tree, the result of which indicates
the predicted label of the data point. To sum up, a new data sample is classified as:

X
m
HðxÞ ¼ arg max Iðhi ðxÞ ¼ cj Þ ð1Þ
cj
i¼1

1
BCD code is the abbreviation of Binary-Coded Decimal.
2
AFN code is the application layer function code in the frame format. More detailed information can
be found in [11].
3
Fn is the information class identification code in 376.1 protocol. More detailed information can be
found in [11].
172 X. Liang et al.

where hi indicates classification result of each single tree model, fc1 ; c2 ;


; cm g

denotes the label set, and . Details of random forest can be

found in [12].

4 Performance Evaluation

4.1 Introduction to the Dataset

In our experiment, the terminal access data set is provided by China Electric Power
Research Institute. After feature extraction and data deduplication, the training dataset
consists of 1270 normal URLs and 390 malicious URLs.

4.2 Model Selection Method

In our experiment, we choose logistic regression, Naïve Bayes, random forest and
support vector machine (SVM) as candidate models. Then the hold-out method is
applied to select the terminal access data anomaly detection model. The hold-out
method divides the dataset D into two exclusive sets. One of them is used as training
set S and the other is used as testing set T. That is to say, D ¼ S [ T and S \ T ¼ ;.
After completing model training on S, the classification error on T is treated as an
estimate of the generalization error. This process is repeated for many times through
random dataset partition and the mean value of each index is eventually returned as the
evaluation result of the hold-out method.

4.3 Model Evaluation Indexes

To evaluate classification performance of candidate models, we use standard evaluation
indexes, precision, recall rate and f1 score. In our experiment, abnormal data instances
are labeled as 0 and normal ones are labeled as 1. The three evaluation indexes are
defined as follows in terms of confusion matrix (see Table 1):
(1) Precision
(a) Abnormal data: Pa ¼ TPTP þ FN
. It indicates the proportion of true positive
samples in the number of samples predicted to be positive.
(b) Normal data: Pn ¼ TNTN þ FP
. It indicates the proportion of true negative
samples in the number of samples predicted to be negative.
(2) Recall rate
(a) Abnormal data: Ra ¼ TPTP þ FP
. It indicates the proportion of the positive
samples which are correctly predicted.
(b) Normal data: Rn ¼ TNTN þ FN
. It indicates the proportion of the negative
samples which are correctly predicted.
 Terminal Access Data Anomaly Detection Based on Random Forest 173

(3) f1 score
(a) Abnormal data: F1a ¼ 2Pa Pa*Ra. It indicates the harmonic mean of the
þ Ra
accuracy and recall rate corresponding to the positive samples.
 Pn  Rn
(b) Normal data: F1n ¼ 2Pn þ Rn
. It indicates the harmonic mean of the accu-
racy and recall rate corresponding to the negative samples.

Table 1. Confusion matrix

Predicted label
0 1
Actual label 0 True positive (TP) False positive (FP)
1 False negative (FN) True negative (TN)

Fig. 4. Precision of different classifiers

4.4 Experimental Results and Analysis

Figures 4, 5 and 6 shows the precision, recall rate and f1 score of the four candidate
models, respectively. The experimental result shows that random forest and SVM
outperform logistic regression and Naïve Bayesian model in terms of the aforemen-
tioned model evaluation indexes. As for the weighted average of three indexes, random
forest performs as same as SVM, the indexes of which are all above 97%. However, in
the actual communication environment of the acquisition system, the access data is
often unbalanced. For example, more than 99% of terminal access data would be
normal. Therefore, we mainly consider the classification performance over abnormal
data in model selection. Both models have same value of f1 score. As for SVM, there is
significant difference in the recall rate of different kinds of data. Besides, when it comes
to the precision of abnormal data, random forest performs better than SVM. Therefore,
random forest is chosen as the terminal access data anomaly detection model.
174 X. Liang et al.

Fig. 5. Recall rate of different classifiers

Fig. 6. F1 score of different classifiers

5 Conclusion

In this paper, we propose a terminal access data anomaly detection model based on
random forest for power user electric energy data acquisition system. We first analyze
the attribute fields of the communication protocol, namely 376.1 master station com-
munication protocol and its packet data. Then we construct a feature set of terminal
access data, which is the input to the subsequent machine models. Finally, we choose
random forest as the access data anomaly detection model by the hold-out method. The
experimental results show that random forest has significant detection performance.
Our work also provides a new idea for the anomaly detection of terminal access data in
the power user electric energy data acquisition system.
 Terminal Access Data Anomaly Detection Based on Random Forest 175

Acknowledgments. This work was supported by Research and Application of Key Technolo-
gies for Unified Data Collection of Multi-meter (JL71-17-007) and National Natural Science
Foundation of China (No. U1536122).

References
1. Ma, J., Dai, G., Xu, Z.: Network anomaly detection using dissimilarity-based one-class SVM
classifier. In: International Conference on Parallel Processing Workshops, pp. 409–414.
IEEE (2009)
2. Bicego, M., Figueiredo, M.A.T.: Soft clustering using weighted one-class support vector
machines. Pattern Recogn. 42(1), 27–32 (2009)
3. Celebi, M.E., Kingravi, H.A., Vela, P.A.: A comparative study of efficient initialization
methods for the k-means clustering algorithm. Expert Syst. Appl. 40(1), 200–210 (2013)
4. Han, Z.: An adaptive K-means Initialization method based on data density. Comput. Appl.
Softw. 2, 182–187 (2014)
5. Zuo, J., Chen, Z.: Anomaly detection algorithm based on improved K-means clustering.
Comput. Sci. 43(8), 258–261 (2016)
6. Ye, X., Lan, J., Guo, T.: Network anomaly detection method based on principle component
analysis and tabu search and decision tree classification. J. Comput. Appl. 33(10), 2846–
2850 (2013)
7. Zhou, Y., Hu, G.: Network-wide anomaly detection based on muter connection relation-
ships. IEICE Trans. Commun. E94B(8), 2239–2242 (2011)
8. Chen, J., Zhang, M., Liu, X., Fan, T.: Distributed intrusion detection model based on
artificial immune. J. Comput. Appl. (1), 86–89 + 94 (2014)
9. Kruegel, C., Mutz, D., Robertson, W., Valeur, F.: Bayesian event classification for intrusion
detection. In: Proceedings of the 19th Annual Computer Security Applications Conference,
p. 12 (2003)
10. Mukkamala, S., Janoski, G., Sung, A.: Intrusion detection using neural networks and support
vector machines. In: Proceedings of the 2002 International Joint Conference on Neural
Networks, IJCNN 2002, p. 5 (2002)
11. Liu, K., Liao, X.: Design and implementation of Q/GDW 376. 1 protocol and DL/T 645
protocol conversion. Adv. Technol. Electr. Eng. Energy 32(2), 72–75 (2013)
12. Liaw, A., Wiener, M.: Classification and regression by random forest. R News 2, 18–22
(2002)
13. Natekin, A., Knoll, A.: Gradient boosting machines, a tutorial. Front. Neurorobotics 7, 21
(2013)
 Multi-level Authentication Protocols Using
Scientific Expertise Approach

Marek R. Ogiela1(&) and Lidia Ogiela2

1
AGH University of Science and Technology Cryptography
and Cognitive Informatics Research Group,
30 Mickiewicza Ave., 30-059 Kraków, Poland
[email protected]
2
Pedagogical University of Cracow,
Podchorążych 2 St., 30-084 Kraków, Poland
[email protected]

Abstract. This paper will describe new approaches for creation of multi-level
user authentication protocols, using hybrid verification CAPTCHA codes.
Proposed codes will define a new subclass of cognitive CAPTCHA, which
require special skills and knowledge for proper user authentication. Such
authentication procedure can be oriented especially to provide access for trusted
group of experts, who represent particular expertise areas, and based of theirs
perception abilities and knowledge. For presented authentication protocols
possible examples of applications will be presented and security features will be
described.

Keywords: Authentication procedures
Cognitive CAPTCHA

Security protocols

1 Introduction

In advanced authentication protocols it is possible to apply selected personal features or

cognitive skills. Such personal characteristics allow to define user oriented authenti-
cation procedures, and also have influence on enriching a new security branch called
cognitive cryptography [1, 2]. Of course among many different authentication
approaches it is possible to find several authentication solutions, which are based on
visual CAPTCHA codes. CAPTCHA codes are very popular on many websites and
remote services. Such authentication approach can guarantee that received response is
originating from human, and not from machine or computing systems. It is possible to
find several classes of such codes, which are based on using texts, audio signals, visual
patterns and others [3, 4]. CAPTCHA verification protocols are very secure, because
noises, which can be introduced to original patterns, makes the interpretation processes
more difficult than the analysis of original patterns. Recently we could also proposed a
new classes of CAPTCHA codes called cognitive CATCHA. In this type of authen-
tication codes we can use human perception capabilities [5–7].
Cognitive CAPTCHA codes can be oriented for specific group of participants, which
possess specific skills and knowledge, which is required for proper authentication.

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 176–180, 2020.
https://doi.org/10.1007/978-3-030-15032-7_15
 Multi-level Authentication Protocols Using Scientific Expertise Approach 177

This type of verification procedures can play very important role in many applications
like data management, Internet of Things, smart contracts etc. [8–10], but of course can
be further extended towards more secure solutions.
In this paper we try to present possible extensions, in which we can perform multi-
level authentication, which will require expert knowledge from several different dis-
ciplines, as well as focusing attention on different items, objects or visual parts. In our
previous paper we defined multilevel authentication approach and also moving atten-
tion visual codes. Now we try to join these procedures and define a new one combining
theirs features.

2 Multi-level Authentication Based on Expert Knowledge

As was mentioned above it is possible to create novel security protocols, which use
specific expert knowledge. Cognitive approaches allow to join security protocols with
memory and mental features, what finally allow to define human oriented security
procedures. Below will be presented new solutions oriented on multi-level authenti-
cation protocols. In general such multi stages verification can be performed in several
iterations, in which attention can be focused on different object, and in each iteration
different questions can be asked. This type of cognitive CAPTCHAs combine several
features like moving attention, answer for different questions, and recognition abilities.
In this type of verification codes, authentication can be performed in several consec-
utive iterations, in which are asked questions connected with different parts of pre-
sented visual pattern, or connected with different objects visible on it. Depending on the
image division for particular CAPTCHA it is possible to find different answers, or
theirs combinations, which may be required for final verification. When we analyse
only one part of the whole verification pattern, it is possible to find answer for par-
ticular question, but for more detailed analysis, it is also possible to find other solutions
based on different regions of interest.
Figure 1 presents an example of multi-level CAPTCHA, in which answers depend
on the displayed structures or marked regions. During verification in this procedure it is
possible to ask different questions about visible patterns, different objects or selected
features important for proper understanding the content of this pattern. In this solution
is possible to quickly verify if authenticated person has an expert knowledge connected
with presented pattern, and also has special mental skills to answer for several ques-
tions with different detail levels. Such protocols allow not only to determine if the
answering is a person, but also determine whether this user represents a particular area
of expertise or not.
During verification it is also possible to recognize patterns with different perception
thresholds. In this approaches it is possible to use different visual objects softened in
such way that it is not quite easy to recognize it. Authenticated user should recognize
the content of fuzzy images, what is possible only in situation when he has some
knowledge, which allow to guess the proper content. In this case verification is also
performed in several iteration with different perception thresholds, which can be
changed according cognitive and perceptual skills.
178 M. R. Ogiela and L. Ogiela

Fig. 1. CAPTCHA with multi-questions and expert knowledge verification. Image presents
brain vessels (A), carotid artery (B), coronary arteries (C) and pancreatic duct (D). The first
example question can be connected with finding coronary structures among these images. Having
detected coronary arteries next questions can be connected with visible pathologies, name of
vessel, type of visualization etc.

Presented multi-level authentication procedures can be applied for verification or

authorization of users in distributed computing infrastructures, secure data repositories,
and Cloud resources. Cognitive procedures allow also to obtain access at different
stages or security levels. To receive higher security it is necessary to perform a higher
number of iteration for verification procedure [11].

3 Security Features of Multi-level Protocols

Comparing with classical CAPTCHA codes, also cognitive verification procedures are
secure from cryptographic point of view. Traditional CAPTCHTA are secure due to the
difficulties in recognition proper items on softened or distorted images. The same
situation we can observe with relation to multi-level and expert knowledge verification.
This type of codes require specific information, knowledge or experiences to
 Multi-level Authentication Protocols Using Scientific Expertise Approach 179

successfully find the proper answers. Questions which can be asked during authenti-
cation requires not only skills with complex analysis, moving attention, fast response,
but also expertise knowledge in selected scientific or professional areas. These features
guarantee the appropriate level of security and allow to successfully pass verification
procedure only for high-qualified users, who possess specific and expertise informa-
tion, which are not available for computer or answering systems.

4 Conclusions

In this paper has been presented new authentication procedure, which is based on
application of cognitive approach and expert knowledge. Such authentication proce-
dure works in several stages, in which different levels of expertise can be checked and
different perception skills of users can be verified.
Security of such protocols are similar to traditional CAPTCHA codes, but lays not
only on difficulties in recognition of patterns, but also on proper semantic analysis of
the content, and proper knowledge application in particular area of expertise. Presented
solutions are oriented for expert groups of people, who have specific expert knowledge
and high cognitive skills. Presented solutions extend traditional protocols used for
authentication, towards application of very specific human mental characteristic what
finally allow to define new areas in security technologies and cryptosystems [12–14].

Acknowledgments. This work has been supported by the National Science Centre, Poland,
under project number DEC-2016/23/B/HS4/00616.

References
1. Ogiela, M.R., Ogiela, L.: On using cognitive models in cryptography. In: IEEE AINA 2016 -
The IEEE 30th International Conference on Advanced Information Networking and
Applications, Crans-Montana, Switzerland, 23–25 March, pp. 1055–1058 (2016)
2. Ogiela, M.R., Ogiela, L.: Cognitive keys in personalized cryptography. In: IEEE AINA 2017
- The 31st IEEE International Conference on Advanced Information Networking and
Applications, Taipei, Taiwan, 27–29 March, pp. 1050–1054 (2017)
3. Alsuhibany, S.: Evaluating the usability of optimizing text-based CAPTCHA generation. Int.
J. Adv. Comput. Sci. Appl. 7(8), 164–169 (2016)
4. Osadchy, M., Hernandez-Castro, J., Gibson, S., Dunkelman, O., Perez-Cabo, D.: No bot
expects the DeepCAPTCHA! Introducing immutable adversarial examples, with applica-
tions to CAPTCHA generation. IEEE Trans. Inf. Forensics Secur. 12(11), 2640–2653 (2017)
5. Ogiela, L.: Cognitive computational intelligence in medical pattern semantic understanding.
In: Guo, M.Z., Zhao, L., Wang, L.P. (eds.) In: ICNC 2008: Fourth International Conference
on Natural Computation, vol. 6, Proceedings, 18–20 October, Jian, Peoples R China,
pp. 245–247 (2008)
6. Ogiela, L.: Cognitive Information Systems in Management Sciences. Elsevier, Academic
Press, UK (2017)
180 M. R. Ogiela and L. Ogiela

7. Ogiela, L., Ogiela, M.R.: Data mining and semantic inference in cognitive systems. In:
Xhafa, F., Barolli, L., Palmieri, F., et al. (eds.) 2014 International Conference on Intelligent
Networking and Collaborative Systems (IEEE INCoS 2014), 10–12 September, Salerno,
Italy, pp. 257–261 (2014)
8. Ogiela, U., Ogiela, L.: Linguistic techniques for cryptographic data sharing algorithms.
Concurr. Comp. Pract. E. 30(3), e4275 (2018). https://doi.org/10.1002/cpe.4275
9. Ogiela, L., Ogiela, M.R.: Insider threats and cryptographic techniques in secure information
management. IEEE Syst. J. 11, 405–414 (2017)
10. Ogiela, M.R., Ogiela, U.: Secure information management in hierarchical structures. In:
Kim, T.-h., et al. (eds.) AST 2011. CCIS, vol. 195, pp. 31–35 (2011)
11. Ogiela, L., Ogiela, M.R., Ogiela, U.: Efficiency of strategic data sharing and management
protocols. In: The 10th International Conference on Innovative Mobile and Internet Services
in Ubiquitous Computing (IMIS-2016), 6–8 July, Fukuoka, Japan, pp. 198–201 (2016),
https://doi.org/10.1109/imis.2016.119
12. Ogiela, L.: Advanced techniques for knowledge management and access to strategic
information. Int. J. Inf. Manag. 35(2), 154–159 (2015)
13. Menezes, A., van Oorschot, P., Vanstone, S.: Handbook of Applied Cryptography. CRC
Press, Waterloo (2001)
14. Schneier, B.: Applied Cryptography: Protocols, Algorithms, and Source Code in C. Wiley,
Chichester (1996)
 Biometric-Based Linguistic Solutions for Data
Encryption and Sharing

Urszula Ogiela1, Makoto Takizawa2, and Lidia Ogiela3(&)

1
Cryptography and Cognitive Informatics Research Group,
AGH University of Science and Technology,
30 Mickiewicza Ave., 30-059 Kraków, Poland
[email protected]
2
Department of Advanced Sciences, Hosei University, 3-7-2, Kajino-cho,
Koganei-shi, Tokyo 184-8584, Japan
[email protected]
3
Pedagogical University of Cracow, Podchorążych 2 Street,
30-084 Kraków, Poland
[email protected]

Abstract. This paper will describe new hybrid classes of secret sharing pro-
tocols joining features of linguistic secret division procedures with biometric
threshold schemes. Such advanced protocols will be defined with application of
different classes of formal grammars with standard or non-standard biometric
patterns. Evaluation of such method will be done, as well as possible application
for data and services management will be presented.

Keywords: Secret sharing protocols
Biometric-linguistic threshold schemes

Cloud computing
Secure service management

1 Introduction

The secret sharing schemes are one of the threshold techniques dedicated to guarantee
data security. The main idea of these techniques is to secure strategic data by its
splitting. Splitting processes guarantee the data security by data division and distri-
bution of all secret parts between participants of cryptographic protocol. In this class of
cryptographic protocols we define two groups of division algorithms – splitting and
sharing threshold schemes.
The main idea of such solutions are data splitting by trusted users (participants of
the protocols). All of them receive secret parts – one or more – what depends on the
sharing procedures. It’s possible to distribute one or more shadows for all participants.
It’s also possible to allocate more than one shadow to all participants, but it depends on
the used division procedures. Also, it’s possible to distribute different number of secret
parts between trustees of the secret. These procedures were describe in the [2, 6, 7]. All
of proposed solutions can be used in threshold schemes. Sharing techniques are used
for data security in selected classes of cryptographic schemes – in threshold schemes.
The latest classes of threshold protocols are the linguistic and biometric threshold
schemes. The main characteristics of these protocols were presented in [6–11].

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 181–187, 2020.
https://doi.org/10.1007/978-3-030-15032-7_16
182 U. Ogiela et al.

Linguistic threshold schemes perform a division of secret information between a

groups of secret participants. This secret contains a linguistic description. The linguistic
description processes provide a full description and interpretation of encrypted
data/secret. This process is introduced at the stage of selecting the secret data, what is
necessary for security [2–5, 12]. This secret is subject to the process of extracting the
characteristic features, typical only for this set. In this procedure are eliminated dif-
ferent features, characteristic for other secret sets. Linguistic description is allocated in
additional shadow and can be divided (or not) by the secret holders [6–11].
The second group of new threshold protocols are the biometric threshold schemes.
In this class of cryptographic schemes, secret is splitting between all participants of the
protocol. But in this class of protocols is possible to biometric marking of individual
secret parts. All participants of this protocols should mark all own secret parts. It’s very
important, because all secret holders receive only this shadows which can confirm the
secret holder and biometric feature. In this class of the protocol each shadows are mark
by the biometric (personal) features. It is possible to use different types of biometrics –
standard or non-standard biometric features. Standard biometrics are the most typical
personal data as well as fingerprint, voice parameters, DNA code, eyes parameters, face
parameters, etc. But there are not all individual personal data sets. We also distinguish
second group of personal data – non-standard biometrics. This class of biometric
features includes for example hands structure, foots structure, head or neck parameters,
coronary arteries structures. All of possible biometrics can be used for mark secret parts
in sharing protocols. It is obvious that the use of non-standard parameters will be
justified in the case of coding extremely important data, because the process of
extracted these features are more complex than the extraction standard parameters.
In biometric threshold schemes personal data are also necessary in secret decoding.
In this process all secret holders should mark own shadows for proper identification.
These both classes of threshold schemes were used independently. In this paper we
propose a new class of threshold schemes – combination of biometric and linguistic
threshold schemes.

2 New Classes of Biometric-Based Threshold Schemes

Described below new classes of extended methods are dedicated for better description
and securing strategic data. In proposed methods we select the biometric-based lin-
guistic threshold schemes. This class of protocols serves for linguistic description of a
shared secret, and parts of this secret are labeled by the selected biometrics.
New classes of threshold schemes consist the solution, in which the linguistic
threshold schemes are extended by the biometric-based description. The algorithms of
this procedure is following.
Algorithm 1. Biometric-based linguistic division procedure
Step 1. Data set selection for encryption process.
Step 2. Determining the degree of secret secrecy – confidential/secret/strategic data,
full confidentiality/partial confidentiality.
Step 3. Selection of the sharing protocol.
 Biometric-Based Linguistic Solutions for Data Encryption and Sharing 183

Step 4. Determining the number of all participants in selected protocol (n).

Step 5. Determining the method of shadow distribution between secret holders – equal
or privileged.
Step 6. Determining the minimum number of secret parts necessary to reproduce the
original data (m).
Step 7. Generating linguistic shadow – containing a linguistic description of classified
data.
Step 8. Choice of secret marking method.
At this stage the shadow marking method is determined. It is possible to
mark only a linguistic shadow that consist important data about the
meaning of the secret, but it is also possible to mark all secret parts.
The marking shadow methods depends also on the degree of secret secrecy
(at this stage it is necessary to use the information from step 2).
Step 9. Selection a biometric group for personal identification – standard or non-
standard biometrics.
Step 10. Choice of type of marking techniques – the same/different biometric type for
all secret parts.
Step 11. Distribution all shadows between all protocol participants.
Step 12. Shadows management by secret trustees.
Step 13. Start the secret reconstruction procedure.
Step 14. Choosing m secret parts to reconstruct the original data – in this set k parts
comes from the division of the linguistic shadow.
Step 15. Verification of compliance of biometric data with personal marked shadow.
Step 16. Submitting the required amount of correctly verified shadows.
Step 17. Secret reconstruction procedure.
All of these steps create a full biometric-based linguistic threshold protocol. The
characteristic features of these schemes are following:
• ensures data division by threshold techniques,
• securing data by dividing them and split separating parts of secret between all
trusted users,
• describe semantic mining of secured information by using linguistic procedures,
• guarantees an unambiguous allocation of secret part to their owners through the use
of biometric procedures,
• guarantee that the computational complexity of the sharing protocols is at the same
level as used security techniques.
Biometric-based linguistic threshold schemes can be defined by application of
different types of linguistic description. Linguistic description of secure data/secret
depends on use of formal grammars. According to Chomsky we distinguish the fol-
lowing types of grammars [1]:
• unrestricted grammars (type 0),
• context grammars (type 1),
• context-free grammars (type 2),
• regular grammars (type 3).
184 U. Ogiela et al.

Linguistic process of secret description in threshold procedures can be implemented

by the context-free and regular grammars. Type of grammar used in the secret
description procedure also determine the type of threshold schemes, so it’s possible to
create the following schemes:
• threshold schemes used sequence grammars,
• threshold schemes used trees,
• threshold schemes used graph.
In the first class of proposed schemes generally we use bit sequences to describe
secret data. In the second class we can use tree grammars to describe secure data. In the
last class of linguistic threshold schemes we define graph grammar to describe secret
data. Types of selected grammars determine not only description of secret data, but also
allow to define the secret reconstruction process.
Figure 1 presents examples of tree and graphs possible to use in secret sharing
processes.

Fig. 1. Examples of structures possible to use in linguistic threshold schemes: (a) tree structure,
(b) graph structure.

Proposed techniques based on different grammar description can be applied for data
security processes in biometric-based threshold schemes.

3 Example of Application of Biometric-Based Threshold

Schemes

Proposed new classes of threshold schemes can be applied in different structure and
levels of data security. Also, possibilities of application of different types of formal
grammar in description processes (in linguistic protocols) creates new development
opportunities for the presented algorithms. The main characteristic features for all
possible classes of threshold schemes presents Table 1.
 Biometric-Based Linguistic Solutions for Data Encryption and Sharing 185

In Table 1 we can see, that the combined protocols – biometric-based linguistic

threshold schemes – are the most universal schemes dedicated to secret security.
Presented techniques of secret protection can be used in different structures – inside
and outside the organization. Generally we distinguish three basic levels – one inside
and two outside the organization. The first level it is the structure of organization, the
second level it is a fog level and the third level it is a Cloud level [11].

Table 1. Comparison of each classes of threshold schemes.

Name of Type of Assessment of Biometric Identification of
threshold scheme grammars the secret marking of the secret holders
meaning shadows
Linguistic Sequence + − −
tree graph
Biometric − − + +
Biometric-based Sequence + + +
linguistic scheme tree graph

In this paper, we focus our attention on data security related to the process of
service management. In this range, we can share data and services using selected
grammars in secret decryption process. Figure 2 presents an example of proposed
application of biometric-based linguistic threshold schemes.

Fig. 2. Examples of biometric-based linguistic threshold schemes created by: (a) tree structure,
(b) graph structure.

Figure 2 present two examples of biometric-based linguistic threshold schemes.

The first one – Fig. 2a presents tree structure, on which node 1 belongs to the arbiter,
node 2 is a linguistic shadow and nodes 3–5 are the normal shadows. Each of them is
marked by selected biometrics (for example finger print). It is possible to reconstruct
original secret at different levels. At organization level secret will be reconstructed by
186 U. Ogiela et al.

any four selected shadows (included linguistic shadow). At fog level the same secret
will be reconstructed by any three secret shadows (also included linguistic shadow). At
Cloud level secret is reconstructed by any two secret shadows (included linguistic
shadow).
Different situation presents Fig. 2b. It is a graph structure, on which node 1 belongs
to the arbiter, node 2 is a linguistic shadow and nodes 3–5 are the normal shadows.
Each of them is marked by personal information – for example biometric finger print.
Also, in this structure is possible to reconstruct secret at different levels. At organi-
zation level is necessary to submit a four parts (included linguistic shadow) connected
with each other (forming a loop). At fog level the secret will be reconstructed by any
three of secret parts (included linguistic shadow and forming a loop). At Cloud level is
enough to join two secret parts, but in this case no loop will occur. So, finally in this
example, the reconstruction process is the same at the fog and at the Cloud levels.

4 Conclusions

This paper present new idea of combined cryptographic threshold schemes. We have
described possibilities of combining two separated threshold procedures, which allow
to create new class of secure division methods. Biometric-based linguistic threshold
schemes extend classical methods of data security, but also allow to learn about the
importance of protected information. In this class of threshold protocols is possible to
describe data by used linguistic techniques and determine their meaning. Also, it is
possible to create a biometric description by individual data included in personal
biometrics. Personal features can mark all secret parts by unique biometrics.
Proposed methods are dedicated to secure information and perform management
services at different structures and levels, inside and outside by the organization.

Acknowledgments. This work has been supported by the National Science Centre, Poland,
under project number DEC-2016/23/B/HS4/00616.
This work was supported by JSPS KAKENHI grant number 15H0295.

References
1. Chomsky, N.: Syntactic Structures. Mouton, London (1957)
2. Gregg, M., Schneier, B.: Security Practitioner and Cryptography Handbook and Study Guide
Set. Wiley, Hoboken (2014)
3. Duolikun, D., Aikebaier, A., Enokido, T., Takizawa, M.: Design and evaluation of a
quorum-based synchronization protocol of multimedia replicas. Int. J. Ad Hoc Ubiquit.
Comput. 17(2/3), 100–109 (2014)
4. Nakamura, S., Ogiela, L., Enokido, T., Takizawa, M.: Flexible synchronization protocol to
prevent illegal information flow in peer-to-peer publish/subscribe systems. In: Barolli, L.,
Terzo, O. (eds.) Complex, Intelligent, and Software Intensive Systems. Advances in
Intelligent Systems and Computing, vol. 611, pp. 82–93. Springer, Cham (2018)
 Biometric-Based Linguistic Solutions for Data Encryption and Sharing 187

5. Ogiela, L.: Cognitive computational intelligence in medical pattern semantic understanding.

In: Guo, M.Z., Zhao, L., Wang, L.P. (eds.) Proceedings of Fourth International Conference
on Natural Computation, ICNC 2008, 18–20 October 2008, Jian, People’s Republic of
China, vol. 6, pp. 245–247 (2008)
6. Ogiela, L.: Cryptographic techniques of strategic data splitting and secure information
management. Pervasive Mobile Comput. 29, 130–141 (2016)
7. Ogiela, L., Ogiela, M.R.: Insider threats and cryptographic techniques in secure information
management. IEEE Syst. J. 11(2), 405–414 (2017)
8. Ogiela, M.R. Ogiela, L., Ogiela, U.: Intelligent bio-inspired approach for secrecy
management in the cloud. In: Ninth International Conference on P2P, Parallel, Grid, Cloud
and Internet Computing, 3PGCIC, 08–10 November 2014, Guangzhou, People’s Republic
of China, pp. 286–289
9. Ogiela, M.R., Ogiela, U.: Secure information splitting using grammar schemes. In: New
Challenges in Computational Collective Intelligence: Studies in Computational Intelligence,
vol. 244, pp. 327–336 (2009)
10. Ogiela, M.R., Ogiela, U.: Secure information management in hierarchical structures. In:
Advanced Computer Science and Information Technology: Communications in Computer
and Information Science, vol. 195, pp. 31–35 (2011)
11. Ogiela, U., Takizawa, M., Ogiela, L.: Classification of cognitive service management
systems in cloud computing. In: Barolli, L., Xhafa, F., Conesa, J. (eds.). Advances on Broad-
Band Wireless Computing, Communication and Applications, BWCCA 2017. Lecture Notes
on Data Engineering and Communications Technologies, vol. 12, pp 309–313. Springer,
Cham (2018). https://doi.org/10.1007/978-3-319-69811-3_28
12. Yan, S.Y.: Computational Number Theory and Modern Cryptography. Wiley, Hoboken
(2013)
 Dependability Analysis for On-Demand
Computing Based Transaction
Processing System

Dharmendra Prasad Mahato1(B) , Jasminder Kaur Sandhu2 ,

Nagendra Pratap Singh1 , and Kamlesh Dutta1
1
Computer Science and Engineering, National Institute of Technology Hamirpur,
Hamirpur 177005, Himachal Pradesh, India
{dpm,nps,kd}@nith.ac.in
2
Computer Science and Engineering, Thapar University,
Patiala 147001, Punjab, India
[email protected]

Abstract. This work is devoted to the dependability modeling and

analysis for on-demand computing based transaction processing system.
This paper proposes a stochastic differential equation (SDE) based on
Ornstein-Uhlenbeck processes. The Wiener processes used in the pro-
posed equations miss some important features of the transactional data.
Jumps based on Gaussian jump diffusion process using normal distribu-
tion are added to the Wiener processes to make the estimation easy. The
paper uses Particle Swarm Optimization (PSO) to estimate the jump
diffusion parameters in the proposed model.

1 Introduction
On-demand computing is a model for enabling ubiquitous, convenient, on-
demand network access to a shared pool of configurable computing resources.
It provides several services. The on-demand computing based transaction pro-
cessing is one of them which deals with the challenge to an enterprise to meet
fluctuating demands of sufficient resources efficiently. Concepts such as grid com-
puting, utility computing, autonomic computing, cloud computing and adaptive
management seem very similar to the concept of the on-demand computing.
The paper proposes a stochastic differential equation (SDE) based on
Ornstein-Uhlenbeck processes for dependability modeling of on-demand comput-
ing based transaction processing system. The paper also considers four types of
faults in the model; program fault, network fault, resource fault, and deadline-
miss fault. The Wiener processes used in the proposed equations miss some
important features of the transactional data. Jumps based on Gaussian jump
diffusion process using normal distribution are added to the Wiener processes to
make the estimation easy. Jump diffusion parameters are estimated using PSO
algorithm. Moreover, the paper formulates a total expected software cost model

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 188–199, 2020.
https://doi.org/10.1007/978-3-030-15032-7_17
 Dependability Analysis for On-Demand Computing 189

for the environment of on-demand computing. The paper applies the proposed
method to construct SDEs-based models from different probability distribution
also.

2 Model Description
Let {N (t), t ≥ 0} is the cumulative number of faults detected up to time t where
N (t) takes on continuous real values. As these faults are detected and eliminated
during the transaction processing, N (t) goes on increasing in a gradual way.
Thus, using the concept of reliability growth modeling, N (t) can be formulated
by using the linear differential equation as

dN (t)
= x(t){D(t) − N (t)} (1)
dt
where x(t) is the fault-detection rate at processing time t and is a non-negative
function and D(t) represents the number of changes in requirement specifications
and is defined as D(t) = αe−βt where α is the number of latent faults in the
system and β is the rate of change in D(t).
On-demand computing based transaction processing system can be modeled
by extending the Eq. 1 to the following stochastic differential equation consider-
ing four Brownian motions:

dN1 (t)
= {x1 (t) + σ1 ν1 (t)}{R1 (t) − N1 (t)} (2)
dt

dN2 (t)
= {x2 (t) + σ2 ν2 (t)}{R2 (t) − N2 (t)} (3)
dt
dN3 (t)
= {x3 (t) + σ3 ν3 (t)}{R3 (t) − N3 (t)} (4)
dt
dN4 (t)
= {x4 (t) + σ4 ν4 (t)}{R4 (t) − N4 (t)} (5)
dt
where σ1 , σ2 , σ3 , and σ4 are the positive constants and represent magnitudes of
the irregular fluctuations. Moreover, ν1 , ν2 , ν3 , and ν4 represent the standardized
Gaussian white noises and x1 (t), x2 (t), x3 (t), and x4 (t) are the fault-detection
rates which depend on the failure-occurrence phenomena in the system of the
software or the program, the network, the resource, and the deadline-miss respec-
tively.

dN1 (t)
= {x(t) + σ1 ν1 (t) + σ2 ν2 (t) + σ3 ν3 (t) + σ4 ν4 (t)}{D(t) − N (t)} (6)
dt
When we extend Eqs. (2)–(5) by using the concept of an Itô stochastic dif-
ferential equation [1], we obtain the following equation,
190 D. P. Mahato et al.

1
dN1 (t) = {x1 (t) − σ12 }{R1 (t) − N1 (t)}dt + σ1 {R1 (t) − N1 (t)}dω1 (t) (7)
2
1
dN2 (t) = {x2 (t) − σ22 }{R2 (t) − N2 (t)}dt + σ2 {R2 (t) − N2 (t)}dω2 (t) (8)
2
1
dN3 (t) = {x3 (t) − σ32 }{R3 (t) − N3 (t)}dt + σ3 {R3 (t) − N3 (t)}dω3 (t) (9)
2
1
dN4 (t) = {x4 (t) − σ42 }{R4 (t) − N4 (t)}dt + σ4 {R4 (t) − N4 (t)}dω4 (t) (10)
2
where ω1 (t), ω1 (t), ω1 (t), and ω1 (t) are the Wiener processes and are the inte-
gration of white noises ν4 (t), ν4 (t), ν4 (t), and ν4 (t) respectively.
We obtain the integrated stochastic differential equation which is based on
the independence of each noise and is expressed as:
1
dN (t) = {x(t) − (σ12 + σ22 + σ32 + σ42 )}{D(t) − N (t)}dt
2
+ σ1 {D(t) − N (t)}dω1 (t)
+ σ2 {D(t) − N (t)}dω2 (t) (11)
+ σ3 {D(t) − N (t)}dω3 (t)
+ σ4 {D(t) − N (t)}dω4 (t)
We define four dimension processes [ω1 (t), ω2 (t), ω3 (t), ω4 (t)] as [2]:
1
ω̃(t) = (σ12 + σ22 + σ32 + σ42 )− 2
(12)
.{σ1 ω1 (t) + σ2 ω2 (t) + σ3 ω3 (t) + σ4 ω4 (t)}
where ω̃(t) is a Gaussian process.
When we solve Eq. 11 by using Itô, s formula [1–4], we obtain the following
solution with the initial condition N (0) = 0 as given by
  
t
N (t) = D(t) 1 − exp − x(s)ds
0
 (13)
− σ1 ω1 (t) − σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t)

We can estimate the reliability of transaction processing system by several

estimation models [7], such as Exponential, Weibull, Delayed S-shaped, and
Inflected S-shaped model.
Let H(t) is the mean value function which is defined as the statically expected
value of N (t) [7]. Assume H(t) as a nondecreasing with the boundary conditions
H(0) = 0 and H(∞) = a, where a is the statistically expected cumulative
number of faults to be detected eventually.
 Dependability Analysis for On-Demand Computing 191

2.1 The Exponential Model of Goel and Okumoto

Definition 2.1
He (t) = a(1 − exp(−xt)) (14)

 t dNe (t) dHe (t)

dt dt
x1 (s)ds = = =x (15)
0 a − Ne (t) a − He (t)

2.2 Delayed S-shaped Model

Definition 2.2
Hd (t) = a(1 − (1 + xt) exp(−xt)) (16)
 t dNd (t) dHd (t)
dt dt x2 t
x2 (s)ds = = = (17)
0 a − Nd (t) a − Hd (t) 1 + xt

2.3 Inflected S-shaped Model

Definition 2.3
1 − exp(−xt)
Hin (t) = a (18)
1 + ψ. exp(−xt)
 t dNin (t) dHin (t)
dt dt x
x3 (in)ds = = = (19)
0 a − Nin (t) a − Hin (t) 1 + ψ. exp(−xt)

2.4 Weibull Model

Definition 2.4
1 − exp((−t)x )
Hw (t) = a (20)
1 + ψ. exp(−xt)

 t dNw (t)
dt
x4 (w)ds =
0 a − Nw (t)
dHw (t)
dt
=
a − Hw (t) (21)
x. exp((−t)x )(−t)x−1 + xψ exp((−t)x )(−t)x−1 exp(−xt)
=
(1 + ψ. exp(−xt))(ψ exp(−xt) + exp((−t)x ))
xψ exp(−xt) − xψ exp(−xt) exp((−t)x )
+
(1 + ψ. exp(−xt))(ψ exp(−xt) + exp((−t)x ))
192 D. P. Mahato et al.

where ψ is the inflection parameter. ψ is defined for given inflection rate r which
is the ratio of the number of detectable faults to the total number of faults by
the following equation as [5]:
1−r x
ψ= =, r > 0 & r = (22)
r a
which gives ψ = a−x x and He (t), Hd (t), Hin (t), and Hw (t) are the mean value
functions for the exponential, delayed s-shaped, inflection s-shaped, and Weibull
softare reliability growth model (SRGM) [5], respectively, based on NHPP.
Therefore, the cumulative number of detected faults for these model can be
obtained as follows (Table 1):


Ne (t) = D(t) 1 − exp − xt
 (23)

− σ1 ω1 (t) − σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t)



Nd (t) = D(t) 1 − (1 + xt) exp − xt
 (24)

− σ1 ω1 (t) − σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t)


1+ψ 
Nin (t) = D(t) 1 − exp − xt − σ1 ω1 (t)
1 + ψ. exp(−xt)
 (25)

− σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t)


ψ. exp(−xt) + exp((−t)x ) 
Nw (t) = D(t) 1 − exp − xt − σ1 ω1 (t)
1 + ψ. exp(−xt)
 (26)

− σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t)

2.5 Jump Diffusion Modeling

2.5.1 The Gaussian Jump Diffusion Process
Definition 2.5. A Gaussian jump diffusion process Jt is defined as

Kt
Jt = Yi (27)
i=1
 Dependability Analysis for On-Demand Computing 193

Table 1. Parameters of the simulated SDE models

He Hd Hin Hw
a = 6 × 10−4 a = 6 × 10−4 a = 6 × 10−4 a = 6 × 10−4
x = 5 × 10−4 x = 5 × 10−4 x = 5 × 10−4 x = 5 × 10−4
M T T F = x1 M T T F = x1 M T T F = x1 M T T F = x1
variance = x1 variance = x1 variance = x1 variance = x1
- - ψ = a−x
x
= 0.2 ψ = a−x
x
= 0.2

where Jt is a compound Poisson process, Yi are the jump sizes which are inde-
pendently and identically distributed with F , and the number of jumps Kt is a
Poisson process with intensity λ. Here Yi , which is a Gaussian process can be
expressed as

λ  (ξ − μ)2 
Yi ≡ fi (ξ) = √ exp − (28)
2πδ 2δ 2
where μ is the mean of the jump J and δ is the standard deviation of jumps.

In this jump diffusion model, the stochastic differential equation to consider

the characteristics of the transaction fault-detection phenomena is given as
1
dN (t) = {x(t) − (σ12 + σ22 + σ32 + σ42 )}{D(t) − N (t)}dt
2
+ σ1 {D(t) − N (t)}dω1 (t)
+ σ2 {D(t) − N (t)}dω2 (t) (29)
+ σ3 {D(t) − N (t)}dω3 (t)
+ σ4 {D(t) − N (t)}dω4 (t)
+ dJt
where Jt is the jump term to incorporate the irregular state around the time t

by a change in the number of log-in users. By using Itô s formula, the solution
of Eq. 29 can be obtained as

ψ. exp(−xt) + exp((−t)x ) 
Nw (t) = D(t) 1 − exp − xt − σ1 ω1 (t)
1 + ψ. exp(−xt)
 (30)
 Kt
− σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t) − log Yi
i=1
194 D. P. Mahato et al.

3 Parameter Estimation
3.1 Method of Maximum-Likelihood
This section presents the estimation method for unknown parameters α, β, x,
σ1 , σ2 , σ3 , and σ4 . Then

P (t1 , y1 ; t2 , y2 ; ...; tm , ym ) ≡ Pr[N (t1 ) ≤ y1 , ..., N (tm ) ≤ ym |N (t0 ) = 0]

(31)
Then we can obtain the probability density of Eq. 31 as
∂ m P (t1 , y1 ; t2 , y2 ; ...; tm , ym )
P (t1 , y1 ; t2 , y2 ; ...; tm , ym ) ≡ (32)
∂y1 ∂y2 ...∂ym
The likelihood function, l, for the data (tm , ym ) can be obtained as follows:

l = P (t1 , y1 ; t2 , y2 ; ...; tm , ym ) (33)

Then logarithmic likelihood function can be obtained as:

L = log l (34)
Now if α∗ , β ∗ , x∗ , σ1∗ , σ2∗ , σ3∗ , and σ4∗ are the maximum likelihood estimates,
then they make L in Eq. 34 maximal. Thus we get
∂L ∂L ∂L ∂L ∂L ∂L ∂L
= = = = = = =0 (35)
∂α ∂β ∂x ∂δ1 ∂δ2 ∂δ3 ∂δ4

3.2 Estimation of the Jump Diffusion Parameters

In the algorithm, the position and velocity of a particle or solution in n-
dimensional search space can be represented as the vectors Sj = (sj1 , sj2 , ..., sjn )
denoting a particle’s position and the vectors Vj = (vj1 , vj2 , ..., vjn ) denot-
ing the velocity of the particle. Let pBestj = (spBest j1 , spBest
j2 , ..., spBest
jn ) and
gBest = (sgBest
1 , sgBest
2 , ..., sgBest
n ) are the position of the j th
particle and the
best position of its neighbors so far respectively. pBest and gBest can be
expressed as

Sj (t), if Sj (t + 1) ≥ Sj (t)
pBest = (36)
Sj (t + 1), if Sj (t + 1) < Sj (t)

gBest ∈ {S0 (t), S1 (t), ..., Ss } | θ(gBest)

(37)
= min{θ(S0 (t)), θ(S1 (t)), ..., θ(Ss (t))}
Each particle consists of: Data which represents a possible solution, a velocity
which indicates how much the data is changed, and a personal best pBest which
indicates the closest the data has ever come to the target
The algorithm also keeps on tracking three global variables: The target value,
global best (gBest) which represents the particle’s data is closest to the target,
 Dependability Analysis for On-Demand Computing 195

and stopping value which indicates when the algorithm stops if the target is not
found.
The fundamental PSO algorithm consists of three steps:

1. Evaluation of fitness of each particle,

2. Update the velocity and position of each particle,
3. Update the individual and the global best values.

3.2.1 Evaluation of Fitness Value

In this paper, the value of fitness is calculated from the evaluated value of
each particle. The following value represents the fitness value which is the error
between the estimated and the actual values:

minimize θ(Sj (t))

Sj (t)

K (38)
subject to θt (Sj (t)) = {N (t) − yt }2
t=0

where N (t) is the estimated faults at time t, yt is the number of actual detected
faults, θ represents the set of parameters ξ, δ, and μ used in Eqs. 27–28.

3.2.2 Updation of Velocity and Position of Each Particle

PSO algorithm works by having a population of candidate solutions. The parti-
cles swarm through the search space to find the maximum value which is eval-
uated by using an objective function. Each solution maintains position in the
search space, velocity, and individual best position. Each particle has its own
determined pBest with which it compares at each point with the fitness value
of its current location. If the fitness value of the current location is higher than
its pBest, the current location becomes its pBest.

3.2.3 Updation of Individual and Global Best Value

Moreover, each particle knows also the highest concentration of the errors dis-
covered by the entire swarm. This highest concentrated location is known as
gBest. There exists one gBest for the entire swarm and each particle is attracted
towards gBest. Along their path, at each point, each particle compares the fit-
ness value of its current location with gBest. If the fitness value of the current
location of any particle is higher than gBest, the current location becomes gBest.
The fundamental pseudocode and structure of PSO is presented in
Algorithm 1.
The velocity and position of each particle are calculated by using the current
velocity and the difference between pBestj and gBest as the following:

Vj (t + 1) = ωVj (t) + c1 Rand1 (pBestj (t) − Sj (t))

(39)
+ c2 Rand2 (gBest(t) − Sj (t))
196 D. P. Mahato et al.

Algorithm 1. PSO to parameter estimation of the Jump Diffusion Model

1: for each particle do
2: generate an initial article
3: end for
4: while maximum iterations or minimum error criteria is not
attained do
5: for each particle do
6: Calculate the fitness value using Eq. 36
7: if the fitness value is better than the best fitness value
(pBest) in history then
8: set current value as the new pBest
9: end if
10: end for
11: Choose the particle with the best fitness value of all the
particles as the gBest
12: for each particle do
13: Calculate particle velocity using Eq.37
14: Update particle position using Eq.38
15: end for
16: end while

Sj (t + 1) = Sj + Vj (t + 1) (40)
where Vj (t) represents the velocity of j th particle at k th iteration, ω represents
the weigh parameter (inertia weight) used to balance the global exploration and
local exploitation, c1 , c2 represent the acceleration coefficients, Rand1 , Rand2
represent random numbers uniformly distributed between 0 and 1, Sjk represents
the position of j th particle at iteration k, pBestj be the best position of j th
particle until iteration k, gbestk be the best position of the group until iteration k.
Equation 39 contains three components. The first component is the previous
velocity of the particle scaled by the inertia weight ω. The second component
which is known as “self-knowledge,” gives the linear attraction value to pBestkj . It
is scaled by product of c1 and Rand1 where c1 denotes the acceleration constant
and Rand1 represents a random number. Similar to the second component, the
third one which is known as “team work”, gives the linear attraction value to
gBestk . This component is scaled by the product of c2 and Rand2 where c2 and
Rand2 represent the acceleration constant and random number respectively.

4 Dependability Assessment Measures

4.1 Reliability in Transaction Processing System

Definition 4.1. Reliability can be defined as the probability that a system will
perform a required function under a stated condition for defined period of time.
 Dependability Analysis for On-Demand Computing 197

To define the reliability this paper, we need to calculate Mean Time Between
Failures (MTBF).
Definition 4.2. MTBF is defined as the mean of a distribution of the system
life or life between successive failures. Therefore, MTBF is useful to calculate
the frequency of failure-occurrence (or fault-detection) [6].
The cumulative MTBF denoted by M T BFC can be computed as
t
M T BFC = (41)
E[N (t)]
where E[N (t)] is the expected number of detected faults up to time t (here we
use Weibull distribution based mean value function) and can be calculated as



E[Ne (t)] = D(t) 1 − exp − xt
 K  (42)
 t

− σ1 ω1 (t) − σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t) − log Yi

i=1



E[Nd (t)] = D(t) 1 − (1 + xt) exp − xt
 (43)
 Kt
− σ1 ω1 (t) − σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t) − log Yi
i=1


1+ψ 
E[Nin (t)] = D(t) 1 − exp − xt
1 + ψ. exp(−xt)
 (44)
 Kt
− σ1 ω1 (t) − σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t) − log Yi
i=1


ψ. exp(−xt) + exp((−t)x ) 
E[N (t)] = D(t) 1 − exp − xt
1 + ψ. exp(−xt)
 (45)
 Kt
− σ1 ω1 (t) − σ2 ω2 (t) − σ3 ω3 (t) − σ4 ω4 (t) − log Yi
i=1

Then, reliability that the system will work for a defined period of time with-
out failure is given by

R(t) = exp(−t/M T BFC ) (46)

198 D. P. Mahato et al.

4.2 Performability
Definition 4.3. Performability is the combination of reliability and perfor-
mance of the system. It is used to quantify the operational quality of service
between failure occurrence and its recovery. It can be computed as [6]:

t − M T T R(t).E[N (t)]
P ER(t) = (47)
t
where M T T R(t) is the mean time taken to recover the system from failure.

The term is useful to assess the dependability of a system.

5 Optimal Maintenance Problem

Maintenance can be done by minimizing the total expected cost of the system
maintenance.
Let cost1 , cost2 , cost3 , and cost4 be the related cost parameters which we
define as follows:
cost1 : the fixing cost of each fault during the transaction processing,
cost2 : the cost per unit time during the transaction processing,
cost3 : the maintenance cost of each fault during the transaction processing,
cost4 : the service cost during the transaction processing.
The expected cost of the system can be computed as [6]:

Cost1 (t) = cost1 .E[N (t)] + cost2 .t (48)

The expected maintenance cost can be calculated as

Cost2 (t) = cost3 {D(t) − E[N (t)]} (49)

The service cost function can be given as

Cost3 (t) = cost4 .U {1 − exp[−P ER(t).t]} (50)

where U is the number of users requesting for the transaction processing.
The total cost [6] can be computed as

Cost(t) = Cost1 + Cost2 + Cost3 (51)

References
1. Allen, E.: Modeling with Itô Stochastic Differential Equations, vol. 22. Springer,
Heidelberg (2007)
2. Arnold, L.: Stochastic differential equations, New York (1974)
3. Øksendal, B.: Stochastic differential equations. In: Stochastic Differential Equations,
pp. 65–84. Springer, Heidelberg (2003)
 Dependability Analysis for On-Demand Computing 199

4. Oksendal, B.: Stochastic Differential Equations: An Introduction with Applications.

Springer, Heidelberg (2013)
5. Sagar, B.B., Saket, R.K., Singh, C.G.: Exponentiated Weibull distribution approach
based inflection S-shaped software reliability growth model. Ain Shams Eng. J. 7(3),
973–991 (2016)
6. Tamura, Y., Yamada, S.: Dependability analysis and optimal maintenance prob-
lem for open source cloud computing. In: 2012 IEEE International Conference on
Systems, Man, and Cybernetics (SMC), pp. 1592–1597. IEEE (2012)
7. Yamada, S., Ohba, M., Osaki, S.: S-shaped software reliability growth models and
their applications. IEEE Trans. Reliab. 33(4), 289–292 (1984)
 A Comparative Analysis of Trust
Requirements in Decentralized Identity
Management

Andreas Grüner(B) , Alexander Mühle(B) , Tatiana Gayvoronskaya(B) ,

and Christoph Meinel(B)

Hasso Plattner Institute (HPI), University of Potsdam, 14482 Potsdam, Germany

{andreas.gruener,alexander.muehle,tatiana.gayvoronskaya,
christoph.meinel}@hpi.uni-potsdam.de

Abstract. Identity management is a fundamental component in secur-

ing online services. Isolated and centralized identity models have been
applied within organizations. Moreover, identity federations connect dig-
ital identities across trust domain boundaries. These traditional mod-
els have been thoroughly studied with regard to trust requirements.
The recently emerging blockchain technology enables a novel decentral-
ized identity management model that targets user-centricity and elim-
inates the identity provider as a trusted third party. The result is a
substantially different set of entities with mutual trust requirements.
In this paper, we analyze decentralized identity management based on
blockchain through defining topology patterns. These patterns depict
schematically the decentralized setting and its main actors. We study
trust requirements for the devised patterns and, finally, compare the
result to traditional models. Our contribution enables a clear view of
differences in trust requirements within the various models.

1 Introduction

In the present, online services and electronic communication pervade everyday

life, both in the private and business area. Banking, shopping, social networks
and cloud services are a few samples in this regard. Securing these services in
various dimensions is essential to prevent misuse, attract customers and finally
create trust in a product or technology. A common factor in terms of security and
a basic prerequisite for personalized use is the establishment and management of
digital identities – commonly referred to as identity management [1]. The online
service needs to differentiate and recognize users, for instance, to selectively
enable functionality or grant access to user-specific resources. At the same time,
fraudulent impersonation must be prevented.
In 2008, Satoshi Nakamoto published the foundations for a peer-to-peer cur-
rency named Bitcoin [2]. Bitcoin is a fully decentralized cash scheme that does
not require a trusted third party. Subsequently, the idea of a decentralized cash

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 200–213, 2020.
https://doi.org/10.1007/978-3-030-15032-7_18
 A Comparative Analysis of Trust Requirements 201

scheme was generalized to obtain a decentralized execution platform for arbi-

trary operations, in general, called blockchain [3]. Hence, a blockchain enables
the implementation of an identity provider that is decentralized and not a trusted
third party [4]. The issuance of self-sovereign identities, which are in the posses-
sion and under true control of the user, is empowered by a blockchain-based iden-
tity provider. Trust requirements may significantly change by applying decen-
tralized identity management without having an identity provider as a trusted
third party.
In this paper, we investigate trust requirements in the decentralized identity
management model and constitute a view on the differences when compared
to traditional models. In this way, we address the points that are still lacking
in the understanding. To achieve this, we outline decentralized identity man-
agement based on blockchain. Subsequently, we define topology patterns that
reflect the relevant actors and their interaction paths. Using these patterns, we
study trust requirements for the different entities and compare them to the trust
prerequisites in the traditional models.
The rest of the paper is structured as follows. Section 2 presents related work
on trust requirements in identity management, and the respective differences
to our work. The general concept of trust and our evaluation methodology is
outlined in Sect. 3. We illustrate in detail patterns and associated trust require-
ments in decentralized identity management in Sect. 4. Finally, we conclude our
analysis with Sect. 5.

2 Related Work

Jøsang et al. [5] researched simple trust requirements in isolated, centralized and
federated identity management as well as personal authentication management.
Trust requirements are analyzed from the perspectives of the user and the ser-
vice provider. The service provider implicitly covers the identity provider as well.
The common identifier domain, meta-identifier domain and single-sign-on are
sub-classes of the centralized identity management model. Moreover, in identity
federation structures the authors distinguish trust requirements between differ-
ent service providers and users. Finally, personal authentication management,
with a trust requirement to the tamper-resistance of the personal authentication
device is evaluated. Kylau et al. [6] studied detailed trust requirements and asso-
ciated risks in identity federation topologies. In general, the entities user, identity
provider and service provider are differentiated. Kylau et al. analyzes the iden-
tity federation structures based on defined patterns. These patterns are clustered
in direct trust topologies and patterns that additionally integrate indirect trust
schemes. The paper concludes by comparing trust requirements and risks of the
federated identity patterns. With the increasing complexity of the pattern, the
trust requirements and the associated risk increase as well. Ferdous and Poet [7]
evaluate attribute aggregation models in federated identity management with
regard to risk as well as trust and functional requirements. These models are
clustered according to a taxonomy that strongly considers the location where
202 A. Grüner et al.

the attribute aggregation occurs. In general, attributes are aggregated at either

the service provider, identity provider or user level.
In contrast to the previous work, we study trust requirements in the decen-
tralized identity management model based on blockchain technology by devising
appropriate patterns. Furthermore, we compare the derived trust prerequisites to
the traditional models in order to evaluate differences. We consider the attribute
aggregation manner and do not concentrate on the aggregation location.

3 Analysing Trust in Identity Management

In this section, we provide a general background on the notion of trust, the
relevant actors and their domains of trusted interactions. Additionally, we outline
our evaluation methodology for analyzing the trust requirements.

3.1 The Notion of Trust

Trust is a pervasive and significant phenomenon in social societies with a diverse
and manifold range of meanings and definitions [8]. In human relationships trust
is fundamental for cooperation, conversation and mutual interactions between
persons or institutions. A definition of trust is subjective in nature and depends
on the contextual setting [9]. Jøsang et al. [8] created a definition based on the
former work of McKnight and Chervany [10] that characterizes decision trust as
The extent to which one party is willing to depend on something or some-
body in a given situation with a feeling of relative security, even though
negative consequences are possible.
First of all, the interpretation emphasizes entities that rely on each other unilat-
erally or bilaterally. Secondly, a certain situation is required that is related to the
dependency. Finally, negative consequences are named in case the dependency
is misused from the perspective of the relying party. The potential occurrence
and impact of the adverse effect reflects the actual risk.
Embedding the definition of decision trust into the context of identity man-
agement, determining the acting parties is the starting point for analyzing
mutual trust relationships. We differentiate the user, the service provider, and
the identity provider. Additionally, we consider the attribute provider as derived
entity from the identity provider.
• User (U). The user is characterized as the subject who is represented by the
digital identity. The user is in possession of a credential to solely control and
use the digital identity.
• Service Provider (SP). The service provider offers a service that requires
users to authenticate and convey attributes. Based on the authenticated user
and the transmitted attributes, the service provider may adopt its service.
• Identity Provider (IdP). The identity provider offers capabilities to cre-
ate, manage and employ digital identities. An additional role of the identity
provider is the supply and verification of user attributes.
 A Comparative Analysis of Trust Requirements 203

– Attribute Provider (AP). The attribute provider offers a subset of

the identity provider’s functionality with the focus on the allocation and
validation of user attributes.

3.2 Trust Domains and Requirements

The trust domains cluster the situations where dependencies between the dif-
ferent actors exist. Aligned with the decision trust definition, a dependency
indicates that an actor relies on another entity. This reliance reflects a trust
requirement of the trustor towards the trustee. The violation of a trust require-
ment implies an adverse impact for the trusting party. We consider the following
domains with the listed trust requirements for the comparative analysis.

• Privacy. A digital identity and its attributes represent personal information

of the user that are protected by a variety of regulations [11]. Additionally,
statistics and profiles can be derived from the use of the digital identity at
a service provider or identity provider. The user expects that the individual-
related information is kept private and is solely evaluated and disclosed for
the intended and consented purpose.
– T1a. The identity resp. attribute provider protects the privacy of the
user.
– T1b. The service provider protects the privacy of the user [5].
• Credential Management. A credential is required to enable a user to con-
trol their digital identity and to prevent impersonation attacks. The identity
provider needs to securely generate, change and store the credential or related
verification information. Moreover, the user has the responsibility to protect
and not to share the credential.
– T2a. The identity provider implements secure credential management
[5].
– T2b. The user protects the credential [5] and does not deliberately dis-
close it.
• Authentication. A service provider requires user authentication. During
authentication, the user demonstrates the possession of the credential to con-
trol the digital identity at the identity provider. The identity provider for-
wards the success or failure result to the service provider. The service provider
maps the logged in digital identity to internally administrated datasets.
– T3a. User authentication is done adequately by the identity provider [6].
– T3b. User mapping is managed correctly by the service provider [6].
• Attribute Management. Attributes are a substantial part of a digital iden-
tity. A service provider may use these properties within the offered service
(e.g. for age verification). Moreover, the characteristics can be applied to
decide if a user is eligible to consume a service. Therefore, the properties must
be correct and reflect reality. Additionally, the attributes must be revoked in
a timely manner if the validity period has expired.
– T4a. Delivered attributes of the users are correct.
– T4b. Invalidated attributes are revoked in a timely manner.
204 A. Grüner et al.

3.3 Objective and Evaluation Methodology

The objective of our contribution is to analyze trust requirements in decentral-

ized identity management and to identify differences and common features with
regard to the traditional identity management models. To achieve this, we elab-
orate bilateral trust relations of the outlined actors according to the described
trust requirements. Moreover, we add an indicator to describe the required level
of trust that is needed. The level of trust relates to the degree of dependency
towards one trusted third party and is differentiated in the following way.

• Absolute. The dependent actor needs to completely rely on a trusted third

party. There is no significant compensating control or trust distribution
between several third parties to reduce the overall required trust allocation.
• Limited. In any given situation, the trust requirement does not apply uncon-
ditionally for an entity. A major compensating control may exist that degrades
the needed trust. Besides that, the required trust might be reduced by dis-
tributing the dependency to several entities in order to remove a single trusted
third party.

4 Trust Requirements in Decentralized Identity

Management

In the following sections, we start with a characterization of decentralized iden-

tity management based on blockchain and outline a respective architecture.
Afterwards, we define patterns in this setting and describe associated trust
requirements between the actors. Finally, we compare the trust requirements
with the traditional types of identity management.

4.1 Decentralization Characteristics

A common characterization of traditional identity management models is the

consolidated implementation of central functions at an identity provider. There-
fore, the identity provider represents a trusted third party towards the user and
the service provider. The decentralization of identity management and there-
fore, the remediation of the identity provider as trusted third party needs to
consider the characteristics execution, storage, attributes, namespace and
organization.
The identity provider is implemented as software and executed in a server
environment under full control of the hosting party. Trust is required to believe
that the implementation works as expected and does not deviate from published
properties. An identity provider stores under its control user information and
specific attributes of digital identities. The decentralization of storage refers to
the relocation of an identity provider-owned storage to a user-defined position.
In case a certain attribute is delivered by a single entity, the attribute’s val-
idation and correctness rely solely on the specific provider. An aggregation of
 A Comparative Analysis of Trust Requirements 205

attributes from various providers drives decentralization [12]. The identifier of

a digital identity is generated within a namespace. An essential prerequisite
for identifiers is their uniqueness that needs to be managed in a decentralized
way. An identity provider is operated by an organization. This organization is
usually in control of the identity provider and reflects a point of centralization.

4.2 Decentralized Identity Management Based on Blockchain

Blockchain enables the implementation of a decentralized identity provider

adhering to the outlined characteristics. Programs are executed decentralized
with a common and transparent consensus by all participating nodes in the net-
work [3,4]. A wide range of projects with different implementation approaches
exists [13]. uPort [14] is a solution based on smart contracts on the public and
unpermissioned blockchain Ethereum [3]. Moreover, Sovrin [15] applies a dis-
tinct set of public and permissioned blockchains to implement decentralized
identity management. A common factor is the decentralization of the organi-
sation, namespace and attributes. Nodes can join an unpermissioned blockchain
independently from any registration process. Therefore, no organization exerts
control over it. In the case of Sovrin, there is a fine-grained trust model with
a voting scheme to enable democratic participation in the blockchain network
[15]. Besides that, the namespace is managed as a decentralized registry on the
blockchain. Furthermore, the attributes are depicted as Verifiable Claims [16]
that are comprised of claims and attestations. A claim is a statement about
a user and the attestation is a verification of the statement. Having multiple
attestation authorities, a service provider does not need to rely solely on a single
attribute provider. The decision about the storage location of the attributes is
conveyed to users and their preference, according to the user controls primitive
in Allen’s principles of self-sovereign identity [17].

4.3 Decentralized Identity Management Patterns

To analyze trust requirements in this model, we abstract from the actual pecu-
liarities of the various blockchain implementations to a generalized architec-
ture that is outlined in different patterns. This architecture comprises the ser-
vice provider and the attribute provider as distinct actors. In Figs. 1 and 2 the
actors are depicted as rectangular shapes with rounded edges. A decentralized
blockchain-based identity provider connects the different entities with regard to
identity management functions. However, attributes are actually provided by the
attribute provider and not by the identity provider itself. The decentralized iden-
tity provider is presented as a dashed circled line in the figures. An arrow between
the service provider and the attribute provider reflects the usage of attributes for
a user upon a service request. An arrow between a service provider and multiple
attribute providers illustrates an aggregated usage of attributes.
206 A. Grüner et al.

4.3.1 Bilateral Integration

The simplest pattern is the bilateral integration comprising a single service and
attribute provider. This pattern is shown in the left section of Fig. 1. The service
provider relies solely on one attribute provider for user attributes. The decen-
tralized identity provider implements functions for credential management and
identifier registration. A user registers a digital identity on the decentralized iden-
tity provider and the attribute provider supplies certain attributes of the digital
identity. Upon requesting access at a service provider, the decentralized identity
provider authenticates the user and mediates the attributes from the attribute
provider. Based on the authentication result and the conveyed attributes, the
service provider conducts an access decision.
The user is required to trust the attribute and the service provider with
regard to adherence to privacy obligations. From the user’s point of view, the
trust requirements T1a and T1b apply absolutely toward the respective party.
The trust prerequisite T2a is not applicable in this context, because the decen-
tralized identity provider transparently implements secure credential manage-
ment that can be publicly verified. Equally alike, therefore the prerequisite T3a
is not appropriate in this environment. The decentralized setting also enables a
public verification of the authentication. Therefore, no actual trust is required
by the user. In contrast, the user needs to rely on the service provider that
the user mapping is conducted appropriately. Therefore, trust requirement T3b
applies absolutely between the user and the service provider. Moreover, the user
depends on the attribute provider to convey correct and valid attributes. Thus,
trust prerequisites T4a and T4b are fully applicable as well.
Analysing the service provider, trust requirements T1a, T4a and T4b apply
absolutely towards the attribute provider. The attribute provider is a trusted
third party for the service provider. The service provider expects the adher-
ence to privacy regulations as well as the correctness and validity of the user’s
attributes. Besides that, the trust demand T2b, the protection and secure cre-
dential storage, exists towards the user. In contrast, demands related secure
credental management (T2a) and authentication (T3a) are not applicable due
to the decentralized nature of the identity provider.
The attribute provider fully expects that the service provider protect the
user’s privacy according to T1b.

4.3.2 Multiple Aggregated Integration

The multiple aggregated integration pattern is presented in the middle section of
Fig. 1. The pattern consists of a service provider and several attribute providers.
The service provider receives aggregated attributes of a user that are composed
of both attribute providers. Attribute aggregation considers the same attribute
from multiple attribute providers to verify the delivered attributes of a single
provider. Thus, trust in one attribute provider is decreased.
Comparable to the bilateral integration, the privacy trust requirements (T1a,
T1b) and T3b regarding user mapping are fully applicable to the user. Moreover,
the service provider requires T1a, T2b, and the attribute provider expects adher-
 A Comparative Analysis of Trust Requirements 207

ence to T1b. On the contrary, the trust requirements T4a and T4b with reference
to the properties of the digital identity apply differently. The user and the ser-
vice provider depend on the attribute providers to provide correct attributes
and that attributes are revoked in due course if they are not valid anymore.
The trust requirements T4a and T4b are solely applicable in a limited manner
in this pattern, because the same attributes are delivered by multiple attribute
providers. Therefore, an absolute dependency towards one attribute provider is
eliminated.

Bilateral Integration Multiple Aggregated Integration Multiple Side-by-Side Integration

AP AP AP AP AP

SP SP SP

Fig. 1. Decentralized identity management patterns - Part 1

4.3.3 Multiple Side-by-Side Integration

The multiple side-by-side integration pattern is presented in the right section of
Fig. 1. The pattern consists of a service provider and several attribute providers.
The service provider aggregates the attributes of a user that are received from
both attribute providers. In this pattern, attribute aggregation combines differ-
ent attributes from the providers to obtain all required properties of a digital
identity for the service provider. There is no decrease in trust for a single prop-
erty that is received from multiple attribute providers. That approach differen-
tiates the multiple side-by-side pattern from the multiple aggregated integration
pattern.
In this pattern, trust requirements for all actors are fully comparable to the
bilateral integration. For the user, the trust requirements T1a, T1b and T3b
are completely applicable. Furthermore, the service provider requires fully T1a,
T2b and the attribute provider expects adherence to T1b. Service provider and
user require absolute trust towards the attribute providers with regard to T4a
and T4b.
208 A. Grüner et al.

4.3.4 Multiple Service Provider Integration

The multiple service provider integration pattern is presented in the left section
of Fig. 2. The pattern consists of several service providers and one attribute
provider. The attribute provider is the single source of attributes of digital iden-
tities. Two service providers are representatively depicted in the pattern. How-
ever, multiple service providers can be assumed.
In general, trust requirements are analogous to the bilateral integration pat-
tern. The user has absolute trust demands according to T1a, T1b and T3b.
The service provider requires absolutely T1a and the attribute provider expects
adherence to T1b. Service provider and the user require absolute trust towards
the attribute providers with regard to T4a and T4b.
A difference exists in the limited applicability of trust requirement T2b with
regard to credential protection. The digital identity of the user can be employed
at a multitude of service providers and has, therefore, a significant importance
for the user. One digital identity enables the user to interact in the online world
in contrast to a large number of service-specific digital identities within other
identity management models. This leads to an increased interest of the user to
protect the credential and lowers the risk of deliberate disclosure. Therefore, a
limited level of trust regarding T2b is needed.

4.3.5 Arbitrary Aggregated and Side-by-Side Integration

The arbitrary aggregated and side-by-side integration pattern is the most com-
plex in nature (see Fig. 2) of all such patterns. The pattern is comprised of
several service providers and manifold attribute providers. Properties from dif-
ferent attribute providers are aggregated to decrease trust into a single provider
and to ensure completeness for service consumption. The pattern most likely
reflects a realistic setup with numerous entities and different interaction paths
that are connected by a common decentralized identity provider.
The applicability of trust requirements within this pattern is a composition
of the previously described structures. The privacy trust prerequisites T1a and
T1b apply absolutely towards the attribute and the service provider from the
user’s perspective. Additionally, the user relies fully on an appropriate iden-
tity mapping by the service provider covered under T3b. Furthermore, attribute
trust requirements T4a and T4b from user and service provider towards the
attribute providers are applicable absolutely or in a limited manner depending
on the attribute aggregation strategy. Requirement T1a applies fully towards
the attribute provider. Moreover, the attribute provider completely expects that
the service provider protects the user’s privacy according to T1b.
The trust requirement T2b to securely protect the credential of the digital
identity exists with limited trust towards the user by the service provider. Due
to the holistic applicability of the digital identity, the user has a strong interest
in protecting the credential.
 A Comparative Analysis of Trust Requirements 209

Multiple Service Provider Integration Arbitrary Aggregated and Side-by-Side Integration

AP AP

AP

AP
AP

SP SP

SP SP

Fig. 2. Decentralized identity management patterns - Part 2

4.4 Comparison to Trust Requirements in Traditional Models

Traditional models comprise isolated, centralized and federated identity man-
agement. The advancement in this area promoted the comprehensive usage of
specific digital identities and fostered the reduction of digital identities with a
simultaneously increasing number of online services.

4.4.1 Isolated Identity Management

Isolated identity management denotes service-specific digital identities at a ser-
vice provider [18]. The digital identities are solely applicable at a determined
service and no overarching usage is possible. In the isolated setting, the identity
provider and the service provider are one entity [5], requiring no trust between
each other.
For the user, the trust requirements T1a, T2a, T3a, T4a and T4b are suited
with an absolute trust level towards the identity provider. Moreover, the pre-
requisites T1b and T3b apply for the user towards the service provider with an
absolute rating as well. In both cases, the user is confronted with a trusted third
party either being the identity or service provider. The trust requirements T1a,
T2a, T3a, T4a and T4b of the service provider towards the identity provider
and the prerequisite T1b in reverse direction are not applicable. In contrast, the
service provider and the identity provider absolutely require T2b towards the
user.

4.4.2 Centralized Identity Management

Centralized identity management moves from a service-specific to an
organization-specific use of digital identities [18]. Therefore, a converged digi-
tal identity can be used at several services within an organization. The service
210 A. Grüner et al.

provider and the identity provider are separate entities belonging to the same
trust domain. Thus, a limited trust level is required for mutual relationships.
Comparable to the isolated model, the service provider and the identity
provider are trusted third parties for the user and vice versa. Therefore, respec-
tive trust requirements apply absolutely. Trust requirements T1a, T2a, T3a,
T4a and T4b that exists from the service provider to the identity provider are
applicable with limited trust rating. Similarly, prerequisite T1b exists in reverse
direction.

4.4.3 Federated Identity Management

In federated identity management, digital identities are used across organiza-
tional and trust boundaries [18]. The identity provider represents a distinct
trusted third party that is independent of a service provider and its organization.
Generally, the trust requirements apply similarly to the centralized identity
management model. However, the trust requirements that exists between identity
provider and service provider in both directions apply with an absolute rating.
Additionally, the trust requirement T2b towards the user is limited in rating
because the federation of an identity provider with several service providers
increases the employability, and thus the value of the digital identity for the
user.

4.4.4 Comparison
During the evolution of traditional identity management models, starting from
isolated over centralized to the federated approach, there was no decrease in
trust between the actors. In contrast, the level of required trust for the various
prerequisites increased overall, leading to the highest trust requirements in the
federated identity model. Additionally, from a user’s perspective the most trust,
in terms of requirements and associated rating, is demanded across all traditional
identity management models. The dependency of the user is significantly higher
in the case of the service provider and identity or attribute provider than vice
versa. In contrast, the identity provider is the target of the majority of trust
requirements by having at the same time the lowest number of trust prerequisites
towards the other entities. Thus, the traditional identity management models
have a very disparate distribution of required trust between the actors with a
detrimental situation for the user.
The application of decentralized identity management based on the
blockchain, reduces the imperative trust prerequisites in general for all enti-
ties when considering the most complex arbitrary aggregated and side-by-side
integration pattern. The blockchain enables the implementation of an identity
provider to replace needed trust by the user and the service provider that is both
transparent and public verifiable. At the same time, trust in a certain attribute
can be limited by aggregation from different providers. Specifically, the following
significant differences in trust requirements exist.
 Table 1. Overview of trust requirements in traditional identity management models and decentralized patterns

T1a T1b T2a T2b T3a T3b T4a T4b

U SP IdP/AP U SP IdP/AP U SP IdP/AP U SP IdP/AP U SP IdP/AP U SP IdP/AP U SP IdP/AP U SP IdP/AP
Isolated U - - • - • - - - • - - - - - • - • - - - • - - •
SP - - - - - - - - - • - - - - - - - - - - - - - -
IdP/AP - - - - - - - - - • - - - - - - - - - - - - - -
Centralized U - - • - • - - - • - - - - - • - • - - - • - - •
SP - - · - - - - - · • - - - - · - - - - - · - - ·
IdP/AP - - - - · - - - - • - - - - - - · - - - - - - -
Federated U - - • - • - - - • - - - - - • - • - - - • - - •
SP - - • - - - - - • · - - - - • - - - - - • - - •
IdP/AP - - - - • - - - - · - - - - - - • - - - - - - -
Bilateral U - - - - • - - - - - - - - - - - • - - - • - - •
SP - - - - - - - - - • - - - - - - - - - - • - - •
IdP/AP - - - - - - - - - • - - - - - - • - - - - - - -
Mult. Aggregated U - - - - • - - - - - - - - - - - • - - - · - - ·
SP - - - - - - - - - • - - - - - - - - - - · - - ·
IdP/AP - - - - - - - - - • - - - - - - • - - - - - - -
Mult. Side-by-Side U - - - - • - - - - - - - - - - - • - - - • - - •
SP - - - - - - - - - • - - - - - - - - - - • - - •
IdP/AP - - - - - - - - - • - - - - - - • - - - - - - -
Mult. Service Provider U - - - - • - - - - - - - - - - - • - - - • - - •
SP - - - - - - - - - · - - - - - - - - - - • - - •
IdP/AP - - - - - - - - - · - - - - - - • - - - - - - -
Arb. Agg./ Side-by-Side U - - - - • - - - - - - - - - - - • - - - ·/• - - ·/•
SP - - - - - - - - - · - - - - - - - - - - ·/• - - ·/•
IdP/AP - - - - - - - - - · - - - - - - • - - - - - - -
A Comparative Analysis of Trust Requirements
211
212 A. Grüner et al.

1. Trust in credential management and authentication of the identity provider

by the user and the service provider (T2a, T3a) is remediated in the decen-
tralized identity setting.
2. User and service provider trust in attribute management (T4a, T4b) is
reduced by applying attribute aggregation for the same attribute utilizing
several attribute providers.

However, even if trust requirements are generally reduced, a disparate trust

distribution between the user and service respective to the identity or attribute
provider still exists in the decentralized identity management model.
A detailed overview of the trust requirements and related ratings in the
various models and patterns are shown in Table 1. A dash (-) reflects no trust.
A small dot (·) and large dot (•) implies limited respectively absolute trust.

5 Conclusion
We outlined decentralized identity management based on blockchain and devised
interaction patterns. Based on these patterns, we analyzed trust requirements
and compared it to the traditional models. In conclusion, a reduction of trust
towards the identity and attribute provider is a significant benefit of applying
the decentralized model based on blockchain.

References
1. Williamson, G., Yip, D., Sharoni, I., Spaulding, K.: Identity Management: A
Primer. MC Press Online, LP (2009)
2. Nakamoto, S.: Bitcoin: A Peer-to-Peer Electronic Cash System (2008). https://
bitcoin.org/bitcoin.pdf. Accessed 18 Jan 2019
3. Wood, G.: Ethereum: a secure decentralised generalised transaction ledger.
https://pdfs.semanticscholar.org/ac15/ea808ef3b17ad754f91d3a00fedc8f96b929.
pdf. Accessed 18 Jan 2019
4. Meinel, C., Gayvoronskaya, T., Schnjakin, M.: Blockchain: hype oder innovation.
Hasso-Plattner Institute, Prof.-Dr.-Helmert-Straße 2-3, 14482 Potsdam, Germany
(2018)
5. Jøsang, A., Fabre, J., Hay, B., Dalziel, J., Pope, S.L.: Trust requirements in identity
management. In: Proceedings of the 2005 Australasian Workshop on Grid Com-
puting and e-Research. ACSW Frontiers 2005, Darlinghurst, Australia, vol. 44,
pp. 99–108. Australian Computer Society, Inc., (2005). http://dl.acm.org/citation.
cfm?id=1082290.1082305
6. Kylau, U., Thomas, Y., Menzel, M., Meinel, C.: Trust requirements in identity
federation topologies. In: 2009 International Conference on Advanced Information
Networking and Applications, pp. 137–145, May 2009
7. Ferdous, M.S., Poet, R.: Analysing attribute aggregation models in federated iden-
tity management. In: Proceedings of the 6th International Conference on Security
of Information and Networks. SIN 2013, pp. 181–188. ACM (2013)
8. Jøsang, A., Ismail, R., Boyd, C.: A survey of trust and reputation systems for
online service provision. Decis. Support Syst. 43(2), 618–644 (2007)
 A Comparative Analysis of Trust Requirements 213

9. Castelfranchi, C., Falcone, R.: Trust Theory: A Socio-Cognitive and Computational

Model, 1st edn. Wiley, Hoboken (2010)
10. McKnight, D.H., Chervany, N.L.: The meanings of trust. University of Minnesota,
Technical report (1996)
11. Satariano, A.: What the G.D.P.R., Europe’s tough new data law, means for
you (2018). https://www.nytimes.com/2018/05/06/technology/gdpr-european-
privacy-law.html. Accessed 18 Jan 2019
12. Grüner, A., Mühle, A., Gayvoronskaya, T., Meinel, C.: A quantifiable trust model
for blockchain-based identity management. In: 2018 International Conference on
Blockchain, August 2018
13. Blockchain and identity (2018). https://github.com/peacekeeper/blockchain-
identity. Accessed 18 Jan 2019
14. Lundkvist, C., Heck, R., Torstensson, J., Mitton, Z., Sena, M.: uPort: a platform for
self-sovereign identity (2016). http://blockchainlab.com/pdf/uPort whitepaper
DRAFT20161020.pdf. Accessed 18 Jan 2019
15. Reed, D., Law, J., Hardman, D.: The Technical Foundations of Sovrin. A white
paper from the Sovrin foundation (2016). https://www.evernym.com/wp-content/
uploads/2017/07/The-Technical-Foundations-of-Sovrin.pdf. Accessed 18 Jan 2019
16. Sporny, M., Longley, D.: W3C community group draft report. Verifiable claims
data model and representations 1.0 (2018). https://www.w3.org/2017/05/vc-data-
model/CGFR/2017-05-01/. Accessed 18 Jan 2019
17. Allen, C.: The path to self-sovereign identity (2016). http://www.lifewithalacrity.
com/2016/04/the-path-to-self-soverereign-identity.html/cointelegraph.com/
news/first-iteration-of-ethereum-metropolis-hard-fork-to-appear-monday.
Accessed 18 Jan 2019
18. Windley, P.: Digital Identity. O’Reilly Media Inc., Newton (2005)
 Combined Methods Based Outlier Detection
for Water Pipeline in Wireless Sensor
Networks

Oussama Ghorbel1,2,4(&), Aya Ayadi1, Rami Ayadi2,

Mohammed Aseeri3, and Mohamed Abid1,4
1
National Engineers School of Sfax, Sfax University, CES-Lab, Sfax, Tunisia
[email protected], [email protected]
2
Jouf University, Sakakah, Kingdom of Saudi Arabia
{oaghorbel,rayadi}@ju.edu.sa
3
King Abdulaziz City for Science and Technology (KACST),
Riyadh, Kingdom of Saudi Arabia
[email protected]
4
Digital Research Center of Sfax, B.P. 275, Sakiet Ezzit, 3021 Sfax, Tunisia

Abstract. In the last years, Wireless Sensor Networks (WSNs) has become a
very interesting field for researchers. They are widely employed to solve several
problems in different domains like agriculture, monitoring, health care. Then,
outlier detection method is considered as a very important step in construction of
sensor network systems to ensure data grade for perfect decision making. So,
this task helps to create a gainful approach to find out if data is normal regular or
an outlier. Therefore, in this paper, a newest outlier’s detection and classification
model has been developed to complement the existing hardware redundancy and
limit checking techniques. To overcome these problems, we present a Combined
Outliers Detection Method (CODM) for water pipeline to detect damaged data
in WSNs. To this end, the application of kernel-based non-linear approach is
introduced. The main objective of this work was to combine the advantages of
Kernel Fisher Discriminant Analysis (KFDA) and Support Vector Machine
(SVM) to enhance the performance of the monitoring water pipeline system.
The accuracy of our Combined Outliers Detection Method for classification was
analyzed and compared with variety of methods. Finally, based on the experi-
mental results, our proposed work shows a better performance to detecting
outliers in the monitoring water pipeline process.

Keywords: Outlier detection method
Wireless Sensor Networks (WSNs)

Monitoring water pipeline
Kernelized techniques

1 Introduction

A wireless sensor network is composed of a large number of low-cost sensor nodes

which form a multi-hop ad hoc network through wireless communication. So, to
classify data, a powerful processing becomes significant for several applications like
outlier detection [1]. Wireless sensor networks (WSNs) are plenty used and have

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 214–224, 2020.
https://doi.org/10.1007/978-3-030-15032-7_19
 Combined Methods Based Outlier Detection for Water Pipeline in WSNs 215

gained attention in various domains including those related to industrial, business,

personal, military domains, health care and recently in monitoring water pipeline.
These applications need real-time sensor data collected by WSNs to control the sur-
rounding environment and detect time-critical events occurred in the physical world
which requires precise and accurate data to provide reliable knowledge to the end user
[2]. The advantage of using WSNs for monitoring water pipeline is that they are
cheaper and handier than wired networks. Compared with the conventional data col-
lection techniques, WSN can provide continuous measurements of physical phenomena
by means of dense deployments of sensor nodes which are uncertain and unreliable.
Hence an efficient processing and analysis of data streams become of greatest impor-
tance for many domains of research like outlier detection. To keep being able to make
effective and correct decisions by using data collected by WSNs, it is crucial to identify
potential events, as well as erroneous and malicious attacks, occurred on the network
[3]. Outlier detection methods aim to clear, refine the collected data and let providing
the most constructive information to end users. Additionally, these methods try to
maintain low energy consumption and preserve high computational efforts due to the
limited energy resources of sensor nodes [4]. However, the development of supervising
and fault detection approaches, especially in large-scale process applications, can be a
challenge. Several constraints exist such as the complex and non-linear interactions
existing between the outliers and their symptoms as well as the high correlation among
the measured variables, and the great number of sensors to be observed. In general,
caused by the high dimensionality database, the processing data would contain
redundant and irrelevant information. Consequently, the systems complexity, as well as
computational requirement, will be increased and the performance of the procedure
monitoring system will degrade. In this paper, a WSN-based real-time monitoring
process of intelligent damages detection on water pipelines has been presented.
Wireless pressure sensors have been equipped at various points of the designed
experimental model of the pipeline system. Then, pressure data have been gathered
from these sensors wirelessly at various periods. After that, these data have been
analyzed in order to detect the damages. The main focus of this paper is the application
of combined outlier detection method (CODM) for outliers detection based WSNs for
monitoring water pipeline. KFDA and KSVM have shown practical relevance for
outlier’s detection and classification for many industrial processes [5]. These tech-
niques, being nonlinear kernel technique, can efficiently model the non-linear beha-
viour in input-output data obtained from the demonstrator to diagnose faults of the
pipe. In addition, we successfully tested the efficiency of using KFDA as dimension-
ality reduction approaches combined with some classifiers including Support vector
machine (SVM); Naive Bayesian Network (NB); Neural Network (NN); Decision Tree
(DT) and K-Nearest Neighbours (KNN) to diagnose the damages in the water pipeline.
As a result, the experimental measurements of CODM demonstrate that our method is
the most efficient and accurate for detecting damages in the context of water pipeline
based on WSNs.
The organization of this article is as follows: Sect. 2, describes both wired and
wireless various monitoring techniques used. Section 3 presents the literature existing
outliers detection research in water pipeline applications. Moreover, Sect. 4 introduces
the proposed CODM framework methodology. Then, the results and the discussion of a
216 O. Ghorbel et al.

case study involving the application of our proposed technique are provided in Sect. 5.
Finally, Sect. 6 summarizes and wrap up the principal outcomes of the paper.

2 Related Works

Based on the definition proposed for outlier, this latter is a pattern that does not match
the expected trend in analyzed data. A useful information about the state of the network
acquired by a Wireless Sensor Networks can be provide correct detection of outliers
data like residual WSN lifetime, failed nodes and unexpected environmental events.
Nowadays, to monitor the cultivation process in various domains they need the
development of methods based on fundamental task. In different applications like fire
detection, leak detection, agriculture, etc., monitoring parameters like temperature is
the key parameter to prevent for fire. Then for soil moisture, is the key parameter to
determine the time to supply water and his pression [6]. To prevent forest from fire
caused by humanity and also by climate change or improper use made by man [7]
present a useful solution based on kernel PCA to classify data and detect the outlier.
Wireless Sensor Networks (WSN), thanks to their nodes equipped with several types of
sensors, help to measure temperature, soil moisture, pression, fertility, etc. [8]. To
detect and locate leaks, many leakage detection methods are developed. This latter have
been studied and developed to quickly detect damages in water pipeline. In 2013,
Mutikanga et al. [9] review the state-of-the-art of current tools and methodologies
applied to quantify, monitor, and control both real losses and apparent losses in water
distribution systems. In addition, Li et al. [10] in 2015, have divided the techniques for
damages detection and location into two main categories. The first one is based on
hardware and the second one is based on software methods.
Demirci et al. [11] have introduced the use of Ground Penetrating Radar (GPR).
This technique is useful to detect the leakage of underground pipelines and is capable
to cover some miles pipeline per day. In addition, the soil properties sensors are
currently prevalent methods [12]. The leaky pipe can be identified through the
detection of outlier value of the transported fluid properties. Thus, the choice of mote
types is correlated with the transported fluid such as soil humidity, hydrocarbon vapor
sensors, and temperature sensors Fig. 1. The supervising application based on soil
property sensors supply accurate and real-time leakage identification and localization.
Others authors have chosen to use monitoring techniques based on sensors inside
pipelines. These kinds of sensors determine many factors such as pressure, velocity as
well as the acoustic vibrations caused by the leakages in the water flow. For under-
ground pipelines, the increase of leakage is caused by the high densities of junctions.
Consequently, the sensors can be only deployed inside the pipeline at the checkpoints
or the pump stations. When damage occurs, noise will be generated as the fluid escapes
from the pipeline. The wave of the noise propagates with a speed determined by the
physical properties of the fluid in the pipeline. The acoustic detectors detect these
waves and consequently the damages. Therefore, acoustic sensors of high density are
usually installed to cover the entire pipeline network [13]. Same authors like Lee et al.
[14] have used Negative Pressure Wave. This technique has been extensively used for
identifying leaks in lengthy space with stationary operation mode. When a leak occurs,
 Combined Methods Based Outlier Detection for Water Pipeline in WSNs 217

Fig. 1. Damaged water pipeline.

a rarefaction wave is produced in the pipeline contents. The wave propagates both
upstream and downstream from the leak site. The wave travels with speed equal to the
speed of sound in the pipeline contents as in Fig. 2.

Fig. 2. Detection damage by means of negative pressure wave

Others techniques based on fluid transients for burst detection have gained popu-
larity over the last decade, for example, inverse transient analysis [15]. Covas et al. [16]
evaluate the presence, location, and size of leaks carried out using collected data. This
technique allowed for the identification of the approximated location of leaks as long as
the physical characteristics of the pipeline are well known such as the transient gen-
erated by a fast change of flow conditions.
Each leak detection method has its advantages and disadvantages. The performance
of each method varies considerably depending on the position of sensors, sensibility
(ability to detect small leak), availability (ability to maintain a continuous 24 h of
monitoring), false alarm (the technique that has the low false alarm is the best) and
maintenance requirements (level of technical expertise required to maintain the system).
218 O. Ghorbel et al.

3 Outlier Detection in Wireless Sensor Networks

To find missing values, inconsistent data, errors, noise or duplicate data, outlier
detection method is used. These abnormal values reduce the performance of the system
and affect the quality of data. In WSN, three sources of outliers are considered: errors,
events and malicious attacks. [17]. Outlier detection technique is very important in use
in several real life applications, such as: surveillance monitors, precision agriculture,
environmental monitoring, health and medical monitoring, fire monitoring, industrial
monitoring, and target tracking [18] as shown in Fig. 3.

Fig. 3. Wireless sensor networks: different application areas

Outliers detection is a long-studied problem in data analysis, and its treatment has
been thoroughly described in several studies. An outlier is an observation which
appears to deviate markedly from some sources of the sample in which it occurs [19].
In the literature, outliers in WSNs are classified into three different types untitled Error,
Event and Malicious attacks.
In the context of water pipeline, abnormal data detected from the normal pressure
profile can be a result of damages [20]. These pipelines are often subject to failure like
erosion and sabotage that can cause high financial, environmental and health risks.
Consequently, detecting damage and esteeming its location is very important. Moni-
toring water pipeline applications are a problem that data mining can address. For this
case, several techniques have been investigated in the research community. We have
classified outlier detection techniques designed for WSNs based on the disciplines like:
• Prediction based approaches: These methods are used to identify areas and pipes
with a high probability of damage.
• Classification-based approaches: the classifier is trained using known training data
patterns and used to classify the unknown patterns into one or more types.
 Combined Methods Based Outlier Detection for Water Pipeline in WSNs 219

4 Data Reduction Techniques

We are aiming to develop a suitable technique to detect damages in water pipeline

based in WSNs by means of pressure measurements. Our technique is a combination
between data reduction technique and classifier to improve the accuracy of the
detection process. Our Combined Outliers Detection Method (CODM) is a hybrid
model of Kernel Fisher Discriminant Analysis (KFDA) and Support Vector Machine
(SVM) classifier based on WSNs data to ensure the safety of a transport pipeline
process. Our study gives an adequate guideline for learning from the environment and
predicts future behavior, which leads to reduce the amount of transmitted data and the
sending of only important information. In addition, we are able to detect important
events, such as leak, and classify the data as normal or outliers. After collecting the
sensing data, we reduce data with high dimensions into data with lower dimensions, to
more easily find patterns in the data without much loss of information. This reduced
data is then fed into to classification algorithms to identify outliers and verify the
network behaviors. We note that an important event may be happened, such as a burst,
leak or defected sensors.
We present above the mathematical procedure involved in computation of KFDA
and SVM.

4.1 Kernel Fisher Discriminant Analysis Technique (KFDA)

Different techniques are used to remove irrelevant, redundant and noisy features in the
high dimensional data sets. In our work, data reduction techniques are explored to
improve the accuracy of detection framework. We have used an algorithms technique
known as Kernel Fisher Discriminant Analysis (KFDA) [21] which is investigated in
our work. Fisher discriminant analysis (FDA) is a data reduction technique and has
been applied for fault detection and classification of many industrial processes. It
projects online data obtained from the process under-consideration into a feature space
in such a way that the separation between same-class data (within-class scatter) is
minimized whereas separation between data of different classes (between-class scatter)
is maximized. However, FDA cannot efficiently classify a data set which has a non-
linear behavior. To overcome this limitation, a nonlinear version of FDA, called kernel
Fisher Discriminant Analysis (KFDA) has been presented in literature. The basic idea
of KFDA is simple, a data set with nonlinear behavior is projected into a higher
dimensional feature space using a nonlinear mapping, and then a linear classification
technique is applied in the feature space. However, performing computations in
implicit feature space is cumbersome as it can have large or even infinite dimensions.
To avoid this difficulty, kernel trick is used to compute dot products of feature space
vectors. The overall procedure to apply KFDA algorithm consists of three main steps;
firstly, offline computation of KFDA vectors utilizing the training data, secondly, the
projection of online data using the computed KFDA vectors, and thirdly, assigning a
class to new online data. These steps are in higher dimensional feature space.
To understand mathematical procedure involved in computation of KFDA vectors,
consider a training data set obtained from process to be monitored stacked in a matrix
X in a particular format as described below:
220 O. Ghorbel et al.

X ¼ ½X1 ; X2 ; . . .Xp T

where p is the total number of classes of data set, which equals one more than the
number of possible faults in the process, Xj represents the training data associated with
the jth class. Each Xj has nj measurements of s process sensors.
2 3
xj
6 1j 7
6 x2 7
Xj ¼ 6
6 ..
7; j ¼ 1; 2; . . .; p
7
4. 5
Xnjj

x1j ¼ ½xi1j ; xi2j ; . . .; xisj 

The original data set X is mapped into higher dimensional feature space through
non-linear function U,

U : xi 2 Rm ! Uðxi Þ 2 Fh ; h [ m

4.2 Support Vectors Machine Technique (SVM)

Support Vector Machines (SVM) is one of the best data classification techniques due to
the optimum selection of decision boundary and less dependence on curse of dimen-
sionality. SVM separates different class of data correctly using optimum separating
plane called hyperplane that maximizes the margin. The overall procedure to apply
SVM for detection outlier consists of three steps. Firstly, computation of optimum
decision boundary by using training data set. Secondly, computation of linear dis-
criminative function of online test data using the modeled weighting vectors and bias.
Thirdly, the assigning of the test data to corresponding class label.
Suppose the training data set X : ðXi ; Yi Þ for all i = 1, …, p of a process, where Yi is
the ith class label. We draw a decision boundary that would separate various class of
data. Any point Xi which lies on the hyper-plane must satisfy the condition:

W T Xi þ b ¼ 0

where w is called weighting vector which is normal to the hyper-plane and b, the bias,
tells position of hyper-plane relative to origin. The distance of the plane from the origin
is kWb k. If we assume that all observations are, at least, one unit apart from the decision
boundary then we have the following condition:

wT xi þ b  1; if yi ¼ 1

wT xi þ b   1; if yi ¼ 1
 Combined Methods Based Outlier Detection for Water Pipeline in WSNs 221

5 Experimental Results

A practical test bed for experimentation has been deployed on Digital Research Center
Technopark Sfax (CRNS) [22]. The laboratory demonstrator in water pipeline is a
rectangular fragment composed of 25 m pipes which are set up as shown in Fig. 4.

Fig. 4. Water pipeline demonstrator.

These pipes have 32 mm as an external pipe. The water is moving along the pipes
using an electrical pump with 1 hp (Horse Power) motor providing up to 6 bar when
the output valve is closed and up to 2.5 bar in open circuit. The supports are designed
to have variable heights that we will explore in the future to see the effect of this
variation on the pressure and to test our algorithm in varied conditions. Using real data
sets, the performance can be improved further. The data samples were collected from
an exploitation of WSN in static and dynamic environments to identify damage in
water pipeline.
The experimentation was performed based on a set of pressure and the pressure
variation curve. In the experiment, we chose the last 345 samples of each class for
monitoring water pipeline aptitudes. In addition, the samples of each group were
labeled accordingly. To calculate the precision of different techniques, we calculated
the accuracy that means how many outliers are correctly detected. The effectiveness
outlier detection technique should reach a high accuracy.
To apply our method named CODM for classification, training data is obtained
from the sensors available in the pipeline. The variables that are considered for the
implementation of kernel approaches are pressure measurements. This dataset is called
fault free class data and the rest are declared as outliers, damages in pipe. The idea
behind nonlinear classification techniques is to map input data, using some nonlinear
mapping, into some high dimensional feature space and then perform linear techniques
in this new feature space. However, performing computations in implicit feature space
is cumbersome because it can have large or even infinite dimensions. To avoid it,
kernel trick is utilized which eliminates the need to compute feature space explicitly.
222 O. Ghorbel et al.

Instead, a kernel function is used to compute only dot products of feature space vectors,
thus reducing the computational cost. It is obvious from [23] that tested data is clas-
sified very efficiently by a varying degree of precision. The performance of CODM for
detection damages in water pipelines is measured by its accuracy rate [24]. Three kinds
of kernel are used to define the best results. For KFDA implementation, polynomial
kernel function of degree five is selected which reaches 0.8269 of accuracy versus
0.8227 for a Gaussian kernel, with sigma equal to 4, and 0.7392 for Linear kernel
transformation. Whereas, these results remain much lower than that found with our
technique which reaches 0.9841 as shown in Fig. 5.

Fig. 5. Evaluation of accuracy by using CODM.

To summarize, the different methods, which aimed to find the optimal transfor-
mation by minimizing the within-class distance and maximizing the between-class
distance, had simultaneously contributed towards improving damages detection. In
addition, using Kernel technique as KFDA like a reduction dimensionality technique is
beneficial and improves the classification. However, the CODM classification tech-
nique had produced averagely higher classification accuracy compared to other net-
work combination approaches.

6 Conclusion

In this work, we presented a useful solution used to detects outlier for water pipeline in
wireless sensor networks untitled CODM. However, pipeline damages result in
tremendous economic losses and pose serious threats to personal safety and environ-
mental pollution. For that, damages in water pipeline systems must be rapidly detected
to minimize the loss of system functionality and recovery time. Although the proposed
 Combined Methods Based Outlier Detection for Water Pipeline in WSNs 223

CODM framework can successfully detect all types of damages in a pipe, with a high
degree of accuracy (average accuracy equal to 98%). Furthermore, by comparing the
performances and effectiveness of our technique, this study has shown that our clas-
sifier was better than the others approaches for outliers detection in the context of water
pipeline monitoring. However, localizing the exact point on the pipe where higher
background damage outflow occurs and the possible position of pressure control ele-
ments along such critical pipes will be addressed in future work.

References
1. Braun, M.L., Buhmann, J.M., Muller, K.R.: On relevant dimensions in kernel feature spaces.
J. Mach. Learn. Res. 9, 1875–1908 (2008)
2. Naumowicz, T., Freeman, R., Heil, A., Calsyn, M., Hellmich, E., Brandle, A., Guilford, T.,
Schiller, J.: Autonomous monitoring of vulnerable habitats using a wireless sensor network.
In: Proceedings of the Workshop on Real-World Wireless Sensor Networks, REALWSN
2008, Glasgow, Scotland (2008)
3. Akyildiz, I.F., Melodia, T., Chowdhury, R.: A survey on wireless multimedia sensor
networks. J. Comput. Netw.: Int. J. Comput. Telecommun. Netw. 51(4), 921–960 (2007)
4. Zhang, Y., Meratnia, N., Havinga, P.: Outlier detection techniques for wireless sensor
networks: a survey. IEEE Commun. Surv. Tutor. 12, 159–170 (2010)
5. Li, J., Cui, P.: Improved kernel fisher discriminant analysis for fault diagnosis. Expert Syst.
Appl. 36(2), 1423–1432 (2009)
6. Fodor, I.K.: A survey of dimension reduction techniques. Lawrence Livermore National
Laboratory, US Department of Energy (2002)
7. Ghorbel, O., Abid, M., Snoussi, H.: Improved KPCA for outlier detection in wireless sensor
networks. In: 1st International Conference on Advanced Technologies for Signal and Image
Processing (ATSIP), pp. 507–511 (2014)
8. Nakanishi, T.: A generative wireless sensor network framework for agricultural use. In:
Makassar International Conference on Electrical Engineering and Informatics (MICEEI),
pp. 205–211 (2014)
9. Mutikanga, H.E., Sharma, S.K., Vairavamoorthy, K.: Methods and tools for managing losses
in water distribution systems. J. Water Resour. Plan. Manag. 139(2), 166–174 (2013)
10. Weinberger, K.Q., Sha, F., Saul, L.K.: Learning a kernel matrix for nonlinear dimensionality
reduction. In: Li, R., Huang, H., Xin, K., Tao, T. (eds.) Proceedings of the 21st (2015).
A review of methods for burst/leakage detection and location in water distribution systems.
Water Sci. Technol.: Water Supply 15(3), 429–441 (2015)
11. Demirci, S., Yigit, E., Eskidemir, I.H., Ozdemir, C.: Ground penetrating radar imaging of
water leaks from buried pipes based on back-projection method. NDT and E Int. 47, 35–42
(2012)
12. Zheng, L., Kleiner, Y.: State of the art review of inspection technologies for condition
assessment of water pipes. Measurement 46(1), 1–15 (2013)
13. Colombo, A.F., Lee, P., Karney, B.W.: A selective literature review of transient-based leak
detection methods. J. Hydroenviron. Res. 2(April), 212–277 (2009)
14. Lee, S.J., et al.: Online burst detection and location of water distribution systems and its
practical applications. J. Water Resour. Planning Manag. (2016)
15. Martins, J.C., Seleghim Jr., P.: Assessment of the performance of acoustic and mass balance
methods for leak detection in pipelines for transporting liquids. J. Fluids Eng. 132(January),
011401 (2010)
224 O. Ghorbel et al.

16. Covas, D., Ramos, H.: Case studies of leak detection and location in water pipe systems by
inverse transient analysis. J. Water Res. Plann. Manage. 136(2), 248–257 (2010)
17. Szewczyk, R., Mainwaring, A., Polastre, J., Anderson, J., Culler, D.: Analysis of a large
scale habitet monitoring application. In: Proceedings of the Second ACM Conference en
Embedded Networked Sensors Systems (SenSys), Baltimore (2004)
18. Barrenetxea, G., Ingelrest, F., Schaefer, G., Vetterli, M., Couach, O., Parlange, M.:
SensorScope: out-of-the-box environmental monitoring. In: Proceeding of the 7th Interna-
tional Conference on Information Processing in Sensor Networks, 22–24 April, pp. 332–343
(2008)
19. Ayadi, A., Ghorbel, O., Obeid, A.M., Abid, M.: Outlier detection approaches for wireless
sensor networks: a survey. Comput. Netw. 129, 319–333 (2017)
20. Ayadi, A., Ghorbel, O., Bensaleh, M.S., Abid, M.: Outlier detection based on data reduction
in WSNs for water pipeline. In: 2017 25th International Conference on Software,
Telecommunications and Computer Networks (SoftCOM), pp. 1–6. IEEE (2017)
21. Breiman, L.: Classication and Regression Trees. Routledge, New York (2017)
22. Karray, F., Garcia-Ortiz, A., Jmal, M.W., Obeid, A.M., Abid, M.: EARNPIPE: a testbed for
smart water pipeline monitoring using wireless sensor network. Procedia Comput. Sci. 96,
285–294 (2016)
23. Srirangarajan, S., Allen, M., Preis, A., Iqbal, M., Lim, H.B., Whittle, A.: Wavelet-based
burst event detection and localization in water distribution systems. J. Signal Process. Syst.
72(1), 1–16 (2013)
24. Kayaalp, F., Zengin, A., Kara, R., Zavrak, S.: Leakage detection and localization on water
transportation pipelines: a multi-label classification approach. Neural Comput. Appl. 28(10),
2905–2914 (2017)
 The Improved Energy-Efficient Quorum
Selection with Excluding Meaningless
Methods

Tomoya Enokido1(B) , Dilawaer Duolikun2 , and Makoto Takizawa2

1
Faculty of Business Administration, Rissho University, Tokyo, Japan
[email protected]
2
Department of Advanced Sciences, Faculty of Science and Engineering,
Hosei University, Tokyo, Japan
[email protected], [email protected]

Abstract. In distributed object-based systems, applications manipu-

late objects distributed on multiple servers. In order to provide reliable
and available application services, each object is replicated on multi-
ple servers and replicas of each object have to be mutually consistent.
On the other hand, since methods issued to each object are performed
on multiple replicas, the total amount of electric energy consumed in a
system increases. In our previous studies, the IEEQS (improved energy
efficient quorum selection) algorithm is proposed to reduce the total elec-
tric energy of servers by omitting only meaningless write methods. In this
paper, the IEEQS-EMM (IEEQS with excluding meaningless methods)
algorithm is proposed to furthermore reduce the total electric energy
of servers by omitting meaningless read and write methods. Evaluation
results show the total electric energy of servers can be reduced in the
IEEQS-EMM algorithm than the IEEQS algorithm.

Keywords: Distributed object-based systems · Locking protocols ·

Data management · Green computing · IEEQS-EMM algorithm

1 Introduction

Distributed applications are composed of multiple objects [1]. Each object is a

unit of computation resource like a database server and is an encapsulation of
data and methods to manipulate the data in the object. Each object is replicated
on multiple servers in order to increase reliability, availability, and performance
of application services [2]. A transaction is an atomic sequence of methods on
objects [3]. A transaction is initiated on a client and issues methods to manip-
ulate objects. Conflicting methods issued by multiple transactions have to be
serialized [4] to keep the replicas of each object mutually consistent. In order to
keep the replicas mutually consistent, one of the replicas of an object for a read
method and all the replicas for a write method are locked before manipulating

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 225–237, 2020.
https://doi.org/10.1007/978-3-030-15032-7_20
226 T. Enokido et al.

the object in the two-phase locking (2PL) protocol [3]. However, the 2PL proto-
col is not efficient in write-dominated applications since every replica has to be
locked for each write method. In order to reduce the overhead to perform write
methods on replicas, the quorum-based locking protocol [5] is proposed. Subsets
of replicas locked for read and write methods are referred to as read and write
quorums, respectively. The quorum numbers nQr and nQw for read and write
methods, respectively, have to be “nQr +nQw > N ” where N is the total number
of replicas. In the quorum-based protocols, the more number of write methods
are issued, the fewer write quorum number nQw can be taken. As a result, the
overhead to perform write methods can be reduced. While the reliability, avail-
ability, and performance can be increased in the replication schemes, the larger
amount of electric energy is consumed in a system than non-replication systems
since each method issued to an object is performed on multiple replicas.
In our previous studies, meaningless write methods [7] which are not required
to be performed on each replica of an object are defined based on the precedent
relation and semantics of methods. Then, the improved energy efficient quorum
selection (IEEQS ) algorithm [7] is proposed to reduce the total electric energy
of a server cluster by omitting meaningless write methods on each replica. In
this paper, we first define meaningless read methods which are not required to
be performed on each replica of an object. Next, the IEEQS-EMM (IEEQS with
excluding meaningless methods) algorithm is proposed to furthermore reduce the
total electric energy of servers by omitting meaningless read and write methods.
We evaluate the IEEQS-EMM algorithm compared with the IEEQS algorithm.
The evaluation results show the total electric energy of a server cluster, the aver-
age execution time of each transaction, and the number of aborted transactions
in the IEEQS-EMM algorithm can be more reduced than the IEEQS algorithm.
In Sect. 2, we discuss the system model. In Sect. 3, we discuss the IEEQS-
EMM algorithm. In Sect. 4, we evaluate the IEEQS-EMM algorithm compared
with the IEEQS algorithm.

2 System Model
2.1 Object-Based Systems
A cluster S of servers is composed of multiple servers s1 , . . . , sn (n ≥ 1). Let O
be a set of objects o1 , . . . , om (m ≥ 1) [1]. Each object oh is a unit of computation
resource and is an encapsulation of data and methods to manipulate the data in
the object oh . Methods are classified into read (r) and write (w) methods. In
this paper, data in an object is fully read by a read method. Write methods are
furthermore classified in to partial write (wp ) and full write (wf ) methods, i.e.
w ∈ {wf , wp }. In a partial write method, a part of data in an object is written
while a whole data is fully written in a full write method. Let op(oh ) be a state
obtained by performing a method op on an object oh . A composition op1 ◦ op2
means that a method op1 is performed prior to op2 . A pair of methods op1 and
op2 on an object oh are compatible if and only if (iff) op1 ◦ op2 (oh ) = op2 ◦ op1 (oh )
for every state of oh . Otherwise, a method op1 conflicts with another method op2 .
 The IEEQS-EMM Algorithm 227

The conflicting relation is symmetric. A pair of read methods are compatible.

On the other hand, a read method conflicts with a write method and a write
method conflicts with another write method.
Each object oh is replicated on multiple servers to make a system more
reliable and available. Let R(oh ) be a set of replicas o1h , . . . , olh (1 ≤ l ≤ n) [2]
of an object oh . Let nR(oh ) be the total number of replicas of an object oh , i.e.
nR(oh ) = |R(oh )|. Let Sh be a subset of servers which hold a replica of an object
oh (Sh ⊆ S).

2.2 Quorum-Based Locking Protocol

A transaction is an atomic sequence of methods [3]. A transaction Ti issues r and
w methods to manipulate replicas of objects. Multiple conflicting transactions
are required to be serializable [3,4] to keep replicas of each object mutually
consistent. Let T be a set of {T1 , . . . , Tk }(k ≥ 1) of transactions. Let H be a
schedule of the transactions in T. A transaction Ti precedes another transaction
Tj (Ti →H Tj ) in a schedule H iff a method opi from the transaction Ti is
performed before a method opj from the transaction Tj and opi conflicts with
opj . A schedule H is serializable iff the precedent relation →H is acyclic [3].
In this paper, multiple conflicting transactions are serialized based on the
quorum-based locking protocol [5]. Let μ(op) be a lock mode of a method op
(∈ {r, w}). If op1 is compatible with op2 on an object oh , the lock mode μ(op1 )
is compatible with μ(op2 ). Otherwise, a lock mode μ(op1 ) conflicts with another
lock mode μ(op2 ). Let Qoph (⊆ R(oh )) be a subset of replicas of an object oh to
be locked by a method op (op ∈ {r, w}), named a quorum of the method op. Let
nQop op
h be the quorum number of a method op on a object oh , i.e. nQh = |Qh |.
op
r w
The quorums have to satisfy the following constraints: (1) Qh ⊆ R(oh ), Qh ⊆
R(oh ), and Qrh ∪ Qw r w r w
h = R(oh ). (2) nQh + nQh > nR(oh ), i.e. Qh ∩ Qh
= φ. (3)
w
nQh > nR(oh )/2.
[Quorum-Based Locking Protocol] A transaction Ti locks replicas of an
object oh by the following procedure [5]:

1. A quorum Qop op
h for a method op is constructed by selecting nQh replicas in
a set R(oh ) of replicas.
2. If every replica in a quorum Qoph can be locked by a lock mode μ(op), the
replicas in the quorum Qoph are manipulated by the method op.
3. When the transaction Ti commits or aborts, the locks on the replicas in the
quorum Qop h are released.

Suppose there are a pair of replicas o1h and o2h of an object oh . If the replica
o1h is updated but o2h is not updated, o1h is newer than o2h . In the quorum-based
locking protocol, every quorum surely includes at least one newest replica. Each
replica oqh has a version number vhq . Suppose a transaction Ti reads an object oh .
The transaction Ti selects nQrh replicas in the set R(oh ), i.e. read (r) quorum
Qrh . If every replica in the r-quorum Qrh can be locked by a lock mode μ(r),
the transaction Ti reads data in a replica oqh whose version number vhq is the
228 T. Enokido et al.

maximum in the r-quorum Qrh . Next, suppose a transaction Ti writes data in an

object oh . The transaction Ti selects nQw h replicas in the set R(oh ), i.e. write
(w) quorum Qw h . If every replica in the w-quorum Qwh can be locked by a lock
mode μ(w), the transaction Ti writes data in a replica oqh whose version number
vhq is maximum in the w-quorum Qw q
h and the version number vh of the replica oh
q
q
is incremented by one. The updated data and version number vh of the replica
oqh are sent to every other replica in the w-quorum Qw h . Then, data and version
number of each replica in the w-quorum Qw h are replaced with the newest values.

2.3 Data Access Model

Methods which are being performed are current at time τ . Let RPt (τ ) and
W Pt (τ ) be sets of current read (r) and write (w) methods on a server st at
time τ , respectively. Let Pt (τ ) be a set of current r and w methods on a server
st at time τ , i.e. Pt (τ ) = RPt (τ ) ∪ W Pt (τ ). Let rti (oqh ) and wti (oqh ) be methods
issued by a transaction Ti to read and write data in a replica oqh on a server st ,
respectively. By each method rti (oqh ) in a set RPt (τ ), data is read in a replica oqh
at rate RRti (τ ) [B/sec] at time τ . By each method wti (oqh ) in a set W Pt (τ ), data
is written in a replica oqh at rate W Rti (τ ) [B/sec] at time τ . Let maxRRt and
maxW Rt be the maximum read and write rates [B/sec] on a server st , respec-
tively. The read rate RRti (τ )(≤ maxRRt ) and write rate W Rti (τ )(≤ maxW Rt )
are f rt (τ ) · maxRRt and f wt (τ ) · maxW Rt , respectively. Here, f rt (τ ) and
f wt (τ ) are degradation ratios. 0 ≤ f rt (τ ) ≤ 1 and 0 ≤ f wt (τ ) ≤ 1. The
degradation ratios f rt (τ ) and f wt (τ ) are 1/(|RPt (τ )| + rwt · |W Pt (τ )|) and
1/(wrt · |RPt (τ )| + |W Pt (τ )|), respectively. 0 ≤ rwt ≤ 1 and 0 ≤ wrt ≤ 1.
The read laxity lrti (τ ) [B] and write laxity lwti (τ ) [B] of methods rti (oqh )
and wti (oqh ) show how much amount of data are read and written in a replica
oqh by the methods rti (oqh ) and wti (oqh ) at time τ , respectively. Suppose that
methods rti (oqh ) and wti (oqh ) start on a server st at time stti , respectively. At
time stti , the read laxity lrti (τ ) = rbqh [B] where rbqh is the size of data in a
replica oqh . The write laxity lwti (τ ) = wbqh [B] where wbqh is the size of data to be
written in a replica oqh . The read laxity lrti (τ ) and write laxity lwti (τ ) at time
τ are rbqh − Σττ =stti RRti (τ ) and wbqh − Σττ =stti W Rti (τ ), respectively.

2.4 Power Consumption Model of a Server

Let Et (τ ) be the electric power [W] consumed by a server st at time τ . maxEt

and minEt show the maximum and minimum electric power [W] of the server st ,
respectively. The power consumption model for a storage server (P CS model)
[6] to perform storage and computation processes are proposed. According to
the PCS model, the electric power Et (τ ) [W] of a server st to perform multiple
r and w methods at time τ is given as follows:
 The IEEQS-EMM Algorithm 229
⎧
⎪
⎪W Et if |W Pt (τ )| ≥ 1 and |RPt (τ )| = 0.
⎪
⎨W RE (α)
t if |W Pt (τ )| ≥ 1 and |RPt (τ )| ≥ 1.
Et (τ ) = (1)
⎪
⎪ REt if |W Pt (τ )| = 0 and |RPt (τ )| ≥ 1.
⎪
⎩
minEt if |W Pt (τ )| = |RPt (τ )| = 0.

A server st consumes the minimum electric power minEt [W] if no method

is performed on the server st , i.e. the electric power in the idle state. The
server st consumes the electric power REt [W] if at least one r method is
performed on the server st . The server st consumes the electric power W Et
[W] if at least one w method is performed on the server st . The server st con-
sumes the electric power W REt (α) [W] = α · REt + (1 − α) · W Et [W] where
α = |RPt (τ )|/(|RPt (τ )| + |W Pt (τ )|) if both at least one r method and at least
one w method are concurrently performed. Here, minEt ≤ REt ≤ W REt (α) ≤
W Et ≤ maxEt .
The processing power P Et (τ ) [W] of a server st at time τ is Et (τ ) − minEt .
The total processing electric energy T P Et (τ1 , τ2 ) [J] of a server st from time τ1 to
τ2 is given as T P Et (τ1 , τ2 ) = Σττ2=τ 1 P Et (τ ). The total processing electric energy
laxity tpeclt (τ ) shows how much electric energy a server st has to consume to
perform every current r and w methods on the server st at time τ . The total
processing energy consumption laxity tpeclt (τ ) of a server st at time τ is obtained
by the following procedure T P ECLt (τ ) [7]:
T P ECLt (τ ) {
if RPt (τ ) = φ and W Pt (τ ) = φ, return(0);
laxity = Et (τ ) - minEt ; /* P Et (τ ) of a server st at time τ */
for each r-method rti (oqh ) in RPt (τ ), {
lrti (τ + 1) = lrti (τ ) - RRti ;
if lrti (τ + 1) = 0, RPt (τ + 1) = RPt (τ ) - {rti (oqh )};
} /* for end */
for each w-method wti (oqh ) in W Pt (τ ), {
lwti (τ + 1) = lwti (τ ) - W Rti ;
if lwti (τ + 1) = 0, W Pt (τ + 1) = W Pt (τ ) - {wit (oqh )};
} /* for end */
return(laxity + T P ECLt (τ + 1));
}
In the TPECLt procedure, if the read laxity lrti (τ + 1) and write laxity
lwti (τ + 1) get 0, every data is read and written in the replica oqh by the methods
rti (oqh ) and wti (oqh ), respectively, and the methods terminate at time τ .

3 The IEEQS-EMM Algorithm

3.1 Quorum Selection

In the IEEQS-EMM (improved energy-efficient quorum selection with excluding

meaningless methods) algorithm, replicas to be members of a quorum of each
230 T. Enokido et al.

method are selected so that the total electric energy of a server cluster S to
perform the method is the minimum. Suppose a transaction Ti issues a method
op (op = {r, w}) to manipulate an object oh at time τ . For each transaction Ti ,
a subset Shop (⊆ Sh ) of nQop
h servers is selected, whose total processing electric
energy laxity is the minimum by following quorum selection (QS) procedure [7]:
QS(op, oh , τ ) { /* op ∈ {r, w} */
Shop = φ;
while (nQop h > 0) {
for each server st in Sh , {
if op = r, RPt (τ ) = RPt (τ ) ∪ {op};
else W Pt (τ ) = W Pt (τ ) ∪ {op}; /* op = w */
T P Et (τ ) = T P ECLt (τ );
} /* for end */
server = a server st where T P Et (τ ) is the minimum;
Shop = Shop ∪ {server}; Sh = Sh - {server}; nQop op
h = nQh - 1;
} /* while end */
return(Shop );
}

3.2 IEEQS-EMM Algorithm

A method op1 precedes op2 in a schedule H (op1 →H op2 ) in a schedule H iff
(1) the methods op1 and op2 are issued by the same transaction Ti and op1 is
issued before op2 , (2) the method op1 issued by a transaction Ti conflicts with
the method op2 issued by a transaction Tj and Ti →H Tj , or (3) op1 →H op3
→H op2 for some method op3 . Let Hh be a local schedule of methods which are
performed on an object oh in a schedule H.
[Definition] A method op1 locally precedes another method op2 in a local sched-
ule Hh (op1 →Hh op2 ) iff op1 →H op2 .
[Definition]

– A full write method op1 absorbs another partial or full write method op2 in
a local subschedule Hh on an object oh if op2 →Hh op1 , and there is no read
method op such that op2 →Hh op →Hh op1 , or op1 absorbs op and op
absorbs op2 for some method op .
– A read method op1 absorbs another read method op2 in a local subschedule
Hh on an object oh if op1 →Hh op2 , and there is no write method op such
that op1 →Hh op →Hh op2 , or op1 absorbs op and op absorbs op2 for some
method op .

[Definition] A method op is meaningless iff the method op is absorbed by

another method op in the local subschedule Hh of an object oh .
Suppose three replicas o11 , o21 , and o31 of an object o1 are stored in three servers
s1 , s2 , and s3 , respectively. The version numbers v11 , v12 , and v13 of replicas o11 ,
o21 , and o31 are 2, 1, and 2, respectively, as shown in Figs. 1 and 2. The quorum
op
numbers nQw r
1 and nQ1 for the object o1 are the same two. Let Ti .Qh be a
 The IEEQS-EMM Algorithm 231

quorum to perform a method op issued by a transaction Ti . Let Ti .Shop be a

subset of servers which hold replicas in a quorum Ti .Qop h . Suppose a pair of
replicas o11 and o21 are locked by a transaction T1 with lock mode μ(r) and a
r (o )
read method r1 (o1 ) is issued to a r-quorum T1 .Q11 1 = {o11 , o21 } as shown in
Fig. 1. The read method r11 (o1 ) is performed on a replica o11 since the version
1
r (o )
number v11 (= 2) is the maximum in the r-quorum T1 .Q11 1 . Here, another
1 2
transaction T2 locks a pair of replicas o1 and o1 with lock mode μ(r) and a read
r (o )
method r1 (o1 ) is issued to a r-quorum T2 .Q12 1 = {o11 , o21 }. The read method
r12 (o11 ) is performed on the replica o11 since the version number v11 (= 2) is the
r (o )
maximum in the r-quorum T2 .Q12 1 , i.e. a pair of read methods r11 (o11 ) and
r12 (o1 ) are concurrently performed on a replica o11 and r11 (o11 ) →H1 r12 (o11 ).
1

Here, the read method r12 (o11 ) issued by the transaction T2 is meaningless since
the read method r11 (o11 ) issued by the transaction T1 is being performed on the
replica o11 and r11 (o11 ) absorbs the read method r12 (o11 ). Hence, the read method
r12 (o11 ) is not performed on a replica o11 and a result obtained by performing the
read method r11 (o11 ) is sent to a pair of transactions T1 and T2 . That is, the
meaningless read method r12 (o11 ) is omitted on the replica o11 .

1 2 3
o11 o21 o31 o1 o1 o1
1 2 3 1 2 3
T1 v =2
1 v =1
1 v =2
1
v1= 2 v1= 1 v1= 2
T2 T1 p 1
r11(o11) w (o )
11 1

1 2
v1= 3 v1= 3
1 version number is replaced
r12(o ) is omitted.
1 but data is not replaced.

2 3
1 T2 v1= 3 v1= 2
a result of r11(o1) is sent to f 2
w22(o1)
a pair of T1 and T2 .
2 3
v1= 4 v=4 1
version number is replaced
but data is not replaced.

time time

Fig. 1. a meaningless read method. Fig. 2. a meaningless write method.

Suppose a pair of replicas o11 and o21 are locked by a transaction T1 with
lock mode μ(w) and a partial write methods w1p (o1 ) is issued to a w-quorum
wp (o ) p
T1 .Q1 1 1 = {o11 , o21 } as shown in Fig. 2. The partial write method w11 (o11 )
is performed on the replica o11 since the version number v11 of the replica o11
wp (o )
is the maximum in the w-quorum T1 .Q1 1 1 . Then, the version number v11 is
incremented by one, i.e. v11 = 3. The updated data and version number v11 are
sent to the replica o21 . In the IEEQS-EMM algorithm, the version number v12 of
the replica o21 is replaced with the newest value but data of the replica o21 is not
replaced with the newest values until the next method is performed on the replica
p
o21 . This means that the partial write method w21 (o21 ) to replace data of a replica
o1 is delayed until the next method is performed on the replica o21 . Suppose a pair
2
232 T. Enokido et al.

of replicas o21 and o31 are locked by a transaction T2 with lock mode μ(w) and a
wp (o )
full write methods w2f (o1 ) is issued to a w-quorum T2 .Q1 2 1 = {o21 , o31 } after the
f
transaction T1 commits. The full write method w22 (o21 ) issued by the transaction
T2 is performed on the replica o1 since the version number v12 is the maximum
2
wp (o ) p
in the w-quorum T2 .Q1 2 1 . The partial write method w21 (o21 ) issued by the
f
transaction T1 is meaningless since the full write method w22 (o21 ) absorbs the
p f
partial write method w21 (o21 ) on the replica o21 . The full write method w22 (o21 )
2
is performed on the replica o1 without performing the partial write method
p
w21 (o21 ) and the version number of the replica o21 is incremented by one. That
p
is, the meaningless write method w21 (o21 ) is omitted, i.e. not performed on the
2
replica o1 .
Suppose a pair of replicas o21 and o31 are locked by a transaction T3 with lock
r (o )
mode μ(r) and a read method r3 (o1 ) is issued to a r-quorum T3 .Q13 1 = {o21 , o31 }
after the transaction T1 commits in Fig. 2. Here, the transaction T3 reads data
in the replica o21 since the version number v12 is the maximum in the r-quorum
r (o ) p
T3 .Q13 1 . Here, the partial write method w21 (o21 ) issued by a transaction T1
has to be performed before the read method r23 (o21 ) is performed since the read
p
method r23 (o21 ) has to read data written by the partial write method w21 (o21 ).
q q
Let oh .CR be a read method rti (oh ) issued by a transaction Ti , which is being
performed on a replica oqh in a server st . Let oqh .DW be a write method wti (oqh )
issued by a transaction Ti to replace data of a replica oqh in a server st with the
updated data dh , which is waiting for the next method op to be performed on
the replica oqh . Suppose a transaction Ti issues a method op to a quorum Qop h
for manipulating an object oh . In the IEEQS-EMM algorithm, the method op
is performed on the replica oqh whose version number is the maximum in the
quorum Qop h by the following IEEQS-EMM Perform procedure:

IEEQS-EMM Perform(op(oqh )) {
if op(oqh ) = r, {
if oqh .DW = φ,
if oqh .CR = φ, {
oqh .CR = op(oqh ); perform(op(oqh )); oqh .CR = φ;
}
else a result of oqh .CR is sent to a transaction Ti ;
else { /* oqh .DW = φ */
perform(oqh .DW ); oqh .DW = φ;
oqh .CR = op(oqh ); perform(op(oqh )); oqh .CR = φ;
}
}
else { /* op = w */
vhq = vhq + 1;
if oqh .DW
= φ and oqh .DW is not meaningless, {
perform(oqh .DW ); perform(op(oqh )); oqh .DW = φ;
}
else perform(op(oqh )); /* oqh .DW = φ or oqh .DW method is omitted */
 The IEEQS-EMM Algorithm 233


vhq and updated data dh are sent to every replica oqh in a quorum Qop
h ;
}
}

Each time a replica oqh in a write quorum Qw
h receives the newest version

number vh and updated data dh from another replica oqh , the replica oqh manip-
q

ulate the version number vhq and updated data dh by the following IEEQS-
EMM Replace procedure:
IEEQS-EMM Replace(vhq , dh , wti (oqh )) {

vhq = vhq ;
 
if oqh .DW = φ, oqh .DW = wti (oqh );

else { /* oqh .DW = φ */
 
if wti (oqh ) absorbs oqh .DW , oqh .DW = wti (oqh );
 
else {perform(oqh .DW ); oqh .DW = wti (oqh ); }
}
}

4 Evaluation
4.1 Environment

We evaluate the IEEQS-EMM algorithm in terms of the total electric energy

of a server cluster S, the average execution time of each transaction, and the
average number of aborted transactions compared with the IEEQS algorithm.
We consider a homogeneous server cluster S which is composed of ten servers
s1 , . . . , s10 (n = 10). Every server st (t = 1, . . . , 10) in the server cluster S follows
the same data access model and power consumption model as shown in Table 1.
Parameters of each server st are given based on the experimentations [6]. There
are thirty objects o1 , . . . , o30 (m = 30). Each object oh supports partial write
(wp ), full write (wf ), and read (r) methods. The size of data in each object oh
is randomly selected between 50 and 100 [MB]. The total number of replicas
for every object is five. Replicas of each object are randomly distributed on five
servers in the server cluster S. The quorum numbers nQw r
h and nQh on every
object oh are three, respectively.
The number nt of transactions are issues to manipulate objects (0 ≤ nt ≤
1, 000). Each transaction issues three methods randomly selected from ninety
methods on the thirty objects. The total amount of data of an object oh is fully
read and written by each read (r) and full write (wf ) methods, respectively. On
the other hand, a half size of data of an object oh is written by each partial write
(wp ) method. The starting time of each transaction Ti is randomly selected in a
unit of one second between 1 and 360 [sec].
234 T. Enokido et al.

Table 1. Parameters of each server st (t = 1, . . . , 10)

server st maxRRt maxW Rt rwt wrt minEt W Et REt

st 80 [MB/sec] 45 [MB/sec] 0.5 0.5 39 [W] 53 [W] 43 [W]

4.2 Average Total Processing Electric Energy Consumption

Let T P ECβ be the total processing electric energy [J] to perform the number
nt of transactions (0 ≤ nt ≤ 1, 000) in the server cluster S obtained in the β-th
simulation. The total processing electric energy T P ECβ is measured ten times
for each number nt of transactions. Then, the average total 10processing electric
energy AT P EC [J] of the server cluster S is calculated as β=1 T P ECβ /10 for
each number nt of transactions.
Average total processing electric energy [KJ]

450
400 IEEQS-EMM
IEEQS
350
300
250
200
150
100
50
0
0 200 400 600 800 1000
Number nt of transactions

Fig. 3. Average total processing electric energy consumption (ATPEC) [KJ].

Figure 3 shows the average total processing electric energy AT P EC [KJ] of

the server cluster S to perform the number nt of transactions in the IEEQS-EMM
and IEEQS algorithms. The average total processing electric energy AT P EC
of the server cluster S can be more reduced in the IEEQS-EMM algorithm
than the IEEQS algorithm for 0 ≤ nt ≤ 1, 000. In the IEEQS algorithm, only
the meaningless write methods are omitted. On the other hand, not only the
meaningless write methods but also the meaningless read methods are omitted
in the IEEQS-EMM algorithm. As a result, the average total processing electric
energy AT P EC of the server cluster S can be more reduced in the IEEQS-EMM
algorithm than the IEEQS algorithm.

4.3 Average Execution Time of Each Transaction

Suppose a transaction Ti starts at time sti and commits at time eti . The exe-
cution time ETi [sec] of the transaction Ti where the transaction Ti commits
is eti - sti [sec]. The execution time ETi for each transaction Ti is measured
 The IEEQS-EMM Algorithm 235

ten times for each total number nt of transactions (0 ≤ nt ≤ 1, 000). Let ETiβ
be the execution time ETi obtained in β-th simulation. The average execution
time AET [sec] of each transaction for each total number nt of transactions is
10 nt β
β=1 i=1 ETi /(nt · 10).
Figure 4 shows the average execution time AET [sec] to perform the number
nt of transactions in the IEEQS-EMM and IEEQS algorithms. In the IEEQS-
EMM and IEEQS algorithms, the average execution time AET increases as the
total number nt of transactions increases since more number of transactions
are concurrently performed. For 0 < nt ≤ 1, 000, the average execution time
AET can be more reduced in the IEEQS-EMM algorithm than the IEEQS algo-
rithm. In the IEEQS-EMM algorithm, not only the meaningless write methods
but also the meaningless read methods are omitted while only the meaningless
write methods are omitted in the IEEQS algorithm. Hence, the more number
of meaningless methods are omitted in the IEEQS-EMM algorithm than the
IEEQS algorithm. Each transaction can commit without waiting for performing
meaningless methods. As a result, the average execution time of each transaction
can be more reduced in the IEEQS-EMM algorithm than the IEEQS algorithm.
Average execution time [sec] of each transaction

Average number of aborts for each transaction

7 120

6 IEEQS-EMM
100 IEEQS
5
80
4
60
3
40
2 IEEQS-EMM
IEEQS
1 20

0 0
0 200 400 600 800 1000 0 200 400 600 800 1000
Number nt of transactions Number nt of transactions

Fig. 4. Average execution time AET Fig. 5. Average number of aborts for
[sec] of each transaction. each transaction

4.4 Average Number of Aborted Transaction Instances

If a transaction Ti could not lock every replica in an r-quorum Qrh or w-quorum

Qwh , the transaction Ti aborts. Then, the transaction Ti is restarted after δ time
units. The time units δ [sec] is randomly selected between twenty and thirty
seconds in this evaluation. Every transaction Ti is restarted until the transac-
tion Ti commits. Each execution of a transaction is referred to as transaction
instance. We measure how many number of transaction instances are aborted
until each transaction commits. Let ATi be the number of aborted instances of
a transaction Ti . The number ATi of aborted instances for each transaction Ti is
measured ten times for each total number nt of transactions (0 ≤ nt ≤ 1, 000).
236 T. Enokido et al.

Let ATiβ be the number ATi of aborted transaction instances obtained in β-th
simulation. The average number AAT of aborted
10  instances of each transaction
nt
for each total number nt of transactions is β=1 i=1 ATiβ /(nt · 10).
Figure 5 shows the average number AAT of aborted transaction instances to
perform the total number nt of transactions in the IEEQS-EMM and IEEQS
algorithms. The more number of transactions are concurrently performed, the
more number of transactions cannot lock replicas. Hence, the number of aborted
transaction instances increases in the IEEQS-EMM and IEEQS algorithms as
the total number nt of transactions increases. For 0 < nt ≤ 1, 000, the average
number AAT of aborted instances of each transaction can be more reduced in
the IEEQS-EMM algorithm than the IEEQS algorithm. The average execution
time of each transaction can be reduced in the IEEQS-EMM algorithm than
the IEEQS algorithm. As a result, the number of aborted transactions can be
more reduced in the IEEQS-EMM algorithm than the IEEQS algorithm since
the number of transactions to be concurrently performed can be reduced.
Following the evaluation, the total electric energy of a server cluster, the
average execution time of each transaction, and the number of aborted transac-
tions in the IEEQS-EMM algorithm can be reduced than the IEEQS algorithm,
respectively. Hence, the IEEQS-EMM algorithm is more useful than the IEEQS
algorithm.

5 Concluding Remarks
In this paper, we newly proposed the IEEQS-EMM (improved energy-efficient
quorum selection algorithm with excluding meaningless methods) algorithm to
reduce the total electric energy of a server cluster in the quorum-based locking
protocol by omitting meaningless read and write methods. We evaluated the
IEEQS-EMM algorithm compared with the IEEQS algorithm. The evaluation
results show the total electric energy of a server cluster, the average execu-
tion time of each transaction, and the number of aborted transactions can be
more reduced in the IEEQS-EMM algorithm than the IEEQS algorithm. Follow-
ing the evaluation, the IEEQS-EMM algorithm is more useful than the IEEQS
algorithm.

References
1. Object Management Group Inc.: Common object request broker architecture
(CORBA) specification, version 3.3, part 1 – interfaces (2012). http://www.omg.
org/spec/CORBA/3.3/Interfaces/PDF
2. Schneider, F.B.: Replication management using the state-machine approach. In:
Distributed systems, 2nd edn. ACM Press (1993)
3. Bernstein, P.A., Hadzilacos, V., Goodman, N.: Concurrency Control and Recovery
in Database Systems. Addison-Wesley, Boston (1987)
4. Gray, J.N.: Notes on database operating systems. In: Lecture Notes in Computer
Science, vol. 60, pp. 393–481 (1978)
 The IEEQS-EMM Algorithm 237

5. Tanaka, K., Hasegawa, K., Takizawa, M.: Quorum-based replication in object-based

systems. J. Inf. Sci. Eng. 16(3), 317–331 (2000)
6. Sawada, A., Kataoka, H., Duolikun, D., Enokido, T., Takizawa, M.: Energy-aware
clusters of servers for storage and computation applications. In: Proceedings of
the 30th IEEE International Conference on Advanced Information Networking and
Applications (AINA-2016), pp. 400–407 (2016)
7. Enokido, T., Duolikun, D., Takizawa, M.: Improved energy-efficient quorum selec-
tion algorithm by omitting meaningless methods. In: Proceedings of the 6th Inter-
national Conference on Emerging Internet, Data and Web Technologies (EIDWT-
2018), pp. 171–182 (2018)
 A Fuzzy-Based System for Actor Node
Selection in WSANs Considering Level of
Received Signal

Donald Elmazi1(B) , Miralda Cuka2 , Makoto Ikeda1 , Leonard Barolli1 ,

and Makoto Takizawa3
1
Department of Information and Communication Engineering,
Fukuoka Institute of Technology (FIT), 3-30-1 Wajiro-Higashi, Higashi-Ku,
Fukuoka 811-0295, Japan
[email protected], [email protected], [email protected]
2
Graduate School of Engineering, Fukuoka Institute of Technology (FIT),
3-30-1 Wajiro-Higashi, Higashi-Ku, Fukuoka 811-0295, Japan
[email protected]
3
Department of Advanced Sciences, Hosei University,
2-17-1 Fujimi, Chiyoda, Tokyo 102-8160, Japan
[email protected]

Abstract. Wireless Sensor and Actor Network (WSAN) is formed by

the collaboration of micro-sensor and actor nodes. The sensor nodes have
responsibility to sense an event and send information towards an actor
node. The actor node is responsible to take prompt decision and react
accordingly. In order to provide effective sensing and acting, a distributed
local coordination mechanism is necessary among sensors and actors. In
this work, we consider the actor node selection problem and propose a
fuzzy-based system (FBS) that based on data provided by sensors and
actors selects an appropriate actor node. We use 4 input parameters:
Distance to Event (DE), Number of Sensors per Actor (NSA), Remaining
Energy (RE) and Level of Received Signal (LRS) as new parameter. The
output parameter is Actor Selection Decision (ASD). Considering NSA
parameter, the ASD has better values when NSA is medium. Thus, when
the NSA value is 0.5 the load is distributed better and in this situation
the possibility for the actor to be selected is high. Also, for higher values
of LRS, the actor is selected with high possibility.

1 Introduction
Recent technological advances have lead to the emergence of distributed Wireless
Sensor and Actor Networks (WSANs) wich are capable of observing the physical
world, processing the data, making decisions based on the observations and
performing appropriate actions [1].
In WSANs, the devices deployed in the environment are sensors able to sense
environmental data, actors able to react by affecting the environment or have
both functions integrated. Actor nodes are equipped with two radio transmitters,
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 238–250, 2020.
https://doi.org/10.1007/978-3-030-15032-7_21
 A Fuzzy-Based System for Actor Nodes Selection 239

a low data rate transmitter to communicate with the sensor and a high data
rate interface for actor-actor communication. For example, in the case of a fire,
sensors relay the exact origin and intensity of the fire to actors so that they
can extinguish it before spreading in the whole building or in a more complex
scenario, to save people who may be trapped by fire [2–4].
To provide effective operation of WSAN, it is very important that sensors and
actors coordinate in what are called sensor-actor and actor-actor coordination.
Coordination is not only important during task conduction, but also during
network’s self-improvement operations, i.e. connectivity restoration [5,6], reliable
service [7], Quality of Service (QoS) [8,9] and so on.
Sensor-Actor (SA) coordination defines the way sensors communicate with
actors, which actor is accessed by each sensor and which route should data pack-
ets follow to reach it. Among other challenges, when designing SA coordination,
care must be taken in considering energy minimization because sensors, which
have limited energy supplies, are the most active nodes in this process. On the
other hand, Actor-Actor (AA) coordination helps actors to choose which actor
will lead performing the task (actor selection), how many actors should perform
and how they will perform. Actor selection is not a trivial task, because it needs
to be solved in real time, considering different factors. It becomes more compli-
cated when the actors are moving, due to dynamic topology of the network.
In order to measure the strength of the signal of communication between
nodes, we use the Level of Received Signal (LRS) parameter. This measurment
is done to have enough signal to get a good wireless connection. Because for
longer distances the signal strength becomes weaker and it’s hard to have a
good communication with the farthest nodes.
The role of Load balancing in WSAN is to provide a reliable service, in
order to have a better performance and effortness. In this paper we consider the
Load Balancing issue. Load balancing identifies the optimal load on nodes of the
network to increase the network efficiency.
In this paper, different from our previous work [10], we propose and imple-
ment a simulation system which considers also the LRS parameter. The system
is based on fuzzy logic and considers four input parameters for actor selection.
We show the simulation results for different values of parameters.
The remainder of the paper is organized as follows. In Sect. 2, we describe
the basics of WSANs including research challenges and architecture. In Sect. 3,
we describe the system model and its implementation. Simulation results are
shown in Sect. 4. Finally, conclusions and future work are given in Sect. 5.

2 WSAN
2.1 WSAN Challenges

Some of the key challenges in WSAN are related to the presence of actors and
their functionalities.
240 D. Elmazi et al.

• Deployment and Positioning: At the moment of node deployment, algorithms

must consider to optimize the number of sensors and actors and their initial
positions based on applications [11,12].
• Architecture: When important data has to be transmitted (an event
occurred), sensors may transmit their data back to the sink, which will con-
trol the actors’ tasks from distance or transmit their data to actors, which
can perform actions independently from the sink node [13].
• Real-Time: There are a lot of applications that have strict real-time require-
ments. In order to fulfill them, real-time limitations must be clearly defined
for each application and system [14].
• Coordination: In order to provide effective sensing and acting, a distributed
local coordination mechanism is necessary among sensors and actors [13].
• Power Management: WSAN protocols should be designed with minimized
energy consumption for both sensors and actors [15].
• Mobility: Protocols developed for WSANs should support the mobility of
nodes [6,16], where dynamic topology changes, unstable routes and network
isolations are present.
• Scalability: Smart Cities are emerging fast and WSAN, as a key technology
will continue to grow together with cities. In order to keep the functionality
of WSAN applicable, scalability should be considered when designing WSAN
protocols and algorithms [12,16].

2.2 WSAN Architecture

A WSAN is shown in Fig. 1. The main functionality of WSANs is to make
actors perform appropriate actions in the environment, based on the data sensed
from sensors and actors. When important data has to be transmitted (an event
occurred), sensors may transmit their data back to the sink, which will control
the actors’ tasks from distance, or transmit their data to actors, which can per-
form actions independently from the sink node. Here, the former scheme is called
Semi-Automated Architecture and the latter one Fully-Automated Architecture
(see Fig. 2). Obviously, both architectures can be used in different applications.
In the Fully-Automated Architecture are needed new sophisticated algorithms
in order to provide appropriate coordination between nodes of WSAN. On the
other hand, it has advantages, such as low latency, low energy consumption, long
network lifetime [1], higher local position accuracy, higher reliability and so on.

3 Proposed Fuzzy-Based System

Based on WSAN characteristics and challenges, we consider the following param-
eters for implementation of our proposed system.
Distance to Event (DE): The number of actors in a WSAN is smaller than
the number of sensors. Thus, when an actor is called for action near an event,
the distance from the actor to the event is important because when the distance
 A Fuzzy-Based System for Actor Nodes Selection 241

Fig. 1. Wireless Sensor Actor Network (WSAN).

Fig. 2. WSAN architectures.

is longer, the actor will spend more energy. Thus, an actor which is close to an
event, should be selected.
Number of Sensors per Actor (NSA): The number of sensors deployed in
an area for sensing any event may be in the order of hundreds or thousands.
So in order to have a better coverage of these sensors, the number of sensors
covered by each actor node should be balanced.
Remaining Energy (RE): As actors are active in the monitored field, they
perform tasks and exchange data in different ways. Thus some actors may have
a lot of remained power and other may have very little, when an event occurs.
It is better that the actors which have more power are selected to carry out a
task.
Level of Received Signal (LRS): Actors are deployed in many different sce-
narios, some of which are hostile with many obstacles. These hostile environ-
ments affect the overall performance of the actor carrying out a task. Depending
on the number of obstacles, the signal strength level that an actor is receiveing
may increase or decrease. When the signal gets weaker, data rates get slower,
leading to a lower throughput or in a worst scenario loosing contact with the
actor, making it unreachable.
242 D. Elmazi et al.

Fig. 3. Proposed System.

Fig. 4. FLC structure.

Actor Selection Decision (ASD): Our system is able to decide the willingness
of an actor to be assigned a certain task at a certain time. The actors respond
in five different levels, which can be interpreted as:
• Extremely Low Selection Possibility (ELSP) - It is in the minimal condition
to be assigned for carrying out a task
• Very Low Selection Possibility (VLSP) - It is not worth assigning the task to
this actor.
• Low Selection Possibility (LSP) - There might be other actors which can do
the job better.
• Middle Selection Possibility (MSP) - The Actor is ready to be assigned a task,
but is not the “chosen” one.
• High Selection Possibility (HSP) - The actor takes responsibility of completing
the task.
• Very High Selection Possibility (VHSP) - Actor has almost all required infor-
mation and potential and takes full responsibility.
• Extremely High Selection Possibility (EHSP)- Actor is in the perfect condi-
tion to carry out the task.

EIDWT

Fig. 5. Triangular and trapezoidal membership functions.

 A Fuzzy-Based System for Actor Nodes Selection 243

Table 1. Parameters and their term sets for FLC.

Parameters Term sets

Distance to Event (DE) Near (Ne), Moderate (Mo), Far (Fa)
Number of Sensors per Actor (NSA) Few (Fw), Medium (Me), Many (My)
Remaining Energy (RE) Low (Lo), Medium (Mdm), High (Hi)
Level of Received Signal (LRS) Low (Lw), Middle (Mi), High (Hg)
Actor Selection Decision (ASD) Extremely Low Selection Possibility
(ELSP), Very Low Selection Possibility
(VLSP), Low Selection Possibility (LSP),
Middle Selection Possibility (MSP), High
Selection Possibility (HSP), Very High
Selection Possibilty (VHSP), Extremely
High Selection Possibility (EHSP)

Fuzzy sets and fuzzy logic have been developed to manage vagueness and
uncertainty in a reasoning process of an intelligent system such as a knowledge
based system, an expert system or a logic control system [17–30].
The structure of the proposed system is shown in Fig. 3. It consists of one
Fuzzy Logic Controller (FLC), which is the main part of our system and its basic
elements are shown in Fig. 4. They are the fuzzifier, inference engine, Fuzzy Rule
Base (FRB) and defuzzifier.
As shown in Fig. 5, we use triangular and trapezoidal membership functions
for FLC, because they are suitable for real-time operation [31]. The x0 in f (x)
is the center of triangular function, x0 (x1 ) in g(x) is the left (right) edge of
trapezoidal function, and a0 (a1 ) is the left (right) width of the triangular or
trapezoidal function. We explain in details the design of FLC in following.
We use four input parameters for FLC:
• Distance to Event (DE);
• Number of Sensors per Actor (NSA);
• Remaining Energy (RE);
• Level of Received Signal (LRS).
The term sets for each input linguistic parameter are defined respectively as
shown in Table 1.

T (DE) = {N ear(N e), M oderate(M o), F ar(F a)}

T (N SA) = {F ew(F w), M edium(M e), M any(M y)}
T (RE) = {Low(Lo), M edium(M dm), High(Hi)}
T (LRS) = {Low(Lw), M iddle(M i), High(Hg)}
244 D. Elmazi et al.

Fig. 6. Fuzzy membership functions.

The membership functions for input parameters of FLC are defined as:

µN e (DE) = g(DE; N e0 , N e1 , N ew0 , N ew1 )

µM o (DE) = f (DE; M o0 , M ow0 , M ow1 )
µF a (DE) = g(DE; F a0 , F a1 , F aw0 , F aw1 )
µF w (N SA) = g(N SA; F w0 , F w1 , F ww0 , F ww1 )
µM e (N SA) = f (N SA; M e0 , M ew0 , M ew1 )
µM y (N SA) = g(N SA; M y0 , M y1 , M yw0 , M yw1 )
µLo (RE) = g(RE; Lo0 , Lo1 , Low0 , Low1 )
µM dm (RE) = f (RE; M dm0 , M dmw0 , M dmw1 )
µHi (RE) = g(RE; Hi0 , Hi1 , Hiw0 , Hiw1 ).
µLw (LRS) = g(LRS; Lw0 , Lw1 , Lww0 , Lww1 )
µM i (LRS) = f (LRS; M i0 , M iw0 , M iw1 )
µHg (LRS) = g(LRS; Hg0 , Hg1 , Hgw0 , Hgw1 )

The small letters w0 and w1 mean left width and right width, respectively.
 A Fuzzy-Based System for Actor Nodes Selection 245

The output linguistic parameter is the Actor Selection Decision (ASD). We

define the term set of ASD as:

{Extremely Low Selection P ossibility (ELSP ),

V ery Low Selection P ossibility (V LSP ),
Low Selection P ossibility (LSP ),
M iddle Selection P ossibility (M SP ),
High Selection P ossibility (HSP ),
V ery High Selection P ossibility (V HSP ),
Extremely High Selection P ossibility (EHSP )}.

The membership functions for the output parameter ASD are defined as:

µELSP (ASD) = g(ASD; ELSP0 , ELSP1 , ELSPw0 , ELSPw1 )

µV LSP (ASD) = f (ASD; V LSP0 , V LSP1 , V LSPw0 )
µLSP (ASD) = f (ASD; LSP0 , LSP1 , LSPw0 )
µM SP (ASD) = f (ASD; M SP0 , M SP1 , M SPw0 )
µHSP (ASD) = f (ASD; HSP0 , HSP1 , HSPw0 )
µV HSP (ASD) = f (ASD; V HSP0 , V HSP1 , V HSPw0 )
µEHSP (ASD) = g(ASD; EHSP0 , EHSP1 , EHSPw0 , EHSPw1 ).

The membership functions are shown in Fig. 6 and the Fuzzy Rule Base
(FRB) is shown in Table 2. The FRB forms a fuzzy set of dimensions |T (DE)| ×
|T (N SA)| × |T (RE)| × |T (LRS)|, where |T (x)| is the number of terms on T (x).
The FRB has 81 rules. The control rules have the form: IF “conditions” THEN
“control action”.

4 Simulation Results
The simulation results for our system are shown in Figs. 7, 8 and 9. One of the
main issues in WSANs is energy dissipation. Higher remaining energy improves
the overall lifetime of the network. In Fig. 7(a), DE and NSA are constant and
for LRS 0.8 we see that ASD is increased 8% when RE increases from 0.1 to 0.5
and 17% when RE increases from 0.5 to 0.9. Higher remaining energy, increases
the possibility for an actor to be selected to carry out a task.
Another important paramet in our system is NSA. To see how NSA affects
ASD, we compare Fig. 7(a) with (b) and Fig. 7(b) with (c). For LRS 0.8 and RE
0.9, we see that ASD is increased 15% with the increasing of NSA from 0.1 to
0.5. When NSA is increased from 0.5 to 0.9, ASD is decreased 30%, this because
higher values of NSA affects the overall performance.
Actors are distributed in the network, so this means, they have different dis-
tances from an event. Actors that are further away, are less likely to be selected.
In Figs. 7(b), 8(b) and 9(b), DE is increased from 0.1 to 0.5 and 0.9 respectively.
246 D. Elmazi et al.

Table 2. FRB of proposed fuzzy-based system.

No. DE NSA RE LRS ASD No. DE NSA RE LRS ASD

1 Ne Fw Lo Lw VLSP 41 Mo Me Mdm Mi MSP
2 Ne Fw Lo Mi LSP 42 Mo Me Mdm Hg HSP
3 Ne Fw Lo Hg MSP 43 Mo Me Hi Lw HSP
4 Ne Fw Mdm Lw LSP 44 Mo Me Hi Mi HSP
5 Ne Fw Mdm Mi MSP 45 Mo Me Hi Hg VHSP
6 Ne Fw Mdm Hg HSP 46 Mo My Lo Lw ELSP
7 Ne Fw Hi Lw MSP 47 Mo My Lo Mi ELSP
8 Ne Fw Hi Mi HSP 48 Mo My Lo Hg VLSP
9 Ne Fw Hi Hg VHSP 49 Mo My Mdm Lw VLSP
10 Ne Me Lo Lw MSP 50 Mo My Mdm Mi VLSP
11 Ne Me Lo Mi MSP 51 Mo My Mdm Hg VLSP
12 Ne Me Lo Hg HSP 52 Mo My Hi Lw MSP
13 Ne Me Mdm Lw HSP 53 Mo My Hi Mi LSP
14 Ne Me Mdm Mi HSP 54 Mo My Hi Hg MSP
15 Ne Me Mdm Hg VHSP 55 Fa Fw Lo Lw MSP
16 Ne Me Hi Lw VHSP 56 Fa Fw Lo Mi ELSP
17 Ne Me Hi Mi EHSP 57 Fa Fw Lo Hg ELSP
18 Ne Me Hi Hg EHSP 58 Fa Fw Mdm Lw VLSP
19 Ne My Lo Lw ELSP 59 Fa Fw Mdm Mi ELSP
20 Ne My Lo Mi VLSP 60 Fa Fw Mdm Hg VLSP
21 Ne My Lo Hg LSP 61 Fa Fw Hi Lw LSP
22 Ne My Mdm Lw LSP 62 Fa Fw Hi Mi VLSP
23 Ne My Mdm Mi MSP 63 Fa Fw Hi Hg LSP
24 Ne My Mdm Hg MSP 64 Fa Me Lo Lw MSP
25 Ne My Hi Lw MSP 65 Fa Me Lo Mi VLSP
26 Ne My Hi Mi MSP 66 Fa Me Lo Hg LSP
27 Ne My Hi Hg HSP 67 Fa Me Mdm Lw MSP
28 Mo Fw Lo Lw ELSP 68 Fa Me Mdm Mi LSP
29 Mo Fw Lo Mi VLSP 69 Fa Me Mdm Hg MSP
30 Mo Fw Lo Hg VLSP 70 Fa Me Hi Lw HSP
31 Mo Fw Mdm Lw VLSP 71 Fa Me Hi Mi MSP
32 Mo Fw Mdm Mi LSP 72 Fa Me Hi Hg HSP
33 Mo Fw Mdm Hg MSP 73 Fa My Lo Lw VHSP
34 Mo Fw Hi Lw LSP 74 Fa My Lo Mi EHSP
35 Mo Fw Hi Mi MSP 75 Fa My Lo Hg ELSP
36 Mo Fw Hi Hg HSP 76 Fa My Mdm Lw ELSP
37 Mo Me Lo Lw VLSP 77 Fa My Mdm Mi ELSP
38 Mo Me Lo Mi MSP 78 Fa My Mdm Hg VLSP
39 Mo Me Lo Hg MSP 79 Fa My Hi Lw LSP
40 Mo Me Mdm Lw MSP 80 Fa My Hi Mi VLSP
81 Fa Hg Hi Hg MSP
 A Fuzzy-Based System for Actor Nodes Selection 247

DE=0.1-NSA=0.1 DE=0.1-NSA=0.5
1 1
RE=0.1
0.9 RE=0.5 0.9
RE=0.9
0.8 0.8
0.7 0.7
0.6 0.6
ASD [unit]

ASD [unit]
0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
RE=0.1
0.1 0.1 RE=0.5
RE=0.9
0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
LRS LRS

(a) NSA=0.1 (b) NSA=0.5

DE=0.1-NSA=0.9
1
0.9
0.8
0.7
0.6
ASD [unit]

0.5
0.4
0.3
0.2
RE=0.1
0.1 RE=0.5
RE=0.9
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
LRS

(c) NSA=0.9

Fig. 7. Results for DE = 0.1.

DE=0.5-NSA=0.1 DE=0.5-NSA=0.5
1 1
RE=0.1 RE=0.1
0.9 RE=0.5 0.9 RE=0.5
RE=0.9 RE=0.9
0.8 0.8
0.7 0.7
0.6 0.6
ASD [unit]

ASD [unit]

0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
LRS LRS

(a) NSA=0.1 (b) NSA=0.5

DE=0.5-NSA=0.9
1
RE=0.1
0.9 RE=0.5
RE=0.9
0.8
0.7
0.6
ASD [unit]

0.5
0.4
0.3
0.2
0.1
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
LRS

(c) NSA=0.9

Fig. 8. Results for DE = 0.5.

248 D. Elmazi et al.

DE=0.9-NSA=0.1 DE=0.9-NSA=0.5
1 1
RE=0.1 RE=0.1
0.9 RE=0.5 0.9 RE=0.5
RE=0.9 RE=0.9
0.8 0.8
0.7 0.7
0.6 0.6
ASD [unit]

ASD [unit]
0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
LRS LRS

(a) NSA=0.1 (b) NSA=0.5

DE=0.9-NSA=0.9
1
RE=0.1
0.9 RE=0.5
RE=0.9
0.8
0.7
0.6
ASD [unit]

0.5
0.4
0.3
0.2
0.1
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
LRS

(c) NSA=0.9

Fig. 9. Results for DE = 0.9.

Comparing Figs. 7(b) with 8(b), for LRS = 0.2 and RE = 0.9, we see that the
ASD is decreased 14%. When DE increases from 0.1 to 0.9 in Figs. 7(b) to 9(b),
ASD is decreased 20%. This happens because longer distances from a task affects
the actor’s resources.
In the three simulation results we can see that, when the LRS parameter
increases also the ASD increases. This means that actor nodes with stronger
signal strength are more involved in the system and have higher possibilites to
be selected to carry out a task.

5 Conclusions and Future Work

In this we propose and implement a FBS for actor node selection in WSANs,
considering LRS as new parameter. Considering that some nodes are more reli-
able than the others, our system decides which actor nodes are better suited to
carry out a task. From simulation results, we conclude as follows.

• When LRS and RE parameters are increased, the ASD parameter is increased,
so the probability that the system selects an actor node for the job is high.
• When the NSA value is 0.5 the load is distributed better and in this situation
the possibility for the actor to be selected is high.
• When the DE parameter is increased, the ASD parameter is decreased, so the
probability that an actor node is selected for the required task is low.
 A Fuzzy-Based System for Actor Nodes Selection 249

In the future work, we will consider also other parameters for actor selection
and make extensive simulations to evaluate the proposed system.

References
1. Akyildiz, I.F., Kasimoglu, I.H.: Wireless sensor and actor networks: research chal-
lenges. Ad Hoc Netw. J. 2(4), 351–367 (2004)
2. Akyildiz, I., Su, W., Sankarasubramaniam, Y., Cayirci, E.: Wireless sensor net-
works: a survey. Comput. Netw. 38(4), 393–422 (2002)
3. Boyinbode, O., Le, H., Takizawa, M.: A survey on clustering algorithms for wireless
sensor networks. Int. J. Space-Based Situat. Comput. 1(2/3), 130–136 (2011)
4. Bahrepour, M., Meratnia, N., Poel, M., Taghikhaki, Z., Havinga, P.J.: Use of wire-
less sensor networks for distributed event detection in disaster managment appli-
cations. Int. J. Space-Based Situat. Comput. 2(1), 58–69 (2012)
5. Haider, N., Imran, M., Saad, N., Zakariya, M.: Performance analysis of reactive
connectivity restoration algorithms for wireless sensor and actor networks. In: IEEE
Malaysia International Conference on Communications (MICC 2013), November
2013, pp. 490–495 (2013)
6. Abbasi, A., Younis, M., Akkaya, K.: Movement-assisted connectivity restoration
in wireless sensor and actor networks. IEEE Trans. Parallel Distrib. Syst. 20(9),
1366–1379 (2009)
7. Li, X., Liang, X., Lu, R., He, S., Chen, J., Shen, X.: Toward reliable actor services
in wireless sensor and actor networks. In: 2011 IEEE 8th International Conference
on Mobile Adhoc and Sensor Systems (MASS), October 2011, pp. 351–360 (2011)
8. Akkaya, K., Younis, M.: Cola: a coverage and latency aware actor placement for
wireless sensor and actor networks. In: IEEE 64th Conference on Vehicular Tech-
nology (VTC 2006) Fall, September 2006, pp. 1–5 (2006)
9. Kakarla, J., Majhi, B.: A new optimal delay and energy efficient coordination algo-
rithm for WSAN. In: 2013 IEEE International Conference on Advanced Networks
and Telecommunications Systems (ANTS), December 2013, pp. 1–6 (2013)
10. Elmazi, D., Cuka, M., Ikeda, M., Barolli, L.: A fuzzy-based system for actor node
selection in WSANs for improving network connectivity and increasing number of
covered sensors. In: The 21st International Conference on Network-Based Informa-
tion Systems (NBiS 2018) (2018)
11. Akbas, M., Turgut, D.: APAWSAN: actor positioning for aerial wireless sensor
and actor networks. In: 2011 IEEE 36th Conference on Local Computer Networks
(LCN), October 2011, pp. 563–570 (2011)
12. Akbas, M., Brust, M., Turgut, D.: Local positioning for environmental monitoring
in wireless sensor and actor networks. In: 2010 IEEE 35th Conference on Local
Computer Networks (LCN), October 2010, pp. 806–813 (2010)
13. Melodia, T., Pompili, D., Gungor, V., Akyildiz, I.: Communication and coordi-
nation in wireless sensor and actor networks. IEEE Trans. Mob. Comput. 6(10),
1126–1129 (2007)
14. Gungor, V., Akan, O., Akyildiz, I.: A real-time and reliable transport (RT2) proto-
col for wireless sensor and actor networks. IEEE/ACM Trans. Netw. 16(2), 359–370
(2008)
15. Selvaradjou, K., Handigol, N., Franklin, A., Murthy, C.: Energy-efficient directional
routing between partitioned actors in wireless sensor and actor networks. IET
Commun. 4(1), 102–115 (2010)
250 D. Elmazi et al.

16. Nakayama, H., Fadlullah, Z., Ansari, N., Kato, N.: A novel scheme for wsan sink
mobility based on clustering and set packing techniques. IEEE Trans. Autom.
Control 56(10), 2381–2389 (2011)
17. Inaba, T., Sakamoto, S., Kolici, V., Mino, G., Barolli, L.: A CAC scheme based on
fuzzy logic for cellular networks considering security and priority parameters. In:
The 9th International Conference on Broadband and Wireless Computing, Com-
munication and Applications (BWCCA 2014), pp. 340–346 (2014)
18. Spaho, E., Sakamoto, S., Barolli, L., Xhafa, F., Barolli, V., Iwashige, J.: A fuzzy-
based system for peer reliability in JXTA-Overlay P2P considering number of
interactions. In: The 16th International Conference on Network-Based Information
Systems (NBiS 2013), pp. 156–161 (2013)
19. Matsuo, K., Elmazi, D., Liu, Y., Sakamoto, S., Mino, G., Barolli, L.: FACS-MP: a
fuzzy admission control system with many priorities for wireless cellular networks
and its performance evaluation. J. High Speed Netw. 21(1), 1–14 (2015)
20. Grabisch, M.: The application of fuzzy integrals in multicriteria decision making.
Eur. J. Oper. Res. 89(3), 445–456 (1996)
21. Inaba, T., Elmazi, D., Liu, Y., Sakamoto, S., Barolli, L., Uchida, K.: Integrating
wireless cellular and ad-hoc networks using fuzzy logic considering node mobility
and security. In: The 29th IEEE International Conference on Advanced Information
Networking and Applications Workshops (WAINA 2015), pp. 54–60 (2015)
22. Kulla, E., Mino, G., Sakamoto, S., Ikeda, M., Caballé, S., Barolli, L.: FBMIS: a
fuzzy-based multi-interface system for cellular and ad hoc networks. In: Interna-
tional Conference on Advanced Information Networking and Applications (AINA
2014), pp. 180–185 (2014)
23. Elmazi, D., Kulla, E., Oda, T., Spaho, E., Sakamoto, S., Barolli, L.: A comparison
study of two fuzzy-based systems for selection of actor node in wireless sensor actor
networks. J. Ambient Intell. Hum. Comput. 6, 1–11 (2015)
24. Zadeh, L.: Fuzzy logic, neural networks, and soft computing. Commun. ACM, 77–
84 (1994)
25. Spaho, E., Sakamoto, S., Barolli, L., Xhafa, F., Ikeda, M.: Trustworthiness in P2P:
performance behaviour of two fuzzy-based systems for JXTA-overlay platform. Soft
Comput. 18(9), 1783–1793 (2014)
26. Inaba, T., Sakamoto, S., Kulla, E., Caballe, S., Ikeda, M., Barolli, L.: An integrated
system for wireless cellular and ad-hoc networks using fuzzy logic. In: International
Conference on Intelligent Networking and Collaborative Systems (INCoS 2014), pp.
157–162 (2014)
27. Matsuo, K., Elmazi, D., Liu, Y., Sakamoto, S., Barolli, L.: A multi-modal simula-
tion system for wireless sensor networks: a comparison study considering stationary
and mobile sink and event. J. Ambient Intell. Hum. Comput. 6(4), 519–529 (2015)
28. Kolici, V., Inaba, T., Lala, A., Mino, G., Sakamoto, S., Barolli, L.: A fuzzy-based
CAC scheme for cellular networks considering security. In: International Conference
on Network-Based Information Systems (NBiS 2014), pp. 368–373 (2014)
29. Liu, Y., Sakamoto, S., Matsuo, K., Ikeda, M., Barolli, L., Xhafa, F.: A comparison
study for two fuzzy-based systems: improving reliability and security of JXTA-
overlay P2P platform. Soft Comput. 20, 1–11 (2015)
30. Matsuo, K., Elmazi, D., Liu, Y., Sakamoto, S., Mino, G., Barolli, L.: FACS-MP: a
fuzzy admission control system with many priorities for wireless cellular networks
and its perforemance evaluation. J. High Speed Netw. 21(1), 1–14 (2015)
31. Mendel, J.M.: Fuzzy logic systems for engineering: a tutorial. Proc. IEEE 83(3),
345–377 (1995)
 Selection of IoT Devices in Opportunistic
Networks: A Fuzzy-Based Approach
Considering IoT Device’s Selfish
Behaviour

Miralda Cuka1(B) , Donald Elmazi2 , Makoto Ikeda2 , Keita Matsuo2 ,

Leonard Barolli2 , and Makoto Takizawa3
1
Graduate School of Engineering, Fukuoka Institute of Technology (FIT),
3-30-1 Wajiro-Higashi, Higashi-Ku,
Fukuoka 811-0295, Japan
[email protected]
2
Department of Information and Communication Engineering,
Fukuoka Institute of Technology (FIT), 3-30-1 Wajiro-Higashi, Higashi-Ku,
Fukuoka 811-0295, Japan
[email protected], [email protected],
{kt-matsuo,barolli}@fit.ac.jp
3
Department of Advanced Sciences, Hosei University, 2-17-1 Fujimi, Chiyoda,
Tokyo 102-8160, Japan
[email protected]

Abstract. In opportunistic networks the communication opportunities

(contacts) are intermittent and there is no need to establish an end-
to-end link between the communication nodes. The enormous growth
of devices having access to the Internet, along the vast evolution of the
Internet and the connectivity of objects and devices, has evolved as Inter-
net of Things (IoT). There are different issues for these networks. One
of them is the selection of IoT devices in order to carry out a task in
opportunistic networks. In this work, we implement a Fuzzy-Based Sys-
tem for IoT device selection in opportunistic networks. For our system,
we use four input parameters: IoT Device’s Selfish Behaviour (IDSB),
IoT Device Remaining Energy (IDRE), IoT Device Storage (IDST) and
IoT Device Contact Duration (IDCD). The output parameter is IoT
Device Selection Decision (IDSD). The simulation results show that the
proposed system makes a proper selection decision of IoT devices in
opportunistic networks. The IoT device selection is increased up to 14%
and decreased 23% by increasing IDRE and IDSB, respectively.

1 Introduction

Future communication systems will be increasingly complex, involving thousands

of heterogeneous devices with diverse capabilities and various networking tech-
nologies interconnected with the aim to provide users with ubiquitous access to
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 251–264, 2020.
https://doi.org/10.1007/978-3-030-15032-7_22
252 M. Cuka et al.

information and advanced services at a high quality level, in a cost efficient man-
ner, any time, any place, and in line with the always best connectivity principle.
The Opportunistic Networks (OppNets) can provide an alternative way to sup-
port the diffusion of information in special locations within a city, particularly
in crowded spaces where current wireless technologies can exhibit congestion
issues. The efficiency of this diffusion relies mainly on user mobility. In fact,
mobility creates the opportunities for contacts and, therefore, for data forward-
ing [1]. OppNets have appeared as an evolution of the MANETs. They are also
a wireless based network and hence, they face various issues similar to MANETs
such as frequent disconnections, highly variable links, limited bandwidth etc. In
OppNets, nodes are always moving which makes the network easy to deploy and
decreases the dependence on infrastructure for communication [2].
The Internet of Things (IoT) can seamlessly connect the real world and
cyberspace via physical objects embedded with various types of intelligent sen-
sors. A large number of Internet-connected machines will generate and exchange
an enormous amount of data that make daily life more convenient, help to make
a tough decision and provide beneficial services. The IoT probably becomes one
of the most popular networking concepts that has the potential to bring out
many benefits [3,4].
OppNets are the variants of Delay Tolerant Networks (DTNs). It is a class
of networks that has emerged as an active research subject in the recent times.
Owing to the transient and un-connected nature of the nodes, routing becomes
a challenging task in these networks. Sparse connectivity, no infrastructure and
limited resources further complicate the situation [5,6]. Routing methods for
such sparse mobile networks use a different paradigm for message delivery. These
schemes utilize node mobility by having nodes carry messages, waiting for an
opportunity to transfer messages to the destination or the next relay rather than
transmitting them over a path [7]. Hence, the challenges for routing in OppNet
are very different from the traditional wireless networks and their utility and
potential for scalability makes them a huge success.
In Oppnets, the topology of the network is the result of cooperation amongst
the nodes themselves, so a successful transmission between a pair of nodes
requires the cooperation of intermediate nodes in order for the transmitted
packets to reach their destination. While this may be guaranteed in networks
with a central authority forcing the nodes to cooperate, in the absence of such
an authority cooperation may not be guaranteed. This may result in selfish
behaviour of nodes, i.e., the effort by the node to maximize its own utility with-
out caring about the results of its actions on the overall network-wide outcome.
For example, if battery life is a valuable resource for a node, forwarding pack-
ets between two other nodes consumes energy that doesn’t result in any kind
of pay-off for this node, and as a result it may decide to stop cooperating in
forwarding packets for others [8].
The Fuzzy Logic (FL) is unique approach that is able to simultaneously
handle numerical data and linguistic knowledge. The fuzzy logic works on the
 A Fuzzy-Based System for Selection of IoT Devices in OppNets 253

levels of possibilities of input to achieve the definite output. Fuzzy set theory
and FL establish the specifics of the nonlinear mapping.
In this paper, we propose and implement a fuzy-based simulation system
for selection of IoT devices in OppNets considering as a new parameter the
selfish behaviour of an IoT device. For our system, we consider four parameters
for IoT device selection. We show the simulation results for different values of
parameters.
The remainder of the paper is organized as follows. In the Sect. 2, we present
IoT and OppNets. In Sect. 3, we introduce the proposed system model and its
implementation. Simulation results are shown in Sect. 4. Finally, conclusions and
future work are given in Sect. 5.

2 IoT and OppNets

2.1 IoT

IoT allows to integrate physical and virtual objects. Virtual reality, which was
recently available only on the monitor screens, now integrates with the real world,
providing users with completely new opportunities: interact with objects on the
other side of the world and receive the necessary services that became real due
the wide interaction [9]. The IoT will support substantially higher number of
end users and devices. In Fig. 1, we present an example of an IoT network archi-
tecture. The IoT network is a combination of IoT devices which are connected
with different mediums using IoT Gateway to the Internet. The data transmitted
through the gateway is stored, proccessed securely within cloud server. These
new connected things will trigger increasing demands for new IoT applications
that are not only for users. The current solutions for IoT application development
generally rely on integrated service-oriented programming platforms. In partic-
ular, resources (e.g., sensory data, computing resource, and control information)
are modeled as services and deployed in the cloud or at the edge. It is difficult to
achieve rapid deployment and flexible resource management at network edges,
in addition, an IoT system’s scalability will be restricted by the capability of the
edge devices [10].

2.2 OppNets
In Fig. 2 we show an OppNet scenario. OppNets comprises a network where
nodes can be anything from pedestrians, vehicles, fixed devices and so on. The
data is sent from the sender to receiver by using communication opportunity that
can be Wi-Fi, Bluetooth, cellular technologies or satellite links to transfer the
message to the final destination. In such scenario, IoT devices might roam and
opportunistically encounter several different statically deployed networks and
perform either data collection or dissemination as well as relaying data between
these networks, thus introducing further connectivity for disconnected networks.
For example, as seen in the figure, a car could opportunistically encounter other
254 M. Cuka et al.

Fig. 1. An Iot network architecture.

IoT devices, collect information from them and relay it until it finds an available
access point where it can upload the information. Similarly, a person might col-
lect information from home-based weather stations and relay it through several
other people, cars and buses until it reaches its intended destination [11].
OppNets are not limited to only such applications, as they can introduce
further connectivity and benefits to IoT scenarios. In an OppNet, due to node
mobility network partitions occur. These events result in intermittent connec-
tivity. When there is no path existing between the source and the destination,
the network partition occurs. Therefore, nodes need to communicate with each
other via opportunistic contacts through store-carry-forward operation. There
are two specific challenges in an OppNet: the contact opportunity and the node
storage.

• Contact Opportunity: Due to the node mobility or the dynamics of wireless

channel, a node can make contact with other nodes at an unpredicted time.
Since contacts between nodes are hardly predictable, they must be exploited
opportunistically for exchanging messages between some nodes that can move
between remote fragments of the network. Mobility increases the chances
of communication between nodes. When nodes move randomly around the
network, where jamming signals are disrupting the communication, they may
pass through unjammed area and hence be able to communicate. In addition,
the contact capacity needs to be considered [12,13].
• Node Storage: As described above, to avoid dropping packets, the interme-
diate nodes are required to have enough storage to store all messages for an
unpredictable period of time until next contact occurs. In other words, the
required storage space increases as a function of the number of messages in
the network. Therefore, the routing and replication strategies must take the
storage constraint into consideration [14].
 A Fuzzy-Based System for Selection of IoT Devices in OppNets 255

Fig. 2. OppNets Scenario.

3 Proposed Fuzzy-Based System

3.1 System Parameters

We consider the following parameters for implementation of our proposed

system.
IoT Device’s Selfish Behaviour (IDSB): The main characteristic of Opp-
Nets, is the cooperation of devices and the offering of their resources for a suc-
cessful transmission. Due to decentralized nature of OppNets and the absence
of an authority, cooperation may not be guaranteed and devices exhibit self-
ish behaviour. A selfish IoT device, utilizes the network resources, but never or
rarely offers its own resources to other devices. If these selfish behaviours happen
often, the network may be disrupted or its overall performance will degrade.
IoT Device Remaining Energy (IDRE): The IoT devices in opportunistic
networks are active and can perform tasks and exchange data in different ways
from each other. Consequently, some IoT devices may have a lot of remaining
power and other may have very little, when an event occurs.
IoT Device Storage (IDST): In delay tolerant networks data is carried by
the IoT device until a communication opportunity is available. Considering that
256 M. Cuka et al.

IDSB

IDRE

FLC IDSD
IDST

IDCD

Fig. 3. Proposed system model.

Input Inference Output

Fuzzifier Engine Defuzzifier

Fuzzy Rule
Base

Fig. 4. FLC structure.

different IoT devices have different storage capabilities, the selection decision
should consider the storage capacity.
IoT Device Contact Duration (IDCD): This is an important parameter in
mobility-assisted networks as contact times represent the duration of message
communication opportunity upon a contact. Contact durations is the time in
which all buffered messages can be transferred within a single contact.

Fig. 5. Triangular and trapezoidal membership functions.

IoT Device Selection Decision (IDSD): The proposed system considers the
following levels for IoT device selection:
• Extremely Low Selection Possibility (ELSP) - The IoT device will have an
extremely low probability to be selected.
• Very Low Selection Possibility (VLSP) - The IoT device will have very low
probability to be selected.
 A Fuzzy-Based System for Selection of IoT Devices in OppNets 257

• Low Selection Possibility (LSP) - There might be other IoT devices which
can do the job better.
• Middle Selection Possibility (MSP) - The IoT device is ready to be assigned
a task, but is not the “chosen” one.
• High Selection Possibility (HSP) - The IoT device takes responsibility of
completing the task.
• Very High Selection Possibility (VHSP) - The IoT device has almost all the
required information and potential to be selected and then allocated in an
appropriate position to carry out a job.
• Extremely High Selection Possibility (EHSP) - The IoT device has all the
required information and the possibility of an IoT device to be selected is
extremely high.

Table 1. Parameters and their term sets for FLC.

Parameters Term sets

IoT Device’s Selfish Behaviour (IDSB) Rarely (Rl), Often (Of), Very Often (VO)
IoT Device Remaining Energy (IDRE) Low (Lo), Medium (Mdm), High (Hgh)
IoT Device Storage (IDST) Small (Sm), Medium (Me), High (Hi)
IoT Device Contact Duration (IDCD) Short (Sho), Medium (Med), Long (Lg)
IoT Device Selection Decision (IDSD) Extremely Low Selection Possibility
(ELSP), Very Low Selection Possibility
(VLSP), Low Selection Possibility (LSP),
Medium Selection Possibility (MSP), High
Selection Possibility (HSP), Very High
Selection Possibility (VHSP), Extremely
High Selection Possibility (EHSP)

3.2 System Implementation

Fuzzy sets and fuzzy logic have been developed to manage vagueness and uncer-
tainty in a reasoning process of an intelligent system such as a knowledge based
system, an expert system or a logic control system [15–28]. In this work, we use
fuzzy logic to implement the proposed system.
The structure of the proposed system is shown in Fig. 3. It consists of one
Fuzzy Logic Controller (FLC), which is the main part of our system and its basic
elements are shown in Fig. 4. They are the fuzzifier, inference engine, Fuzzy Rule
Base (FRB) and defuzzifier.
As shown in Fig. 5, we use triangular and trapezoidal membership functions
for FLC, because they are suitable for real-time operation [29]. The x0 in f (x)
is the center of triangular function, x0 (x1 ) in g(x) is the left (right) edge of
trapezoidal function, and a0 (a1 ) is the left (right) width of the triangular or
trapezoidal function. We explain in details the design of FLC in following.
258 M. Cuka et al.

Table 2. FRB.

No IDSB IDRE IDST IDCD IDSD No IDSB IDRE IDST IDCD IDSD
1 Rl Lo Sm Sho VLSP 41 Of Mdm Me Med vHSP
2 Rl Lo Sm Med MSP 42 Of Mdm Me Lg LSP
3 Rl Lo Sm Lg VLSP 43 Of Mdm Hi Sho MSP
4 Rl Lo Me Sho LSP 44 Of Mdm Hi Med VHSP
5 Rl Lo Me Med VHSP 45 Of Mdm Hi Lg MSP
6 Rl Lo Me Lg LSP 46 Of Hgh Sm Sho LSP
7 Rl Lo Hi Sho MSP 47 Of Hgh Sm Med VHSP
8 Rl Lo Hi Med VHSP 48 Of Hgh Sm Lg LSP
9 Rl Lo Hi Lg MSP 49 Of Hgh Me Sho MSP
10 Rl Mdm Sm Sho MSP 50 Of Hgh Me Med VHSP
11 Rl Mdm Sm Med VHSP 51 Of Hgh Me Lg HSP
12 Rl Mdm Sm Lg MSP 52 Of Hgh Hi Sho LSP
13 Rl Mdm Me Sho HSP 53 Of Hgh Hi Med EHSP
14 Rl Mdm Me Med EHSP 54 Of Hgh Hi Lg HSP
15 Rl Mdm Me Lg HSP 55 VO Lo Sm Sho ELSP
16 Rl Mdm Hi Sho VHSP 56 VO Lo Sm Med VLSP
17 Rl Mdm Hi Med EHSP 57 VO Lo Sm Lg ELSP
18 Rl Mdm Hi Lg VHSP 58 VO Lo Me Sho ELSP
19 Rl Hgh Sm Sho HSP 59 VO Lo Me Med LSP
20 Rl Hgh Sm Med EHSP 60 VO Lo Me Lg ELSP
21 Rl Hgh Sm Lg HSP 61 VO Lo Hi Sho ELSP
22 Rl Hgh Me Sho VHSP 62 VO Lo Hi Med MSP
23 Rl Hgh Me Med EHSP 63 VO Lo Hi Lg ELSP
24 Rl Hgh Me Lg VHSP 64 VO Mdm Sm Sho ELSP
25 Rl Hgh Hi Sho EHSP 65 VO Mdm Sm Med MSP
26 Rl Hgh Hi Med EHSP 66 VO Mdm Sm Lg ELSP
27 Rl Hgh Hi Lg EHSP 67 VO Mdm Me Sho VLSP
28 Of Lo Sm Sho ELSP 68 VO Mdm Me Med HSP
29 Of Lo Sm Med VLSP 69 VO Mdm Me Lg VLSP
30 Of Lo Sm Lg ELSP 70 VO Mdm Hi Sho LSP
31 Of Lo Me Sho ELSP 71 VO Mdm Hi Med VHSP
32 Of Lo Me Med MSP 72 VO Mdm Hi Lg MSP
33 Of Lo Me Lg ELSP 73 VO Hgh Sm Sho VLSP
34 Of Lo Hi Sho VLSP 74 VO Hgh Sm Med HSP
35 Of Lo Hi Med HSP 75 VO Hgh Sm Lg VLSP
36 Of Lo Hi Lg VLSP 76 VO Hgh Me Sho LSP
37 Of Mdm Sm Sho VLSP 77 VO Hgh Me Med HSP
38 Of Mdm Sm Med HSP 78 VO Hgh Me Lg LSP
39 Of Mdm Sm Lg VLSP 79 VO Hgh Hi Sho MSP
40 Of Mdm Me Sho LSP 80 VO Hgh Hi Med MSP
81 VO Hgh Hi Lg MSP
 A Fuzzy-Based System for Selection of IoT Devices in OppNets 259

Fig. 6. Fuzzy membership functions.

We use four input parameters for FLC: IoT Device’s Selfish Behaviour
(IDSB), IoT Device Remaining Energy (IDRE), IoT Device Storage (IDST),
IoT Device Contact Duration (IDCD).
The term sets for each input linguistic parameter are defined respectively as
shown in Table 1.

T (IDSB) = {Rarely(Rl), Of ten(Of ), V eryOf ten(V O)}

T (IDRE) = {Low(Lo), M edium(M dm), High(Hgh)}
T (IDST ) = {Small(Sm), M edium(M e), High(Hi)}
T (IDCD) = {Short(Sho), M edium(M ed), Long(Lg)}

The membership functions for input parameters of FLC are defined as:

µRl (IDSB) = g(IDSB; Rl0 , Rl1 , Rlw0 , Rlw1 )

µOf (IDSB) = f (IDSB; Of0 , Ofw0 , Ofw1 )
µV O (IDSB) = g(IDSB; V O0 , V O1 , V Ow0 , V Ow1 )
µLo (IDRE) = g(IDRE; Lo0 , Lo1 , Low0 , Low1 )
µM dm (IDRE) = f (IDRE; M dm0 , M dmw0 , M dmw1 )
µHgh (IDRE) = g(IDRE; Hgh0 , Hgh1 , Hghw0 , Hghw1 )
260 M. Cuka et al.

µSm (IDST ) = g(IDST ; Sm0 , Sm1 , Smw0 , Smw1 )

µM e (IDST ) = f (IDST ; M e0 , M ew0 , M ew1 )
µHi (IDST ) = g(IDST ; Hi0 , Hi1 , Hiw0 , Hiw1 )
µSho (IDCD) = g(IDCD; Sho0 , Sho1 , Show0 , Show1 )
µM ed (IDCD) = f (IDCD; M ed0 , M edw0 , M edw1 )
µLg (IDCD) = g(IDCD; Lg0 , Lg1 , Lgw0 , Lgw1 )

The small letters w0 and w1 mean left width and right width, respectively.
The output linguistic parameter is the IoT device Selection Decision (IDSD).
We define the term set of IDSD as:

{Extremely Low Selection P ossibility (ELSP ),

V ery Low Selection P ossibility (LSP ),
Low Selection P ossibility (LSP ),
M iddle Selection P ossibility (M SP ),
High Selection P ossibility (HSP ),
V ery High Selection P ossibility (V HSP ),
Extremely High Selection P ossibility (EHSP )}.

The membership functions for the output parameter IDSD are defined as:

µELSP (IDSD) = g(IDSD; ELSP0 , ELSP1 , ELSPw0 , ELSPw1 )

µV LSP (IDSD) = f (IDSD; V LSP0 , V LSPw0 , V LSPw1 )
µLSP (IDSD) = f (IDSD; LSP0 , LSPw0 , LSPw1 )
µM SP (IDSD) = f (IDSD; M SP0 , M SPw0 , M SPw1 )
µHSP (IDSD) = f (IDSD; HSP0 , HSPw0 , HSPw1 )
µV HSP (IDSD) = f (IDSD; V HSP0 , V HSPw0 , V HSPw1 )
µEHSP (IDSD) = g(IDSD; EHSP0 , EHSP1 , EHSPw0 , EHSPw1 ).

The membership functions are shown in Fig. 6 and the Fuzzy Rule Base
(FRB) for our system is shown in Table 2.
The FRB forms a fuzzy set of dimensions |T (IDSB)| × |T (IDRE)| ×
|T (IDST )| × |T (IDCD)|, where |T (x)| is the number of terms on T (x). We
have four input paramteres, so our system has 81 rules. The control rules have
the form: IF “conditions” THEN “control action”.

4 Simulation Results
We present the simulation results in Figs. 7, 8 and 9. In these figures, we show
the relation between the probability of an IoT device to be selected (IDSD) to
carry out a task, versus IDSB, IDRE, IDST and IDCD. We consider IDSB and
IDRE as constant parameters and change the values of IDST and IDCD. In
 A Fuzzy-Based System for Selection of IoT Devices in OppNets 261

Fig. 7. Results for different values of IDRE, IDST and IDCD when IDSB = 0.1.

Fig. 7(a), we see the effect of storage on IoT Device Selection Decision for IDSB
0.1 and IDRE 0.1. For IDCD 0.2 and IDST 0.1, IDSD is 0.25. For IDST 0.5,
IDSD is 0.39 and for IDST 0.9, IDSD is 0.5. We see that we have an increase
of 14% and 25% of IDSD for IDST 0.5 and IDST 0.9, respectively. We see this
increase in IDSD because devices with more storage capacity are more likely to
carry the message until there is a contact opportunity. This parameter directly
influences the capacity of OppNets because lower storage limits the amount of
data that can be transferred between nodes.
In Fig. 7(a) and (b), we increase the IDRE value to 0.1 and 0.9, respectively,
while keeping IDSB constant. From the figures we can see that for IDCD 0.5 and
IDST 0.9, IDSD is increased 14%. We see that IoT devices with more remaining
energy, have a higher possibility to be selected for carrying out a job because
they stay connected longer.
Contact duration between two devices has a major impact on selection deci-
sion. In Fig. 8(a), for IDST 0.9, when IDCD is increased from 0.2 to 0.5, IDSD
is increased 36%. But when IDCD further increases from 0.5 to 0.8, IDSD is
decreased 36%. This is because a short time of contact means that two devices
may not have enough time to establish a connection. While, when the contact
time is long, the OppNet loses the mobility.
Another parameter that affects the possibility of the selection of an IoT
device, is IDSB. Selfish behaviour of a device has a negative impact on IDSD,
because it affects network performance and equal distribution of resources among
devices. To see this effect on IDSD, we compare Figs. 8(b) with 7(b) and
Figs. 9(b) with 8(b). Comparing Figs. 8(b) with 7(b), for IDCD 0.8, IDST 0.1
and IDRE 0.9, we see a decrease of 23%. When IDSB increases to 0.5 and 0.9 in
Figs. 8(b) and 9(b), IDSD is decreased 15%.
262 M. Cuka et al.

Fig. 8. Results for different values of IDRE, IDST and IDCD when IDSB = 0.5.

Fig. 9. Results for different values of IDRE, IDST and IDCD when IDSB = 0.9.

5 Conclusions and Future Work

In this paper, we proposed and implemented a fuzzy-based IoT device selection

system for OppNets considering the failure rate of an IoT device.
We evaluated the proposed system by computer simulations. The simulation
results show that less selfish IoT devices, are more likely to be selected for
carrying out a job, so with the decrease of IDSB the possibility of an IoT device
to be selected increases.
We can see that by increasing IDST and IDRE, the IDSD is also increased.
But for the IDCD parameter, we need to find an optimal time that IoT devices
have contact with each other.
In the future work, we will also consider other parameters for IoT device
selection such as Node Computational Time, Interaction Probability and make
extensive simulations to evaluate the proposed system.
 A Fuzzy-Based System for Selection of IoT Devices in OppNets 263

References
1. Mantas, N., Louta, M., Karapistoli, E., Karetsos, G.T., Kraounakis, S., Obaidat,
M.S.: Towards an incentive-compatible, reputation-based framework for stimulat-
ing cooperation in opportunistic networks: a survey. IET Netw. 6(6), 169–178
(2017)
2. Sharma, D.K., Sharma, A., Kumar, J., et al.: KNNR: K-nearest neighbour classi-
fication based routing protocol for opportunistic networks. In: 10th International
Conference on Contemporary Computing (IC3), pp. 1–6. IEEE (2017)
3. Kraijak, S., Tuwanut, P.: A survey on internet of things architecture, protocols, pos-
sible applications, security, privacy, real-world implementation and future trends.
In: 16th International Conference on Communication Technology (ICCT), pp. 26–
31. IEEE (2015)
4. Arridha, R., Sukaridhoto, S., Pramadihanto, D., Funabiki, N.: Classification exten-
sion based on IoT-big data analytic for smart environment monitoring and analytic
in real-time system. Int. J. Space-Based Situated Comput. 7(2), 82–93 (2017)
5. Dhurandher, S.K., Sharma, D.K., Woungang, I., Bhati, S.: HBPR: history based
prediction for routing in infrastructure-less opportunistic networks. In: 27th
International Conference on Advanced Information Networking and Applications
(AINA), pp. 931–936. IEEE (2013)
6. Spaho, E., Mino, G., Barolli, L., Xhafa, F.: Goodput and PDR analysis of AODV,
OLSR and DYMO protocols for vehicular networks using CAVENET. Int. J. Grid
Utility Comput. 2(2), 130–138 (2011)
7. Abdulla, M., Simon, R.: The impact of intercontact time within opportunistic
networks: protocol implications and mobility models. TechRepublic White Paper
(2009)
8. Karakostas, G., Markou, E.: Emergency connectivity in ad-hoc networks with self-
ish nodes. In: Latin American Symposium on Theoretical Informatics, pp. 350–361.
Springer (2008)
9. Popereshnyak, S., Suprun, O., Suprun, O., Wieckowski, T.: IoT application testing
features based on the modelling network. In: The 14th International Conference
on Perspective Technologies and Methods in MEMS Design (MEMSTECH), pp.
127–131. IEEE (2018)
10. Chen, N., Yang, Y., Li, J., Zhang, T.: A fog-based service enablement architecture
for cross-domain IoT applications. In: 2017 IEEE Fog World Congress (FWC), pp.
1–6. IEEE (2017)
11. Pozza, R., Nati, M., Georgoulas, S., Moessner, K., Gluhak, A.: Neighbor discovery
for opportunistic networking in internet of things scenarios: a survey. IEEE Access
3, 1101–1131 (2015)
12. Akbas, M., Turgut, D.: APAWSAN: actor positioning for aerial wireless sensor and
actor networks. In: IEEE 36th Conference on Local Computer Networks (LCN-
2011), pp. 563–570, October 2011
13. Akbas, M., Brust, M., Turgut, D.: Local positioning for environmental monitoring
in wireless sensor and actor networks. In: IEEE 35th Conference on Local Computer
Networks (LCN 2010), pp. 806–813, October 2010
14. Melodia, T., Pompili, D., Gungor, V., Akyildiz, I.: Communication and coordi-
nation in wireless sensor and actor networks. IEEE Trans. Mob. Comput. 6(10),
1126–1129 (2007)
264 M. Cuka et al.

15. Inaba, T., Sakamoto, S., Kolici, V., Mino, G., Barolli, L.: A CAC scheme based on
fuzzy logic for cellular networks considering security and priority parameters. In:
The 9th International Conference on Broadband and Wireless Computing, Com-
munication and Applications (BWCCA 2014), pp. 340–346 (2014)
16. Spaho, E., Sakamoto, S., Barolli, L., Xhafa, F., Barolli, V., Iwashige, J.: A fuzzy-
based system for peer reliability in JXTA-overlay P2P considering number of inter-
actions. In: The 16th International Conference on Network-Based Information Sys-
tems (NBiS 2013), pp. 156–161 (2013)
17. Matsuo, K., Elmazi, D., Liu, Y., Sakamoto, S., Mino, G., Barolli, L.: FACS-MP: a
fuzzy admission control system with many priorities for wireless cellular networks
and its performance evaluation. J. High Speed Netw. 21(1), 1–14 (2015)
18. Grabisch, M.: The application of fuzzy integrals in multicriteria decision making.
Eur. J. Oper. Res. 89(3), 445–456 (1996)
19. Inaba, T., Elmazi, D., Liu, Y., Sakamoto, S., Barolli, L., Uchida, K.: Integrating
wireless cellular and ad-hoc networks using fuzzy logic considering node mobility
and security. In: The 29th IEEE International Conference on Advanced Information
Networking and Applications Workshops (WAINA 2015), pp. 54–60 (2015)
20. Kulla, E., Mino, G., Sakamoto, S., Ikeda, M., Caballé, S., Barolli, L.: FBMIS: a
fuzzy-based multi-interface system for cellular and ad hoc networks. In: Interna-
tional Conference on Advanced Information Networking and Applications (AINA
2014), pp. 180–185 (2014)
21. Elmazi, D., Kulla, E., Oda, T., Spaho, E., Sakamoto, S., Barolli, L.: A comparison
study of two fuzzy-based systems for selection of actor node in wireless sensor actor
networks. J. Ambient Intell. Hum. Comput. 6(5), 635–645 (2015)
22. Zadeh, L.: Fuzzy logic, neural networks, and soft computing. ACM Commun. 37(3),
77–85 (1994)
23. Spaho, E., Sakamoto, S., Barolli, L., Xhafa, F., Ikeda, M.: Trustworthiness in P2P:
performance behaviour of two fuzzy-based systems for JXTA-overlay platform. Soft
Comput. 18(9), 1783–1793 (2014)
24. Inaba, T., Sakamoto, S., Kulla, E., Caballe, S., Ikeda, M., Barolli, L.: An integrated
system for wireless cellular and ad-hoc networks using fuzzy logic. In: International
Conference on Intelligent Networking and Collaborative Systems (INCoS 2014), pp.
157–162 (2014)
25. Matsuo, K., Elmazi, D., Liu, Y., Sakamoto, S., Barolli, L.: A multi-modal simula-
tion system for wireless sensor networks: a comparison study considering stationary
and mobile sink and event. J. Ambient Intell. Hum. Comput. 6(4), 519–529 (2015)
26. Kolici, V., Inaba, T., Lala, A., Mino, G., Sakamoto, S., Barolli, L.: A fuzzy-based
CAC scheme for cellular networks considering security. In: International Conference
on Network-Based Information Systems (NBiS 2014), pp. 368–373 (2014)
27. Liu, Y., Sakamoto, S., Matsuo, K., Ikeda, M., Barolli, L., Xhafa, F.: A comparison
study for two fuzzy-based systems: improving reliability and security of JXTA-
overlay P2P platform. Soft Comput. 20(7), 2677–2687 (2015)
28. Matsuo, K., Elmazi, D., Liu, Y., Sakamoto, S., Mino, G., Barolli, L.: FACS-MP: a
fuzzy admission control system with many priorities for wireless cellular networks
and its perforemance evaluation. J. High Speed Netw. 21(1), 1–14 (2015)
29. Mendel, J.M.: Fuzzy logic systems for engineering: a tutorial. Proc. IEEE 83(3),
345–377 (1995)
 Detecting Mental Health Illness Using
Short Comments

Takahiro Baba1(B) , Kensuke Baba2 , and Daisuke Ikeda1

1
Kyushu University, Fukuoka 819-0395, Japan
[email protected]
2
Fujitsu Laboratories, Kawasaki 211-8588, Japan

Abstract. Mental health illness has become a serious public problem.

Finding changes in everyday behavior is a demand. This paper tries to
detect persons who have mental health illness using their short comments
posted to social network systems. The novelty of this study is using com-
ments in a system for communication between users with mental health
illness, in order to prepare a sufficient amount of supervised data for
machine learning. The authors used approximately 120,000 comments in
the system as positive samples and 120,000 comments in Twitter as neg-
ative samples for detecting mental health illness. Both data are posted
short comments on a daily basis. The authors conducted a straightfor-
ward classification of the comments using a support vector machine and
surface-level features of the comments. The accuracy of the classification
is 0.92 and the characteristic phrases used for the classification are related
to troubles in mental health. The ability to classify everyday statements
can be expected to lead to the early detection of mental disorders.

Keywords: Mental health illness · Social network system ·

Machine learning · Data mining · Text processing

1 Introduction
Mental health illness, especially depression, has become a serious public problem.
According to statistics released by the World Health Organization (WHO) [2],
more than 350 million people suffer from depression. Many people do not think
that their disorder is caused by depression, and sometimes they are in a serious
condition when they are noticed. Therefore, we need technologies for automatic
detection of mental health illness including depression. Especially, detecting men-
tal health in its early stages is a demand. Some kinds of the illness are expected
to be detected from changes in everyday behavior which is observed using social
network system (SNS).
The target problem of this study is detecting mental health illness using short
comments posted to SNSs. A number of existing studies have similar approaches
to the problem. Wongkoblep et al. [9] explore the range and limit of state-of-the-
art technologies used by researchers in predictive analysis of mental health, and
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 265–271, 2020.
https://doi.org/10.1007/978-3-030-15032-7_23
266 T. Baba et al.

consider relevant issues such as ethical concerns in research in this field. They
carefully analyzed the titles, abstracts, and main texts for 48 papers. Articles
were coded based on key features, technology used for data collection, data pre-
processing, feature extraction, feature selection, model construction, and model
validation. Guntuku et al. [5] review recent research aimed at predicting psychi-
atric disorders using SNSs. Symptoms associated with psychosis are observable
on Twitter [4], Facebook [3], and other Web forums, and automated methods
are increasingly able to detect depression and other psychoses.
The novelty of our study is that we used comments posted to a Web system
for communication between users who are expected to have illness with their
mental health. We tried to detect mental health illness by applying machine
learning to the comments and other usual comments in Twitter, which solved
the problem of preparing a sufficient amount of suitable supervised data. Existing
studies require confirmation of patients with mental health illness for obtaining
training data. Reece et al. [8] investigated a calculation model to predict the
appearance of depression and post-traumatic stress disorder of Twitter users.
Details of Twitter data and depression history were collected from 204 people
(105 in depressed state and 99 in health). From 279,951 comments of the partici-
pants, they extracted the prediction function which measures influence, language
style, and context, and then constructed these models using supervised learning
algorithm.
We conducted a classification of short comments posted to two SNSs for
detecting mental health illness. We used comments in a Web system Cocooru [1]
as positive samples, that is, comments posted by persons who have mental health
illness, and comments in Twitter as negative samples. In the classification, we
used a standard method; we applied the support vector machine (SVM) with a
linear kernel to surface-level features of the comments. Additionally, we extracted
which words were used as feature words.

2 Methods

This section formalizes the problem we tackle for detecting mental health ill-
ness and proposes a solution for the problem. This section also describes the
experimental methods for evaluating the method.

2.1 Data

We reduced the detection of mental health illness to a classification of short com-

ments in two SNSs. We used comments posted to Cocooru and Twitter for our
experiments. Cocooru is a Japanese Web site that provides users with counsel-
ing for mental health from clinical psychologists, and has a Twitter-like message
board system for communication between users. Therefore, the comments in the
system of Cocooru are supposed to be posted by persons who have troubles with
their mental health, while the comments in Twitter are about general topics.
 Detecting Mental Health Illness Using Short Comments 267

In twitter and cocooru, casual remarks are posted on a regular basis. There-
fore, if these statements can be categorized, it becomes possible to judge mental
disorders from everyday speech, and it can be expected to lead to early detection.
We generated two sets of short comments for the classification. We gathered
the total comments posted to Cocooru as of September, 2018, as positive data.
As for negative data, we randomly gathered comments written in Japanese from
Twitter. The number of the comments in the positive and negative data are
118,551 and 116,561, respectively. The average length (number of characters) of
the comments in the positive and negative data are 57.06 and 75.36, respectively.
The accuracy of a naive classification with the two data set using the length and
the optimized threshold was approximately 0.5.

2.2 Classification

We conducted a classification of the short comments using their surface-level

features. We used the character-level 1-, 2-, 3-, and 4-grams of comments as
features for the classification, where an n-gram of a sequence is a sequence of
contiguous n elements of the sequence. For example, the 1- and 2-grams of a
comment “YES” are the strings “Y”, “E”, “S”, “YE”, and “ES”.
The accuracy measures for our experiments are defined as follows. The accu-
racy of a classification is defined to be the ratio of the number of the correct
predictions to the number of the total predictions examined in a validation. The
precision and the recall are defined, for each class, to be the ratio of the num-
ber of the correct predictions to a class to the number of the predictions to the
class, and the ratio of the number of the correct predictions to a class to the
actual number of samples of the class, respectively. The F1-score for a class is
the harmonic mean of the precision and the recall for the class.

3 Results

Table 1 shows the accuracy generated by the classification of short comments.

As shown in the table, the classification could be conducted with high accuracy.
Figures 1, 2, and 3 show the changes of precision, recall, and F1-score against
the cost parameter C of the linear SVM. C represents the allowable amount
of misclassification. The smaller the value of C is, the more misclassification is
permitted, and the larger C is, the more superficial it is determined not to allow
misclassification.
Table 2 shows distinctive strings of the comments posted by persons with
mental health illness. The listed strings correspond to the top 20 elements of
the separating hyperplane used in the classifier in the order of the coefficients.
Therefore, the strings are supposed to be distinctive of the positive data. Some
of the strings do not completely correspond to their translation because we
conducted the character-level analysis.
268 T. Baba et al.

Table 1. Accuracy of the classification of short comments posted in two Web systems.

Dataset Precision Recall F1-score

Positive (Cocooru) 0.92 0.93 0.92
Negative (Twitter) 0.93 0.92 0.92
Total 0.92 0.92 0.92

Fig. 1. Precision against the cost parameter of the linear SVM.

4 Discussion
4.1 Main Conclusion
We found that persons who have mental health illness can be distinguished from
other persons using their comments posted to SNSs. The comments obtained
from Twitter and Cocooru were classified with the accuracy 0.92. The precision
of positive comments is 0.81 when the recall is 0.99. On the assumption that the
positive comments were posted by persons with mental health illness, we can
accurately find the persons. Additionally, as shown in Table 2, the distinctive
phrases of the positive data used in the classification are related to troubles in
mental health. Also, by being able to classify with a short comment that is said
on a daily basis, it can be expected to lead to early detection of mental disorders.

4.2 Future Directions

The accuracy of the classification is expected to be improved by using vector
representation of words which includes meanings of words. Some kinds of vector
representation of words can be obtained by using machine learning from actual
documents [6] instead of using man-made knowledge, such as dictionaries.
Using the kind of vector representation of words is also expected to extend
the target of our analysis to other data. We can apply our method to documents
 Detecting Mental Health Illness Using Short Comments 269

Fig. 2. Recall against the cost parameter of the linear SVM.

written in other languages and other kinds of data including images using vector
representation of words obtained from suitable training data. Reece and Dan-
forth [7] report that mental health of a person can be estimated using photos
uploaded by the person in SNS.
Predicting mental health illness in its early stages is also one of our future
work. We can put scores, which represent a continuous degree of mental health,
to comments using the classifier used in the classification. Predicting some kinds
of changes in mental health is possible by analyzing multiple comments ordered
as a time series with the scores.

Fig. 3. F1-score against the cost parameter of the linear SVM.

270 T. Baba et al.

Table 2. Distinctive phrases of positive data for detecting mental health illness.

5 Conclusion
Persons who have illness with their mental health can be detected using their
short comments posted to social network systems. We classified comments
obtained from two systems, one is for communication between users with men-
tal health illness. The classification accuracy was 0.92 using a standard method
with the bag-of-words model and a support vector machine. Therefore, it can be
found that it is possible to classify as having mental disorders with daily short
comments, which may lead to early detection. Additionally, we found that the
distinctive phrases of the positive data were related to troubles in mental health.

Acknowledgement. This work was supported by JSPS KAKENHI Grant Number

15H02787.

References
1. Cocooru. https://cocooru.com. Accessed 13 Sept 2018
2. Depression, World Health Organization. http://www.who.int/en/news-room/fact-
sheets/detail/depression. Accessed Nov 2018
3. Facebook. https://www.facebook.com. Accessed 6 Dec 2018
4. Twitter. https://twitter.com. Accessed 13 Sept 2018
 Detecting Mental Health Illness Using Short Comments 271

5. Guntuku, S.C., Yaden, D.B., Kern, M.L., Ungar, L.H., Eichstaedt, J.C.: Detecting
depression and mental illness on social media: an integrative review. Curr. Opin.
Behav. Sci. 18, 43–49 (2017). Big data in the behavioural sciences
6. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representa-
tions of words and phrases and their compositionality. In: Proceedings of the 26th
International Conference on Neural Information Processing Systems, NIPS 2013,
vol. 2, pp. 3111–3119. Curran Associates Inc., USA (2013)
7. Reece, A.G., Danforth, C.M.: Instagram photos reveal predictive markers of depres-
sion. EPJ Data Sci. 6(1), 15 (2017)
8. Reece, A.G., Reagan, A.J., Lix, K.L.M., Dodds, P.S., Danforth, C.M., Langer, E.J.:
Forecasting the onset and course of mental illness with twitter data. Sci. Rep. 7(1)
(2017)
9. Wongkoblap, A., Vadillo, M.A., Curcin, V.: Researching mental health disorders in
the era of social media: systematic review. J. Med. Internet Res. 19(6), e228 (2017)
 On the Weakest Failure Detector
for Read/Write-Based Mutual Exclusion

Carole Delporte-Gallet1 , Hugues Fauconnier1 , and Michel Raynal2,3(B)

1
IRIF, Université Paris 7 Diderot, Paris, France
2
IRISA, Université de Rennes, 35042 Rennes, France
[email protected]
3
Department of Computing, Polytechnic University, Kowloon, Hong Kong

Abstract. Failure detectors are devices (objects) that provides the pro-
cesses with information on failures. They were introduced to enrich
asynchronous systems so that it becomes possible to solve problems (or
implement concurrent objects) that are otherwise impossible to solve in
pure asynchronous systems where processes are prone to crash failures.
The most famous failure detector (which is called “eventual leader” and
denoted Ω) is the weakest failure detector which allows consensus to be
solved in n-process asynchronous systems where up to t = n−1 processes
may crash in the read/write communication model, and up to t < n/2
processes may crash in the message-passing communication model.
When looking at the mutual exclusion problem (or equivalently the
construction of a lock object), while the weakest failure detectors are
known for both asynchronous message-passing systems and read/write
systems in which up to t < n processes may crash, for the starvation-
freedom progress condition, it is not yet known for weaker deadlock-
freedom progress condition in read/write systems. This paper extends
the previous results, namely, it presents the weakest failure detector
that allows mutual exclusion to be solved in asynchronous n-process
read/write systems where any number of processes may crash, whatever
the progress condition (deadlock-freedom or starvation-freedom).

1 Introduction

Concurrent Objects. When considering multiprocess programming, concur-

rent objects are the objects that can be simultaneously accessed by several pro-
cesses. Examples of such objects are the classical objects encountered in sequen-
tial computing (such as stacks, queues, graphs, sets, trees, etc.) now shared by
several processes to communicate and cooperate to a common goal, and objects
targeting new concurrency-related issues (such as rendezvous and non-blocking
atomic commitment objects). When there are no failures the implementation of
such objects are usually based on underlying locks, which can be built from base
read/write or read/modify/write registers (see e.g., [18,20]).
In an n-process failure-prone system (where a failure is a process crash),
the situation is different, and many concurrent objects (as simple as stacks
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 272–285, 2020.
https://doi.org/10.1007/978-3-030-15032-7_24
 On the Weakest Failure Detector for Read/Write-Based Mutual Exclusion 273

and queues) can no longer be implemented. This impossibility follows from

the famous impossibilities to build a consensus object in the presence of asyn-
chronous and process crashes [13,17] (pedagogical presentations of these results
can be found in textbooks such as [18,20]).
Impossibility Results and Failure Detectors. Several approaches have been
proposed and investigated to circumvent the previous impossibilities. One of
them, which is system-oriented, consists in enriching the system with failure-
related objects providing each process individually with information on failures.
This is the failure detector-based approach introduced in [5]. More precisely, a
failure detector provides each process with one or several read-only local vari-
ables, containing information on failures. According to the type and the quality
of this information, different failure detector classes can be defined. As a simple
example, the class of perfect failure detectors (denoted P ) provides each process
pi with a read-only set suspectedi that (i) never contains a process identity
before it crashes, and (ii) eventually contains the identities of all the processes
that crashed. It is easy to see that a perfect failure detector allows a process that
does not crash not to remain blocked forever because another process is crashed.
A fundamental notion associated with failure detectors is the notion of weak-
est failure detector for a given concurrent object. Intuitively, weakest failure
detector means that, a failure detector D is the weakest failure detector to imple-
ment an object O, if (1) there is a D-based algorithm implementing O, and (2)
any other failure detector that allows us to implement O, provides each process
with enough information on failures that allows to build D.
The weakest failure detector notion was introduced in [4], where it is shown
that the eventual leader failure detector (denoted Ω) is the weakest failure detec-
tor which allows consensus to be implemented in an n-process asynchronous
message-passing system where up to t < n/2 processes may crash. Ω provides
each process pi with a read-only variable leaderi such that, after an arbitrarily
long but finite time, the variables leaderi of the non-crashed processes contain
the same process identity, which is the identity of a non-crashed process. Before
this time occurs, the variables leaderi can contain different, and varying with
time, process identities. It is shown in [16] that Ω allows consensus to be solved in
asynchronous read/write systems prone to any number t < n of process crashes.

Table 1. Weakest failure detectors cited in this article

Crash-prone model Atomic register Consensus Starvation-free mutual exclusion

Shared memory Given for free Ω [4] Γ 1 [2]
Message-passing with t < n/2 ∃ algorithms Ω [16] T (Trusting) [9]
Message-passing with t < n Quorums Σ [7] Σ + Ω [8] T + Σ [1]

Another important failure detector is the quorum failure detector introduced

in [7]. Denoted Σ, this failure detector provides each process pi with a read-only
set variable quorumi that always contains the identity of a (possibly changing)
274 C. Delporte-Gallet et al.

correct process, and such that any pair of quorums, each taken at any time,
always intersect. It is shown in [7] that (i) Σ is the weakest failure detector to
implement a read/write register in asynchronous message-passing systems prone
to any number t < n of process crashes, and (ii) the pair (Ω, Σ) is the weakest
failure detector to implement consensus in asynchronous message-passing sys-
tems prone to any number of process crashes t < n. (Implementation of failure
detectors in asynchronous crash-prone read/write and message-passing systems
can be found in [12,18,19]. All these implementations rely on underlying behav-
ioral assumptions.) Table 1 presents a global view of failure detectors cited in the
present article. The failure detector T (trusting) is the weakest failure detector
for starvation-free mutual exclusion in asynchronous message-passing systems
where a majority of processes do not crash. In [1], this extended with appropri-
ate quorums (Σ) to obtain the weakest failure detector for starvation-free mutual
exclusion in message-passing systems where any number of processes may crash.
Content of the Paper. As shown in Table 1, while the weakest failure detec-
tors for starvation-free mutual exclusion are known for both crash-prone asyn-
chronous message-passing systems and read/write systems, it is not known when
we consider the weaker deadlock-freedom progress condition. This article answers
this question. It presents a simple failure detector, denoted QP , and shows that
it is the weakest for both deadlock-freedom and starvation-freedom. The article
is composed of two main parts.

– The first is the presentation of the failure detector QP (Quasi-Perfect) that

can be used to implement a starvation-free (hence deadlock-free) lock object
despite any number of process crashes. To this end, a version of Lam-
port’s Bakery mutual exclusion algorithm [15] (based on underlying atomic
read/write registers) is extended with this failure detector.
– The second is the proof that QP is necessary to implement a lock object
despite any number of process crashes, for the deadlock-freedom (hence star-
vation free) progress conditions.
– It follows from the previous items that QP is the weakest failure detector
to implement a lock object despite any number of process crashes, for both
the deadlock-freedom and starvation-freedom progress conditions. A direct
consequence is that, while the definitions of Γ 1 (as defined in [2]) and QP are
different, they are equivalent (in the sense they have the same computability
power).

Roadmap. The paper is made up of 7 sections. Section 2 introduces the basic

crash-prone read/write model, and Sect. 3 presents the quasi-perfect failure
detector QP . Then Sect. 4 show how crash-prone mutual exclusion can be solved
with the help of QP , while Sect. 5 shows that QP is the weakest information on
failure which allow to build a crash-prone deadlock-free mutual exclusion (lock)
object. Section 6 briefly discusses the respective computability power of QP and
Ω. Finally, Sect. 7 concludes the paper.
 On the Weakest Failure Detector for Read/Write-Based Mutual Exclusion 275

2 Basic Computing Model

The system is made up of a set Π of n sequential asynchronous processes denoted
p1 , ..., pn . “Asynchronous” means that each process proceeds at its own speed,
which can vary with time and remains always unknown to processes.
The processes communicate by accessing a shared read/write memory made
up of atomic read/write registers. From 0 to (n − 1) processes can commit crash
failures. A process commits a crash when it halts prematurely. Before halting
(if it ever halts), a process executes correctly its algorithm. After it crashed,
it executes no more steps. Given an execution, a process that crashes in this
execution is said to be faulty. Otherwise, it is correct.
The time used to specify the failure pattern, and the behavior of the failure
detector, is a conceptual time perceived by an omniscient external observer.
This time is not accessible by the processes. For a time τ , let F(τ ) be the set of
processes
  crashed at time τ , and thus the set of correct processes C is
that are
Π \ ∪τ >0 F(τ ) .
From a notational point of view, atomic read/write registers are denoted with
uppercase letters. Differently, local variables of each process are denoted with
lowercase letters.

3 The Quasi-Perfect Failure Detector

3.1 Definition
The notion of a failure detector was introduced in [5]. A failure detector is a
device that provides processes with information on failures. The failure detector
QP , introduced in [10], provides each process pi with two sets of processes,
denoted trustedi and crashedi . These sets are both initially empty, and are
then subsets of Π. Let initi = Π\ (trustedi ∪crashedi ) (hence, initially initi
= Π).
Intuition. Intuitively, the aim of the set trustedi is to eventually contain
the set of correct processes, while the aim of the set initi ∪ crashedi is to
eventually contain the set of faulty processes.
The set initi can only decrease, while the set crashedi can only increase.
Moreover, QP is such that, at any time, we have trustedi ∩ crashedi = ∅.

TRUSTED i

INIT i CRASHED i

Fig. 1. Transitions of the failure detector QP at process pi

Let us consider a process pi , while pj is any process. As previously indicated,

initially j ∈ initi . If pj is correct, or before it crashes (if it is faulty), pj can
276 C. Delporte-Gallet et al.

move from initi to trustedi . Moreover, if it is correct, pj will go and remain

forever in trustedi . If it crashes, pj either remains forever in initi , or goes
directly from initi to crashedi , or goes from trustedi to crashedi . Figure 1
describes this behavior (at each process pi , initially initi contains all process
identities; then a process identity can move along the arrows.)
By convention, if a process pi crashes at time τ , its failure detector outputs
after time τ are both the set trustedi (τ ) and the set crashedi (τ ).
Formal Definition. Formally, the behavior of QP is defined by the following
properties, for every process pi . To simplify the reading of notations, we write
k ∈ seti instead of pk ∈ seti , where seti is trustedi , crashedi , F(τ ), or C.
– ∀ i, ∀ τ : trustedi (τ ) ∩ crashedi (τ ) = ∅.
– ∀ i: j ∈ trustedi (τ ) ⇒ 
  
∀ τ ≥ τ : j ∈ trusted i (τ ) ∪ crashed i (τ ) ∧ 
∀ k ∈ C : ∃τ  : j ∈ trustedk (τ  ) ∪ crashedk (τ  ) .
(A trusted process has to be eventually observed by all correct processes.)
– j ∈ crashedi (τ ) ⇒ j ∈ F(τ ). (crashedi contains  only crashed processes.)
– j ∈ crashedi (τ ) ⇒ ∀ τ  ≥ τ : j ∈ crashedi (τ  ) . (crashes are stable.)

Moreover, if pi is correct:
 
– j ∈ F(τ ) ⇒ ∃τ  ≥ τ : j ∈ trustedi (τ  ) . (Eventually, no faulty process ∈
trustedi .) 
– j ∈ C ⇒ ∃τ : j ∈ trustedi (τ ) . (Eventually, every correct process ∈
trustedi .)
It is easy to see that a correct process pj never appears in the set crashedi .
The failure detector QP is realistic [6] (which means that it can be implemented
with appropriately defined underlying synchrony assumptions).

3.2 QP with Respect to P and 3P

Failure Detector Reductions. A failure detector D is weaker than a failure
detector D (denoted D  D ) if there is a reduction algorithm from D to D,
i.e, an algorithm based on D whose outputs satisfy the properties of D. If D is
weaker than D , any problem that can be solved with D can be solved with D .
If D  D but D  D, D is strictly weaker than D (D ≺ D ).
QP with respect to P and 3P Both the perfect failure detector P and the
eventually perfect failure detector 3P were introduced in [5]. Each outputs at
each process pi a set of processes, denoted suspectedi . The lists output by
the perfect failure detector P satisfy the following properties: Strong Accuracy:
no process belongs to suspectedi before it crashes, and Strong Completeness:
eventually every process that crashes belongs forever to suspectedi .
The sets suspectedi output by 3P satisfy a weakened version of P , namely,
its accuracy and completeness properties are required to hold eventually only.
The power of a perfect failure detector is investigated in [14], and the following
theorem is proved in [10].
 On the Weakest Failure Detector for Read/Write-Based Mutual Exclusion 277

Theorem 1. 3P ≺ QP ≺ P .
Remark. Let us notice that P and QP are very close. Considering P , let
othersi = Π\ suspectedi . Initially othersi = Π, and then this set can only
decrease, while suspectedi can only increase.
When considering QP , a local set initi contains initially all the processes,
and can then only decrease. But a faulty process that does not participate in the
computation can remain forever in this set. This is a main difference between P
and QP , which makes QP weaker than P .
It follows that, from a practical point of view, despite not being the weakest,
P is a good candidate to solve crash-prone mutual exclusion.

4 Mutual Exclusion with the Help of the Failure

Detector QP
4.1 Mutual Exclusion
Mutual exclusion is the oldest (and one of the most important) concurrency-
related problem. Formalized by Dijkstra in the mid-sixties [11], it consists in
building what is called a lock (or mutex) object, defined by two operations,
denoted entry() and exit(). (Recent textbooks including mutual exclusion and
variants of it are [18–20].)
The invocation of these operations by a process pi always follows the following
pattern: “entry(); critical section; exit()”, where “critical section” is any sequence
of code. A process that is not in the critical section and has no pending entry()
or exit() invocation, is said to be in the remainder section. A mutual exclusion
object (also called a lock) is defined by the following properties.
– Mutual exclusion: No two processes are simultaneously in their critical
section.
– Deadlock-freedom: If a correct process pi has a pending entry() operation
and no process is in the critical section, eventually some process pj (possibly
pj = pi ) returns from its entry() operation.
– Wait-free exit: If a correct process invokes exit(), it returns from its invocation.

In the failure model considered in this paper, it is possible that a process

crashes while it is in the critical section. It is assumed that such a crash implicitly
releases of the critical section. (Without such an assumption, the crash of a
process inside a critical section would prevent any other process from entering
the critical section.) Hence, from a conceptual point view, no crashed process is
inside the critical section (it is the role of QP to inform the other processes of
it).
While some mutual exclusion algorithms consider the deadlock-freedom
progress condition, other mutual exclusion algorithms consider the stronger
starvation-freedom progress condition, namely, if a correct process invokes the
operation entry(), it eventually returns from its invocation (i.e., it enters its
critical section).
278 C. Delporte-Gallet et al.

Fig. 2. QP -based version of Lamport’s bakery mutual exclusion algorithm

4.2 Enriching Lamport’s Bakery Mutual Exclusion Algorithm

Lamport’s Bakery Mutual Exclusion Algorithm. This section shows

how, assuming underlying atomic read/write registers, Lamport’s Bakery algo-
rithm [15] can be enriched with access to the failure detector QP to build a
lock object which copes with asynchrony and any number of process crashes.
Lamport’s basic algorithm is pretty simple and elegant. It appears in Fig. 2 from
which the boxes at line 1, line 6, and line 7 are suppressed. When a process pi
wants to enter the critical section it first computes a label (its ticket number) at
line 3. As this computation is asynchronous, pi informs the other processes it is
computing its ticket value by raising its local flag FLAG[i] (lines 2 and 4). The
order on the labels is the classical lexicographical ordering on pairs of integers.
When this “doorway” statements are executed, pi waits until it is the pro-
cess with the smallest labels among all the non-crashed processes (waiting
room implemented by lines 5–8). (Pedagogical presentations and proofs of Lam-
port’s Bakery algorithm can be found in [18,20].) This algorithm satisfies the
starvation-freedom property.
Exploiting the Power Provided by the Failure DetectorQP . The enrich-
ment appears in the three boxes. First, when pi invokes entry(), it waits until its
local failure detector module trusts it (line 1). This ensures that, if pi is correct,
it will appear in all the sets trustedj , and, if it crashes, it will eventually appear
in all the sets crashedj (before crashing it was then in initj or trustedj ).
The two other boxes (lines 6–7, allows a process not to be blocked forever by a
crashed process.
It follows From these observations that the proof is essentially the same that
the bakery algorithm. Moreover as in the Bakery algorithm, due to the fact
 On the Weakest Failure Detector for Read/Write-Based Mutual Exclusion 279

that crashes are stable (once crashed, a process remains crashed forever), this
fault-tolerant version satisfies the starvation-freedom property (which is stronger
than deadlock-freedom) of the basic bakery algorithm, and we have the following
theorem.

Theorem 2. The fault-tolerant QP -based mutual exclusion algorithm described

in Fig. 2 satisfies the mutual exclusion, starvation-freedom, and wait-free exit
properties.

5 Extracting QP from Crash-Tolerant Mutex Algorithms

This section shows that QP is optimal, i.e., it provides the processes with the
weakest information on failures that allows the mutual exclusion problem to be
solved in the read/write crash-prone model. To that end, it describes an algo-
rithm that builds (“extracts” in the failure detector parlance) a failure detector
satisfying the properties defining QP from ANY algorithm that solves deadlock-
free mutual exclusion in presence of process crashes. See Fig. 3. (Other failure
detector reductions are described in [3,4,7]).

Any crash-prone mutex algorithm A Extraction algorithm QP failure detector

Fig. 3. Extracting QP from any crash-prone mutual exclusion algorithm

5.1 Extraction Algorithm

Let A be any algorithm that solves the deadlock-free mutual exclusion prob-
lem in the presence of any number of process crashes. An algorithm E that
“extracts” QP from A is presented in Fig. 4. “Extract” means that, at any time,
the algorithm E outputs the sets trustedi and crashedi at every process pi ,
and these sets satisfy the properties defining the failure detector QP .
A Set of n Lock Objects. In E, the processes access n different instances of a
deadlock-free lock object, denoted MT [1..n]. Let MT [i].entry() and MT [i].exit()
be the code of A associated with the lock object MT [i]. The code defining the
critical section protected by MT [i] is the following.

– If the invoking process is process pi , the code inside the critical section is
made up of the lines 2–4 of Fig. 4.
– If the invoking process is pj , 1 ≤ j ≤ n ∧ j = i, the code is the no-op (empty)
statement (line 8 of Fig. 4).
280 C. Delporte-Gallet et al.

Fig. 4. Extracting QP from a crash-prone mutual exclusion algorithm A (code for

process pi )

A Task per Object. The extraction algorithm E manages, at every process, n

parallel tasks t1 , ..., tn , the task ti being associated with the deadlock-free lock
object MT [i]. We have the following for this lock object and its associated task
ti .

– Process pi invokes first MT [i].entry(), and then the associated critical section
code (lines 2–4), and finally invokes MT [i].exit(). Let us notice that, if pi is
correct the code in critical section never terminates.
– Each process pj , 1 ≤ j ≤ n ∧ j = i, executes the sequence of statements
“MT [i].entry(), no-op; MT [i].exit() (line 8).

At the Operational Level. The n tasks are executed in parallel. SU CC[1..n]

is an array of Boolean shared registers, such that, when true, SU CC[i] indicates
that pi is inside the critical section protected by the lock object MT [i]. For each
task ti , the extraction algorithm E works as follows.

– Process pi invokes first MT [i].entry() (line 1), while (due to the Boolean
SU CC[i], which is set to true at line 2 only by pi ) any other process pj
waits until pi terminates its invocation before invoking its own invocation of
MT [i].entry() (line 8). As no process pj invokes MT [i].entry() before pi returns
from its invocation, it follows from the deadlock-freedom property of the lock
object MT [i], that (if it does not crash) pi returns from its invocation. When
this occurs, pi informs the other processes by writing true in the atomic
register SU CC[i] (line 2). Finally, pi enters an endless wait statement (i.e.,
it never invokes MT [i].exit()).
 On the Weakest Failure Detector for Read/Write-Based Mutual Exclusion 281

– As just indicated, every process pj = pi first waits until process pi returns

from its invocation of MT [i].entry() and writes true in the atomic register
SU CC[i] (line 2). If pi crashes before this write, it will be added neither to
trustedj nor to crashedj . If pi writes true in the atomic register SU CC[i],
pj adds it to trustedj (line 7). After that, pj monitors pi to detect its possible
future crash. To that end, pj uses the critical section protected by MT [i].
• If pi is correct, it never executes MT [i].exit(). It then follows from the
mutual exclusion realized by MT [i] that no other process pj will be able
to complete its invocation of MT [i].entry() (line 8). Consequently, pi will
never be removed from the set trustedj .
• If pi crashes, we have the following.
∗ If pi crashes before executing line 2, SU CC[i] is never set to true,
and pi remains in the set initj =Π \ (trustedj ∪ crashedj ), at
each pj .
∗ If pi crashes after SU CC[i] has been assigned to the value true at
line 2 (then its sets trustedi and crashedi are meaningless), it
follows from the deadlock-freedom property that at least one of the
other processes pj executes MT [i].entry() and the “no-op” correspond-
ing internal operation (line 8). As the number of processes pj = pi
is bounded, and each of them invokes MT [i].entry() and MT [i].exit()
(line 8), it follows from the deadlock-freedom property of MT [i] that
each process pj (that does not crash) executes the line 8. Hence, pj
removes pi from trustedj and adds it to crashedj (line 9).

5.2 Optimality of the Failure Detector QP

Theorem 3. The algorithm described in Fig. 4 extracts a failure detector sat-

isfying the properties of QP from any algorithm solving deadlock-free mutual
exclusion in any read/write n-process system where any number of processes
may crash.

Proof. The three following properties are direct consequences of the text of the
extraction algorithm. They are left to the reader.

– ∀ τ : trustedi (τ ) ∩ crashedi (τ ) = ∅. 
– j ∈ trustedi (τ ) ⇒ ∀ τ  ≥ τ : j ∈ trustedi (τ  ) ∪ crashedi (τ  ) .
– j ∈ crashedi (τ ) ⇒ ∀ τ  ≥ τ : j ∈ crashedi (τ  ) .

The rest of the proof addresses the other properties defining QP .

If j ∈ trustedi (τ ), pj was added to trustedi before time τ (at line 3 if
i = j, and at line 7 if i = j). Hence, SU CC[j] was previously set to true (line 2).
Thus, when a correct process pk (k = j) executes task tj , it finds SU CC[j] equal
to true, and consequently adds j to trustedk at line 7. Hence, we have
 
– j ∈ trustedi (τ ) ⇒ ∀k ∈ C : ∃τ  : j ∈ trustedk (τ  ) ∪ crashedk (τ  ) .
282 C. Delporte-Gallet et al.

If j ∈ crashedi (τ ), at some time τ  ≤ τ , pi inserted pj in crashedi (τ  )

(line 9). Hence, pi previously executed lines 6 and 7 of task tj . At line 6, pi
found SU CC[j] = true. Process pj is the only process that can set SU CC[j]
to true, and it does it at line 2, after having completed MT [j].entry(). After
that, pj remains in its critical section controlled by MT [j]. As due to the mutual
exclusion property of MT [j], the processes pi and pj cannot be together in the
critical section, it follows that pj crashed before pi enters in critical section
protected by MT [j]. Hence, it follows that
– i ∈ C, j ∈ crashedi (τ ) ⇒ j ∈ F(τ ).

 us now assume that pi is correct,

Let  and pj crashed before time τ . We show
that ∃τ  ≥ τ : j ∈ / trustedi (τ  ) . There are two cases, either (1) ∀τ  : j ∈
/
trustedi (τ  ), and the property holds, or (2) ∃ τ  : j ∈ trustedi (τ  ) and pi
inserts pj in trustedi (line 7).
Assume that j ∈ F(τ ) and ∃ τ  : j ∈ trustedi (τ  ). As pi inserted pj in
trustedi (line 7), it previously found SU CC[j] equal to true (line 6). As pj
is the only process that can assign SU CC[j] to true, it does it at line 2 in the
critical section controlled by MT [j] after having executed MT [j].entry(). Any
correct process that finds SU CC[j] equal to true, invokes MT [j].entry(). As the
number of processes is bounded, by the deadlock freedom property of MT [j], pi
executes line 8 and terminates it. It then moves j from trustedi to crashedi ,
and never inserts it again in trustedi . Hence, we have
 
– i ∈ C, j ∈ F(τ ) ⇒ ∃τ  ≥ τ : j ∈ trustedi (τ  ) .
Let us finally consider the case where both pi and pj are correct. In the extraction
algorithm, pj executes MT [j].entry() of task tj . Until pj sets SU CC[j] to true,
all the other processes that execute task tj remain blocked at line 6 and do
not execute MT [j].entry(). Due to the deadlock-freedom property of MT [j], pj
will enter in critical section and set SU CC[j] to true. Then, pj is added to
trustedi (τ ) at line 3 if i = j, or at line 7 if i = j. Hence, it follows that
 
– i ∈ C, j ∈ C ⇒ ∃τ : j ∈ trustedi (τ ) . 


From two Observations to a Theorem.

– Observation O1. Theorem 3 shows that QP can be obtained from any crash-
tolerant deadlock-free mutual exclusion algorithm. This means that, if a fail-
ure detector D allows us to solve deadlock-free mutual exclusion, QP can be
built from D. (This means that D contains at least as much information on
failures than the one provided by QP .)
– Observation O2. On another side, the algorithm described in Fig. 2 shows
that, in a read/write n-process system where any number of processes may
crash, starvation-free (hence deadlock-free) mutual exclusion can be solved
from QP .
As starvation-freedom is a stronger progress condition than deadlock-freedom,
the following theorem is a consequence of the previous observations.
 On the Weakest Failure Detector for Read/Write-Based Mutual Exclusion 283

Theorem 4. QP is the weakest failure detector for both deadlock-free and

starvation-free mutual exclusion in read/write n-process systems where any num-
ber of processes may crash.
From the fact that Γ 1 (as defined in [2]) is the weakest failure detector
for starvation-free mutual exclusion in read/write n-process systems where any
number of processes may crash, we get:
Corollary 1. Γ 1 and QP are equivalent.

6 Weakest Failure Detectors in RW Systems:

QP Versus Ω
As indicated in Table 1, the eventual leader failure detector Ω (whose definition
appears in the Introduction) is the weakest failure detector to solve the consensus
problem in asynchronous read/write systems where any number of processes may
crash [4,16].
Let suspectedi be the local output of the eventually perfect failure detector
3P at process pi . Defining leaderi as being max(Π\suspectedi ) (where Π is
the set of all the processes) provides us with the failure detector Ω, from which
we have at Ω  3P . Moreover, as Ω provides a correct eventual leader only, it
is not powerful enough to implement 3P . Hence, Ω ≺ 3P . The next theorem
follows directly this observation and Theorem 1 (which states 3P ≺ QP ≺ P ,
see Sect. 3.2).
Theorem 5. Ω ≺ 3P ≺ QP .
It follows from this theorem that, from an information on failure point of view,
consensus is a weaker problem than crash-prone mutual exclusion (“weaker”
meaning here “requiring less information on failures to be solved”).

7 Conclusion
This paper was on the information on failures which allows mutual exclusion to
be solved in asynchronous systems where (i) processes communicate by accessing
atomic read/write registers and (ii) any number of processes may crash. It has
presented the weakest failure detector solving this problem. Called QP (for quasi-
perfect), this failure detector is very close to the perfect failure detector (denoted
P ). This means that, from a practical point of view, it is reasonable to use P
to solve read/write mutual exclusion in the presence of process crashes. Let us
remind that, as P is a realistic failure detector, it can be easily implemented
by an underlying synchronous system, which always remains unknown (hence
inaccessible) to both the upper layer asynchronous application and the mutual
exclusion algorithm.
As far as liveness is concerned, the modified Lamport’s mutual exclusion
algorithm presented in Sect. 4.2 satisfies the starvation-freedom property. Dif-
ferently, the algorithm described in Fig. 4 extracts the failure detector QP from
284 C. Delporte-Gallet et al.

any deadlock-free crash-tolerant mutual exclusion algorithm. It follows that QP

is the weakest failure detector for both deadlock-free and starvation-free mutual
exclusion.

Acknowledgments. This work was partially supported by the French ANR project
16-CE40-0023-03 (16-CE40-0023-03) devoted to layered and modular structures in dis-
tributed computing.

References
1. Bhatt, V., Christman, N., Jayanti, P.: Extracting quorum failure detectors. In:
Proceedings of the 28th ACM Symposium on Principles of Distributed Computing
(PODC 2009), pp. 73–82. ACM Press (2009)
2. Bhatt, V., Jayanti, P.: On the existence of weakest failure detectors for mutual
exclusion and k-exclusion. In: 23rd International Symposium on Distributed Com-
puting (DISC 2009), LNCS, vol. 5805, pp. 325–339. Springer (2009)
3. Bonnet, F., Raynal, M.: A simple proof of the necessity of the failure detector
Σ to implement an atomic register in asynchronous message-passing systems. Inf.
Process. Lett. 110(4), 153–157 (2010)
4. Chandra, T., Hadzilacos, V., Toueg, S.: The weakest failure detector for solving
consensus. J. ACM 43(4), 685–722 (1996)
5. Chandra, T., Toueg, S.: Unreliable failure detectors for reliable distributed systems.
J. ACM 43(2), 225–267 (1996)
6. Delporte-Gallet, C., Fauconnier, H., Guerraoui, R.: A Realistic look at failure
detectors. In: Proceedings of the 43rd Annual IEEE/IFIP International Conference
on Dependable Systems and Networks (DSN 2002), pp. 345–353. IEEE Computer
Press (2002)
7. Delporte-Gallet, C., Fauconnier, H., Guerraoui, R.: Tight failure detection bounds
on atomic object implementations. J. ACM 57(4), 32 (2010). Article 22
8. Delporte-Gallet, C., Fauconnier, H., Guerraoui, R., Hadzilacos, V., Kouznetsov, P.,
Toueg, S.: The weakest failure detectors to solve certain fundamental problems in
distributed computing. In: Proceedings of the 23th ACM Symposium on Principles
of Distributed Computing (PODC 2004), pp. 338-346. ACM Press (2004)
9. Delporte-Gallet, C., Fauconnier, H., Guerraoui, R., Kouznetsov, P.: Mutual exclu-
sion in asynchronous systems with failure detectors. J. Parallel Distrib. Comput.
65, 492–505 (2005)
10. Delporte-Gallet, C., Fauconnier, H., Raynal, M.: Fair synchronization in the pres-
ence of process crashes and its weakest failure detector. In: 33rd IEEE Interna-
tional Symposium on Reliable Distributed Systems, (SRDS 2014), pp. 161–170.
IEEE Press (2014)
11. Dijkstra, E.W.: Solution of a problem in concurrent programming control. Com-
mun. ACM 8(9), 569 (1965)
12. Fernández, A., Jiménez, E., Raynal, M., Trédan, G.: A timing assumption and two
t-resilient protocols for implementing an eventual leader service in asynchronous
shared-memory systems. Algorithmica 56(4), 550–576 (2010)
13. Fischer, M.J., Lynch, N.A., Paterson, M.S.: Impossibility of distributed consensus
with one faulty process. J. ACM 32(2), 374–382 (1985)
14. Hélary, J.-M., Hurfin, M., Mostéfaoui, A., Raynal, M., Tronel, F.: Computing global
functions in asynchronous distributed systems with perfect failure detectors. IEEE
Trans. Parallel Distrib. Syst. 11(9), 897–909 (2000)
 On the Weakest Failure Detector for Read/Write-Based Mutual Exclusion 285

15. Lamport, L.: A new solution of Dijkstra’s concurrent programming problem. Com-
mun. ACM 17(8), 453–455 (1974)
16. Lo, W.-K., Hadzilacos, V.: Using failure detectors to solve consensus in asyn-
chronous shared memory systems. In: Proceedings of the 8th International Work-
shop on Distributed Algorithms (WDAG 1994), LNCS, vol. 857, pp. 280–295.
Springer (1994)
17. Loui, M., Abu-Amara, H.: Memory requirements for agreement among unreliable
asynchronous processes. Adv. Comput. Res. 4, 163–183 (1987)
18. Raynal, M.: Concurrent Programming: Algorithms, Principles, and Foundations,
p. 515. Springer (2013). ISBN 978-3-642-32027-9
19. Raynal, M.: Fault-Tolerant Message-passing Distributed Systems: An Algorithmic
Approach, p. 492. Springer (2018). ISBN: 978-3-319-94140-0
20. Taubenfeld, G.: Synchronization Algorithms and Concurrent Programming, p. 423.
Pearson Education/Prentice Hall (2006). ISBN 0-131-97259-6
 An Active Signaling Mechanism
to Reduce Access Collisions in a
Distributed TDMA Based MAC Protocol
for Vehicular Networks

Fouzi Boukhalfa1(B) , Mohamed Hadded1(B) , Paul Muhlethaler2(B) ,

and Oyunchimeg Shagdar1(B)
1
VEDECOM, Versailles, France
{fouzi.boukhalfa,mohamed.elhadad,oyunchimeg.shagdar}@vedecom.fr
2
INRIA, Paris, France
[email protected]

Abstract. A Vehicular Ad-Hoc NETwork (VANET) is an emerging

technology consisting in a set of vehicles, mobile devices and an infras-
tructure, which uses wireless transmission technologies to enable real-
time communication between them. VANETs help to improve traffic
safety and efficiency by enabling each vehicle to exchange information
about vehicle positions, speeds and the state of the road. Due to its
promising applications, this type of network has attracted a lot of atten-
tion in the research community, including those in the domain of Medium
Access Control (MAC). In fact, an efficient MAC protocol design in
VANETs is crucial to guarantee safety-critical message broadcasting with
high reliability and minimum delay. A Time Division Multiple Access
(TDMA)-based MAC protocol is advantageous in VANETs thanks to its
ability to prevent the hidden node problem, and to guarantee high qual-
ity of service for real-time applications. However, time slot assignments
to vehicles could suffer from the access collision problem which can fre-
quently occur between vehicles trying to access the same time slots. This
problem is mainly due to the lack of infrastructure and potentially high
density of vehicles in VANETs. In this paper, we focus on the problem
of access collision in the time slot assignments of the DTMAC protocol,
and present an enhanced version based on active signaling (AS-DTMAC,
i.e. Active Signaling-DTMAC). Extensive simulations are conducted
considering various traffic densities to demonstrate the performance of
AS-DTMAC.

Keywords: VANETs · MAC · TDMA · Active signaling ·

Low latency · 5G · Network simulation

1 Introduction and Motivation

The automotive field is currently undergoing a massive transformation with the
car becoming a connected device and efforts being made for autonomated driv-
ing vehicles. In this context, connectivity will become a key issue for the future
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 286–300, 2020.
https://doi.org/10.1007/978-3-030-15032-7_25
 An Active Signaling Mechanism to Reduce Access Collisions 287

of the automotive industry, allowing new usages and new services to be offered.
To ensure this new functionality, the future services need to benefit from the
best connectivity with high transmission rates for huge data exchange and low
latency which will enable real time services with immediate reaction, high levels
of scalability, quality of service (QoS) and greater reliability. All of these features
will be included in the future 5G standard [1]. This new network is very impor-
tant for the automotive sector because it enables direct communication between
vehicles (V2V), this leads to a better reaction time which means that collisions
between vehicles can be avoided and hence passenger safety will be increased.
V2V will complete the information obtained from sensors (radar, camera, etc.)
with more real time information such as location, heading speed and intention.
This promising evolution cannot come about without a new medium access
control (MAC) design that can support all these requirements. The difficulty
being that VANETs are very different from traditional networks due to their spe-
cial features like a dynamic topology, a high mobility of the connected device [2].
Many studies have been proposed to address these issues [3,4]. The main goal of
MAC is to provide an efficient way to share the channel between nodes and cover
the largest possible number of users without decreasing the QoS. MAC protocols
are classified into two categories: contention-based and contention-free. The for-
mer allows multiple access to the channel based on channel sensing, however the
collision can be high due to the high probability that neighbors detect the chan-
nel being idle at the same time. Contention-based approaches use CSMA/CA
scheme which is used in the current standard for vehicular communication (IEEE
802.11p) [5]. Recently, many scientific studies proposed contention-free protocols
that try to reduce the collision probability, essentially by remoring the sensing
phase and enhancing robustness, the number of collisions even in high density
scenarios. Contention-free protocols enable multiple access by dividing the chan-
nel in various ways: code sequence, frequency band or time slot which correspond
respectively to these three protocols: Code Division Multiple Access (CDMA),
Frequency Division Multiple Access (FDMA), Time Division Multiple Access
(TDMA) [6].
TDMA-based MAC are one of the most appreciated protocols that can enable
a multiple access to the channel by dividing it efficiently. The use of this kind
of access protocols in vehicular networks will allow the vehicles to access to
the channel by using the same channel frequency. Consequently, each time slot
will be assigned to a vehicle depending on the priority of its message. However,
these protocols suffer from the access collision problem which can frequently
occur between vehicles trying to access the same time slots. Motivated by this
observation, we propose in this paper an enhanced technique to reduce access
collisions for a fully distributed and location-based TDMA scheduling scheme
for VANETs named DTMAC [7], with the aim to meet the requirements of the
emerging applications. DTMAC takes into consideration the linear topology of
VANETs and provides an improved broadcast coverage due to the spatial reuse
of the slots, which gives a lower rate of collision compared to other protocols.
288 F. Boukhalfa et al.

The remainder of the paper is organized as follows. In the next section, we

present related work. Section 3 contains a brief recall of the DTMAC protocol
and its specification. Section 4 introduces the active signaling technique and the
proposed protocol. Section 5 presents our simulation studies that evaluate and
campare the proposed AS-DTMAC protocol, which is an enhanced version of
DTMAC with the active siganling. Finally, Sect. 6 concludes this paper.

2 Related Work
MAC protocols generally fall into one of two broad categories: contention-based
and contention-free. In contention-based protocols, each node can try to access
the channel when it has data to transmit using the carrier sensing mechanism
[11]. The IEEE 802.11p [12], which is the emerging standard deployed to enable
vehicular communication, is a contention-based MAC protocol which uses a
priority-based access scheme that employs both Enhanced Distributed Chan-
nel Access (EDCA) and Carrier Sense Multiple Access with Collision Avoidance
(CSMA/CA) mechanisms [13]. Since the IEEE 802.11p standard is a contention-
based MAC, it cannot provide a reliable broadcast mechanism with bounded
communication delay. This disadvantage is particularly important in VANETs
which are specifically designed to improve road safety.
In Contention-free MAC protocols, only one vehicle can access the channel
at any given time within a given neighborhood. Therefore, these protocols pro-
vide collision-free transmission with bounded access delay for real-time applica-
tions. In recent years, many distributed TDMA-based MAC protocols have been
proposed to guarantee real-time and reliable communications in VANETs while
avoiding the access collision and the merging collision problems [10]. Each proto-
col has been proposed for a particular problem in a specific mobility scenario. For
instance, the authors in [14] have proposed an ADHOC Medium Access Control
(ADHOC MAC) to provide an efficient broadcast service for inter-vehicle com-
munications and solve the issues such as the hidden-exposed terminal problem
and QoS provision. ADHOC MAC is a contention-free medium access protocol
which implements a dynamic TDMA mechanism that is able to provide prompt
access based on a distributed access technique, R-ALOHA (Reliable R-ALOHA
[14]). Each vehicle can access the channel at least once in each frame by randomly
selecting a time slot as its Basic CHannel (BCH). In [17] Omar et al. devel-
oped and evaluated a contention-free multi-channel MAC protocol proposed for
VANETs, called VeMAC. This protocol supports efficient one-hop and multi-hop
broadcast services on the control channel without the hidden terminal problem
caused by node mobility. VeMAC reduces the collision rate by assigning disjoint
sets of time slots to vehicles moving in opposite directions (Left;Right) and to
Roadside Units (RSUs). Since ADHOC MAC and VeMAC are fully distributed
protocols, an access collision problem can occur between vehicles trying to access
the same time slots. A completely distributed TDMA scheduling scheme, named
DTMAC which exploits the linear topology of VANETs is proposed and pre-
sented in [10]. The scheme is based on the assumption that the road is divided
 An Active Signaling Mechanism to Reduce Access Collisions 289

into small fixed areas and the time slots in each TDMA frame are partitioned
into three sets associated with vehicles in three contiguous areas. The ways that
slots are allocated and reused between vehicles in DTMAC are designed to avoid
collisions caused by the hidden node problem.
Although the distributed-TDMA-based MAC protocols show improvements
compared to the IEEE 802.11p standard and can support QoS requirements of
safety applications, as the size of the VANET grows, the access collision problem
frequently occurs between vehicles trying to access the same time slots. Moti-
vated by this observation, we mainly focus in this paper on the problem of access
collision in the DTMAC protocol, and we propose an enhanced version based on
active signaling (AS-DTMAC) to effectively alleviate the access collision problem
in the slot assignment process.

3 System Model and DTMAC Protocol

In this section, we present the system model and we briefly describe the DTMAC
protocol [15]. DTMAC is the scheme on which, we present an enhancement in
Sect. 4 operates.
A VANET in a highway scenario consists of sets of vehicles moving in directio
same or different ns and under varying traffic conditions (speed, density). In
this paper, we suppose that each vehicle in a VANET is equipped with a GPS
(Global Positioning System) or a GALLILEO receiver so that it can obtain
an accurate real-time three-dimensional geographic position (latitude, longitude
and altitude), speed and exact time. Moreover, synchronization between vehicles
may be performed by using GPS timing information. As shown in Fig. 1, the road
is divided into N small fixed areas, denoted by xi , i = 1, . . . , N of length equal to
R, where R is the communication range of the vehicles. Area IDs can be easily
derived using maps and GPS devices. In the following, we will briefly describe
the TDMA scheduling principle of the DTMAC protocol.
DTMAC is a distributed TDMA-based MAC protocol for VANETs which
exploits the linear topology of VANETs. The time slots in each TDMA frame
are partitioned into three sets, S0 , S1 and S2 , associated with vehicles in three
contiguous areas: xi , xi+1 and xi+2 , respectively, see Fig. 1. Each frame consists
of a constant number of time slots, denoted by τ and each time slot is of a
fixed time duration, denoted by s. Each vehicle can detect the start time of
each frame as well as the start time of a time slot. In the VANET studied here,
all the vehicles are equipped with a GPS and thus the one-Pulse Per-Second
(1PPS) signal that a GPS receiver gets from GPS satellites can be used for slot
synchronization. In DTMAC, each vehicle moving in area xi knows only the slot
scheduling information of its neighboring vehicles moving in areas xi+1 and xi+2 .
The TDMA scheduling mechanism of the DTMAC protocol uses the vehicle
location and slot reuse concept to ensure that vehicles in adjacent areas have
collision-free schedules in which the three subsets of time slots will be reused
between neighboring areas in such a way that no vehicles in different adjacent
areas can access the channel at the same time, and thus no interference can
290 F. Boukhalfa et al.

Fig. 1. TDMA slots scheduling principle.

Fig. 2. DTMAC protocol principle

occur. To acquire this information, messages are exchanged between neighbor-

ing vehicles. Furthermore, specific information, called frame information (FI) is
added to each transmitted packet to notify the neighboring vehicles of a time
slot assignment. Each time slot is dynamically reserved by an active vehicle (a
vehicle whose communication device is transmitting) for collision-free delivery of
safety messages or other control messages. The FI consists of a set of ID Fields
(IDFs) of size equal to the number of time slots per frame, τ . Each IDF con-
sists of three fields: VEC ID, SLOT STATUS and PKT TYPE. The VEC ID
field contains the ID of the vehicle that is accessing the corresponding slot. The
SLOT STATUS field contains the status of the slot which indicates whether the
slot is idle, busy or in collision. Finally, the PKT TYPE field indicates the type
of packet transmitted by the vehicle, i.e. periodic information or event-driven
safety messages. When an active vehicle v moving within the area xi needs to
 An Active Signaling Mechanism to Reduce Access Collisions 291

acquire a time slot on the transmission channel, it starts listening to the channel
during the set of time slots reserved for the area in which it is traveling, this set
is Sj (v), where j = (i + 2) mod 3. At the end of the frame the vehicle v can
determine the set of its neighbors N (v), the set of busy slots in Sj (v), denoted by
B(v) and the set of available time slots denoted F (v). Vehicle v selects one slot
in F (v) at random. Figure 2 shows a flowchart summarizing the slot reservation
mechanism of the DTMAC protocol which is executed by each vehicle x that
needs to acquire a time slot.

4 Active Signaling Based DTMAC Protocol

(AS-DTMAC)
4.1 General Scheme
Although the concept of active signaling is not very well-known in the research
community, it has been shown to improve the IEEE 802.11 access scheme. How-
ever, to our best knowledge, we are the first to combine this technique with
a fully distributed and location-based TDMA scheduling protocol. Active sig-
naling is used to improve the slot selection process, mostly by avoiding access
collisions, but it can also increase the probability of successful transmission when
these slots have been acquired. The main idea is to introduce a scheme which,
with a high probability, ensures that a slot will be successfully acquired by a
single vehicle to send its packet.
To integrate the active signaling scheme with DTMAC, the slot is divided
into two time intervals as shown in Fig. 3. The first interval is reserved for the
selection process based on our active signaling scheme. During this time interval,
the nodes compete in order to access the channel and hopefully at this end of
time interval only one node remains as a winner. The second interval is used by
the winning node to transmit its payload packet.

Fig. 3. Structure of the AS-DTMAC slot

The selection interval contains a set of mini-slots where a competing node

can be either in transmission mode or in listening mode. In transmission mode,
the node sends a signaling burst whereas in listening mode the node listens for
292 F. Boukhalfa et al.

Fig. 4. Active signaling mechanism

possible concurrent transmissions. We summarize the transmission and reception

activity of a competing node by a binary transmission key e.g. ‘1010101100...10’.
By convention ‘1’ corresponds to a mini-slot in transmission and ’0’ to mini-slot
in reception mode. When nodes compete for a given slot, they use their binary
transmission key. The key is consumed from left to right and we assume that
the binary transmission key has n digits.
To compete for a slot, each node with a message to send shows its presence by
sending a burst signal, which is generated by drawing a random key. If we suppose
that the first value is ‘0’ the vehicle will sense the channel during this mini-slot.
If during this listening period the node senses a transmission it withdraws from
the competition. If no competing transmission is sensed, the process can continue
and the next digit of the key is used. Let us assume that is the case, then the
vehicle reads the next bit of its transmission key. If this value is 1, then the
vehicle will transmit his signaling burst during this next mini slot. The process
will continue following this algorithm until the last bit of the key is reached. In
this way, there are fewer and fewer vehicles in competition from one mini-slot
to the next. The description of this technique is summarized in the flowchart in
Fig. 4.
The vehicle that successfully gets a slot will keep it for the next frames, and
free the slot only if a collision occurs or when the vehicle changes the zone. In
these two cases the active signaling technique is needed to reserve again. Thus,
the competition is limited to the node that are trying to get a slot for the reasons
described above.
A very important property of this selection process is that at least one node
reads all the bits of its selection key and successfully completes the selection
 An Active Signaling Mechanism to Reduce Access Collisions 293

process. It can be shown that with a high probability there is one and only
selected node in the selection process and thus we have no collision.

4.2 Using Active Signaling with Priority and for Emergency Traffic

4.2.1 Priority
The default functioning of the active signaling scheme is with a random key.
However it is possible to divide the transmission key into a priority part and
a random part. In contrast to the random part, the priority part is the deter-
ministic part of the transmission key. If we allocate n1 bits to the priority part
(n2 = n − n1 bits being left to the random part), we can handle 2n1 different
priority levels. The n2 bits of the transmission key are left to the random part
and are used as in the default functioning.

4.2.2 Emergency Traffic

The idea is to build a special access scheme for emergency packets. We assume
that these packets are sporadic packets and produce a small additional channel
load. Examples of such packets can be DENMs (Decentralized Environmental
Notification Messages).
In the default functioning of the AS-DTMAC protocol, the vehicles use the
active signaling to decrease the access collision probability and, after the initial
access, to overcome the merging collision probability. However, the first attempt
to reserve a slot is performed on a free slot according to the DTMAC procedure,
see Sect. 3. In some cases, one may wish almost immediate access to a very next
slot without doing the DTMAC listening procedure. In that case, the priority
scheme can be used. To illustrate, let us suppose that we have n1 = 1 priority
bit, the default traffic will set its priority bit to 0 whereas the emergency packet
will set its priority bit to 1. By doing so the emergency packet will eliminate any
other usual packet on the coming slot. However, in order to save the reservation
granted in the slot to a node, we use the following technique. The last bit of the
n2 of the random part will be deterministic. For default packets this bit will be
set to 0 and this bit will be set to 1 by an emergency packet. Using this coding for
the transmission key, a node with a usual packet and with a slot already reserved
can determine if it has been eliminated by an emergency packet. In that case,
this node will continue to use this slot in the next frame and will hopefully be
able to successfully transmit since the vehicle with emergency packets will use
another slot.

5 Performance Evaluation
In this section, we evaluate and compare the performance of DTMAC and AS-
DTMAC. The methodology and the scenarios are taken from [15].
294 F. Boukhalfa et al.

5.1 Simulation Scenarios and Parameters

We use MOVE and SUMO [20] to generate vehicular traffic scenarios and to
perform real vehicular mobility simulations, respectively.
In our simulations we consider a digital map to build a VANET environment
close to real highway configurations taking into account lane with different direc-
tions. In Fig. 5, we can see a metropolitan area taken from a Map of San Jose
(California) of size 3000 m × 100 m. This map was exported from OpenStreetMap
(OSM) and adapted with the help of OpenStreetMap Editor (JOSM). The result-
ing roads are then populated with vehicles traveling in both directions. Each flow
of vehicles is characterized by a set of parameters which consist of the starting
and ending time of the flow, the initial point and the destination of the flow and
the maximum number of vehicles. In this environment, each vehicle is assigned
a random speed between 120 km/h and 150 km/h. The resulting traffic traces
generated by MOVE were injected into the Network Simulator ns2.34. Table 1
summarizes the simulation parameters used in our scenarios.

Fig. 5. VANET network topology captured from Google MAP.

Table 1. Simulation parameters

Parameter Value
Simulation duration 120 (s)
Speed 120 (km/h)
Speed standard deviation 30 (km/h)
Number of slots per frame (τ ) 100
Slot duration 0.001 (s)
Mini slot duration 0.000025 (s)
Highway length 2.5 (km)
The number of lanes per direction 2
The radio range (R) 250 (m)

5.2 Simulation Results

We will keep the same parameters as those used in the previous study [15] and
we vary the density of vehicles in the network. This parameter is called area
occupancy (AO); the definition of this metric is given in [17]. We also introduce
the following metrics:
 An Active Signaling Mechanism to Reduce Access Collisions 295

• Overhead: is the total number (in bytes) of packets transmitted from one
vehicle to another in order to maintain a collision-free schedule. It consists
of the overhead of the basic protocol plus the additional bytes created in the
burst of the signaling process. In our case, only the latter is computed.
• Number of slots acquired: this is the number of vehicles that have successfully
acquired a time slot per frame.
• Average Access Time: this is the average time for a vehicle to access a slot in
a given frame.
• Access collision rate: is defined as the average number of access collisions per
slot and per area.

5.2.1 Best Length of the Signaling Interval

To tune the performance of our AS-DTMAC protocol, a preliminary study is
necessary to determine the adequate number of mini-slots that will be used
during the selection phase described in Sect. 4. In order to find that optimal
number which guarantees a good compromise between the metrics, we plot the
curves of the access collision rate, the overhead and the average access time versus
the number of the mini-slots for different network loads. In fact, on the one hand,
a high number provides a small access collision rate but, on the other hand, it
significantly increases the number of control messages needed to establish and
maintain a collision-free schedule.
The results of our study are given in Figs. 6, 7 and 8. For example if we
take the value 4 as a number of mini slots, we have a lower access time and
overhead, while the access collision is high. Also if we take a large n = 14, the
access collision rate is very low but the overhead is significantly high. It is clear
from these observations that the best compromise is located approximatively for
the number of bits in the transmission key: n ∈ [6; 10], as there is an acceptable
access latency, low access collision rate and it generates of a reasonable overhead.
For the rest of this paper, using 9 mini-slots seems appropriate, as it requires
less computing and presents good performance. In Figs. 6, 7 and 8, we can also
see that the network load significantly affects the above metrics, the more we
increase it, the larger are the access time, the overhead, and the access collision
rate.

5.2.2 Comparison of DTMAC and AS-DTMAC

The aim of this sub-section is to compare the performance of DTMAC and AS-
DTMAC in terms of the number of vehicles acquiring a time slot, access collision
rate and, finally, the average access delay.
Figure 9 shows the slot allocation for the 10 first frames of DTMAC and AS-
DTMAC with AO is equal to 0.9. From this graph, we could conclude that
DTMAC requires approximatively twice as many frames as AS-DTMAC to
obtain a reservation for each vehicle in the network. In fact, when the num-
ber of frames is equal to 2 in AS-DTMAC, 93% of vehicules have successfully
reserved a slot, whereas only 5% of vehicles were able to reserve a slot using the
DTMAC protocol.
296 F. Boukhalfa et al.

0.07
AO = 0.96
AO = 0.8
AO = 0.4

Average access time (s) 0.06

0.05

0.04

0.03

0.02
2 4 6 8 10 12 14
Number of mini slots

Fig. 6. Average access time versus the number of mini slots

20000
AO = 0.96
AO = 0.8
18000 AO = 0.4

16000

14000
Overhead (bytes)

12000

10000

8000

6000

4000

2000

0
2 4 6 8 10 12 14
Number of mini slots

Fig. 7. The overhead in bytes versus the number of mini slots

Figure 10 (plotted in logscale) presents the percentage of access collisions for

DTMAC and AS-DTMAC. It’s clear that the difference between the two curves
is roughly of three orders of magnitude, which is evidently a very significant
result. For instance, if we take the [0.4, 0.7] interval of AO, we observe that
the difference in the access collision rate using DTMAC and AS-DTMAC is
very large and almost the same for all the values (i.e. at the value 0.4, the
 An Active Signaling Mechanism to Reduce Access Collisions 297

0.03
AO = 0.96
AO = 0.8
AO = 0.4
0.025

Access Collision Rate (%)

0.02

0.015

0.01

0.005

0
2 4 6 8 10 12 14
Number of mini slots

Fig. 8. Access collision versus the number of bits of the transmission key

200

180
Number of vehicules acquirinng a time slot

160

140

120

100

80

60

40

20 AS-DTMAC
DTMAC
0
0 5 10 15 20
Frames

Fig. 9. Number of vehicles acquiring a time slot within N frames.

access collision rate is 0.00015 for AS-DTMAC whereas it’s nearly 0.03 when
the DTMAC protocol is used).
Figure 11 illustrates that AS-DTMAC provides a significantly smaller access
delay than DTMAC. As we can see, the difference in time between the two
protocols stays relatively the same except for the two last values where the load
of the network AO is very high (0.9 and 0.96).
298 F. Boukhalfa et al.

10.000000
ASDTMAC
DTMAC

1.000000
Access Collision Rate (%)

0.100000

0.010000

0.001000

0.000100

0.000010
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
AO

Fig. 10. Access collision versus channel occupancy for DTMAC and AS-DTMAC with
error bar (95% confidence interval).

0.07
ASDTMAC
DTMAC
0.065

0.06

0.055
Average access time (s)

0.05

0.045

0.04

0.035

0.03

0.025

0.02
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
AO

Fig. 11. Average access time versus channel occupancy for DTMAC and AS-DTMAC
with error bar (95% confidence interval).

In [15] it is shown that DTMAC significantly outperforms VeMAC [17] which

is very well-known TDMA protocol for VANETs in the literature. As we have
used the same simulation tools and the same scenarios used in [15], we can
ensure by transitivity that AS-DTMAC outperforms VeMAC and similar TDMA
schemes.
 An Active Signaling Mechanism to Reduce Access Collisions 299

6 Conclusion
In this paper, we have proposed to enhance DTMAC protocol by integrat-
ing active signaling. The simulation results show that AS-DTMAC drastically
reduces the access collision rate and allocates slots to all the vehicles in the
network in half the time it takes DTMAC to do so. We also presented a use
case in the V2V for urgent and high priority traffic message like DENM, that
can help to avoid an accident, all these new features are very important for the
future technology described in the beginning of this paper. As future work, we
will do additional simulations to compare with the standard used in V2V (IEEE
802.11p) and we plan to develop an analytical model for AS-DTMAC as well as
to investigate further advanced access features that could be provided using the
active signaling scheme.

References
1. Shah, S.A.A., Ahmed, E., Imran, M., Zeadally, S.: 5G for vehicular communica-
tions. IEEE Commun. Mag. 56(1), 111–117 (2018)
2. Jakubiak, J., Koucheryavy, Y.: State of the art and research challenges for
VANETs. In: IEEE Consumer Communications and Networking Conference
(CCNC), Las-Vegas, USA, pp. 912–916 (2008)
3. Jiang, X., Du, D.H.: PTMAC: a prediction-based TDMA MAC protocol for reduc-
ing packet collisions in VANET. IEEE Trans. Veh. Technol. 65(11), 9209–9223
(2016)
4. Huang, J.J., Chiu, Y.S.: A scheme to reduce merging collisions in TDMA-based
VANETs. In: International Symposium on Wireless and Pervasive Computing
(ISWPC), pp. 1–4, November 2013
5. Bilstrup, K., Uhlemann, E., Strom, E.G., Bilstrup, U.: Evaluation of the IEEE
802.11 p MAC method for vehicle-to-vehicle communication. In: IEEE 68th Vehic-
ular Technology Conference in Wireless and Pervasive Computing (ISWPC), VTC
2008-Fall, pp. 1–5. IEEE, September 2008
6. Dharsandiya, A.N., Patel, R.M.: A review on MAC protocols of vehicular ad hoc
networks. In: International Conference on Wireless Communications, Signal Pro-
cessing and Networking (WiSPNET), pp. 1040–1045. IEEE, March 2016
7. Hadded, M., Laouiti, A., Muhlethaler, P., Saidane, L.A.: An infrastruture-free slot
assignment algorithm for reliable broadcast of periodic messages in vehicular ad
hoc networks. In: Proceedings of VTC Fall 2016, Montréal, Canada, September
2016
8. Baccouch, H., Adjih, C., Muhlethaler, P.: Active signaling for 802.11 networks in
the NS-3 simulator. In: PEMWN 2012: First International Workshop on Perfor-
mance Evaluation and Modeling in Wireless Networks, Tunis, Tunisia, November
2012
9. Hu, F.: Security and Privacy in Internet of Things (IoTs): Models, Algorithms,
and Implementations. ISBN 978-1-4987-2318-3
10. Hadded, M., Muhlethaler, P., Laouiti, A., Zagrouba, R., Saidane, L.A.: TDMA-
based MAC protocols for vehicular ad hoc networks a survey, qualitative analysis
and open research issues. IEEE Commun. Surv. Tutor. 17(4), 2461–2492 (2015)
300 F. Boukhalfa et al.

11. Ye, F., Yim, R., Zhang, J., Roy, S.: Congestion control to achieve optimal broad-
cast efficiency in VANETs. In: IEEE International Conference on Communications
(ICC), South Africa, Cape Town, pp. 1–5, May 2010
12. 802.11p-2010: IEEE standard for information technology - Telecommunications
and information exchange between systems - local and metropolitan area networks
- specific requirements part 11 : Wireless LAN medium access control (MAC)
and physical layer (PHY) and physical layer (PHY) specifications amendment 6 :
Wireless access in vehicular environments Std. (2010)
13. Uzcategui, R., Acosta-Marum, G.: Wave: a tutorial. IEEE Commun. Mag. 47(5),
126–133 (2009)
14. Borgonovo, F., Capone, A., Cesana, M., Fratta, L.: ADHOC MAC: new MAC
architecture for Ad Hoc networks providing efficient and reliable point-to-point
and broadcast services. Wirel. Netw. 10(4), 359–366 (2004)
15. Hadded, M., Laouiti, A., Muhlethaler, P., Saidane, L.A.: An infrastructure-free slot
assignment algorithm for reliable broadcast of periodic messages in vehicular ad
hoc networks. In: Vehicular Technology Conference, VTC-Fall, Montreal, Canada
(2016)
16. Borgonovo, F., Capone, A., Cesana, M., Fratta, L.: RR-ALOHA, a reliable R-
ALOHA broadcast channel for Ad-Hoc inter-vehicle communication networks. In:
IEEE IFIP Annual Mediterranean Ad Hoc Networking Workshop (Med-Hoc-Net),
Baia Chia, Italy (2002)
17. Zhuang, W., Omar, H.A., Lio, L.: VeMAC: a novel multichannel MAC protocol for
vehicular ad hoc networks. In: IEEE Conference on Computer Communications
Workshops (INFOCOM WKSHPS), Shanghai, China, pp. 413–418, August 2011
18. Weidong, Y., Pan, L., Yan, L., Hongsong, Z.: Adaptive TDMA slot assignment
protocol for vehicular ad-hoc networks. J. China Univ. Posts Telecommun. 20(1),
11–18 (2013)
19. Ke, W., Weidong, Y., Pan, L., Hongsong, Z.: A decentralized adaptive TDMA
scheduling strategy for VANET. In: IEEE Wireless Communications and Network-
ing Conference Workshops (WCNCW), Shanghai, China, pp. 216–221, April 2013
20. Karnadi, F., Mo, Z., Lan, K.C.: Rapid generation of realistic mobility models for
VANET. In: IEEE WCNC, Hong Kong, China, pp. 2506–2511, March 2007
 Estimation of Tags Using Various Data
for Online Videos

Hiroki Sakaji1(B) , Akio Kobayashi2 , Masaki Kohana3 , Yasunao Takano4 ,

and Kiyoshi Izumi1
1
The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8654, Japan
{sakaji,izumi}@sys.t.u-tokyo.ac.jp
2
RIKEN AIP Center, Nihonbashi 1-chome Mitsui Building, 15th floor,
1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan
[email protected]
3
Ibaraki University, 4-12-1 Nakanarusawa, Hitachi, Ibaraki 316-8511, Japan
[email protected]
4
Aoyama Gakuin University, 5-10-1 Fuchinobe, Chuo-ku, Sagamihara-shi,
Kanagawa 252-5258, Japan
[email protected]

Abstract. We propose a method for estimating new tags to enrich folk-

sonomy on video sharing sites using various video information. Folkson-
omy is a process by which users tag videos to facilitate searches. One
such website is Nico Nico Douga, although users of the site cannot post
more than 12 tags to a video. Consequently, although some important
tags will be posted, others might be missing. We present a method for
acquiring some of these missing tags by choosing new tags. The method
uses comments, tags, titles, descriptions, and thumbnails to estimate new
tags. Using these data, we develop various algorithms, including neural
networks, word embeddings, and K-means. Our method indicated high
performance in experimental results.

1 Introduction
Online videos have become an indispensable part of our daily lives. Every day,
millions of people watch videos on video-sharing websites such as Nico Nico
Douga1 and post comments. Moreover, since Internet election campaigns became
available in Japan in April 2013, the demand for video sharing websites has been
increasing. On Nico Nico Douga, users can post tags and comments on videos.
Moreover, the comments are overlaid directly on the video and synchronized to
a specific playback time. Figure 1 shows examples of user tags and comments.
Nico Nico Douga is a Japanese video sharing website owned by the DWANGO
Co., Ltd. It was launched on December 12, 2006 and currently has over
70,000,000 registered users. Currently, it has over 70,000,000 registered users.
As of 2018, the site has over 16,147,000 videos and 5,544,805,000 comments.
1
http://www.nicovideo.jp/.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 301–312, 2020.
https://doi.org/10.1007/978-3-030-15032-7_26
302 H. Sakaji et al.

Fig. 1. Example of video on Nico Nico Douga

On the site, tags are essential for characterizing, retrieving, and clustering
videos. For example, the tag “game” indicates that videos tagged with it are
related to games. Users tag videos so that they can be searched for easily, a
process known as folksonomy. Moreover, some tags express user ratings. For
example, the tag “should be valued” expresses a positive user rating. However,
users cannot post more than 12 tags to a video on Nico Nico Douga. As such,
whereas some important tags are posted, others might be missing. Further, new
videos do not fetch enough tags. Consequently, it is relatively difficult to retrieve
new videos from among all the videos on the website.
Given these circumstances, Wu et al. [1] and Lv et al. [2] proposed methods
that extract tags from comments on time-synced videos such as those on Nico
Nico Douga. However, their methods do not consider existing tags, which are
pertinent to video cataloging. Therefore, we previously proposed a method for
estimating new tags for online videos using both comments and existing tags [3].
Despite exploiting more information, our method was limited to these two
sources when estimating tags. Indeed, the use of additional information would
improve its performance. Thus, we considered various kinds of information to
estimate tags for videos. In this paper, in addition to tags and comments, we
consider thumbnail images, video titles, and video descriptions.
In what follows, we propose a method for estimating new tags to enrich folk-
sonomy on video sharing sites using various video information. We first calculate
the scores of pairs comprising a tag and a comment. These scores are derived
using information theory and statistics. Then, we make vectors from titles and
descriptions using a word embedding technique and generate tag clusters using
these vectors. Next, a neural network model classifies thumbnail images using
the tag clusters. Finally, we estimate new tags based on the calculated scores,
tag vectors, neural network, and the co-occurrence of existing tags.
 Estimation of Tags Using Various Data for Online Videos 303

Contributions of this research are followings:

• For estimating tags on online videos, we made clear what kind of information
is useful.
• Various algorithms for processing various data were developed.

Furthermore, we consider that our method can be applied other video sites such
as YouTube2 .

2 Related Work
Wu et al. proposed a method for tagging videos using comments and a temporal
and personalized topic model [1]. Their method tags scenes of videos instead of
videos. Ikeda et al. proposed a method for annotating comments by referring to
the content of the comments [4]. Additionally, Ikeda et al. structured a tag set
based on the content of the comments. Lv et al. proposed a video-understanding
framework for assigning temporal labels on highlighted video shots [2]. Their
method assigns tags or labels to comments or scenes of videos. Conversely, our
task involves assigning tags to videos. Moreover, Lin et al. proposed a statistical
method to identify whether a Chinese word is a loanword from Japanese [5].
They focused on the website Bilibili3 . Nevertheless, their method identifies tags,
but cannot estimate them.
Hotho et al. proposed a method that recommends tags for users [6]. They
call this method FolkRank. FolkRank is an improvement of the Adapted PageR-
ank approach. This approach adapts the Google PageRank algorithm to rank the
nodes within the graph structure of a folksonomy on the basis of their importance
in the network [7]. A modified form of the k-nearest neighbors algorithm for tag
recommendation in folksonomies is presented in Gemmell et al. [8]. Additionally,
Gemmell et al. also proposed a linear-weighted hybrid algorithm for tag-based
resource recommendation [9]. Budura et al. proposed a method for predicting
tags on the basis of the neighbourhood of a document-tag graph [10]. They pro-
posed harvesting adjacent metadata in large-scale tagging systems (HAMLET),
a suite of principles, scoring models, and algorithms for propagating metadata.
However, it is unknown whether their approach can predict more than 40,000
tags or not.
Krestel et al. proposed a method based on latent Dirichlet allocation for
recommending tags of resources in order to improve searches [11]. Ramage et
al. proposed a topic model labeled LDA that considers a one-to-one correspon-
dence between LDA’s latent topics and user tags [12]. In a recent study, Kowald
proposed a modelling method based on cognitive processes for improving tag
recommendations [13]. His study uses LDA for analysing micro-level processes.
Since we have confirmed that LDA-based methods are not suitable for predicting
a large number of tags in past research [14], it is difficult to apply these methods.

2
https://www.youtube.com/.
3
http://www.bilibili.com/.
304 H. Sakaji et al.

3 Score Calculation
In this section, the method of scoring the pairs of tags and comments is presented.

3.1 Category Tags

A category tag is part of a complete tag and indicates the genre of a video
on Nico Nico Douga. Many category tags are used with high frequency. For
example, the frequency of the category tags “game” and “music” on the website
was 3,593,302 and 874,535, respectively. Such high frequency occurs because the
website recommends that users give at least one category tag for each video.
Therefore, category tags must be taken into account when calculating the scores
of comments and tags using the proposed method.

3.2 Scoring

In this section, we explain the formulas used in the proposed method to score
the comments and tags. Before calculating the scores of pairs of tags and com-
ments, we normalized all comments using Normalization Form Compatibility
Composition4 .
A measurement of the correlation between a tag (t) and a comment (c) is
needed, expressed by the score S(t, c), which is calculated as follows:

tdf (t)
S(t, c) = P M I(t, c) × IT F (c) × (0.5 + 0.5 × ), (1)
maxt tdf (t)

where P M I(t, c) denotes the pointwise mutual information (PMI) [15] of a pair
comprising t and c. The PMI is a commonly used metric for measuring the cor-
relation between two events. In this study, we measured the correlation between
tags and comments using the PMI. Additionally, IT F (c) represents the inversed
tag frequency. Here, tag frequency refers to the frequency of the types of tags
appearing in a comment. High weight in IT F (c) is reached by the low tag fre-
quency of a comment. Moreover, tdf (t) is the frequency of tag t in a set com-
prising all videos on the website. Herein, we improved tdf (t) to prevent a bias
towards category tags. P M I(t, c) and IT F (c) are calculated using Eqs. 2 and 3,
respectively.
p(t, c)
P M I(t, c) = log2 , (2)
p(t)p(c)
where p(t, c) is the probability of t and c occurring on the same video, p(t) is
the probability of tag t occurring in a set of videos, p(c) is the probability of
comment c occurring in a set of videos.

maxc tf (c)
IT F (c) = log2 , (3)
tf (c)
4
http://www.unicode.org/reports/tr15/.
 Estimation of Tags Using Various Data for Online Videos 305

where tf (c) is the number of the types of t occurring in c.

Initially, the factor tdf (t)/maxt tdf (t) was used to prevent the acquisition of
rare tags when estimating new tags. For example, our method estimated “Tajima
cattle” as a new tag but this was inappropriate owing to its rarity and because
the scores for pairs comprising any general comment with this rare tag are high.
Therefore, we normalized tdf (t) by dividing it by maxt tdf (t).
Nevertheless, this method estimated category tags as inappropriate tags
because the scores for the category tags as general tags were high, whereas
the scores of rare tags were low. Therefore, we constructed a weight factor, viz.,
(0.5 + 0.5 × (tdf (t)/maxt tdf (t))), for Eq. 1 with its value restricted to 0.5–1.

4 Tag Vectors

In this section, we explain the generation of tag vectors. To make tag vectors,
we first make vectors from titles and descriptions. Our previous method did
not use this information. To distinguish these vectors from tag vectors, vectors
derived from titles and descriptions are hereafter referred to as TD vectors. The
algorithm for making TD vectors is as follows:

Step 1: create word embeddings from titles and descriptions.

Step 2: make a TD vector by averaging the value of the word embeddings of the
word included in the title and the description in a video.

Then, our method generates tag vectors using TD vectors. The algorithm for
making tag vectors is as follows:

Algorithm 1. Making tag vectors

Input: Video set V = (v0 , v1 , ..., vn ) and TD vectors T DV
v: Video (T, vid)
T : Tag set (t0 , t1 , ..., tm )
vid: Video id
Output: Tag Vectors T V
1: T V ← ∅
2: T Vcount ← ∅
3: for each v ∈ V do
4: T, vid ← V
5: for each t ∈ T do
6: T V [t] ← T DV [vid]
7: T Vcount [t] ← T Vcount [t] + 1
8: end for
9: end for
10: for each t ∈ T V do
11: T V [t] ← T V [t]/T Vcount [t]
12: end for
13: return T V
306 H. Sakaji et al.

In Algorithm 1, T V is an associative array comprising tags as keys and vectors

as values. Additionally, T Vcount is an associative array comprising tags as keys
and the number of tags as values. The algorithm calculates the average TD
vectors for each tag in Algorithm 1. Then, the values are used as tag vectors in
this research.

5 Neural Network Model for Classifying Thumbnails

In our previous research, we did not use image data such as video data and
images, despite targeting video sharing sites. Indeed, it is difficult to gather a
satisfying amount of learning data from video data, given the size of video files.
However, thumbnail images are small in size, and Nico Nico Douga offers an API
that can gather these. Therefore, our proposed method uses thumbnail images
to estimate tags, rather than video data.
Furthermore, because neural network models perform well with images [16–
18], we employ a neural network. However, there are many tags to learn using
thumbnail images, with more than 40,000 tags on Nico Nico Douga. Thus, we
created a neural network model to classify tags into 1,000 tag clusters. We created
these tag clusters using the tag vectors explained in Sect. 4. Since this task
required processing more than 40,000 tags, we employed Mini Batch K-Means
[19]. Mini Batch K-Means is often used for clustering such quantities.

5.1 Neural Network Model

In this section, we introduce our neural network model for classifying tags into
1,000 tag clusters. To train our model using thumbnail images, we needed strong
GPUs and time to learn models such as ResNet [18]. However, we could not
accomplish this because we did not have strong GPUs and enough time. There-
fore, we choose to tune our model based on previously learned published models.
We employed ResNet-50 in Pytorch5 to use the learned model. The output
size of the last layer was changed to 1,000, and the model was relearned. Here,
only the weight of the last layer was learned. The weights of other layers were
fixed such that they did not update.
The model learns by considering the task as a multilabel problem. Therefore,
we made labels for learning the tag clusters using Algorithm 2. Beforehand, serial
numbers were added to the tag clusters. The labels were generated by applying
Algorithm 2 to videos included in the learning data. In Algorithm 2, L is gener-
ated as an array with 1,000 elements of 0. As a result of a grid search, we decide
to use the following parameters. The learning rate is 0.1, and the batch size is
64 and epoch is 300.

5
https://pytorch.org/docs/stable/torchvision/models.html#torchvision.models.
resnet50.
 Estimation of Tags Using Various Data for Online Videos 307

Algorithm 2. Making labels

Input: Tag Set of a video T = (t0 , t1 , ..., tn ) and tag cluster list T CL = (tl0 , tl1 , ..., tl999 )
tlm : the tag set included in the m-th tag cluster
Output: Labels L
1: L ← 0
2: i ← 0
3: for each tl ∈ T CL do
4: for each t ∈ T do
5: if t ∈ tl then
6: L[i] ← 1
7: i←i+1
8: end if
9: end for
10: end for
11: return L

6 Estimation

In this section, we introduce our method for estimating new tags using the cal-
culated scores, the tag vectors, the neural network model, and the co-occurrence
of existing tags. First, the databases that are used for the estimation of new tags
are explained.

6.1 Databases

We created two databases for estimating new tags. One database, Dc , is used
to search for tags with high scores S(t, c) when they appear with a user-entered
comment. The other database, Dt , is used to search for tags that frequently
co-occur with the entered tag. These databases were constructed by Kyoto
Cabinet6 .

6.2 Estimation Algorithm

Our method estimates tags using Algorithm 3. Here, Tsc is an associative array
comprising tags as keys and scores as values. Additionally, our method corrects
the scores given in Line 11 of Algorithm 3.
In Algorithm 3, Dc and Dt indicate the two databases used in this study.
Score(t, c) is a function that computes the score S(t, c) of a pair comprising a t
and a c; ExtractTags(Tsc ) is a function for acquiring N tags with a high score.
In this study, N = 5.
In Algorithm 3, ApplyOtherScores(Tsc ) is a function that applies the values
of the tag vectors, the neural network model, and the co-occurrence of existing
tags. In this research, we evaluate all combinations of the above three values.
We now discuss how these values are applied to Tsc .

6
http://fallabs.com/kyotocabinet/.
308 H. Sakaji et al.

Algorithm 3. Estimation of tags

Input: A video V = (C, T )
C: comment set (c0 , c1 , ..., cn )
T : tag set (t0 , t1 , ..., tm )
Output: New tag set T́
1: T́ ← ∅
2: Tsc ← ∅
3: for each c ∈ C do
4: for each t ∈ Dc (c) do
5: Tsc [t] ← Tsc [t] + Score (t, c)
6: end for
7: end for
8: Tsc ← ApplyOtherScores(Tsc )
9: T́ ← ExtractTags(Tsc )
10: return T́

Algorithm 4. Applying co-occurrence of existing tags

Input: Score Tsc and tag set T = (t0 , t1 , ..., tn )
Output: New score Tsc
1: for each t ∈ T do
2: for each t́ ∈ Dt (t) do
3: if t́ ∈ Tsc then
4: Tsc [t́] ← Tsc [t́]× Co-occurrence(t, t́)
5: end if
6: end for
7: end for
8: return Tsc

Algorithm 5. Applying tag vectors

Input: Score Tsc , TD vector tdv, and tag vectors T V
Output: New score Tsc
1: for each t ∈ Tsc do
2: Tsc [t] ← Tsc [t]× Cosine(tdv, T V [t])
3: end for
4: return Tsc

Algorithm 6. Applying the neural network model

Input: Score Tsc , thumbnail tn, and tag cluster list T CL = (tl0 , tl1 , ..., tl999 )
Output: New score Tsc
1: P rob ← model(tn)
2: for each t ∈ Tsc do
3: for each tli ∈ T CL do
4: if t ∈ tli then
5: Tsc [t] ← Tsc [t] × P rob[i]
6: end if
7: end for
8: end for
9: return Tsc
 Estimation of Tags Using Various Data for Online Videos 309

Algorithm 4 applies the co-occurrence of existing tags to scores. Algorithm 5

applies a cosine value of the TD vector and tag vectors to scores. Algorithm 6
applies the output of the neural network model to the scores. In Algorithm 4,
Co-occurrence(t, t́) is a function that calculates the number of co-occurrences of
tag t and tag t́ in a single video that appears in a set of videos. In Algorithm 5,
Cosine(tdv, T V [t]) is a function that calculates the cosine value of the TD vector
tdv and the tag vector T V [t]. In Algorithm 6, P rob is the output of the neural
network model and the predicted probability. Additionally, Model(tn) a function
that applies the thumbnail tn to the neural network model.

7 Evaluation
In this section, we describe our evaluation of the proposed method. Nico Nico
datasets7 were used for the evaluation. Scores were calculated using 779,154
videos, which were extracted randomly. In addition, 26 test videos (exclusively
from the abovementioned 779,154 videos) were randomly selected from the Nico
Nico dataset for estimating new tags. For learning CNN (Sect. 5.1 model), we
extracted 50,000 videos as training data, 10,000 videos as validation data, 10,000
videos as test data, from 779,154 videos that were used for calculating scores.
Moreover, a baseline was evaluated that estimates tags on the basis of the
frequency of tags and comments. We manually evaluated the estimated new

Fig. 2. Evaluation results

7
http://www.nii.ac.jp/cscenter/idr/nico/nico.html.
310 H. Sakaji et al.

Table 1. Example of estimated tags

Video ID Existing tags Estimated tags

sm10864587 Nature BGM:1, Entertainment:1,
Science:1, Wind:1, Cooking:0
sm13139215 VOCALOID, Ren Kagamine, etc. Miku Hatsune:1, Kaito:1,
Luka Megurine:1, Gumi:1,
Game:0
sm10036862 Domestic trip, Diary, Trip, etc. Nature:1, Entertainment:0,
Cooking:0, Overseas:0,
In-vehicle Video:1
sm191300 Anime Lucky star:1, BGM:1, Last of
impact:0, MAD:0, Nanoha:0

tags for the 11 test videos. Fig. 2 shows the results for our method and for the
baseline. Table 1 shows an example of the estimated tags.
In Fig. 2, “Tag” refers to applying Algorithm 4 to the scores. “Vec” refers to
applying Algorithm 5 to the scores. “CNN” refers to applying Algorithm 6 to the
scores. “All” refers to applying Algorithms 4, 5, and 6 to the scores. “Normal”
indicates that Algorithm 3 does not apply ApplyOtherScores(Tsc ). “Frequency”
denotes the baseline.
In Fig. 2, “Pre@1” indicates that the highest-scored new tag for each video
was evaluated as an appropriate tag. “Pre@3” means that all of the top three
highest-scoring tags were evaluated as appropriate tags. “Pre@5” means that all
of the top five highest-scoring tags were evaluated as appropriate tags. “Top@3”
indicates that the estimation of tags was correct when an appropriate tag existed
within the top three highest-scoring new tags. “Top@5” indicates that the esti-
mation of the tags was correct when an appropriate tag existed within the top
five highest-scoring new tags.
In Table 1, “:1” refers to an appropriate tag, and “:0” refers to an inappro-
priate tag. These marks were assigned manually by an evaluator.

8 Discussion
As shown in Fig. 2, the proposed method (Tag Vec) achieved a precision of 0.65
when the evaluation criterion was Pre@1. However, the precision of the baseline
was only 0.19 when the evaluation criterion was Pre@1. All the evaluations
indicate that the proposed method outperformed the baseline. Our previous
work [20] showed that the frequency was not useful when analyzing comments
on Nico Nico Douga. Therefore, we built new methods that did not employ the
frequency, instead applying new tasks to analyze the comments on Nico Nico
Douga.
Figure 2 shows that the neural network model for thumbnails was ineffective
for this task. On the other hand, existing tags were indeed useful when estimat-
ing tags. Additionally, the experimental results show that Tag Vec achieved the
 Estimation of Tags Using Various Data for Online Videos 311

best Pre@1. From these results, we conclude that comments, tags, titles, and
descriptions are useful when estimating tags, whereas thumbnails are not.
As shown in Table 1, the proposed method estimated appropriate new tags on
videos that have just one user-entered tag. For example, the method estimated
the appropriate new tags “BGM,” “Entertainment,” “Science,” and “Wind” for
the video “sm10864587,” which has just one user-entered tag, viz., “Nature.”
Consequently, the proposed method can appropriately estimate new tags for
videos that have only one or a few user-entered tags.
Moreover, the proposed method estimated additional new tags for videos
that have 11 user-entered tags. This means that new tags can completely fill in
all the available tag slots for a video on Nico Nico Douga, recalling that users
are collectively able to enter up to 12 tags for each video on Nico Nico Douga.
For example, for the video “sm13139215,” four additional estimated tags were
evaluated as appropriate, viz., “Miku Hatsune,” “Kaito,” “Luka Megurine,” and
“Gumi.” These results indicate that if the proposed method estimates a new
tag, the users can retrieve their favorite videos using a newly estimated tag even
for videos where users have not entered a tag manually.

9 Conclusions
In this paper, we proposed a method that estimates new tags for videos. First,
the method calculates the scores for pairs of a tag and a comment. We used the
pointwise mutual information to calculate scores. Then, we made vectors from
titles and descriptions using a word embedding technique, and generated tag
clusters using these vectors. Further, we created a neural network model that
classifies thumbnail images into tag clusters. We estimated new tags using an
algorithm we developed. The results of our evaluation showed that a precision of
0.65 was achieved for the highest-scored new tags for each of the videos evaluated
using the proposed method.
In future research, we shall employ audio data and moving image data to
estimate new tags. Additionally, we have plan to apply our method to other
video sharing sites such as YouTube, Bilibili.

References
1. Wu, B., Zhong, E., Tan, B., Horner, A., Yang, Q.: Proceedings of the 20th
ACM SIGKDD International Conference on Knowledge Discovery and Data Min-
ing, KDD 2014, pp. 721–730. ACM, New York (2014). https://doi.org/10.1145/
2623330.2623625
2. Lv, G., Xu, T., Chen, E., Liu, Q., Zheng, Y.: The 30th AAAI Conference on
Artificial Intelligence (2016)
3. Sakaji, H., Kohana, M., Kobayashi, A., Sakai, H.: 2016 19th International Confer-
ence on Network-Based Information Systems (NBiS), pp. 550–553. IEEE (2016)
4. Ikeda, A., Kobayashi, A., Sakaji, H., Masuyama, S.: 18th International Conference
on Network-Based Information Systems (NBiS), pp. 673–678 (2015). https://doi.
org/10.1109/NBiS.2015.101
312 H. Sakaji et al.

5. Lin, X., Ito, E., Hirokawa, S.: 13th International Conference on Computer and
Information Science (ICIS), pp. 163–166 (2014). https://doi.org/10.1109/ICIS.
2014.6912126
6. Hotho, A., Jäschke, R., Schmitz, C., Stumme, G.: European Semantic Web Con-
ference, pp. 411–426. Springer (2006)
7. Jäschke, R., Marinho, L., Hotho, A., Schmidt-Thieme, L., Stumme, G.: European
Conference on Principles of Data Mining and Knowledge Discovery, pp. 506–514.
Springer (2007)
8. Gemmell, J., Schimoler, T., Ramezani, M., Mobasher, B.: Proceedings of the 7th
International Conference on Intelligent Techniques for Web Personalization & Rec-
ommender Systems, vol. 528, pp. 69–80. CEUR-WS.org (2009)
9. Gemmell, J., Schimoler, T., Mobasher, B., Burke, R.: International Conference on
User Modeling, Adaptation, and Personalization, pp. 111–122. Springer (2011)
10. Budura, A., Michel, S., Cudré-Mauroux, P., Aberer, K.: European Semantic Web
Conference, pp. 608–622. Springer (2009)
11. Krestel, R., Fankhauser, P., Nejdl, W.: Proceedings of the Third ACM Conference
on Recommender Systems, pp. 61–68. ACM (2009)
12. Ramage, D., Hall, D., Nallapati, R., Manning, C.D.: Proceedings of the 2009 Con-
ference on Empirical Methods in Natural Language Processing, vol. 1, pp. 248–256.
Association for Computational Linguistics (2009)
13. Kowald, D.: Modeling cognitive processes in social tagging to improve tag recom-
mendations (2018)
14. Sakaji, H., Kohana, M., Akio, K., Sakai, H.: Int. J. Grid Utility Comput. (2018).
https://www.inderscience.com/info/ingeneral/forthcoming.php?jcode=ijguc
15. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley-Interscience,
New York (1991)
16. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D.,
Vanhoucke, V., Rabinovich, A.: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pp. 1–9 (2015)
17. Simonyan, K., Zisserman, A.: arXiv preprint arXiv:1409.1556 (2014)
18. He, K., Zhang, X., Ren, S., Sun, J.: Proceedings of the IEEE Conference on Com-
puter Vision and Pattern Recognition, pp. 770–778 (2016)
19. Sculley, D.: Proceedings of the 19th International Conference on World Wide Web,
pp. 1177–1178. ACM (2010)
20. Sakaji, H., Ishibuchi, J., Sakai, H.: Int. J. Space-Based Situated Comput. 6(3), 165
(2016)
 Framework for Feature Selection in Health
Assessment Systems

Ayesha Ubaid(&), Fan Dong(&), and Farookh Kadeer Hussain

Centre for Artificial Intelligence, UTS, Sydney, Australia

[email protected],
{Fan.Dong,Farookh.Hussain}@uts.edu.au

Abstract. Anomaly detection in health assessment systems has gained much

attention in the recent past. Various feature selection techniques have been
proposed for successful anomaly detection. However, these methods do not
cater for the need to select features in health assessment systems. Most of the
present techniques are data dependent and do not offer an option for incorpo-
rating domain information. This paper proposes a novel domain knowledge-
driven feature selection framework named domain-driven selective wrapping
(DSW) that can help in the selection of a correlated feature subset. The proposed
framework uses an expert’s domain knowledge for the selection of subsets. The
framework uses a custom-designed logic-driven anomaly detection block
(LDAB) as a wrapper. The experiment results show that the proposed frame-
work is able to select feature subsets more efficiently than traditional sequential
selection methods and is very successful in detecting anomalies.

Keywords: Features
Health assessment
Smart questionnaire

1 Introduction

New innovations in technology have led to the collection and storage of massive
amounts of data. This data can be used for knowledge discovery and based on this data,
different hypothesis can be made. The stored data can be used in machine learning
(ML) algorithms to make predictions for future events. To use the data in ML algo-
rithms, we need to select some of the most influential data elements called features.
Features may be defined as the quantitative property of the process being observed.
Feature selection (FS) is the process of selecting subsets of relevant variables from the
original data set which represent the data effectively. It helps in understanding data and
reducing computation time, noise and data dimensionality, resulting in increased pre-
diction performance [1, 2]. FS does not create new features, rather it helps in obtaining
a subset of existing features. If two features represent the same meaning of data, only
one of them is selected. Figure 1 shows the generic feature selection process.

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 313–324, 2020.
https://doi.org/10.1007/978-3-030-15032-7_27
314 A. Ubaid et al.

Dataset

Feature Selection
Technique β

Information Feature Subset

Objective Function

Final Feature set

Fig. 1. Generic feature selection process

At the start of the feature selection processes, the full dataset is fed into the FS
algorithm which splits the dataset into subsets of selected features. These subsets are
used as input to the objective function to evaluate their effectiveness. If an objective
function is achieved, the selected subsets become part of the final feature set. However,
if not, the FS process continues until it is achieved. The final feature set is then used in
respective machine learning algorithms for algorithm training.
Current FS algorithms tend to select variant data. These algorithms are highly data
dependent i.e. the presence of data is necessary for analysis to observe the relationship
between the features. Additionally, these techniques are unable to incorporate any extra
information into their process. However, there are certain applications in which no data
are available to create a mathematical/probabilistic relationship between features and
some flexibility is needed to incorporate some inputs into the FS process.
To overcome these shortcoming, we propose a novel hybrid feature selection
framework named domain-driven selective wrapping (DSW). DSW is a unique
framework that combines the advantages of filters and wrappers while reducing their
disadvantages. The framework is capable of incorporating input from domain experts
into the feature selection procedure. The approach is highly dependent on domain
information but can easily be replaced or combined with mathematical models. DSW is
able to extract features more efficiently than traditional sequential FS techniques. It uses
a logic-driven anomaly detection block (LDAB) as a wrapper to measure the effec-
tiveness of the selected features. LDAB is a customized anomaly detection block
especially designed to detect anomalies in health assessment questionnaires. The
implementation details however are not in the scope of this paper.
 Framework for Feature Selection in Health Assessment Systems 315

This paper is organized as follows. In Sect. 2, the related work is summarized.

Section 3 explains the proposed feature selection framework for health assessment
systems followed by its analysis in Sect. 4. The paper is concluded in Sect. 5.

2 Related Work

There are different methods through which to achieve feature selection. These include
filters, wrappers, embedded methods and classification algorithms [1]. A feature is said
to be useful if it has unique information about the classes in the data set [2]. Features
can be discarded only if they have no influence on class labels [3]. Filters are pre-
processors for feature ranking. Features with high ranking are selected as input to the
prediction algorithms. Wrappers, on the other hand, depend on the performance of the
predictors [1, 2]. In addition to these two methods, there are embedded and classifi-
cation algorithms which are also useful for feature selection.
Filter methods are used as primary selection criteria and are also known as ranking
methods [1, 5, 6]. They are used to rank a variable’s usefulness. They set up a
threshold, the features below which are discarded by filtering out irrelevant data.
Correlation criteria is the most well-known and commonly used filter method. The
Pearson correlation coefficient is used to detect the linear dependencies between input
(variable) and output (class). Mutual information (MI) is another filtration technique. It
is a measure of dependencies between two variables. MI can be calculated for both
discrete and continuous variables. MI between X and Y is zero if they are independent
and is greater if dependent. MI alone provides poor results, so it is used in combination
with embedded methods to achieve better results.
Wrapper methods rely on classification algorithms to produce results. Wrappers use
classification models as a black box and their performance as the objective function to
select a subset of features [2]. These can be divided into two classes, sequential
selection algorithms and heuristic search algorithms [11]. Sequential selection algo-
rithms are iterative in nature. Sequential forward selection (SFS) starts with an empty
feature set in the first iteration and the empty feature set is filled with a single feature to
obtain the maximum value of the objective function [4]. Starting from the second
iteration, each individual feature is added into the subset and its effectiveness in relation
to the objective function is measured. Added features take a permanent position in the
feature set if they provide the maximum value of the objective function. Sequential
backward selection (SBS) starts with a full data set and sequentially removes from the
feature set those features whose removal has a minimum effect on predictor perfor-
mance. The sequential floating forward selection (SFFS) algorithm, proposed in [4],
adds a new step in basic SFS, which is to remove a feature from the existing subset and
check if it provides the maximum value of the objective function. Another variant of
SFS is the adaptive SFFS algorithm (ASFFS), which reduces the correlation amongst
the feature set. A parameter r is calculated which specifies the number of features to be
added in the next inclusion [2]. The parameter r is selected randomly. The plus-L-
minus-r search algorithm is similar to naïve SFS. At each cycle, L variables are added
and r variables are removed from the subset of the dataset. L and r are selected
randomly. Heuristic search algorithms include genetic algorithms (GA) as explained in
316 A. Ubaid et al.

[13] and can be used for feature selection. In GAs, chromosome bits show if a feature
should be included or not in the feature set. The parameters of GA can be changed
based on the data and the application. A modification of GA called CHCGA is pro-
posed in [14]. The CHCGA selects the best N individuals from a pool of parents. Better
offspring replace less fit parents.
The embedded methods explained in [1, 9, 10] reduce the computation required to
classify different subsets. They combine feature selection and the training process. In
[12], a greedy search algorithm is used to find MI between the features and output class
as well as between the subsets. Max-relevancy min-redundancy (mRMR) is another
embedded method, similar to the greedy search algorithm.
Classifiers can also be used for feature selection. Weights are assigned to the
features and the linear relation between input and output is established. Such a feature
removal technique is called recursive feature elimination. It is used in SVM and is
called SVM-RFE. The same methodology can be adopted in neural networks
(NNs) and linear regression. Clusters e.g. K-nearest neighbors (KNNs) help in
grouping features that are naturally related to each other. Details of unsupervised
methods for feature selection can be found in [7, 16–19]. Semi-supervised methods for
feature selection are investigated in [7, 20, 21].
In addition to these techniques, the role of the domain expert is also of great
importance in certain areas, such as the medical and finance fields etc. Input from
domain experts while selecting features can improve the selection process exponen-
tially [8, 15]. But no formal framework for incorporating the domain expert has been
designed as yet.
To address this research gap, a new feature selection framework is proposed in the
next section which can incorporate input from the domain expert at each stage of the
feature selection process.

3 The Proposed Framework – Domain-Driven Selective

Wrapping (DSW)

DSW is a hybrid framework that offers a twofold method for the selection of data in
conjunction with domain knowledge. DSW also allows the incorporation of user
defined parameters in FS to increase feature selection efficiency. It is suitable for
applications where correlated data is required to detect anomalies when there is no
previous data for developing mathematical models.
DSW is applied to select features that can help detect anomalies in smart ques-
tionnaires in health assessment systems. It is capable of selecting subsets of correlated
features from a feature set. The selected features can help identify possible medical
risks for a potential candidate by identifying anomalies in the provided information.
The dataset used comprises questions which help in the assessment of different medical
conditions. The selected feature set is represented by a group of subsets. Each subset
has a collection of questions that are highly correlated and can represent a single
medical condition. Thus, in the final feature set, each medical condition under obser-
vation is represented by a subset of features that collectively can provide insight into it.
DSW is a two-phased framework, as shown in Fig. 2.
 Framework for Feature Selection in Health Assessment Systems 317

Phase-1 Phase-2

Fig. 2. Domain-driven selective wrapping

Phase 1: Domain Knowledge-Based Filtration: In this phase, preliminary filtration

is performed based on the domain expert’s knowledge. At the start of this phase, the
whole data set is fed into the filtration block for analysis. Depending on the data
content, a parameter b is also initialized and is given as input to the filtration phase. The
value of b is set by the domain expert depending on the application and depicts the
number of groups in which the data has to be filtered for further processing. Let D be
the original data set and d represents the filtered subsets, then

D ¼ d1 [ d2 [ . . . [ db
di \ dj ¼ /; i 6¼ j; i; j ¼ 1; 2; . . .; b ð1Þ

Phase 2: Domain Knowledge-Based Correlation Analysis and Wrapping: The

second phase is wrapping one of the filtered datasets d using parameters defined by the
domain expert. These parameters include n, representing the total number of features in
a subset where n  2 and 0 gi representing the subset identifier. The subset gi depicts a
correlated group of features in the subset. Thus, in an ith iteration, n number of features
are selected related to the same gi . n and gi are decided by analyzing the correlation
between features by the domain expert. A subset is selected from the input dataset d for
the ith iteration. Hence, dataset d can be expressed as


d ¼ g1; ; g2 ; . . .:gn ð2Þ
318 A. Ubaid et al.

Also


g1; ; g2 ; . . .:gn  d ð3Þ

Features belonging to the same group become part of the same subset. The selected
subset is used is used to check the effectiveness of the objective function. If the subset
is able to provide the maximum value of the objective function, all the features forming
the subsets are removed from the original dataset. Hence, the remaining dataset can be
described as

d ¼ d  gi ð4Þ

If the objective function is not achieved, the subset remains part of the original
dataset. In this case, Eq. (4) becomes

d ¼ d þ gi ð5Þ

The selected subset is described by Eq. (6), where x represents the features in the ith
subset derived from d.

  x 2 gi ; 0\r ð xÞ\1
gi ¼ x1; ; x2 ; . . .:xn ; where gðiÞ ¼ ð6Þ
x 62 gi ; r ð xÞ\0

In Eq. 6, r ð xÞ represents the Pearson correlation coefficient, which can be calcu-

lated using Eq. (7). Equation (6) shows that all the questions with a positive correlation
belong to the same question group. Any value between 0 and 1 indicates a positive
correlation between the set of questions. Values less than or equal to 0 indicate a
negative correlation between the questions. If no data is available, the r ð xÞ calculation
is highly dependent on the experience of the domain expert. Once the data is generated
by the initial setup, it can be incorporated into the system. This combination of domain
knowledge and data is important because the domain expert may not be able to predict
hidden relationships in the data.

covðxi ; Y Þ
r ð xÞ ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð7Þ
var ðxi Þ  var ðY Þ

Application of DSW on Health Assessment Dataset for FS: DSW was applied on a
dataset from Australia’s biggest pre-employment Health Assessment Company. The
dataset has two parts, one related to the medical assessor and other to the candidate.
The assessor part consists of questions which are asked by the health assessor to the
candidate. These include inquiries related to clinical and physical examinations etc. The
candidate part contains group of questions to summarize the candidate’s medical his-
tory. To filter these two groups, the domain expert’s input is incorporated into the
filtration block. Thus, in the first phase, the dataset is preliminarily filtered and is
 Framework for Feature Selection in Health Assessment Systems 319

partitioned into subsets defined by b In our application, b ¼ 2, and Eq. (1) can now be
expressed as
[
D ¼ QA QC ð8Þ

According to Eq. (8), the original data set D is divided into two subsets. QA is the
assessor dataset and QC is the candidate dataset. Furthermore, QA  D and QC  D.
Both groups are filtered such that QA \ QC ¼ 0. Once QA and QC are separated, QC is
fed into the second phase of DSW for correlation analysis. At the start of phase 2, the
value of n and the subset identifier is set. In our application, the limits of n are
described as 2  n  5 and the feature group identifier is named as the question id (qi ).
Each question in QC is assigned a unique gi which depicts its relation to a specific
medical condition. Questions with the same qi relate to the same medical condition and
have a positive correlation with each other. Once n and q Id are specified, the domain
expert selects a subset from QC specific to qi for the ith iteration. Thus, in every
iteration, using n and qi , a group of questions is selected until QC is grouped into
subsets. Equations (2) and (3) can become Eqs. (9) and (10) respectively

QC ¼ fq1 ; q2 ; . . .:qn g ð9Þ

Also


q1; ; q2 ; . . .:qn  d ð10Þ

  x 2 qi ; 0\r ð xÞ\1
qi ¼ x1; ; x2 ; . . .:xn where qð i Þ ¼ ð11Þ
x 62 qi ; r ð xÞ\0

To be part of the ith subset, the criteria described in Eq. (11) must be satisfied. The
value of the Pearson correlation coefficient can be calculated using Eq. (7). If the
objective function is achieved (i.e. detection of anomalies), then the remaining dataset
can be defined as

QC ¼ QC  qi ð12Þ

On the other hand, if the subset is found to be ineffective in detecting anomalies, it

is merged back into the dataset. This can be expressed as

QC ¼ QC þ qi ð13Þ

Objective Function: To check the validity of the selected features, subsets are fed into
the LDAB which is designed to detect abnormal data. LDAB is a domain knowledge-
based logic-driven anomaly detection block that detects abnormal data based on the
logical evaluation of features. This block is independent of the probabilistic calcula-
tions and relies solely on the medical relationship between the questions. To be part of
the final feature set, the selected features from a particular question group should
320 A. Ubaid et al.

provide insight into the anomalies present in the candidate’s answers. If the objective
function is achieved, i.e. selected qi provides insight into intentional or unintentional
anomalies in the candidate’s answers, qi is removed from QC and is added into the final
feature set. The remaining QC starts over the DSW phase 2 for the next iteration along
with new n and qi values. The process continuous until all the required subsets are
formed from the candidate dataset. Thus, at each ith iteration, Qi is formed based on n
until the whole dataset is partitioned into subsets. The effectiveness of the subsets is
analyzed using LDAB. The selection criteria of the subset are highly dependent on the
domain knowledge of the expert. The greater the knowledge and experience of the
domain expert, the more refined the subsets will be.

4 Results and Discussion

In order to observe the effectiveness of DSW, a dataset from one of Australia’s largest
pre-employment health assessment companies is used. LDAB is used as a wrapper. The
results from LDAB finalize the selection of the subsets.
At the start of phase I, parameter b is set to 2. DSW divides the data set into two
subsets, QA and QC respectively. This filtration is done by the domain expert. We used
QC with 50 variable feature sets as input for phase 2. To analyze the correlation,
parameters n and qi are initialized. The domain expert selects n number of questions
related to a medical condition specified by q Id and forms a subset. The subset
achieved is used as input to the objective function (LDAB) to measure its effectiveness.
The process continues until the whole dataset (QC ) is grouped into the subsets.
To measure the effectiveness of the proposed framework, three iterations of phase 2
are executed using (n, qi ) set to find the anomalies related to a candidate’s vision, their
capability of operating machinery and their history of surgeries. If LDAB is able to find
anomalies for the said question group, then the subsets are added into the final feature
set and removed from QC as described in Eq. (12). Output from LDAB serves as a final
deciding factor for feature selection.
Any deviation from the ideal behavior indicates anomalies. The presented frame-
work is highly domain-knowledge dependent and is useful in applications where no
previous data/information is available to perform mathematical modeling. In such
cases, an experienced domain expert can undertake this requirement and domain
knowledge-dependent methodologies based on logic can be developed to get the
system started. Once the system is live, data can be captured for further analysis.
Furthermore, DSW is flexible enough to fit in data models as well by replacing domain-
driven filtration and wrapping with any probabilistic models.
Traditional comparative analysis on different data sets with DSW is not possible
because no such domain knowledge-driven approach has been proposed so far for
feature selection. Thus, to analyze the efficiency of the proposed approach, QC is
exposed to three FS wrapper algorithms, SFS, SBS and then to phase II of DSW. At
each iteration, a number of relevant features are observed. The graphs in Figs. 3, 4 and
5 show the number of relevant features extracted by the FS methods per iteration.
 Framework for Feature Selection in Health Assessment Systems 321

4
Poor Vision

3
Relevant Features

1
1 6 11 16 21 26 31 36 41 46
Iterations
SFS SBS DSW

Fig. 3. The number of relevant features selected per iteration (question group: poor vision)

4
Capability to Operate Machinery
Relevant Features

0
1 6 11 16 21 26 31 36 41 46
Iterations
SFS SBS DSW

Fig. 4. The number of relevant features selected per iteration (question group: capability to
operate machinery)

4 History of Surgeries
Relevant Features

0
1 6 11 16 21 26 31 36 41 46
Iterations
SFS SBS DSW

Fig. 5. The number of relevant features selected per iteration (question group: history of surgeries)
322 A. Ubaid et al.

From the graphs, it can be observed that DSW is able to populate relevant features
more efficiently than the other two techniques. The knowledge of the domain expert is
the key success criteria in DSW. In SBS and SFS, features were extracted after a
number of iterations. However, the relevant features are picked out from a pool of
features in just one iteration using the domain expert’s knowledge, n and qi . DSW is a
light weight FS framework that helps in selecting the required data and is also capable
of discarding useless features from the final feature set.
The features selected by DSW are passed through LDAB to identify any anomalies
in the data. The percentage success rate of the anomaly detection of LDAB using
features extracted by DSW is shown in Fig. 6.
Percentage of detected anomalies

100

80

60

40

20

0
2 3 4 5
No of selected features (n)

Fig. 6. Anomaly detection rate using FS from DSW

5 Conclusion and Future Work

In this paper, we introduced FS and overviewed the existing FS techniques. Differ-

ent FS methods behave differently on different types of data. Thus, a balance has to be
achieved, depending on data type, the application under consideration and the under-
lying machine learning algorithm. Also, stability, flexibility, computation requirements,
storage and the number of reduced features etc. are some of the metrics that can affect
the selection of a feature selection technique. A novel hybrid feature selection
framework was also proposed which is capable of selecting features in a minimum
number of iterations with equitable classification accuracy and computation time.
Furthermore, the framework can incorporate input from the domain expert for selecting
feature subsets. This offers great flexibility in the proposed framework. Also, the
proposed framework has the capability of discarding useless features from the
remaining feature set.
 Framework for Feature Selection in Health Assessment Systems 323

This research has also opened pathways for modifying traditional feature selection
methods based on the application requirement and showed that correlated data is
sometimes of utmost importance. In the future, the proposed domain knowledge-based
approach can be combined with a data-driven approach to achieve higher confidence.
This is because the domain expert may not be able to predict a hidden relationship in
the data. Thus, the advantages of both domains can be used to fine tune the system,
which will be part of our future work. Furthermore, the combination of data, the
domain expert and traditional feature selection algorithms to find an optimal dataset is a
new research direction by which to obtain a superlative feature set.

Acknowledgment. The work presented in this paper is supported by the SIEF STEM+ funding.
Fan Dong is the recipient of a SIEF STEM+ Business Fellowship.

References
1. Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artif. Intell. 97(1), 273–324
(1997)
2. Chandrashekar, G., Sahin, F.: A survey on feature selection methods. Comput. Electr. Eng.
40(1), 16–28 (2014)
3. Law, M.H.C., Figueiredo, M.A.T., Jain, A.K.: Simultaneous feature selection and clustering
using mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1154–1166 (2004)
4. Reunanen, J.: Overfitting in making comparisons between variable selection methods.
J. Mach. Learn. Res. 3, 1371–1382 (2003)
5. Vergara, J.R., Estévez, P.A.: A review of feature selection methods based on mutual
information. Neural Comput. Appl. 24(1), 175–186 (2014)
6. Rodriguez-Galiano, V.F., Luque-Espinar, J.A., Chica-Olmo, M., Mendes, M.P.: Feature
selection approaches for predictive modelling of groundwater nitrate pollution: an evaluation
of filters, embedded and wrapper methods. Sci. Total Environ. 624, 661–672 (2018)
7. Inbarani, H.H., Azar, A.T., Jothi, G.: Supervised hybrid feature selection based on PSO and
rough sets for medical diagnosis. Comput. Methods Programs Biomed. 113(1), 175–185
(2014)
8. Shilaskar, S., Ghatol, A.: Feature selection for medical diagnosis: Evaluation for
cardiovascular diseases. Expert Syst. Appl. 40, 4146–4153 (2013)
9. Langley, P.: Selection of relevant features in machine learning. In: Proceedings of the AAAI
Fall Symposium on Relevance, pp. 140–144 (1994)
10. Blum, A.L., Langley, P.: Selection of relevant features and examples in machine learning.
Artif. Intell. 97(1), 245–271 (1997)
11. Vieira, S.M., Mendonça, L.F., Farinha, G.J., Sousa, J.M.C.: Modified binary PSO for feature
selection using SVM applied to mortality prediction of septic patients. Appl. Soft Comput.
13(8), 3494–3504 (2013)
12. Xue, B., Zhang, M., Browne, W.N.: Particle swarm optimisation for feature selection in
classification: novel initialisation and updating mechanisms. Appl. Soft Comput. 18, 261–
276 (2014)
13. Goldberg, D.E.: Genetic Algorithms in Search, Optimization and Machine Learning.
Addison-Wesley Longman Publishing Co., Inc., Boston (1989)
14. Cordón, O., Damas, S., Santamaría, J.: Feature-based image registration by means of the
CHC evolutionary algorithm. Image Vis. Comput. 24(5), 525–533 (2006)
324 A. Ubaid et al.

15. Wilcox, A.B., Hripcsak, G.: The role of domain knowledge in automating medical text
report classification. J. Am. Med. Inform. Assoc. JAMIA 10(4), 330–338 (2003)
16. Xing, E.P., Karp, R.M.: CLIFF: clustering of high-dimensional microarray data via iterative
feature filtering using normalized cuts. Bioinformatics 17(Suppl. 1), S306–S315 (2001)
17. Pudil, P., Novovičová, J., Choakjarernwanit, N., Kittler, J.: Feature selection based on the
approximation of class densities by finite mixtures of special type. Pattern Recogn. 28(9),
1389–1398 (1995)
18. Mitra, P., Murthy, C., Pal, S.K.: Unsupervised feature selection using feature similarity.
IEEE Trans. Pattern Anal. Mach. Intell. 24(3), 301–312 (2002)
19. Pal, S.K., De, R.K., Basak, J.: Unsupervised feature evaluation: a neuro-fuzzy approach.
IEEE Trans. Neural Netw. 11(2), 366–376 (2000)
20. Xu, Z., King, I., Lyu, M.R.-T., Jin, R.: Discriminative semi-supervised feature selection via
manifold regularization. IEEE Trans. Neural Netw. 21(7), 1033–1047 (2010)
21. Zhao, Z., Liu, H.: Semi-supervised feature selection via spectral analysis. In: SDM (2007)
 A Framework to Achieve Full Waypoint
Enforcement in Hybrid SDN Networks

Sandhya Rathee(B) , T. Dinesh Ram Kumar, K. Haribabu,

and Ashutosh Bhatia

CSIS Department, Birla Institute of Technology and Science,

Pilani 333031, Rajasthan, India
{p2015007,f2014302,khari,ashutosh.bhatia}@pilani.bits-pilani.ac.in

Abstract. Software Defined Networking (SDN) addresses the opera-

tional challenges faced in maintaining legacy networks by decoupling the
data plane and control plane. Overhauling of the network to replace
legacy switches with SDN switches is not economically viable. Incremen-
tal deployment of SDN devices is the most promising approach. Hybrid
SDN network contains both legacy and SDN devices in a network. One of
the challenges in hybrid SDN network is to enforce traffic to go through
some of the SDN switches (waypoints) present in the network. There
exist a few solutions in the literature to achieve waypoint enforcement,
but practically none of them provide full waypoint enforcement. In this
paper, we propose a novel framework to achieve full waypoint enforce-
ment. The proposed framework uses virtual IP addresses to divert the
complete network traffic towards the SDN switches present in the net-
work. We also provide an analysis of the framework.

Keywords: Hybrid SDN · Waypoint enforcement · Virtual IP ·

SDN Switch

1 Introduction
Enterprise networks have several operational challenges. The network must be
reliable and provide high performance connectivity while enforcing the organisa-
tional policy. The network should be easy to manage with minimum operational
and capital cost. Software defined networking (SDN) has the potential to resolve
these network challenges by decoupling the data plane and control plane. Thus,
the underlying network merely becomes a means of packet forwarding and the
network control and intelligence are centralized. This eases programming and
controlling of the network. SDN claims independence from current static nature
of networks arising due to vendor specific protocols and implementation. It does
away the tedious task of manual configuration of individual network hardware
and makes provision of a controller running a network operating system [7]. How-
ever, overhauling of the network to replace legacy switches with SDN switches is
not a viable solution, due to major economic hurdles and also challenges faced
in training network administrators to efficiently manage new infrastructure [8].
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 325–340, 2020.
https://doi.org/10.1007/978-3-030-15032-7_28
326 S. Rathee et al.

This has sprung up a new research area in hybrid networks, Hybrid SDN [15],
the co-existence of legacy and SDN devices.
In the past few years, researchers have proposed multiple solutions [1,2,5,6,
8,9,13,14,16] to achieve hybridization of SDN and legacy networks. It has been
shown that by deploying a few SDN switches in a legacy network, routing perfor-
mance can be improved [2], network management can be enhanced and the way-
point enforcement [5,6,8] can be achieved to different degrees. Waypoint enforce-
ment is defined as forcing a packet to go through certain checkpoints [10]. In
hybrid SDN networks, the checkpoints are SDN switches. The works on waypoint
enforcement can be categorized into partial waypoint enforcement and full way-
point enforcement. Partial waypoint enforcement does not achieve 100% waypoint
enforcement. In other words, in partial waypoint enforcement, there exists some
traffic in the network which does not traverse any SDN switch. Whereas full way-
point enforcement achieves 100% waypoint enforcement. Telekinesis and Magento
[5,6] are probabilistic solutions resulting in partial waypoint enforcement. They
require at least one SDN switch to be present in each subnet for network-wide
deployment. The work proposed in [14] overcomes this limitation by using a sin-
gle SDN switch and VLANs. Panopticon [8] achieves full waypoint enforcement
for a switch port designated as SDNc port by the administrator. As the number
of SDNc port increases, the number of SDN switches required also increases.
In this paper, we aim to achieve full waypoint enforcement for all switch ports
even with a single SDN switch. We propose a novel framework to bring entire
network traffic under the SDN controller by redirecting the traffic such that
every packet goes through at-least one SDN switch. We use unused IP addresses
as virtual IP addresses to achieve full waypoint enforcement. These virtual IP
addresses are accessible only via SDN switch. The idea is, when a source wants
to communicate with a destination in the network, the source is provided with
virtual IP address of the destination. Thus, traffic destined to any of virtual IP
addresses is forwarded to the SDN switch/es.
The rest of the paper is organised as follows, in Sect. 2, we discuss some of
the research works on achieving waypoint enforcement. In Sect. 3 we define the
problem statement and propose a framework to achieve full waypoint enforce-
ment based on the idea of virtual IP addresses. In Sect. 4.1, we discuss different
methods to map virtual IP addresses to real IP addresses. In Sect. 5, we analyse
the proposed framework and subsequently discuss the results in Sect. 6. Finally
in Sect. 7, we conclude our work.

2 Related Work

In this section, we discuss the previous works done to achieve waypoint enforce-
ment in hybrid SDN networks. The solutions proposed in [5,6,8,14] provide par-
tial waypoint enforcement. Panopticon [8] proposes an architecture for partial
deployment of SDN devices in traditional networks. The network is divided into
cell blocks by deploying few SDN switches in the network. A few ports on legacy
devices or hosts are designated as SDN-controlled port, called SDNc ports. A
per-VLAN spanning tree is constructed with SDNc port as the root. The traffic
 A Framework to Achieve Full Waypoint Enforcement 327

from an SDNc port to an SDN switch is tagged with a VLAN ID unique to that
path. This ensures that all traffic going out and coming in goes through an SDN
switch. The scope of a VLAN ID ends in a cell block, making it reusable in other
cell blocks. SDN switch can forward the packet to a different cell block, thereby
acting as a gateway. This design faces an inherent issue in the implementation
of waypoint enforcement, due to its inability to control the traffic between two
non-SDNc ports. In other words, this solution only controls the traffic which is
either going out of the cell blocks or the traffic going from or to an SDNc port.
The solution is proposed for enterprise network, which is quite reliant on VLANs
to operate the network. This can disrupt the existing VLAN configurations of
the enterprise network.
In [5], the authors propose an approach called Telekinesis, which exploits
MAC (Media Access Control) learning present in legacy switches and packet out
feature of OpenFlow [11] switches to forward traffic packets towards the SDN
switch. The controller sends a seed packet through an SDN switch to populate
MAC learning table of the legacy switches, thereby tricking the source host to
send the traffic packets to the SDN switches. However, such paths are unstable,
as entries in MAC learning table keep aging out. This necessitates re-sending of
seed packets at intervals, which can be a network overhead. The authors have
recently proposed an improvement over this work called magneto [6]. In this
solution, for every path between a legacy switch and an SDN switch, or between
a pair of SDN switches, they associates a magnet MAC address. A magnet MAC
address can’t be found in the network. The controller sends gARP (Gratuitous
Address Resolution Protocol) messages with magnet MAC addresses and poison
the ARP (Address Resolution Protocol) table of end-hosts, such that, the traffic
is forwarded towards an SDN switch. This solution requires at least one SDN
switch in each subnet for waypoint enforcement. Another limitation is that, if the
ARP entries are not used for a specific time, then the ARP entries are removed
from the ARP table, which makes the traffic to take original path which may
not include any SDN switch.
In [14], the authors propose a solution to eliminate the need of having an
SDN switch in each subnet by attaching a work-station, which is part of every
VLAN. This work-station generates gARP packets to poison the ARP table of
end-hosts and make them to send the traffic packets towards the SDN switch.
This solution overcome the requirement of one SDN switch per-subnet, but only
provides partial waypoint enforcement.

3 Proposed Framework
In this section, we discuss our proposed framework to achieve full waypoint
enforcement in an enterprise network.

3.1 Problem Statement

Our goal is to achieve full waypoint enforcement for complete network traffic.
For example, given a pair of communication host, h1 and h2 (access traffic)
328 S. Rathee et al.

as shown in Fig. 1, and a path between them (say P ), our goal is to enforce
a new path P  between h1 and h2 which consists of at least one SDN switch.
As shown in Fig. 1, the original path between h1 and h2 is P : (h1 , S1 , h2 )
and our objective is to enforce a new path between h1 and h2 which is P  :
(h1 , S1 , R3 , R1 , SDN, R1 , R3 , S1 , h2 ).

3.2 Framework to Achieve Waypoint Enforcement

We take a set of IP addresses that are not assigned to any host in the network,
called virtual IP addresses. For every host in the network the controller maps a
virtual IP address with their real IP address. Before any communication starts in
the network, the source hosts are required1 to send a DNS (Domain Name Server)
query to resolve the IP address of the destination host. DNS server maintains
the mapping of real IP address of the host to its virtual IP address. When a DNS
query reaches the DNS server, it sends DNS response with virtual IP address
of the destination host instead of real IP address. DNS server is supervised
by the SDN controller. The mapping between virtual IP addresses and real IP
addresses is known only to the SDN switch. For all virtual IP addresses SDN
switch acts as gateway. The source host creates packets with virtual IP address
as the destination, which gets diverted towards the SDN switch. Once a packet
reaches to the SDN switch it is forwarded towards the intended destination
through underlying network (consists of both legacy as well as SDN devices),
using the real IP address of the destination. To achieve this, the SDN controller
adds the flow entries in SDN switches. The flow entries have actions to modify the
packet’s destination virtual IP address with it’s corresponding real IP address. It
also modifies the packet’s source real IP address with it’s corresponding virtual
IP address. Thus, the destination host receives the packet which has virtual IP
address of source host, it responds to source host using the virtual IP address.
The same procedure applies when the destination replies.

4 Packet Forwarding
In this section, we discuss different models to allocate virtual IP addresses to
the hosts in the network and show how inter subnet and intra subnet traffic is
forwarded towards an SDN switch. All the virtual IP addresses are accessible via
SDN switch, as the SDN switch acts as gateway to the virtual IP addresses. We
use GNS3 (Graphical Network Simulator-3) [12] to emulate the proposed frame-
work. GNS3 allows to design complex network and emulate it by configuring the
devices ranging from simple workstation machines to powerful Cisco routers. We
use the topology given in Fig. 2 for emulations which consists of three subnets
10.0.0.0/24 (Subnet 1), 10.0.1.0/24 (Subnet 2) and 10.0.2.0/24 (Subnet 3). We
use Cisco 2691 [4] switch at core layer and distribution layer which contains R1 ,
R2 , R3 , and R4 . Cisco IOSvL2 switch is used at access layer which contains S1 ,
S2 , and S3 .
1
To force the host to use virtual IP address, ACLs are configured in the network to
drop all packets which do not have virtual IP address as destination.
 A Framework to Achieve Full Waypoint Enforcement 329

Fig. 1. Path setup in Hybrid SDN network.

Fig. 2. Topology for GNS3 emulations.

330 S. Rathee et al.

Table 1. One-to-one Table 2. Cell block based Table 3. Subnet based

mapping mapping mapping

Real IP Virtual IP Real IP Virtual IP Real IP Virtual IP

10.0.0.0/24 172.16.0.0/24 10.0.0.0/24 172.16.0.0/24 10.0.0.0/24 172.16.0.0/24
10.0.1.0/24 172.16.1.0/24 10.0.1.0/24 172.16.1.0/24 10.0.1.0/24 172.16.0.0/24
10.0.2.0/24 172.16.2.0/24 10.0.2.0/24 172.16.0.0/24 10.0.2.0/24 172.16.0.0/24

4.1 Virtual IP Address Mapping

Now we discuss the different methods to assign virtual IP addresses to the hosts
in the network. We also discuss the network configurations for each method
which we need to enable waypoint enforcement.

(a) One-to-One Mapping. Simplest mapping method is to use a one-to-one

mapping between host’s real IP address and virtual IP address. In this model, we
have two domains called real IP address domain and virtual IP address domain
and each domain have same number of IP addresses. The mapping is a one-
to-one function from real IP address to virtual IP address. We consider three
virtual IP subnets one for each of the host subnet given in Fig. 2. Let the virtual
subnets be 172.16.0.0/24 (Virtual subnet 1), 172.16.1.0/24 (Virtual Subnet 2)
and 172.16.2.0/24 (Virtual Subnet 3). Table 1 shows the mapping of real IP
addresses to virtual IP addresses.
Now we need to configure the network to handle these virtual IP addresses.
Whenever any legacy device encounters a packet with destination address in
virtual IP domain, it must forward the packet towards the SDN switch. In the
topology given in Fig. 2, R1 and R3 must forward the packet to R2, R2 and R4
must forward the packet to OVS1. Thus, we define routes in R1 and R2 as given
in Fig. 3, similarly we define the routes in R3 and R4.
The configuration of OVS1 is given in Fig. 4. In OVS1, we check if the des-
tination IP address belongs to virtual IP address space. If not, then the packets
match the first flow entry, which has action to process the packets as normal
L2/L3 device. If yes, then we modify the source MAC to that of OVS1 and we
submit the packet to Table 1 for further processing. In Table 1, we modify the
source IP address to the corresponding virtual IP address, and destination IP
address to the corresponding real IP address and sends the packet to Table 2 to
decide where to send the resultant packet. In Table 2, based on the destination
IP address, we decide the next hop of the packet and modify the destination
MAC address accordingly.

(b) Cell Block Based Mapping. Consider a graph modelling the network,
where nodes are hosts/devices and edges are links. To minimize the virtual IP
 A Framework to Achieve Full Waypoint Enforcement 331

Fig. 3. Router R1 and R2 configuration for one-to-one mapping model.

Fig. 4. OVS1 configuration for one-to-one mapping model.

addresses required for the proposed framework, we divide the whole network into
cell blocks.

Definition 1 (Cell Blocks). Given network graph G, cell blocks of G is defined

as set of connected components obtained after removal of SDN switches from G
along with all the links incident on SDN switches.

Consider a pair of hosts which wants to communicate and both hosts belong
to a different cell block. In this case, the path between given pair of hosts consists
of at least one SDN switch which connects both the cell blocks. Thus, any pair
of hosts which belong to different cell blocks does not require virtual IP address
for waypoint enforcement and can communicate with each other with real IP
addresses. However, hosts belonging to same cell block have to use virtual IP
addresses to achieve waypoint enforcement. Thus, we need virtual IP addresses
only for the communications within the cell blocks and use real IP addresses to
communicate with the hosts outside the cell block. Thus, we can reuse virtual IP
addresses across cell blocks. In this model, the number of virtual IP addresses
required is equal to the number of hosts in the largest connected component
(cell block). We get two cell block from the topology given in Fig. 2, cell block 1
contains subnet 1 and subnet 2 and cell block 2 contains subnet 3. Cell block 1
contains the largest number of hosts (hosts in subnet 1 and subnet 2). Let the
virtual IP address space be 172.16.0.0/23. In this scheme, we can re-use virtual
IP addresses across cell blocks. The mapping of real IP address to virtual IP
address for each subnet is given in Table 2. As it is evident from Table 2, the
virtual IP addresses in 172.16.0.0/24 are used in cell block 1 as well as in cell
block 2.
332 S. Rathee et al.

We need to configure the network to handle packets with virtual destination

IP addresses and forward these packets to the closest SDN switch. Like in one-
to-one mapping, R1 and R3 must forward packets to R2, and R2 and R4 must
forward the packet to OVS1. The configurations of R1, R2, R3, and R4 are
similar to one-to-one mapping.

Fig. 5. OVS1 Configuration for cell block based mapping model.

The flow entries required at the OVS1 switch to handle packets with virtual
IP addresses are given in Fig. 5. We check if the destination IP address belongs
to virtual IP address space. If not, then the packets match the first flow entry,
which has action to process the packets as normal L2/L3 device. If yes, then
we send the packets to Table 1 for further processing. In Table 1, based on the
source IP address’ subnet, we modify the source address IP address (convert it
to virtual IP address) and destination IP address (convert it to real IP address)
and send it to Table 2. In Table 2, we swap the source and destination MAC
addresses and send it back along the incoming port.
In one-to-one mapping and cell block based mapping, we add routes to the
routing table of legacy switches (Layer 3 switches) to forward the packets with
virtual destination IP addresses to the closest SDN switch. Further, multiple
subnets can be present in one cell block, which may increase the virtual addresses
as well as routing entries. To solve this problem we can do mapping based on
subnets and use policy based routing to further reduce the virtual IP addresses
required.

(c) Subnet Based Mapping. Layer 3 switches in the network support policy
based routing (route-maps, source based routing and ACLs), we can leverage it
to lower the virtual IP address requirement. Once the packet reaches the layer
3 switch, we can then use policy based routing (using ACLs and route-maps)
to forward the packets towards the SDN switch. Once the packets reach SDN
switch and they are to be sent back towards the intended destination using IP
based routing instead of policy based routing. When the destination host lies
in the same subnet as the source host, the traffic packets directly reach the
destination via the access switch using MAC-based forwarding and may not
reach the layer 3 switch. Thus, we need to use virtual IP addresses for the
 A Framework to Achieve Full Waypoint Enforcement 333

communications within the same subnet. Any packet with virtual IP address is
sent to the default gateway of the subnet to which the communication pair hosts
belong. Thus, packets eventually reach the SDN switch. SDN switch converts
the source IP address to virtual IP address and virtual IP address of destination
back to real and the packets are sent back to the intended destination using
destination based IP routing. In the topology given in Fig. 2, all subnets are of
same size. Let the virtual IP address space be 172.16.0.0/24. In this method, we
re-use the virtual IP addresses across subnets as given in Table 3. The policies
and configurations of R1 and R2 are given in Fig. 6. Similarly, we configure R3
and R4 . The configuration of OVS1 is given in Fig. 7.

Fig. 6. Router R1 and R2 configuration for subnet based mapping model.

Fig. 7. OVS1 configuration for subnet based mapping model.

The virtual IP address is required only for the communications within the
subnet. Therefore, the virtual IP addresses can be reused for communications in
334 S. Rathee et al.

other subnets. SDN switch must identify the subnet to which the host belongs
and must know the mapping from host real IP address to virtual IP address and
vice versa based on the subnet it identified. The maximum number of virtual
IP addresses required in this model is equal to the size of the largest subnet.
This model lowers the number of virtual IP addresses required but at the cost
of increased policy definitions and is not easy to comprehend and visualize.
Difference between subnet based mapping and cell block based mapping is that,
virtual IP addresses can be reused across the subnets in case of subnet based
mapping instead of cell blocks.

(d) Dynamic IP Mapping. We can use IP based forwarding (routing tables)

and still lower the virtual IP addresses by defining a dynamic mapping between
hosts real IP addresses and virtual IP addresses. Hosts are given virtual IP
addresses which expire over time. Heuristic: Hosts only communicate with fewer
number of hosts in the network. Using this heuristic, in a given window of time,
we can determine the maximum number of hosts to which a given host may
communicate and define requirement for the number of virtual IP addresses.
When hosts demand for more virtual IP addresses than either deny the request
or give the real IP address of the host (this can result in no waypoint enforcement
for that flow). Similar to cell block based mapping, mapping of destination IP
address depends on source IP address, however virtual IP address can change
over time. For example, host A wants to communicate with host B. Host A
requests for the IP address of Host B. Based on virtual IP address availability
for Host A, SDN controller allocates virtual IP address for host B with respect
to host A. When host A sends a packet with virtual IP address of host B as
destination address, the packet reaches the SDN switch, now SDN controller
allocates a virtual IP for host A with respect to host B and now changes the
source address to this virtual IP address and destination address to that of host
B’s real IP address and forwards the packet to host B. Now host A and host
B can communicate with each other using these virtual IP addresses. Dynamic
virtual IP involves mapping which changes over time and mapping is defined
per host per destination basis, thus the number of flow entries in SDN switch
increases.

5 Performance Analysis

5.1 Network Set-Up

To analyse the performance of the proposed solution, we consider a 3-tier hier-

archical network topology suggested by Cisco guideline [3], as shown in Fig. 8. In
network topology, layer 2 switches are used at access layer and layer 3 switches
are used at core and distribution layers. The network has a large number of
access layer devices and a few high-end core devices. This indicates that a few
legacy switches of distribution layer can be replaced with SDN switches. We
consider a symmetric topology for ease of analysis.
 A Framework to Achieve Full Waypoint Enforcement 335

5.2 Assumptions
All links in the network are fully duplex, and the capacity of all the links at a
given layer is same.

Fig. 8. 3-tier Network Topology.

Host Traffic Composition. All hosts generate same unit amount of traffic.
We divide the traffic generated by the hosts into three different categories:
1. Access Traffic (Ta ): The percentage of total traffic generated by hosts, which
is resolved by the access layer switch, is called access traffic.
2. Distribution Traffic (Tb ): The percentage of total traffic generated by hosts,
which is resolved by the distribution layer switch, is called distribution traffic.
3. Core Traffic (Tc ): The percentage of total traffic generated by hosts, which is
resolved by the core layer switch, is called core traffic.

5.3 Load on Link

In our analysis, we consider that traffic composition (Ta , Td , Tc ) for all hosts
is same. Also, when packets of given type are generated by a host, all possible
destinations are equally likely. When a host generates a packet, based on the
traffic composition, the likelihood that it travels through a given link varies
from link to link. For each host, we can find the likelihood that the packet flows
through a given link. We consider, that the traffic flow through the given link
in a given direction when all hosts generate unit amount of traffic and no link
behaves as bottleneck. Due to the symmetry in the network topology and traffic
composition, traffic flow is same in both directions of the link. A summary of
symbols used in anaylsis is provided in Table 4.

Definition 2. Load on a link is defined as the total amount of traffic flowing

through a given link in unit time when all host generates unit amount of traffic.
336 S. Rathee et al.

Table 4. List of symbols used in analysis

Symbol Meaning
Sc Number of core switches
Sd Number of distribution switch pairs
Sa Number of access switches connected to one distribution switch pair
Ah Number of hosts connected to one access switch
Ta Percentage of total traffic resolved at access layer
Td Percentage of total traffic resolved at distribution layer
Tc Percentage of total traffic resolved at core layer
HAload Load on the links in host-access layer
ADload Load on the links in access-distribution layer
DCLoad Load on the links in distribution-core layer
N Total number of distribution switches.
S Total number of SDN switches at distribution layer
L Total number of legacy switches at distribution layer
Aplen Average path length of all traffic
ATplen Average path length of access traffic
DTplen Average path length of distribution traffic
CTplen Average path length of core traffic

First, we calculate the load on the links in each layer of network when there
is no waypoint enforcement. Let HAlaod , ADload and DCload denotes the load
on the links in host-access layer, access-distribution layer and distribution-core
layer respectively.

HAload = Ta + Td + Tc
= 1.0 (1)

Td + Tc
ADload = Ah × (2)
2
Tc
DCload = Sa × Ah × (3)
4
Now consider the case when SDN switches are placed in the distribution layer
and waypoint enforcement mechanism is applied. Now, the traffic is diverted
from its native path to towards the SDN switches. The amount of traffic passing
through the host-access layer links remains the same. All traffic reaches the access
switches, including the access traffic, must be forwarded towards the distribution
layer as SDN switches exist at the distribution layer. There are two kinds of links
in distribution-core layer- (1) the link which is connected to the SDN switch
(SDN Distribution-Core (SDNdc ) link) and (2) the link which is not connected
 A Framework to Achieve Full Waypoint Enforcement 337

to SDN switch (Legacy Distribution-Core (Legacydc ) link). Load on these links

is denoted by S DCload and L DCload respectively.
HAload = Ta + Td + Tc = 1.0 (4)

Ta + Td + Tc Ah
ADload = Ah × = (5)
2 2
Ta + Td + Tc Sa × Ah
L DCload = Sa × Ah × = (6)
4 4
Sa × Ah L−1 L
S DCload = × {(Ta + Td + Tc × ) × + Tc } (7)
4 N −1 S

5.4 Path Length

Definition 3. Path length is defined as average number of hops a packet tra-
verses before reaching its destination. Total average path length is given by,
Aplen = ATplen × Ta + DTplen × Td + ATplen × Tc (8)
where, ATplen , DTplen andCTplen are the average path length of access traffic,
distribution traffic and core traffic respectively. When waypoints are enforced
the average path length of each type of traffic is a function of number of SDN
switches in the network. When waypoints are enforced, the packets are diverted
from its native path and forwarded towards the SDN switch/es, thus increases
the path length. For a traffic type X, the average path length is calculated as
follows,
Xplen = DPplen × P (DX ) + N Pplen × (1 − P (DX )) (9)
where, DPplen is the diverted path length and N Pplen is the native path length.
P (DX ) is the probability that packet of a given traffic type X is diverted for
waypoint enforcement. Equations (10), (11), (12) gives the probability of diver-
gence for each layer of traffic. P (Da ), P (Dd ), P (Dc ) denotes the probability of
divergence of access layer traffic, distribution layer traffic, and core layer traffic
respectively.
L
P (Da ) = (10)
L+S
L
P (Dd ) = (11)
L+S
L L−1
P (Dc ) = × (12)
L+S L+S−1

6 Results
In this section, we present the analytical results of proposed solution. To perform
the analysis we consider 3-tier architecture suggested by Cisco with following
topology and traffic composition parameters: Sc = 2, Sd = 20, Sa = 10, Ah = 20
and Ta = 0.5, Td = 0.25, Tc = 0.25.
338 S. Rathee et al.

6.1 Load on Different Links

In our solution, we replace the legacy switches with SDN switches at distribu-
tion layer. Figure 9(a), shows that the SDN distribution-core links experience
high load (as defined in definition 2) due to the divergence from other legacy
distribution-core links. The additional load on SDN distribution-core links can
further be divided into two types- (1) the access traffic and distribution traf-
fic that reaches the distribution switch which is not an SDN switch and must
be forwarded to an SDN switch, (2) the core traffic between two distribution
switches when neither of the distribution switch is an SDN switch, must be for-
warded to an SDN switch. We call the load exerted by these traffic on the SDN
distribution-core links as internal deviation load (as given in Eq. 6) and core
deviation load (as given in Eq. 7) respectively. As shown in Fig. 9(a), these loads
decrease with increase in number of SDN switches and eventually reduce to zero
when all distribution switches are replaced with SDN switches.

Fig. 9. Load on the links at each layer

Figure 9(b) shows the plot of maximum load on the links in different network
layers. The load on host-access link and access-distribution link remain same
whereas the load on distribution-core link decreases drastically as we increase
the number of SDN switches.
 A Framework to Achieve Full Waypoint Enforcement 339

6.2 Path Length

Figure 10, shows the effect on average path length (in terms of number of hops)
with respect to number of SDN switches deployed in the network. The average
path length is calculated using Eqs. 8 and 9. We observe that as we increase the
number of SDN switches the average path length decreases. Thereby reducing
the propagation delay. Since we are replacing the distribution switches with SDN
switches, all access traffic still must be forwarded towards distribution switch.
So, even after replacement of all distribution switches with SDN switches, the
path length of waypoint enforced traffic does not become equal to the path length
of the traffic in legacy network.

Fig. 10. Average path length with respect to number of SDN switch.

7 Conclusion

Waypoint enforcement of network packets is one of the essential requirements

to enforce various policies to regulate the manner in which these packets are
allowed to move in a given network. Achieving full waypoint enforcement in
a hybrid SDN network becomes challenging as all the packets traveling in a
network may not necessarily pass through any of the SDN switch present in the
network, and therefore the path of the packets needs to be diverted during their
flight from a source to a destination. In this paper, we introduced a framework
to achieve full waypoint enforcement in hybrid SDN network that uses a novel
concept of providing virtual IP address of the hosts to other hosts instead of
their real IP addresses. The performance analysis of the proposed framework
depicts that replacement of single SDN switch to achieve waypoint enforcement
can severely restrict the capacity of the network. However, by introducing a
few SDN switches in the network and perform load balancing among them can
quickly improve the situation.
340 S. Rathee et al.

References
1. Hand, R., Keller, E.: ClosedFlow: OpenFlow-like control over proprietary devices.
In: Proceedings of the Third Workshop on Hot Topics in Software Defined Net-
working, pp. 7–12. ACM (2014)
2. Hong, D.K., Ma, Y., Banerjee, S., Morley Mao, Z.: Incremental deployment of SDN
in hybrid enterprise and ISP networks. In: Proceedings of the Symposium on SDN
Research, SOSR 2016, p. 1. ACM (2016)
3. https://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Campus/
campover.html. Accessed 08 Dec 2018
4. https://www.cisco.com/c/en/us/td/docs/ios/12 2/12 2z/release/notes/rn2600zj.
html. Accessed 24 Dec 2018
5. Jin, C., Lumezanu, C., Xu, Q., Zhang, Z.-L., Jiang, G.: Telekinesis: controlling
legacy switch routing with OpenFlow in hybrid networks. In: Proceedings of the
1st ACM SIGCOMM Symposium on Software Defined Networking Research, p.
20. ACM (2015)
6. Jin, C., Lumezanu, C., Xu, Q., Mekky, H., Zhang, Z.-L., Jiang, G.: Magneto: unified
fine-grained path control in legacy and OpenFlow hybrid networks. In: Proceedings
of the Symposium on SDN Research, pp. 75–87. ACM (2017)
7. Kreutz, D., Ramos, F.M.V., Verissimo, P.E., Rothenberg, C.E.: Software-defined
networking: a comprehensive survey. Proc. IEEE 103(1), 14–76 (2015)
8. Levin, D., Canini, M., Schmid, S., Schaffert, F., Feldmann, A.: Panopticon: reaping
the benefits of incremental SDN deployment in enterprise networks. In: USENIX
Annual Technical Conference, pp. 333–345. USENIX Association (2014)
9. Lu, H., Arora, N., Zhang, H., Lumezanu, C., Rhee, J., Jiang, G.: Hybnet: network
manager for a hybrid network infrastructure. In: Proceedings of the Industrial
Track of the 13th ACM/IFIP/USENIX International Middleware Conference, p.
6. ACM (2013)
10. Ludwig, A., Rost, M., Foucard, D., Schmid, S.: Good network updates for bad
packets: waypoint enforcement beyond destination-based routing policies. In: Pro-
ceedings of the 13th ACM Workshop on Hot Topics in Networks, p. 15. ACM
(2014)
11. McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford,
J., Shenker, S., Turner, J.: OpenFlow: enabling innovation in campus networks.
ACM SIGCOMM Comput. Commun. Rev. 38(2), 69–74 (2008)
12. Neumann, J.C.: The Book of GNS3: Build Virtual Network Labs Using Cisco,
Juniper, and More. No Starch Press (2015)
13. Parniewicz, D., Corin, R.D., Ogrodowczyk, L., Fard, M.R., Matias, J., Gerola, M.,
Fuentes, V., et al.: Design and implementation of an OpenFlow hardware abstrac-
tion layer. In: Proceedings of the 2014 ACM SIGCOMM Workshop on Distributed
Cloud Computing, pp. 71–76. ACM (2014)
14. Sandhya, Swamy, M., Haribabu, K., Bhatia, A.: Achieving waypoint enforcement
in multi-VLAN hybrid SDN. In: Proceedings 10th International Conference on
Communication Systems and Networks (COMSNETS), pp. 519–521. IEEE (2018)
15. Sandhya, Sinha, Y., Haribabu, K.: A survey: hybrid SDN. J. Netw. Comput. Appl.
100, 35–55 (2017)
16. Vanbever, L., Vissicchio, S.: Enabling SDN in old school networks with software-
controlled routing protocols. In: ONS (2014)
 A New Fuzzy Logic Based Model
for Location Trust Estimation in Electric
Vehicular Networks

Ilhem Souissi1(B) , Nadia Ben Azzouna1 , Tahar Berradia2 ,

and Lamjed Ben Said1
1
SMART Lab, ISG, Universite de Tunis, Tunis, Tunisia
[email protected], [email protected],
[email protected]
2
IRSEEM Lab, ESIGELEC, Universite de Rouen, Rouen, France
[email protected]

Abstract. Vehicular networks are designed to provide a set of appli-

cations in order to maintain the traffic safety and efficiency. The effec-
tiveness of these applications depends on the accuracy of the location
information provided by vehicles. Accordingly, the presence of malicious
vehicles that broadcast fake location information can threaten the traffic
safety. In this paper, we propose a new fuzzy-based trust model that
aims at detecting the wrong position information that mismatches with
the vehicle’s behavior. Simulation results prove the performance of the
proposed model to detect the erroneous location information with high
precision.

1 Introduction
In the last decade, the Internet of Vehicles (IoVs) becomes of interest since it
aims at reducing the traffic-related crashes and ensuring the traffic safety [11].
To reach these purposes, the IoV incorporates different kinds of communica-
tions mainly: Vehicle-to-Sensor (V2S), Vehicle-to-Infrastructure (V2I), Vehicle-
to-Vehicle (V2V), Vehicle-to-Network (V2N), Vehicle-to-Human (V2H), etc. [1].
The Internet of Electric Vehicles (IoEVs) supports the same kinds of commu-
nications. It further incorporates the Electric Vehicle-to-Electric Vehicle Supply
equipments’ communications [2]. Electric Vehicles (EVs) are cost effective and
aim at reducing the environmental pollution and coping with the lack of natural
resources. Alike the IoV, the IoEV intends to enhance the driving experience
and help the driver to make the right decision in the right place and at the right
moment. However, the open, distributed and dynamic nature of vehicular net-
works makes them vulnerable to multiple security threats [10]. These latter can
intentionally (e.g. due to selfish reasons) or unintentionally (e.g. due to sensors’
failure) broadcast wrong traffic information to affect the others’ decisions.
The quality of applications highly depends on the reliability and accuracy
of location data. In electric vehicular networks, it is required to evaluate the
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 341–352, 2020.
https://doi.org/10.1007/978-3-030-15032-7_29
342 I. Souissi et al.

trustworthiness of location information in order to prevent the malicious vehicles

from misbehaving and guarantee the network’s security.
In the present paper, we introduce a new solution for the position trust eval-
uation that supports the V2N communications. This solution is based on the
similarity assessment between the vehicles’ detections in order to ensure that
only plausible and credible location information are considered by the central-
ized processing entities. We deploy the fuzzy logic theory due to its capacity
to meet with information uncertainties in vehicular networks. Therefore, the
contributions of this paper can be summarized as follows:
• Propose a new trust solution that supports the V2N communications.
• Introduce a new fuzzy logic based similarity model to evaluate the consistency
of the vehicle-related location information.
• Conduct experiments to validate the performance of the proposed solution in
regard to a well-defined scenario.
The rest of the paper will be structured as follows. Section 2 presents the
related work. Section 3 introduces the problem formulation. Section 4 provides
an overview about the proposed model for the position trust evaluation. Section 5
details the proposed solution. Section 6 presents the simulation results and dis-
cussion and Sect. 7 concludes the paper.

2 Related Work
Multiple research studies focused on the vehicles’ localization problem in
VANETs. The adopted approaches that aim at overcoming the localization issues
can be classified into three well-known classes [12]: localization-based trusted
certification, localization-based resource testing and localization based ranging
techniques.
The trusted certification approach is the most known one [12]. The aim of this
technique is to ensure trusted relationship between different entities based on
multiple techniques such as cryptography, digital signature, certificate author-
ity, etc. However, the implementation of cryptographic algorithms requires high
computation complexity, great memory to save all the security information and
large bandwidth for key distribution. Besides, the trusted certification approach
cannot resist to internal malicious threats that may provide wrong information
due to unintentional or selfish reasons.
The resource testing approach stands on the evaluation of the entity’s
resources [12]. Usually, a malicious node requires extra resources to accomplish
its malicious tasks. The resource testing approach can particularly meet with
the Sybil attacks. Often, this approach is not adopted in VANETs since some
vehicles are already designed to expend more computational resources than the
ordinary vehicles.
The localization based ranging techniques adopt physical measurements such
as Time Difference of Arrival (TDoA) method, Angle of Arrival (AoA) method,
Received Signal Strength Indicator (RSSI) method, etc. [3,6,12]. These methods
 A New Fuzzy Logic Based Model for Location Trust Estimation 343

depend on the adopted hardware as well as the physical environment. Time-based

localization model is strongly related to the signal propagation time. This model
performs well to estimate the distance between the transmitter and the receiver,
especially when they are equipped with synchronized clocks. However, the mul-
tipath inference may affect the accuracy of the estimated propagation time.
AoA method estimates the distance between both entities based on the relative
angle of the received signal from neighboring nodes. Nonetheless, this method
requires high cost to equip vehicles with directional antenna. AoA cannot also
provide accurate position information in high distributed environments when
multiple paths exist and the number of nodes is extremely low. Regarding the
RSSI-based method, it allows the receiver of a message to estimate the distance
that separates him from the transmitter using the theoretical radio propagation
models. However, similarly to the two previous methods, this one cannot pro-
vide accurate localization information in multipath environments. Besides, the
shadowing effects and the presence of building blocks can affect the attenuation
of the received signal.
In this paper, we present a new approach to evaluate the data trust in regard
to the provided position information in electric vehicular networks. Alike the
model presented in [9], this approach relies on the similarity assessment between
messages. However, in the present paper, we focus on another aspect of EVs
(i.e. trustworthiness of location information). Indeed, we aim at filtering the
inappropriate location information in order to ensure that only plausible packets
are considered during the service delivery. In the next sections, we describe the
proposed solution in more details.

3 Problem Formulation

Over the past decade, cellular technologies (e.g. 3G, 4G and 5G) and open Wi-Fi
access points have emerged to support the vehicle-to-network communications
(V2N) [5]. These technologies enable vehicles to directly communicate with cen-
tralized processing servers.
The electric vehicular networks mainly include: Electric Vehicles (EVs),
charging stations and servers. The latter entities are dedicated to analyze and
process the big amount of received data in order to present a set of applications
to the final users. These applications are usually classified into three main classes,
safety, traffic management and infotainment applications [8]. The main purposes
of such applications are to reduce the accidents, improve the driving experience
and maintain the security and comfort of drivers and passengers. Each kind of
applications presents particular features with different severity levels. Indeed, the
quality of these applications depends, mainly, on the quality of the transmitted
messages. Accordingly, the trust of these messages should be preserved in order
to filter the erroneous and inaccurate information. Position-related information
trustworthiness is one of the main factor for the success of these applications.
In this paper, we pay a special attention toward the position-related threats as
follows [10]:
344 I. Souissi et al.

• Sybil attack: the same vehicle may create multiple fake identities for other
vehicles. These latter present themselves under different locations in order to
mislead the network.
• False data injection attack: an EV lies about its real position due to selfish
(e.g. prevent the server from estimating the real traffic state in a specific lane)
or involuntary reasons (e.g. sensor’s failure).
• Global Positioning System (GPS) spoofing attack: the attacker convinces the
legitimate entities that they are in a wrong position, other than the real one.

Moreover, some vehicles may collude to convince the server that they follow a
particular lane in order to affect the identification of the optimal path and redi-
rect the legitimate vehicles towards another path. This fact may result in risky
situations (e.g. the actual battery state does not support the whole distance)
due to the wrong estimation of the traffic state.

4 Similarity-Based Position Trust Estimation Model

In this paper, we focus on the position-related information trustworthiness for

V2N communications. The proposed solution is described in Fig. 1. The server
receives multiple messages from dispersed vehicles in the network. Each packet
includes the next parameters: vehicle’s ID, lane ID, position (x, y), transmis-
sion time, average speed, energy consumption. The lane ID provides additional
information such as start and end coordinates, speed limit, etc. Whenever a
vehicle reaches the end of the lane, it transmits the travel time, average speed
and energy consumption throughout the involved lane.
As the focus of this paper is on the position trust assessment, we suppose that
all of the other parameters are credible. Therefore, the server aims at filtering
the inappropriate messages in terms of position. It starts by identifying the
corresponding lane that matches with the provided position. It verifies whether
the previous indicated lane and the present one are consecutive or not. If they
are not adjacent then, the message will be classified as untrustworthy. Otherwise,
the server looks for any alternative lanes. On the one hand, if the EV has the
opportunity to follow a single way then, the server will only check whether the
lane length, speed and time coincide with the provided location or not. If they
match then, the message will be accepted. On the other hand, if other potential
lanes exist then, the server compares their lengths. If these latter are similar
then, it refers to the others’ detections (e.g. energy consumption, speed, etc.) to
validate the position trustworthiness. Otherwise, if length of lanes differs then the
server can easily evaluate the consistency of the received position by estimating
whether the provided average speed and travel time allows the vehicle to browse
the whole lane or not. Finally, the accepted data is supposed to be kept in
memory for future use such as estimation of the traffic state, identification of
the optimal path, etc.
 A New Fuzzy Logic Based Model for Location Trust Estimation 345

Fig. 1. Flowchart of the position trust estimation model

5 Fuzzy-Logic Based Position Trust Computation

In this section, we detail the proposed solution for the position trust assessment
in vehicular networks.

5.1 Scenario and Assumptions

Our solution considers two main graph topologies as depicted in Fig. 2. In both
scenarios, we vary the lengths of lanes to show the impact of this parameter on
the identification of wrong position information.
Figure 2 shows that a vehicle may follow different paths to reach its desti-
nation. We suppose that the vehicle’s ID remains static during its journey and
the start point is definitely trustworthy. Each vehicle is supposed to transmit its
data packet whenever it moves from one lane to another. All of the transmit-
ted information are supposed to be trustworthy except the vehicle’s position. In
Fig. 2(a), the EV can, initially, move from the start point to A, B or C. If a
malicious vehicle is in A and it indicates that it is in B or C then, the server
checks whether the received vehicle’s information (i.e. average speed, travel time,
etc.) allows him to reach the indicated position or not. In this case, the length of
the corresponding lanes are not similar. This fact allows the server to detect any
wrong position information that mismatches with the other parameters. More-
over, the server can easily estimate whether the involved lanes are adjacent or
346 I. Souissi et al.

Fig. 2. Graphical representation of the vehicle trajectory scenarios

not based on the graph topology. For example, if it says that it follows the path
Start − B − J and thereafter it moves to G then, it is evident that the vehicle
cannot reach this location without going through K.
Figure 2(b) presents another graph topology: homogeneous topology. Similar
lengths of adjacent lanes characterize this kind of graph. If a vehicle is in the
start point then, it browses the same distance to reach A and B. Thus, we cannot
follow the same approach as in the first scenario. Therefore, we refer to other
information to estimate the position trustworthiness. We assume that each lane
has its own features at a given time mainly; the average speed, average energy
consumption, travel time, etc. Accordingly, to meet with the characteristics of
the homogeneous graph topology, we adopt the similarity assessment between
the vehicle’s readings and the others’ measurements. For example, if a malicious
vehicle was in A and it indicates that it reaches E, then the server looks for all
potential paths (i.e. A − D, A − C, A − E). Thereafter, it has to identify the
most consistent lane that meets with time, speed and energy information.

5.2 Description of the Proposed Solution

In this paper, we pay special attention towards the second scenario (refer to
Fig. 2(b)). We use the fuzzy logic theory in order to enable the server to estimate
the similarity between its acquired knowledge and the vehicle’s data. We should
state that the server has to be able to dynamically update its knowledge database
whenever the traffic state changes. To reach this purpose, we suppose that the
server adopts the majority strategy [7].
Similarity and trust are strongly associated with each other. The more the
similarity between messages is high the more trust increases. We create a Fuzzy
Inference System (F IS) that aims at comparing the vehicle’s information with
the server data knowledge. The proposed F IS considers three inputs and one
output as depicted in Fig. 3.
The input variables are: Time Difference Level (T DL), Speed Difference Level
(SDL) and Energy Consumption Difference Level (ECDL) and the output vari-
able is the Similarity Level (SL). The T DL denotes the difference between the
transmitted travel time and the server’s estimated travel time. The SDL refers
to the difference between the vehicle’s average speed and the server’s estimated
 A New Fuzzy Logic Based Model for Location Trust Estimation 347

Fig. 3. Fuzzy classes and membership functions for similarity assessment

speed. The ECDL denotes the difference between the vehicle’s energy consump-
tion and the expected energy.
The membership functions for the inputs and output are low, medium and
high. Fuzzy classes are determined to provide optimal output values. After
the fuzzification step, the correlation between the inputs and output should be
defined. Fuzzy inference engine is devoted to build the knowledge-based fuzzy
rules in order to describe the relationship between variables. We define 27 rules
that aim at supporting the decision-making system as depicted in Table 1. The
first rule in the table can be expressed as follows: If T DL, SDL and ECDL
are low then, the SL is high. In the presented F IS, we assume that all of the
inputs have the same importance and the same weight. Regarding the defuzzifi-
cation step, it aims at calculating the output value. In this paper, we consider
the centroid strategy to measure the SL as follows:

xi µ(xi )
SL = 
µ(xi )

xi denotes the fuzzy variable and µ(xi ) refers to the corresponding mem-
bership function. Finally, we should state that if the computed SL is below a
defined Threshold T h then, the message will be considered as untrustworthy.
Only trustworthy messages that match with the others’ detections will be kept
in the server database for future use.

6 Performance Evaluation
In this section, the performance of the proposed solution is evaluated in regard
to the second scenario. We use SU M O and M AT LAB to generate traffic data
and create a fuzzy-based inference engine, respectively. We configure SU M O in
such a way to provide a xml file that includes the vehicles-related information
348 I. Souissi et al.

Table 1. Rule base for similarity assessment

Rule Input Input Input Output Rule Input Input Input Output
T DL SDL ECDL SL T DL SDL ECDL SL
1 Low Low Low High 15 Medium Medium High Low
2 Low Low Medium Medium 16 Medium High Low Low
3 Low Low High Low 17 Medium High Medium Low
4 Low Medium Low Medium 18 Medium High High Low
5 Low Medium Medium Low 19 High Low Low Low
6 Low Medium High Low 20 High Low Medium Low
7 Low High Low Low 21 High Low High Low
8 Low High Medium Low 22 High Medium Low Low
9 Low High High Low 23 High Medium Medium Low
10 Medium Low Low Medium 24 High Medium High Low
11 Medium Low Medium Low 25 High High Low Low
12 Medium Low High Low 26 High High Medium Low
13 Medium Medium Low Low 27 High High High Low
14 Medium Medium Medium Low

Table 2. Simulation parameters

Parameter Value
Simulation time 100 s
Number of lanes (6)
Length of lanes 100 m
Number of vehicles 50 EV
Velocity range 7 km/h–37 km/h
Traffic density Random
Th 0.5

such as ID, consumed energy, speed, position, etc. The simulation parameters
are described in Table 2.
We suppose that a set of malicious vehicles may provide fake position infor-
mation to mislead the server. The proposed similarity-based model can iden-
tify such threats based on the correlation between the inputs (T DL, SDL and
ECDL) and the output (SL) as shown in Fig. 4.
The T DL is setting to 0.01 and 0.99 in Fig. 4(a) and (b), respectively.
Figure 4(a) shows that the SL can reach 0.83 when T DL, SDL and ECDL
are all very low. That is to say that the similarity level is increasingly high
when the difference between the vehicle’s information and the server knowledge
database is low. Figure 4(b) shows that SL decreases if the offset between the
received measurement and the expected one, for one parameter (T DL, SDL or
 A New Fuzzy Logic Based Model for Location Trust Estimation 349

Fig. 4. Correlation between inputs and output for the similarity assessment. (a)
T DL = 0.01 and (b) T DL = 0.99

ECDL), becomes wide. Therefore, the SL cannot exceed 0.35 if one parameter
differs from the expected value.
To validate the performance of the proposed solution, we perform a set of
simulations under different perspectives:
1. Simulate different mobility models to prove that the proposed solution is
always effective to detect the wrong position information.
2. Illustrate the impact of each parameter on the similarity level to prove that
the combination of the time, speed and energy information achieves more
accurate results than the similarity assessment in regard to one parameter.
3. Vary the lane characteristics, at each simulation, to show that whenever the
vehicle’s information mismatches with the expected one then, the similarity
level fails below the defined threshold.
We perform a set of experiments for two mobility models as depicted in
Fig. 5(a) and (b). In the first model, we suppose that the speed factor is pro-
portional to the acceleration rate. Indeed, the speed factor reflects the speed
limit per lane [4]. In such case, if the speed factor increases then, the speed limit
increases, as well. Regarding the second model (Fig. 5(b)), we set the speed fac-
tor into a defined value which means that all vehicles cannot exceed the same
speed limit.
Figure 5(a) shows that the energy consumption is the most influential factor
on the SL. This result is explained by the fact that whenever the acceleration
rate and the speed factor increase then, the consumed energy increases, cor-
respondingly. In Fig. 5(b), the SL, in regard to the consumed energy, remains
static (SL(Energy) = 0.84) as the speed factor is a constant. That is to say that
a vehicle that chooses to gradually accelerate (i.e. the speed and energy increase
a little bit, correspondingly, at each timestamp) consumes the same amount of
energy as a vehicle that aggressively accelerates and thereafter maintains the
same speed level. This remains true as long as the speed limit is preset in both
cases. Therefore, in the last case, the energy information cannot provide addi-
tional information about the similarity level.
In both mobility models, we assume that the average speed in L1 is equal to
7 km/h, the consumed energy is equal to 57.14 W and the travel time is equal
350 I. Souissi et al.

Fig. 5. Impact of time, speed and energy on the similarity assessment. (a) SL for
dynamic speed factor and (b) SL for static speed factor

to 51.42 s. Therefore, if a node lies about its position (e.g. indicates that it was
in L1 while it was really in L2) then, its data will not match with the lane
characteristics at a given moment. Accordingly, we conduct multiple simulations
under different L2-related information to evaluate the capacity of the proposed
solution to effectively detect the wrong position information. We suppose that
the average speed, varies from 9 km/h to 37 km/h. Regarding the other infor-
mation, it varies correspondingly. Simulation results prove that if a malicious
node indicates that it was in L1 instead of L2 then it will be detected since its
information mismatches with the lane characteristics.
In brief, we deduce that the integration of time, speed and energy consump-
tion information improves the accuracy of the detection of malicious vehicles that
provide wrong position information compared with the use of a single parameter.
 A New Fuzzy Logic Based Model for Location Trust Estimation 351

7 Conclusion
The position trustworthiness is a key element to ensure a high quality of the
provided applications in vehicular networks. In this paper, we introduced a new
fuzzy-based similarity model for location trust assessment in the context of elec-
tric vehicular networks. The proposed model focuses on vehicle-to-network com-
munications in order to cope with fake position information and ensure that
only credible and plausible information are considered by the servers. As future
work, we mainly intend to evaluate the performance of the proposed solution in
regard to the position-related threats mainly the Sybil attack. We further intend
to consider a real city map in order to validate the performance of the proposed
solution. Finally, an extensive comparison with other models will be performed
in regard to many criteria such as effectiveness, energy consumption, etc.

Acknowledgments. This work was supported by the PHC Utique program of the
French Ministry of Foreign Affairs and Ministry of higher education and research and
the Tunisian Ministry of higher education and scientific research in the CMCU project
number 16G1404.

References
1. Abboud, K., Omar, H.A., Zhuang, W.: Interworking of DSRC and cellular network
technologies for V2X communications: a survey. IEEE Trans. Veh. Technol. 65(12),
9457–9470 (2016)
2. Bayram, I.S., Papapanagiotou, I.: A survey on communication technologies and
requirements for internet of electric vehicles. EURASIP J. Wirel. Commun. Netw.
2014(1), 223 (2014)
3. Chen, Y., Yang, J., Trappe, W., Martin, R.P.: Detecting and localizing identity-
based attacks in wireless and sensor networks. IEEE Trans. Veh. Technol. 59(5),
2418–2434 (2010)
4. Krajzewicz, D., Bonert, M., Wagner, P.: The open source traffic simulation package
sumo. In: RoboCup 2006 (2006)
5. Lu, N., Cheng, N., Zhang, N., Shen, X., Mark, J.W.: Connected vehicles: solutions
and challenges. IEEE Internet of Things J. 1(4), 289–299 (2014)
6. Qureshi, K.N., Abdullah, A.H.: Localization-based system challenges in vehicular
ad hoc networks: survey. Smart CR 4(6), 515–528 (2014)
7. Raya, M., Papadimitratos, P., Gligor, V.D., Hubaux, J.-P.: On data-centric trust
establishment in ephemeral ad hoc networks. In: The 27th Conference on Computer
Communications, INFOCOM 2008, pp. 1238–1246. IEEE (2008)
8. Souissi, I., Azzouna, N.B., Berradia, T.: Towards a self-adaptive trust management
model for vanets. In: Proceedings of the 14th International Joint Conference on e-
Business and Telecommunications, SECRYPT, (ICETE 2017), vol. 6, pp. 513–518.
INSTICC, SciTePress (2017)
9. Souissi, I., Azzouna, N.B., Berradia, T., Said, L.B.: Fuzzy logic based model for
energy consumption trust estimation in electric vehicular networks. In: Proceedings
of the 15th International Joint Conference on e-Business and Telecommunications,
SECRYPT, vol. 2, pp. 221–233. INSTICC, SciTePress (2018)
352 I. Souissi et al.

10. Sumra, I.A., Hasbullah, H.B., AbManan, J.B.: Attacks on security goals (confiden-
tiality, integrity, availability) in VANET: a survey. In: Vehicular Ad-Hoc Networks
for Smart Cities, pp. 51–61. Springer (2015)
11. Sun, S., Hu, J., Peng, Y., Pan, X., Zhao, L., Fang, J.: Support for vehicle-to-
everything services based on LTE. IEEE Wirel. Commun. 23(3), 4–8 (2016)
12. Yao, Y., Xiao, B., Wu, G., Liu, X., Yu, Z., Zhang, K., Zhou, X.: Voiceprint: a
novel sybil attack detection method based on RSSI for VANETs. In: 2017 47th
Annual IEEE/IFIP International Conference on Dependable Systems and Net-
works (DSN), pp. 591–602. IEEE (2017)
 Angular Histogram-Based Visualisation
of Network Traffic Flow Measurement
Data

Adrian Pekar(B) , Mona B. H. Ruan, and Winston K. G. Seah

Victoria University of Wellington, Wellington, New Zealand

{adrian.pekar,winston.seah}@ecs.vuw.ac.nz, [email protected]

Abstract. Knowledge of the traffic that is being carried within a net-

work is critical for ensuring the network’s smooth operation, and net-
work traffic measurement has provided an effective means to achieve
this. However, network traffic volume has substantially increased over
the last decades. Combine that with the traffic heterogeneity from a
diverse range of new, connected devices, we have reached a point where
the response to any outage or anomalous event is simply beyond human
ability. Network information visualisation is a useful tool to help network
administrators deal with this problem. While this approach is not new,
traditional approaches do not scale well with the increasing volume and
heterogeneity of network traffic. In this paper, we propose the applica-
tion of angular histogram visualisation to provide an information-rich
overview of large network traffic data sets to improve the interpretation
and understanding of network traffic flow measurement data. We evalu-
ate our approach experimentally using live network traffic to demonstrate
its efficacy and provide suggestions on how it can be further improved.

1 Introduction

Network traffic measurement is an essential part of computer networks for a

variety of network managerial tasks such as monitoring for anomaly, intrusion
and security violation detection, accounting, profiling, troubleshooting, as well
as, traffic classification and engineering [16]. Traffic measurement requires fast
techniques for processing hundreds of thousands of packets per second and effi-
cient methods for storing the measured data for future use. However, as new
devices are connected, services launched, and applications being used, the traffic
that flows through the network is continuously growing in volume and diversity.
By 2021, network traffic volume is forecasted to increase 127-fold since 2005 [3].
Consequently, making operational sense out of the massive measurement data is
a challenge.
A tool that is aimed at streamlining and simplifying the analysis of the net-
work traffic measurement data is information visualisation [7,11]. Information
visualisation is gaining popularity as turning all forms of potentially obscure data
into an image-based representation has a number of advantages. For example, it
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 353–365, 2020.
https://doi.org/10.1007/978-3-030-15032-7_30
354 A. Pekar et al.

can help to reveal relationships and properties implicit and inherent in the under-
lying data. Information visualisation can also provide a great level of reduction
using various compression techniques. Furthermore, information visualisation is
not only efficient but also very effective at communicating information.
Existing information visualisation approaches can be categorised into value
and relation visualisations [4]. The former focuses on the representation for each
data point via visual mappings. It usually utilises colour maps and parallel coor-
dinates. The advantage of value visualisation approaches is that they allow for a
detailed analysis of network traffic insights. However, they lack an overview and
are usually subject to strong visual clutter [4]. The latter, relation visualisation,
convey data relationships. Relation visualisation approaches are usually point
mappings that project high-dimensional data into the low-dimensional presenta-
tion space. However, as network traffic measurement data are often too complex
to be conveyed in presentation space, relationship visualisations often result in
ambiguous projections [4].
To make accurate reasoning about the behaviour of network traffic (e.g.
bursty or elephant flow) and its motive (e.g. network attack or normal traf-
fic flow), value and relation visualisations are often used in a complementary
manner. In practice, this means that in multi-view network monitoring systems
the same data (or features) is often re-queried/recalculated to be used for differ-
ent views (graphs). This, however, radically increases the computing demands of
the monitoring system as well as the time costs. Our experience has shown that
state-of-the-art network traffic data visualisation tools are often incapable of effi-
ciently visualise the massive volume of network traffic measurement data [17].
Related works [7,15] also report on similar experiences. In addition, most infor-
mation visualisation approaches in the domain of network and service manage-
ment focus only on the security aspects, such as [9,10], rather than the global
interpretation of the network and its traffic. To ensure the quality and reliability
of the networks and their applications, smarter and more efficient approaches
must be developed for traffic monitoring, especially in the field of visualisation.
In this paper, we apply the angular histogram visualisation technique [6] on
network traffic flow measurement data in order to derive an information-rich
overview of large data sets for improving the interpretation and understanding
of network traffic flow measurement data. To the best of our knowledge, no
such work currently exists in the domain of network and traffic management.
The rest of this paper is organised as follows: Sect. 2 provides an overview of
flow measurement concepts and the visualisation process. The evaluation of the
proposed approach is discussed in Sect. 4, followed by discussion on salient issues
to be addressed by further research, before concluding.

2 Network Traffic Flow Measurement Data

A flow refers to a set of packets sharing a common key that traverse an obser-
vation point during a certain period of time. This flow key is usually defined by
a five-tuple, namely, the source and destination IP addresses and port numbers,
 Angular Histogram-Based Visualisation 355

and the protocol identifier. Traffic information such as flow features (e.g. the
total number of bytes of all packets belonging to a certain flow) and the flow key
(also referred to as flow properties) is carried in flow records. Network manage-
ment and monitoring related activities are based on the analysis of the exported
flow records while the most significant information resides in the flow features.
The computation of the flow features usually involves analysis-centric methods
such as feature extraction, time stamping, sampling, and filtering.

2.1 Flow Records

An example of flow records stored in the flow cache is shown in Table 1. Depend-
ing on the properties of the observed (incoming) packets, either new flow records
are created or the flow features of existing ones are updated. The flow records
in Table 1 were created using the flowRecorder tool [5]. flowRecorder is a pro-
gram written in Python that turns IP packets, either in the form of PCAP1
files or sniffed live from a network interface, into flow records and stores them
in a CSV2 file. Flows composed only of packets sent from a single endpoint to
another single endpoint are usually termed as unidirectional flows. When infor-
mation about flows composed of packets sent in both directions between two
endpoints are required (two unidirectional flows in opposite directions), bidirec-
tional flows may be considered. flowRecorder supports the measurement of flow
features in both unidirectional and bidirectional modes.

Table 1. Example of two unidirectional flow records pertaining to two traffic flows

Flow property/feature Flow record 1 Flow record 2

Observation point ID 123 123
Flow ID 26a167f622gs349 704aw8r582467be
Source IP 192.168.11.5 172.16.0.5
Source port 1319 443
Destination IP 192.168.68.72 192.168.68.55
Destination port 1319 44893
Protocol identifier 17 6
Flow start nanosec. 1400508636284335332 1400508899789470101
Flow end nanosec. 1400508901800109719 1400508900578408002
Octet delta count 84 118
Octet total count 4536 118
Packet delta count 1 2
Packet total count 54 2

1
Packet CAPture.
2
Comma-Separated Values.
356 A. Pekar et al.

2.2 Flow Feature Selection

Generating a subset (derivation) of the original data is a common practice in

network and service management. The main reason for creating a subset is the
extremely large number of flow features that can be measured for every single
network traffic flow. An extensive list of 249 flow features (often referred to as the
full feature set) can be observed from the network traffic [12], the computation of
which is a resource and time demanding process. For example, the computation
of all the 249 features for all traces on a dedicated system area network was
shown to take over 40 h [1]. In addition, not only is the feature computation a
challenge, the visual mapping and view transformation of this relatively large
number of flow features are also complex.
The computation of the full feature set is, however, not necessary for achiev-
ing accurate and reliable network traffic monitoring and related tasks [8,13,14].
As such, for our prototype implementation, we have decided to reduce the full
flow feature set to a small but representative number that is sufficient to test the
viability of the solution, and we selected commonly used features, viz. packet and
octet counts, flow duration, average packet inter-arrival time and packet size (as
shown in bold in Table 2, which lists the 22 statistical flow features supported
by flowRecorder.).

Table 2. List of the flow features supported by flowRecorder

Feature name Feature description Direction

Forward Backward
min ps Packet with minimum size in the flow  
max ps Packet with maximum size in the flow  
avg ps Average packet size  
std ps Standard deviation of packet sizes  
min piat Minimum packet inter-arrival time  
max piat Maximum packet inter-arrival time  
avg piat Average packet inter-arrival time  
std piat Standard deviation of packet  
inter-arrival times
octTotCount Number of transmitted data (in  
bytes) in the flow
pktTotCount Number of transmitted packets in the  
flow
flowDur Duration of the flow (usually in
milliseconds)
proto Protocol identifier (e.g. TCP/UDP)
 Angular Histogram-Based Visualisation 357

3 Angular-Histogram-Based Visualisation of Flow

Records
Our traffic flow visualisation technique is based on the angular histogram app-
roach [6]. An angular histogram is a vector-based binning approach. It has several
advantages over traditional point-based histogram visualisations. First, its space
complexity is lower compared to line-based visualisation approaches. Second, it
reveals the relationship of the plots between neighbouring axes providing thus
a rich overview of the data. Third, the interaction with the visualisation (e.g.
by selecting and brushing) can provide a better understanding of the underlying
data and its semantics.

Fig. 1. Break down of an angular histogram [6] styled graph and its transformation
into a bar graph. The resulting visualisation aims to provide the network administrator
with a rich overview of the underlying data and better understanding that cannot be
gained from a traditional point-based histogram.

Figure 1 shows an example of an angular histogram. Our design begins with

a break down of the cohesive nature of an angular histogram styled graph and
its transformation into a bar graph as shown in Figs. 2 and 3. Whilst a standard
bar graph is not a new design, the application of flow records within the bar
graph is novel. Typically, a bar graph uses the same quantities and variables for
comparison between the bars, however within this implementation, each graph
is compared to the other. Rather than comparing individual bars, the collective
of bars can be compared against other graphs.
Each bar, as shown in Fig. 2, can map one flow feature within the flow record.
Given that the graph is separated by a vertical axis, the flow and its features can
be effectively interpreted in different directions. The positive part can represent
the flow statistics in one direction while the negative part can show the flow
behaviour in the opposite direction as per Fig. 3.
However, as the individual flow features are in different metrics (e.g. flow
duration is usually in milliseconds, the transferred data in bytes, and the num-
ber of packets in integers), each features mapping must be multiplied by a certain
factor. Otherwise, some features may cause clutter due to their high values com-
pared to other features. The bars can be also coloured (e.g. with green and red)
358 A. Pekar et al.

Fig. 2. One of the desired capabilities of network traffic visualisation is the ability
to individually traverse through the provided dataset, whilst observing each flow on
the screen. This allows network administrators to watch for changes as the graphs are
generated. Each bar represents a different flow feature that is deemed important to
flow assessment.

Fig. 3. The negative values on the graph represent the flow statistics obtained from
packets passed in one direction of the network and the positive values represent the
statistics obtained from packets passed in the other direction. As the individual flow
features are in different metrics, these must be multiplied by a certain factor.

to show whether or not they have reached a certain threshold that is of concern
(see Fig. 4). The threshold is a value that needs to be specifically calculated for
the specific flow feature. This is because other features will not have the same
values of concerns.
Figure 5 provides an example of multiple graphs that can be compared to
each other. Such a comparison can be useful to spot outliers. To further asses
the suspicious flows, the administrator, for example, can select specific flows and
investigate them in more details.
Lastly, a visualisation of all the flow records can be also useful. By mapping
multiple visualisations into one graph, as shown in Fig. 6, network administrators
can identify common traits between flows. This can be achieved through the
overlapping organisation of flow record visualisations.
 Angular Histogram-Based Visualisation 359

Fig. 4. The bars can be coloured with green and red to show whether or not they have
reached a certain threshold that is of concern. The threshold needs to be specifically
calculated for the specific flow feature as other features may have different levels/values
of concerns.

Fig. 5. Multiple flow visualisation is used for comparing flows against one another. It
is useful to use in scenarios where the network administrator has identified a possible
malicious series of flows and wishes to analyse further. For example, using the individual
flow visualisation as described in Figs. 2 and 3, one can identify the flows that are
of concern and then, proceed to use multiple flow visualisation to further assess the
surrounding flows. Colour intensity can help to identify outliers more easily.
360 A. Pekar et al.

Fig. 6. This visualisation allows the interpretation of all the flow records. The network
administrator can identify common traits between flow records as each flow overlaps
the others. The denser areas show the most common values of the flow features and
demonstrate possible outliers.

4 Evaluation and Results Analysis

Listing 1. The pseudo-code for data visualisation

Open the data set
For each line in data set
Parse in the data
Add data into an array
Set Flow Features for the Y axis
For each flow record
generate a graph displaying appropriate data

The visualisation approach was implemented in the Python language using

the bar graph function of the matplotlib library. The pseudo-code in Listing 1
illustrates the procedure used in generating the visualisation to create an ani-
mated effect that allows the user to quickly identify possible outliers. Using
flowRecorder, we performed a short network traffic measurement on a standard
network connected device. The packets were organised into flow records and then
saved in a CSV file.
Figure 7 shows visualisation outputs for two arbitrary flows. The negative
axis displays data being forwarded on from the network and the positive axis
shows data being received for the same flow record. Different colours are used
to identify features in the flow record that have surpassed a certain threshold.
In our experiments, when a feature exceeds the predefined threshold and thus
needs further investigation, it is shown in red. The specific thresholds were set
at 50% of the maximum value received in the file. Whilst this may not be the
optimal way of identifying risky flow records, it allowed us to test and validate
the functionality of our solution. While single flow record visualisation provides
insight into flow details, plotting the flows adjacent to another is also useful
 Angular Histogram-Based Visualisation 361

(a) Flow ID: 2 (b) Flow ID: 15

Fig. 7. Visualisation of individual flow records

Fig. 8. Comprehensive comparisons of flow records

for comparing them, as shown in Fig. 8. The red and green coloured bars help
identify which flow may have an outlier.
Lastly, Fig. 9 shows visualisations of overlapping flow records. Figure 9(a)
shows 100 flow records that have overlapped each other. It is simple to identify
362 A. Pekar et al.

(a) Multiple colour visualisation of 100 flows (b) Single colour visualisation of 632 flows

Fig. 9. Visualisation of flow records that overlap

the common traits between the 100 flows, i.e. the denser an area is, the more
common a flow record matches that range. This visualisation combines colours
and densities, and utilises a single axis for all the data. If the data were overlap-
ping in a graph, the use of colours would not be necessary since one can easily
identify the most common ranges from colour density. As such, a single colour
variant of the visualisation was also implemented. Figure 9(b) shows an overlap
of 632 flows mapped onto a single graph as discussed in Fig. 6. The uniform red
colour helps to visualise the most dense areas. The Y axis also allows the user
to identify where the outgoing and incoming traffic begins.

5 Discussion
The experiments performed using our prototype showed promising results. First,
the overlapping visualisation of all the flow records is a useful method to detect
certain anomalies in the network. Assuming the colour have a certain level of
transparency, the overlapping bars will yield a colour that is ever-increasing
in intensity. This can be used, for example, to detect port scanning that is a
precursor of brute force attacks.
Second, the side-by-side visualisation of a number of flows helps to highlight
anomalies or outliers. In real-world network environments, hundreds of thou-
sands of flows exist per every single second. While the side-by-side visualisation
of such a large volume of flows can make the overall perception of the traf-
fic behaviour highly cluttered, this can be alleviated with appropriate colouring
and thresholds. The current implementation of our visualisation approach is still
useful, especially when used as a meta-visualisation tool, i.e. a preliminary visu-
alisation to identify the appropriate features and setting for other visualisation
tools. A further approach to reducing the clutter in the side-by-side visualisation
of a large number of flows can be to plot an iconic version of the visualisation
generated in a scatter plot view.
 Angular Histogram-Based Visualisation 363

The colouring decision of the individual flow features is currently threshold-

based. However, due to the dynamic nature of network traffic, estimating
and selecting an appropriate threshold is a challenge itself. The challenge of
threshold-based decisions is also highly relevant to in-situ visualisation. In-
situ visualisations incrementally generate visual representations when new data
arrives. With network traffic measurement data two problems emerge. First, new
data can come from a number of different measurement points which may observe
the same packets, thus generating duplicate data. This creates a great challenge
in streaming data analytics as it becomes difficult to quickly detect various events
in the huge amounts of streaming data and understand the information diffu-
sion patterns in them. Second, the flow records and their values dynamically
change depending on the network traffic. In practice, the flow records are stored
in the flow cache sequentially and do not have a fixed position. This raises the
complexity when mapping information to the visualisations.

Fig. 10. Adaptation of the information visualisation pipeline [2]

To enhance the proposed approach, the visual mapping and view transfor-
mation functions (as shown in Fig. 10) of the information visualisation pipeline
will be implemented as our ongoing and future research. The information visu-
alisation pipeline [2] is a widely-recognised and referenced model used for trans-
forming data into images as it captures the key elements of visualisation across
a variety of domains and visual techniques. Visual mapping is the process of
encoding the information (data) into a visual representation while view trans-
formation focuses on screen space and rendering.

6 Conclusion

Data visualisation is an important aspect of network and service monitoring

and management. Network traffic flow measurement data provides significant
information about network traffic semantics. It has been shown to be useful in
a variety of network managerial related tasks. However, with the increase in the
volume of network traffic, making an operational sense out the measurement
data is a challenge, from data processing to information visualisation.
To ensure the quality and reliability of the networks, smart and efficient flow
measurement data visualisation techniques must be developed. In this paper,
we presented our angular histogram-based visualisation technique that aims to
improve the analysis of traffic flow measurement data. Preliminary visualisations
364 A. Pekar et al.

produced by the implemented prototype show promising results. Future work will
focus on the optimisation of the approach, specifically, on visual mapping and
view transformation.

Acknowledgement. A. Pekar and W. Seah are supported by VUW’s Huawei NZ

Research Programme, Software-Defined Green Internet of Things (project #E2881).

References
1. Auld, T., Moore, A.W., Gull, S.F.: Bayesian neural networks for internet traffic
classification. IEEE Trans. Neural Netw. 18(1), 223–239 (2007). https://doi.org/
10.1109/TNN.2006.883010
2. Card, S.K., Mackinlay, J.D., Shneiderman, B. (eds.): Readings in Information Visu-
alization: Using Vision to Think. Morgan Kaufmann Publishers Inc., San Francisco
(1999)
3. Cisco and/or its affiliates: Cisco visual networking index: forecast and
methodology, 2016–2021. Technical report C11-481360-01. Cisco Systems,
Inc. (2017). https://www.cisco.com/c/en/us/solutions/service-provider/visual-
networking-index-vni/index.html
4. Engel, D., Hagen, H., Hamann, B., Rosenbaum, R.: Structural decomposition trees:
semantic and practical implications. In: Csurka, G., Kraus, M., Laramee, R.S.,
Richard, P., Braz, J. (eds.) Computer Vision, Imaging and Computer Graphics.
Theory and Application, pp. 193–208. Springer, Berlin (2013)
5. flowRecorder: A network traffic flow feature measurement tool (2018). https://
github.com/drnpkr/flowRecorder
6. Geng, Z., Peng, Z., Laramee, R.S., Roberts, J.C., Walker, R.: Angular histograms:
frequency-based visualizations for large, high dimensional data. IEEE Trans. Vis.
Comput. Graph. 17(12), 2572–2580 (2011). https://doi.org/10.1109/TVCG.2011.
166
7. Guimaraes, V.T., Freitas, C.M.D.S., Sadre, R., Tarouco, L.M.R., Granville, L.Z.:
A survey on information visualization for network and service management. IEEE
Commun. Surv. Tutor. 18(1), 285–323 (2016). https://doi.org/10.1109/COMST.
2015.2450538
8. Kim, H., Claffy, K., Fomenkov, M., Barman, D., Faloutsos, M., Lee, K.: Inter-
net traffic classification demystified: myths, caveats, and the best practices. In:
Proceedings of the 4th ACM Conference on Emerging Network Experiment and
Technology (CoNEXT), Madrid, Spain, pp. 11:1–11:12 (2008). https://doi.org/10.
1145/1544012.1544023
9. Kim, S.S., Reddy, A.L.N.: A study of analyzing network traffic as images in real-
time. In: Proceedings of the 24th Annual Joint Conference of the IEEE Computer
and Communications Societies, vol. 3, pp. 2056–2067 (2005). https://doi.org/10.
1109/INFCOM.2005.1498482
10. Kim, S.S., Reddy, A.L.N.: Image-based anomaly detection technique: algorithm,
implementation and effectiveness. IEEE J. Sel. Areas Commun. 24(10), 1942–1954
(2006). https://doi.org/10.1109/JSAC.2006.877215
11. Liu, S., Cui, W., Wu, Y., Liu, M.: A survey on information visualization: recent
advances and challenges. Vis. Comput. 30(12), 1373–1393 (2014). https://doi.org/
10.1007/s00371-013-0892-3
 Angular Histogram-Based Visualisation 365

12. Moore, A.W., Zuev, D., Crogan, M.L.: Discriminators for use in flow-based classi-
fication. Technical report RR-05-13, Queen Mary University of London (2005)
13. Nguyen, T.T.T., Armitage, G.: A survey of techniques for internet traffic classifi-
cation using machine learning. IEEE Commun. Surv. Tutor. 10(4), 56–76 (2008).
https://doi.org/10.1109/SURV.2008.080406
14. Pekár, A., Chovanec, M., Vokorokos, L., Chovancová, E., Feciľak, P., Michalko,
M.: Adaptive aggregation of flow records. Comput. Inform. 37(1), 142–164 (2018).
https://doi.org/10.4149/cai 2018 1 142
15. Shiravi, H., Shiravi, A., Ghorbani, A.A.: A survey of visualization systems for
network security. IEEE Trans. Vis. Comput. Graphi. 18(8), 1313–1329 (2012).
https://doi.org/10.1109/TVCG.2011.144
16. Valenti, S., Rossi, D., Dainotti, A., Pescapé, A., Finamore, A., Mellia, M.: Review-
ing traffic classification, pp. 123–147. Springer, Berlin (2013). https://doi.org/10.
1007/978-3-642-36784-7 6
17. Vokorokos, L., Pekar, A., Adam, N.: Data preprocessing for efficient evaluation of
network traffic parameters. In: Proceedings of the 16th IEEE International Con-
ference on Intelligent Engineering Systems, INES, pp. 363–367 (2012). https://doi.
org/10.1109/INES.2012.6249860
 Performance Analysis of WMNs
by WMN-PSODGA Simulation System
Considering Weibull and Chi-square
Client Distributions

Admir Barolli1 , Shinji Sakamoto2(B) , Leonard Barolli3 , and Makoto Takizawa4

1
Department of Information Technology, Aleksander Moisiu University of Durres,
L.1, Rruga e Currilave, Durres, Albania
[email protected]
2
Department of Computer and Information Science, Seikei University,
3-3-1 Kichijoji-Kitamachi, Musashino-shi, Tokyo 180-8633, Japan
[email protected]
3
Department of Information and Communication Engineering,
Fukuoka Institute of Technology, 3-30-1 Wajiro-Higashi,
Higashi-Ku, Fukuoka 811-0295, Japan
[email protected]
4
Department of Advanced Sciences, Faculty of Science and Engineering,
Hosei University, Kajino-Machi, Koganei-Shi, Tokyo 184-8584, Japan
[email protected]

Abstract. The Wireless Mesh Networks (WMNs) are becoming an

important networking infrastructure because they have many advantages
such as low cost and increased high speed wireless Internet connectivity.
In our previous work, we implemented a Particle Swarm Optimization
(PSO) based simulation system, called WMN-PSO, and a simulation
system based on Genetic Algorithm (GA), called WMN-GA, for solving
node placement problem in WMNs. Then, we implemented a hybrid sim-
ulation system based on PSO and distributed GA (DGA), called WMN-
PSODGA. In this paper, we analyze the performance of WMNs using
WMN-PSODGA simulation system considering Weibull and Chi-square
client distributions. Simulation results show that a good performance is
achieved for Chi-square distribution compared with the case of Weibull
distribution.

1 Introduction
The wireless networks and devices are becoming increasingly popular and they
provide users access to information and communication anytime and any-
where [2,6–9,12,17,23–25,28]. Wireless Mesh Networks (WMNs) are gaining a
lot of attention because of their low cost nature that makes them attractive for
providing wireless Internet connectivity. A WMN is dynamically self-organized
and self-configured, with the nodes in the network automatically establishing
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 366–375, 2020.
https://doi.org/10.1007/978-3-030-15032-7_31
 Performance Analysis of WMNs by WMN-PSODGA Simulation System 367

and maintaining mesh connectivity among them-selves (creating, in effect, an

ad hoc network). This feature brings many advantages to WMNs such as low
up-front cost, easy network maintenance, robustness and reliable service cover-
age [1]. Moreover, such infrastructure can be used to deploy community networks,
metropolitan area networks, municipal and corporative networks, and to support
applications for urban areas, medical, transport and surveillance systems.
Mesh node placement in WMN can be seen as a family of problems, which are
shown (through graph theoretic approaches or placement problems, e.g. [4,13])
to be computationally hard to solve for most of the formulations [29]. We consider
the version of the mesh router nodes placement problem in which we are given a
grid area where to deploy a number of mesh router nodes and a number of mesh
client nodes of fixed positions (of an arbitrary distribution) in the grid area. The
objective is to find a location assignment for the mesh routers to the cells of the
grid area that maximizes the network connectivity and client coverage.
Node placement problems are known to be computationally hard to solve [10,
11,30]. In some previous works, intelligent algorithms have been recently inves-
tigated [3,5,14,15,18–20,26,27].
In [21], we implemented a Particle Swarm Optimization (PSO) based sim-
ulation system, called WMN-PSO. Also, we implemented another simulation
system based on Genetic Algorithm (GA), called WMN-GA [16], for solving
node placement problem in WMNs.
In our previous work, we designed and implemented a hybrid simulation
system based on PSO and distributed GA (DGA). We called this system WMN-
PSODGA. In this paper, we evaluate the performance of WMNs using WMN-
PSODGA simulation system considering Weibull and Chi-square client distribu-
tions.
The rest of the paper is organized as follows. The mesh router nodes place-
ment problem is defined in Sect. 2. We present our designed and implemented
hybrid simulation system in Sect. 3. The simulation results are given in Sect. 4.
Finally, we give conclusions and future work in Sect. 5.

2 Node Placement Problem in WMNs

For this problem, we have a grid area arranged in cells we want to find where to
distribute a number of mesh router nodes and a number of mesh client nodes of
fixed positions (of an arbitrary distribution) in the considered area. The objective
is to find a location assignment for the mesh routers to the area that maximizes
the network connectivity and client coverage. Network connectivity is measured
by Size of Giant Component (SGC) of the resulting WMN graph, while the user
coverage is simply the number of mesh client nodes that fall within the radio
coverage of at least one mesh router node and is measured by Number of Covered
Mesh Clients (NCMC).
An instance of the problem consists as follows.
• N mesh router nodes, each having its own radio coverage, defining thus a
vector of routers.
368 A. Barolli et al.

• An area W ×H where to distribute N mesh routers. Positions of mesh routers

are not pre-determined and are to be computed.
• M client mesh nodes located in arbitrary points of the considered area, defin-
ing a matrix of clients.

It should be noted that network connectivity and user coverage are among
most important metrics in WMNs and directly affect the network performance.
In this work, we have considered a bi-objective optimization in which we first
maximize the network connectivity of the WMN (through the maximization of
the SGC) and then, the maximization of the NCMC.
In fact, we can formalize an instance of the problem by constructing an
adjacency matrix of the WMN graph, whose nodes are router nodes and client
nodes and whose edges are links between nodes in the mesh network. Each mesh
node in the graph is a triple v = < x, y, r > representing the 2D location point
and r is the radius of the transmission range. There is an arc between two nodes
u and v, if v is within the transmission circular area of u.

3 Proposed and Implemented Simulation System

3.1 WMN-PSODGA Hybrid Simulation System

Distributed Genetic Algorithm (DGA) has been focused from various fields of sci-
ence. DGA has shown their usefulness for the resolution of many computationally
hard combinatorial optimization problems. Also, Particle Swarm Optimization
(PSO) has been investigated for solving NP-hard problem.

Fig. 1. Model of WMN-PSODGA migration.

PSO Part
WMN-PSODGA decide the velocity of particles by a random process consid-
ering the area size. For instance,
√ √ area size is W × H, the velocity is
when the
decided randomly from − W 2 + H 2 to W 2 + H 2 . Each particle’s velocities
are updated by simple rule [22].
 Performance Analysis of WMNs by WMN-PSODGA Simulation System 369

Fig. 2. Relationship among global solution, particle-patterns and mesh routers in PSO
part.

DGA Part
Population of individuals: Unlike local search techniques that construct a path
in the solution space jumping from one solution to another one through local
perturbations, DGA use a population of individuals giving thus the search a
larger scope and chances to find better solutions. This feature is also known
as “exploration” process in difference to “exploitation” process of local search
methods.
Selection: The selection of individuals to be crossed is another important
aspect in DGA as it impacts on the convergence of the algorithm. Several selec-
tion schemes have been proposed in the literature for selection operators trying
to cope with premature convergence of DGA. There are many selection methods
in GA. In our system, we implement 2 selection methods: Random method and
Roulette wheel method.
Crossover operators: Use of crossover operators is one of the most impor-
tant characteristics. Crossover operator is the means of DGA to transmit best
genetic features of parents to offsprings during generations of the evolution pro-
cess. Many methods for crossover operators have been proposed such as Blend
Crossover (BLX-α), Unimodal Normal Distribution Crossover (UNDX), Simplex
Crossover (SPX).
Mutation operators: These operators intend to improve the individuals of a
population by small local perturbations. They aim to provide a component of
randomness in the neighborhood of the individuals of the population. In our
system, we implemented two mutation methods: uniformly random mutation
and boundary mutation.
Escaping from local optima: GA itself has the ability to avoid falling prema-
turely into local optima and can eventually escape from them during the search
process. DGA has one more mechanism to escape from local optima by consid-
ering some islands. Each island computes GA for optimizing and they migrate
its gene to provide the ability to avoid from local optima.
Convergence: The convergence of the algorithm is the mechanism of DGA
to reach to good solutions. A premature convergence of the algorithm would
cause that all individuals of the population be similar in their genetic features
and thus the search would result ineffective and the algorithm getting stuck
into local optima. Maintaining the diversity of the population is therefore very
important to this family of evolutionary algorithms.
370 A. Barolli et al.

In following, we present our proposed and implemented simulation sis-

tem called WMN-PSODGA. We show the fitness function, migration function,
particle-pattern, gene coding and client distributions.
Fitness Function
The determination of an appropriate fitness function, together with the chromo-
some encoding are crucial to the performance. Therefore, one of most important
thing is to decide the determination of an appropriate objective function and its
encoding. In our case, each particle-pattern and gene has an own fitness value
which is comparable and compares it with other fitness value in order to share
information of global solution. The fitness function follows a hierarchical app-
roach in which the main objective is to maximize the SGC in WMN. Thus, the
fitness function of this scenario is defined as

Fitness = 0.7 × SGC(xij , y ij ) + 0.3 × NCMC(xij , y ij ).

Migration Function
Our implemented simulation system uses Migration function as shown in Fig. 1.
The Migration function swaps solutions between PSO part and DGA part.
Particle-Pattern and Gene Coding
In ordert to swap solutions, we design particle-patterns and gene coding carefully.
A particle is a mesh router. Each particle has position in the considered area
and velocities. A fitness value of a particle-pattern is computed by combination
of mesh routers and mesh clients positions. In other words, each particle-pattern
is a solution as shown is Fig. 2.
A gene describes a WMN. Each individual has its own combination of mesh
nodes. In other words, each individual has a fitness value. Therefore, the combi-
nation of mesh nodes is a solution.
Client Distributions
Our proposed system can generate many client distributions. In this paper, we
consider Weibull and Chi-square distributions as shown in Fig. 3.

(a) Weibull (b) Chi-square

Fig. 3. Clients distributions.

 Performance Analysis of WMNs by WMN-PSODGA Simulation System 371

Fig. 4. System structure for web interface.

Fig. 5. WMN-PSODGA Web GUI tool.

3.2 WMN-PSODGA Web GUI Tool

The Web application follows a standard Client-Server architecture and is imple-

mented using LAMP (Linux + Apache + MySQL + PHP) technology (see
Fig. 4). We show the WMN-PSODGA Web GUI tool in Fig. 5. Remote users
(clients) submit their requests by completing first the parameter setting. The
parameter values to be provided by the user are classified into three groups, as
follows.

• Parameters related to the problem instance: These include parameter values

that determine a problem instance to be solved and consist of number of
router nodes, number of mesh client nodes, client mesh distribution, radio
coverage interval and size of the deployment area.
• Parameters of the resolution method: Each method has its own parameters.
• Execution parameters: These parameters are used for stopping condition of
the resolution methods and include number of iterations and number of inde-
pendent runs. The former is provided as a total number of iterations and
372 A. Barolli et al.

depending on the method is also divided per phase (e.g., number of iterations
in a exploration). The later is used to run the same configuration for the same
problem instance and parameter configuration a certain number of times.

Table 1. WMN-PSODGA parameters.

Parameters Values
Clients distribution Weibull, Chi-square
Area size 32.0 × 32.0
Number of mesh routers 16
Number of mesh clients 48
Number of migrations 200
Evolution steps 9
Number of GA islands 16
Radius of a mesh router 2.0–3.5
Replacement method LDIWM
Selection method Roulette wheel method
Crossover method SPX
Mutation method Boundary mutation
Crossover rate 0.8
Mutation rate 0.2

4 Simulation Results

In this section, we show simulation results using WMN-PSODGA system. In this

work, we analyse the performance of WMNs considering Weibull and Chi-square
distributions of mesh clients. The number of mesh routers is considered 16 and
the number of mesh clients 48. We conducted simulations 100 times, in order to
avoid the effect of randomness and create a general view of results. We show the
parameter setting for WMN-PSODGA in Table 1.
We show simulation results in Figs. 6 and 7. We see that for both SGC and
NCMC, the performance of Chi-square distribution is better than Weibull dis-
tribution. Thus, we conclude that the performance is better when the clients
distribution is Chi-square distribution compared with the Weibull distribution.
 Performance Analysis of WMNs by WMN-PSODGA Simulation System 373

Distribution of Clients=Weibull distribution Best Solutions Distribution of Clients=Weibull distribution Best Solutions

Number of Covered Mesh Clients[%]

Average of Best Solutions Average of Best Solutions
Replacement Method=LDIWM Replacement Method=LDIWM
Number of GA Islands=16 Number of GA Islands=16
Giant Component[%] Evolution Steps=9
Population Size=32
Evolution Steps=9
Population Size=32
100 100

75 75

50 50

25 25

0 50 100 150 200 0 50 100 150 200

Number of Phases Number of Phases
(a) Size of Giant Component. (b) Number of Covered Mesh Clients.

Fig. 6. Simulation results of WMN-PSODGA for Weibull distribution of mesh clients.

Distribution of Clients=Chi-square distribution Best Solutions Distribution of Clients=Chi-square distribution Best Solutions

Number of Covered Mesh Clients[%]

Average of Best Solutions Average of Best Solutions
Replacement Method=LDIWM Replacement Method=LDIWM
Number of GA Islands=16 Number of GA Islands=16
Giant Component[%]

Evolution Steps=9 Evolution Steps=9

Population Size=32 Population Size=32
100 100

75 75

50 50

25 25

0 50 100 150 200 0 50 100 150 200

Number of Phases Number of Phases
(a) Size of Giant Component. (b) Number of Covered Mesh Clients.

Fig. 7. Simulation results of WMN-PSODGA for Chi-square distribution of mesh

clients.

5 Conclusions

In this work, we evaluated the performance of WMNs using a hybrid simulation

system based on PSO and DGA (called WMN-PSODGA). Simulation results
show that the performance is better for Chi-square distribution compared with
the case of Weibull distribution.
In our future work, we would like to evaluate the performance of the proposed
system for different parameters and patterns.

References
1. Akyildiz, I.F., Wang, X., Wang, W.: Wireless mesh networks: a survey. Comput.
Netw. 47(4), 445–487 (2005)
2. Barolli, A., Sakamoto, S., Barolli, L., Takizawa, M.: Performance analysis of simula-
tion system based on particle swarm optimization and distributed genetic algorithm
for WMNs considering different distributions of mesh clients. In: International Con-
ference on Innovative Mobile and Internet Services in Ubiquitous Computing, pp.
32–45. Springer (2018)
374 A. Barolli et al.

3. Barolli, A., Sakamoto, S., Ozera, K., Barolli, L., Kulla, E., Takizawa, M.: Design
and implementation of a hybrid intelligent system based on particle swarm opti-
mization and distributed genetic algorithm. In: International Conference on Emerg-
ing Internetworking, Data & Web Technologies, pp 79–93. Springer (2018)
4. Franklin, A.A., Murthy, C.S.R.: Node placement algorithm for deployment of two-
tier wireless mesh networks. In: Proceedings of Global Telecommunications Con-
ference, pp. 4823–4827 (2007)
5. Girgis, M.R., Mahmoud, T.M., Abdullatif, B.A., Rabie, A.M.: Solving the wireless
mesh network design problem using genetic algorithm and simulated annealing
optimization methods. Int. J. Comput. Appl. 96(11), 1–10 (2014)
6. Goto, K., Sasaki, Y., Hara, T., Nishio, S.: Data gathering using mobile agents for
reducing traffic in dense mobile wireless sensor networks. Mobile Inf. Syst. 9(4),
295–314 (2013)
7. Inaba, T., Elmazi, D., Sakamoto, S., Oda, T., Ikeda, M., Barolli, L.: A secure-aware
call admission control scheme for wireless cellular networks using fuzzy logic and
its performance evaluation. J. Mobile Multimed. 11(3&4), 213–222 (2015)
8. Inaba, T., Obukata, R., Sakamoto, S., Oda, T., Ikeda, M., Barolli, L.: Performance
evaluation of a QoS-aware fuzzy-based CAC for LAN access. Int. J. Space-Based
Situated Comput. 6(4), 228–238 (2016)
9. Inaba, T., Sakamoto, S., Oda, T., Ikeda, M., Barolli, L.: A testbed for admission
control in WLAN: a fuzzy approach and its performance evaluation. In: Interna-
tional Conference on Broadband and Wireless Computing, Communication and
Applications, pp. 559–571. Springer (2016)
10. Lim, A., Rodrigues, B., Wang, F., Xu, Z.: k-center problems with minimum cov-
erage. In: Computing and Combinatorics, pp. 349–359 (2004)
11. Maolin, T., et al.: Gateways placement in backbone wireless mesh networks. Int.
J. Commun. Netw. Syst. Sci. 2(1), 44 (2009)
12. Matsuo, K., Sakamoto, S., Oda, T., Barolli, A., Ikeda, M., Barolli, L.: Performance
analysis of WMNs by WMN-GA simulation system for two WMN architectures
and different TCP congestion-avoidance algorithms and client distributions. Int.
J. Commun. Netw. Distrib. Syst. 20(3), 335–351 (2018)
13. Muthaiah, S.N., Rosenberg, C.P.: Single gateway placement in wireless mesh net-
works. In: Proceedings of 8th International IEEE Symposium on Computer Net-
works, pp. 4754–4759 (2008)
14. Naka, S., Genji, T., Yura, T., Fukuyama, Y.: A hybrid particle swarm optimization
for distribution state estimation. IEEE Trans. Power Syst. 18(1), 60–68 (2003)
15. Sakamoto, S., Kulla, E., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: A comparison
study of simulated annealing and genetic algorithm for node placement problem
in wireless mesh networks. J. Mobile Multimed. 9(1–2), 101–110 (2013)
16. Sakamoto, S., Kulla, E., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: A compar-
ison study of hill climbing, simulated annealing and genetic algorithm for node
placement problem in WMNs. J. High Speed Netw. 20(1), 55–66 (2014)
17. Sakamoto, S., Kulla, E., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: A simulation
system for WMN based on SA: performance evaluation for different instances and
starting temperature values. Int. J. Space-Based Situated Comput. 4(3–4), 209–216
(2014)
18. Sakamoto, S., Kulla, E., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: Performance
evaluation considering iterations per phase and SA temperature in WMN-SA sys-
tem. Mobile Inf. Syst. 10(3), 321–330 (2014)
 Performance Analysis of WMNs by WMN-PSODGA Simulation System 375

19. Sakamoto, S., Lala, A., Oda, T., Kolici, V., Barolli, L., Xhafa, F.: Application
of WMN-SA simulation system for node placement in wireless mesh networks: a
case study for a realistic scenario. Int. J. Mobile Comput. Multimed. Commun.
(IJMCMC) 6(2), 13–21 (2014)
20. Sakamoto, S., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: An integrated simulation
system considering WMN-PSO simulation system and network simulator 3. In:
International Conference on Broadband and Wireless Computing, Communication
and Applications, pp. 187–198. Springer (2016)
21. Sakamoto, S., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: Implementation and evalu-
ation of a simulation system based on particle swarm optimisation for node place-
ment problem in wireless mesh networks. Int. J. Commun. Netw. Distrib. Syst.
17(1), 1–13 (2016)
22. Sakamoto, S., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: Implementation of a
new replacement method in WMN-PSO simulation system and its performance
evaluation. In: The 30th IEEE International Conference on Advanced Information
Networking and Applications (AINA-2016), pp. 206–211 (2016). https://doi.org/
10.1109/AINA.2016.42
23. Sakamoto, S., Obukata, R., Oda, T., Barolli, L., Ikeda, M., Barolli, A.: Performance
analysis of two wireless mesh network architectures by WMN-SA and WMN-TS
simulation systems. J. High Speed Netw. 23(4), 311–322 (2017)
24. Sakamoto, S., Ozera, K., Barolli, A., Ikeda, M., Barolli, L., Takizawa, M.: Imple-
mentation of an intelligent hybrid simulation systems for WMNs based on particle
swarm optimization and simulated annealing: performance evaluation for different
replacement methods. Soft Comput., pp 1–7 (2017)
25. Sakamoto, S., Ozera, K., Ikeda, M., Barolli, L.: Implementation of intelligent hybrid
systems for node placement problem in WMNs considering particle swarm opti-
mization, hill climbing and simulated annealing. Mobile Netw. Appl. 23(1), 27–33
(2018). https://doi.org/10.1007/s11036-017-0897-7
26. Sakamoto, S., Ozera, K., Oda, T., Ikeda, M., Barolli, L.: Performance evaluation
of intelligent hybrid systems for node placement in wireless mesh networks: a com-
parison study of WMN-PSOHC and WMN-PSOSA. In: International Conference
on Innovative Mobile and Internet Services in Ubiquitous Computing, pp. 16–26.
Springer (2017)
27. Sakamoto, S., Ozera, K., Oda, T., Ikeda, M., Barolli, L.: Performance evaluation of
WMN-PSOHC and WMN-PSO simulation systems for node placement in wireless
mesh networks: a comparison study. In: International Conference on Emerging
Internetworking, Data & Web Technologies, pp. 64–74. Springer (2017)
28. Sakamoto, S., Ozera, K., Barolli, A., Barolli, L., Kolici, V., Takizawa, M.: Perfor-
mance evaluation of WMN-PSOSA considering four different replacement methods.
In: International Conference on Emerging Internetworking, Data & Web Technolo-
gies, pp. 51–64. Springer (2018)
29. Vanhatupa, T., Hannikainen, M., Hamalainen, T.: Genetic algorithm to optimize
node placement and configuration for WLAN planning. In: Proceedings of the 4th
IEEE International Symposium on Wireless Communication Systems, pp. 612–616
(2007)
30. Wang, J., Xie, B., Cai, K., Agrawal, D.P.: Efficient mesh router placement in wire-
less mesh networks. In: Proceedings of IEEE International Conference on Mobile
Adhoc and Sensor Systems (MASS-2007), pp. 1–9 (2007)
 An NSH-Enabled Architecture
for Virtualized Network Function
Platforms

Vinı́cius F. Garcia1(B) , Leonardo C. Marcuzzo1 , Giovanni V. Souza2 ,

Lucas Bondan3 , Jéferson C. Nobre4 , Alberto E. Schaeffer-Filho3 ,
Carlos R. P. dos Santos1 , Lisandro Z. Granville3 , and Elias P. Duarte Jr.2
1
Federal University of Santa Maria, Santa Maria, Brazil
{vfulber,lmarcuzzo,csantos}@inf.ufsm.br
2
Federal University of Paraná, Curitiba, Brazil
{gvsouza,elias}@inf.ufpr.br
3
Federal University of Rio Grande do Sul, Porto Alegre, Brazil
{lbondan,alberto,granville}@inf.ufrgs.br
4
University of Vale do Rio dos Sinos, Porto Alegre, Brazil
[email protected]

Abstract. The proper execution of Virtualized Network Functions

(VNFs) depends on the employment of platforms specifically created
to fulfill multiple Network Function Virtualization (NFV) requirements
(e.g., performance, integration, and management). However, existing
VNF platforms implement different architectures, thus resulting in pro-
prietary or limited solutions that do not always support important NFV
specifications, such as Network Service Header (NSH). In this work, we
introduce a comprehensive architecture for VNF platforms that supports
the NFV requirements defined by the European Telecommunications
Standards Institute (ETSI), while also enabling the execution of NSH.
We implemented a VNF platform prototype, on which we conducted a
case study, and report a series of performance evaluation experiments.
Results demonstrate the intrinsic advantages of supporting NSH and
show the flexibility of our architecture in distinct NFV scenarios.

1 Introduction
Computer networks typically rely on dedicated equipment (i.e., middleboxes) to
perform common network functions (e.g., NAT translation, intrusion detection
and load balancing). However, despite the usual benefits of such middleboxes,
including performance and reliability, they offer limited flexibility regarding the
design and deployment of advanced network services. Network Functions Virtu-
alization (NFV), in turn, is a networking paradigm that leverages virtualization
technologies to decouple network functions from the physical equipment and run
such functions as software that executes on commodity hardware [5].
The adoption of NFV presents several benefits, including higher cus-
tomization and the reduction of Capital and Operational Expenditures
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 376–387, 2020.
https://doi.org/10.1007/978-3-030-15032-7_32
 An NSH-Enabled Architecture for Virtualized Network Function Platforms 377

(CAPEX/OPEX). Multiple efforts are being conducted to foster the adoption

of NFV technologies with the ultimate goal of encouraging the development of
solid foundations that support advanced NFV solutions.
Essentially, a VNF is divided into two main parts: the Network Function
(NF) itself and the VNF platform. NF corresponds to the software implemen-
tation responsible for packet processing, while the VNF platform is the envi-
ronment that supports the execution of NFs. VNF platforms are designed tak-
ing into account the need to enable the creation of multiple network functions
while consuming few computing resources. However, existing VNF platforms
(e.g., ClickOS [12] and OpenNetVM [15]) are not created using standardized
architectures, thus resulting in solutions that are either proprietary or present
serious limitations, such as the lack of support for advanced NFV specifications,
in particular Network Service Header (NSH) – a packet header that enables the
creation of dynamic service planes [13].
In this paper, we introduce a comprehensive architecture for VNF platforms
that strictly adheres to ETSI requirements and provides support for NSH. Our
key contributions are: (i) the development of a VNF platform prototype based
on ETSI requirements; and (ii) the identification of critical features provided by
NSH that enables the development of advanced network services.
The remaining of this paper is organized as follows. Section 2 presents the
background on NFV along with its main components, basic requirements, and a
review of the literature. In Sect. 3, we propose a architecture for developing VNFs
with support for NSH, and instantiate this architecture by describing a running
prototype platform implemented to support virtualized network functions. In
Sect. 4, we evaluate the performance of our prototype. Finally, in Sect. 5, we
conclude this paper with final remarks and an outline of future work.

2 Background and Related Work

In this section, we present the background on Network Functions Virtualization
(NFV) and Virtualized Network Function (VNF). We also discuss the charac-
teristics and limitations of existing VNF platforms.

2.1 Network Function Virtualization in a Nutshell

NFV aims to implement Network Functions (NFs) in software, so they can run
on commodity hardware by employing common virtualization technologies [5].
Two of the most relevant standardization bodies are expending considerable
energy in defining models, methodologies, and concepts in this area: the Euro-
pean Telecommunications Standards Institute (ETSI) and the Internet Engi-
neering Task Force (IETF).
The NFV architectural framework [7] defined by ETSI is composed of three
functional blocks: NFV Infrastructure (NFVI), NFV Management and Orches-
tration (NFV MANO), and Virtualized Network Functions (VNF). NFVI is the
collection of physical resources (e.g., processing, storage, and network) needed
378 V. F. Garcia et al.

to execute VNFs. NFV MANO in turn encompasses: the Virtualized Infrastruc-

ture Manager (VIM), which controls the physical/software infrastructure; the
VNF Manager (VNFM), which is responsible for VNF lifecycle operations (e.g.,
instantiation, termination, scaling, and migrations); and the NFV Orchestra-
tor (NFVO), which enables the management of network services. Finally, the
VNF functional block itself represents the network functions that run on VNF
platforms.
The IETF has been working on standardizing the Service Function Chain
(SFC) [8], which consists of multiple VNFs working together in a composition
that provides a network service. In addition to VNFs, a SFC also includes bound-
ary nodes (i.e., incoming and outgoing points of traffic) and steering specifica-
tions. One way to implement those SFCs is by using the architecture defined
in the IETF’s Request for Comments (RFC) number 7665 [8]. In that RFC, a
Network Service Header (NSH) [13] is employed to enable packets to traverse
a specific path of VNFs. Each VNF, in turn, can either be NSH-aware or not.
NSH-aware VNFs make a request to a proxy element to remove the NSH before
processing, after which the proxy element is again invoked to update/reinsert
the NSH element. Alternatively, VNFs can process NSH themselves, without
relying on external elements.
An NSH encapsulates L3 packets as they are processed by the SFC, and car-
ries information about the Service Function Path (SFP), the current packet loca-
tion in the SFP, and meta-data provided by the VNFs during packet processing.
NSH is subdivided into three meta-headers: Base Header, Service Path Header,
and Context Header. The Base Header provides general information about the
protocol itself and the next meta-headers data. This meta-header (4 bytes), in
turn, carries 5 fields (i.e., version, O bit, TTL, length, and meta-data type), in
addition to reserved space for future protocol use. The Service Path Header is
also formed by 4 bytes and has 2 fields: Service Path Identifier (identifies the
SFC’s SFP) and the Service Index (provides the location of the packet in the
SFP). Finally, the Context Header can be fixed (16 bytes) or variable to carry
meta-data information as the SFC is processed.
VNF platforms are specially designed to host VNFs and their associated
components. The ETSI lists the basic requirements for developing such platforms
(e.g., hardware independence, elasticity, reliability) [14]. The ETSI also specifies
that a VNF platform must host VNFs independently of the underlying hardware.
Furthermore, VNF platforms must provide the flexibility expected from the NFV
paradigm to support the operations of deployment, scaling, and migration.
Several aspects of VNF platforms, however, are still unexplored in the lit-
erature. For example, the lack of standardization on existing VNFs platforms,
mostly created without a well defined architecture and missing features such as
NSH and SFC support. Existing platforms, detailed next, usually support only a
single programming language, run on very specific hypervisors, do not describe
a formal architecture, and do not fulfill all the above requirements defined by
the ETSI.
 An NSH-Enabled Architecture for Virtualized Network Function Platforms 379

2.2 VNF Platforms

Many research efforts concentrate on creating VNF platforms that are capable of
executing NFs. However, these VNF platforms do not provide support for more
recent NFV specifications, such as NSH. Still, two of the most important VNF
platforms today are ClickOS and OpenNetVM, described next.
ClickOS [12] is an optimized platform for running NFs based on Mini-OS,
netmap, and Click Modular Router. ClickOS uses paravirtualization techniques
along with several modifications in both Xen and VALE to support fast packet
I/O, being able to saturate 10GbE links with a single processing core. Alas,
the ClickOS architecture is monolithic and inflexible and supports only a single
packet acceleration tool for sending and receiving network traffic to an indivisible
NF.
OpenNetVM [15] is a simplified platform that uses containers to execute
VNFs on commodity servers along with the packet acceleration framework
DPDK [9]. OpenNetVM meets the ETSI scalability requirements because of
its lower overhead due to the use of containers, which are a lightweight solution
in comparison with virtual machines. The OpenNetVM architecture consists of a
packet acceleration tool interconnecting VNFs with Virtualized NICs (VNICs).
NFs are created as a single component within a proprietary framework and
are deployed on a container core. OpenNetVM also provides an internal router
implemented using shared memory, which steers network traffic between multiple
VNFs.
Both ClickOS and OpenNetVM unfortunately lack support for important
VNF elements (e.g., VNFCs and NSH). The ClickOS architecture is straight-
forward and does not have any native management method, providing only a
minimalist environment to execute simple NFs. Despite OpenNetVM’s internal
traffic router, the platform is restricted to a single packet acceleration tool and
a single packet processing framework, both deployed on a container core and
managed by an external native agent. Furthermore, these platforms do not meet
all of the requirements specified by the ETSI. ClickOS, for example, can only
be executed on the Xen hypervisor and every control operation is performed
locally through the XenStore. Also, OpenNetVM presents limitations related
to NF instantiation (due to hardware device sharing), portability (migrations
are only possible between compatible infrastructures), and security issues with
containers (single kernel sharing) [11].

3 VNF Platform Architecture and Prototype

In this section, we introduce an architecture for VNF platforms which supports

SFC chaining using NSH. The proposed architecture is designed to be flexible
and technology agnostic. Ultimately, we expect this architecture to serve as a
template for designing new systems and re-engineering existing ones.
380 V. F. Garcia et al.

3.1 Architecture Overview

Currently, there is no de facto standard for the design and development of VNF
platforms, from neither industry nor academia. VNF platforms, however, must
be developed to meet multiple strict requirements (e.g., portability, performance,
integration, management, and scalability), in order to fulfill the needs of modern
networks. Furthermore, the NFV area is evolving, with new technologies being
created continuously. Therefore, it is essential to design flexible solutions that
support new NFV Enablers (i.e., existing frameworks and technologies that con-
tribute to the development and implementation of NFV) from an ever increasing
number of players in the NFV market. VNF platforms must also be created with
integration in mind. There are several systems (OSS/BSS, Hypervisors) and ele-
ments (NFVI, VNFM, EMS) that must work together with multiple VNFs in
order to adequately provide virtualized network services [7].

Fig. 1. VNF platform architecture

In this context, we propose a generic and flexible architecture for VNF plat-
forms, (Fig. 1). The proposed platform consists of multiple modules, deployed
on a host operating system (called here VNF Core): an module responsible for
management access of Virtual Network Interfaces (vNIC), being the main point
of input/output of packets from the architecture, a module consisting of a frame-
work for development and execution of network functions, a management agent
responsible for configuration of the host operating system and the other mod-
ules and, finally, a NSH Processor which provides an abstraction for the network
functions regarding the existence of NSH packets.
Network Service Header (NSH) is a new service plane protocol, specified
by IETF, inserted onto packets/frames to provide service function paths [13].
 An NSH-Enabled Architecture for Virtualized Network Function Platforms 381

Despite its advantages, NSH is not always employed to steer traffic across mul-
tiple VNFs. In order to support the cases where NSH is employed or not, we
introduced the NSH Processor, which is responsible for manipulating the NSH
fields that may be modified when a packet is traversing a network path (i.e.,
the Service Index - SI; and Context Header - CH). The proposed NSHP also
provides the following operations: NSH removal, NSH reinsertion, CH retrieval,
and CH update.
Packets are then received by the development frameworks used to implement
network functions. Basically, these frameworks include applications (e.g., Click
Modular Router [10] and Vector Packet Processing [3]), programming languages
(e.g., C, C++, Python), libraries (e.g., Scapy, libtins, libnet), or even single
routines that support the construction and handling of network packets.
All the described modules are controlled by an management agent, which is
responsible for monitoring and controling the execution of VNFs. Once a VNF
is executing, the retrieve operations can be used to gather information about
the VNF instance (e.g., VNF ID, network interfaces), measuring performance
indicators from the VNF Core (e.g., CPU, memory, and network usage), and
providing information from the extended agents deployed in the VNF platform.

3.2 NFV Enablers

As a proof of concept of the proposed architecture previously presented, we

implemented a prototype that consists of a VNF platform that employs modern
and well-accepted NFV enablers. In this context a NFV enabler is any technology
used as a basis for the development of NFV ecosystems [5], such as hypervisors,
packet acceleration frameworks, virtual routers, and operating systems. Specifi-
cally, we employed the following technologies in our solution:

• Click Modular Router – a well-known packet processing framework that

provides an extensive list of network elements, native support for packet accel-
eration frameworks, and built-in control methods. The Click framework has
been extensively investigated in academia and employed in several efforts
to develop VNF platforms, such as ClickOS [12], Click-on-OSv [4], and the
platform proposed by Bu et al. [2];
• Intel Data Plane Development Kit (DPDK) – a packet acceleration
framework that provides high throughput with reduced resource consump-
tion (by using PCI passthrough and zero-copy) and supports multiple net-
work interface cards (including paravirtualized ones, such as virtio and
netfront);
• RESTful Web Services – an architectural style for the provisioning of Web
Services (WSs) with simplified access to resources. RESTful web services are
lighter than traditional SOAP-based WSs and are used here as the basis for
the management agent to export performance statistics regarding the VNF
platform, and to receive management requests from an external VNFM or
EMS.
382 V. F. Garcia et al.

Furthermore, we have chosen Debian as the operating system for the VNF
Core. Although this operating system is not explicitly designed to support the
NFV requirements (e.g., performance), this decision enabled us to focus on the
development of the internal modules without concerns about software compati-
bility, thus enabling the analysis of the architecture from the functional point-of-
view. However, the development of NFV platforms for production environments
should benefit from more recent solutions such as OSv, CoreOS, MiniOS, and
Alpine.

3.3 Platform Prototype

For handling the vNICs, we initially chose DPDK. However, the platform also
supports L2 Sockets to provide increased flexibility. These two options are
available and can be selected according to the specific needs of the operator1 .
Although we have employed these two solutions, other solutions (e.g., PF RING,
and netmap) could be used without significant implementation efforts.
The NSH Processor operates as a proxy during the platform execution and
can be enabled by the network operator. When enabled, packets first pass
through the NSHP in order to remove NSH before steering the packets to the
network function (i.e., CMR or Python-based NFs/VNFCs), and for NSH rein-
sertion before forwarding processed packets back to the vNIC (i.e., DPDK or
L2 Sockets). The NFs, when necessary, can access the NSH Context Header to
retrieve or replace content using specific libraries we developed (both in Python
and CMR)2 . Notice that, during the NSHP reinsertion operation, the Service
Index field is updated.
The Management Agent was developed as a RESTful WS and is able to
control (through system calls and CMR’s control socket) all the internal compo-
nents of the platform. For monitoring, Glances3 was used to recover system-wide
statistics, while a custom REST API was developed to meet the management
requirements as specified in [1], such as providing information regarding the VNF
(e.g., function description, system state, logs) and supporting the reconfigura-
tion of running functions. These operations can be accessed either directly by
network operators (through the integrated EMS) or by external EMS and VNF
Managers (by using REST calls to the management interface).
To initiate the platform, the network operator must first provide (through
the Management Agent) a VNF Package that contains the network function
implementation, its descriptor, and settings to initialize the internal modules
properly. These settings are used, for example, to enable/disable the NSH Pro-
cessor. After the initial platform configuration, the NF lifecycle operations (e.g.,
start, stop, status, and monitoring) become available to the network operator.

1
The polling method of DPDK is inefficient regarding to CPU resources, while L2
Sockets are not able to achieve high throughput.
2
By the time of writing, we opted to support only Context Headers of fixed-length
[13].
3
https://nicolargo.github.io/glances/.
 An NSH-Enabled Architecture for Virtualized Network Function Platforms 383

The platform prototype supports three execution scenarios: NSH packets with
NSH unaware NF, NSH packets with NSH aware NF, and packets without NSH.

4 Evaluation

To evaluate our proposed architecture, a case study of the prototype presented in

Sect. 3.2 are presented. An Intel Core [email protected] Ghz server with 8 GB RAM
DDR4 running Debian 8 was used. The prototype platform was configured to
use L2 Sockets for input/output and NFs developed using both Python3 and/or
CMR frameworks. All the experiments were repeated 30 times, considering a
confidence level of 95%.

4.1 NSH for VNFs Intercommunication and SFC Steering

We used NSH to improve an existing NFV-based solution to mitigate DDoS
attacks called DeMONS [6]. In DeMONS, six separate VNFs were employed
to detect malicious traffic and steer it through separate channels with different
bandwidth capacities. These VNFs are Manager, Priority Classifier, Firewall,
Allocator, Traffic Policing, and Router. The Manager is responsible for orches-
trating the environment execution, for example, by monitoring the network load
to scale running VNFs. The Priority Classifier, the Firewall, and the Allocator
are responsible for identifying and classifying benign traffic, and blocking mali-
cious traffic. Finally, both the Traffic Policing and the Router are responsible for
applying user-defined policies (e.g., partial dropping and traffic shaping) on the
suspicious flows.
In the original DeMONS, the first three VNFs (i.e., Priority Classifier, Fire-
wall, and Allocator) are connected through a static path and share the traffic
reputation through the Manager (acting as a central point of communication).
The Priority Classifier uses Intrusion Detection System (IDS) techniques to gen-
erate the traffic reputation, and classifies the incoming flows with values ranging
from 0 to 1. Reputation 0 means a malicious flow, reputation 1 indicates a benign
flow, and values between those limits indicate unclassified traffic. After the clas-
sification, the traffic is forwarded to the Firewall, which queries the Manager
to verify the traffic reputation, blocks 0 marked flows, and forwards the rest to
the Allocator. The Allocator, in turn, steers the traffic to a high priority tunnel
(that guarantees QoS for high reputation flows) or to a low priority tunnel (that
can be overloaded with low reputation flows). Figure 2-A presents the original
DeMONS implementation design.
DeMONS was deployed using the IETF’s SFC architecture (i.e., Classifier
and Service Function Forwarder) [8] and the VNF Platform prototype developed
in this work. NSH was used to enable in-band control for sharing flow reputa-
tions and to orchestrate the traffic steering across all the employed NSH aware
VNFs (i.e., the SFC). The reputation table (originally at the Manager) is now
maintained by the Priority Classifier and shared between the VNFs through the
NSH Context Header. In case of benign traffic, the Service Index is decremented
384 V. F. Garcia et al.

Fig. 2. DDoS mitigation NFV solution (DeMONS)

by two in order to forward the packets directly to the Allocator, thus skipping
the Firewall. Finally, the Allocator gets the reputation value directly from the
flow packet by using the NSH’s Context Header. Figure 2-B presents the adapted
DeMONS implementation design.
The use of NSH led to performance improvements in DeMONS due to
redesign and the embedded mechanism to exchange control data. The first
experiment was executed to evaluate the execution time overhead introduced
by retrieving and processing the reputations in both original and modified
DeMONS. Two versions of the Packet Filter VNFC were developed in order
to operate in both scenarios (NSH and Non-NSH), and were instrumented to
measure the elapsed processing time for each packet. Iperf was used to generate
the network traffic (UDP packets of 1470 Bytes), with the results presented in
Fig. 3.
 An NSH-Enabled Architecture for Virtualized Network Function Platforms 385

100
Non-NSH DeMONS
NSH Demons

10
Time (ms)

0.1
100 200 500 700 1000
Packets

Fig. 3. Reputation retrieval processing time overhead

In the Non-NSH DeMONS, the reputation retrieval process consists in con-

tacting the Manager (through a UDP Socket) to discover the reputations. This
operation is performed for all the network packets traversing the VNFC. When
NSH is employed, on the other hand, the reputation value is included in the NSH
Context Header of each packet by the Priority Classifier. In this case, the Packet
Filter just needs to open this header to retrieve the information. As suspected,
the same operation (reputation retrieval) leads to significant differences in terms
of processing time (in favor to NSH-based), which may affect other performance
indicators (e.g., throughput and packet loss). An important observation is that
no major differences occurred for packets with different sizes.
The use of a Service Function Forwarder in the NSH DeMONS scenario also
demonstrated interesting opportunities for improving the traffic steering overall.
This SFC element (e.g., an OpenFlow Switch or a P4-enabled device) steers the
network traffic according to the Service Index value present in the NSH Service
Path Header. In this way, it is possible to manipulate the VNF execution order
by updating the NSH Service Index according to decisions taken during VNF
processing. For our case study, the Firewall VNF only processes the malicious
traffic in order to collect statistics (e.g., number of discarded packets) and then
discards the packets. In addition to not processing benign flows – when malicious
traffic is nonexistent (i.e., no attack occurring) – it is possible to temporarily
disable the Firewall VNF, thus saving computational resources.
386 V. F. Garcia et al.

5 Conclusion and Future Work

Network Functions Virtualization (NFV) has been attracting great interest from
both industry and academia. However, despite all the advancements in the field,
there are still opportunities for research, development, and standardization. For
example, there are no widely accepted definitions for the internal architecture of
the platforms responsible for executing NFs, neither the processing of NSH inside
those VNFs. This leads to a scenario where several platforms (e.g., ClickOS and
OpenNetVM) have been created without integration concerns in mind, the result
is that none fulfills the complete set of NFV requirements.
This work proposed an architecture for VNF Platforms with NSH support.
We specified the basic modules for building a platform, as well as identified
existing NFV enablers that can be employed during such development. We also
presented a platform prototype that employs the proposed architecture to sup-
port the execution of disparate network functions. Finally, we conducted a per-
formance evalution to identify the advantages of employing NSH. The exper-
iment shows the advantages of using NSH when designing SFCs. First, the
NSH Context-Header enables the functions to communicate and change infor-
mation in-band. Second, the Service Index field allows the creation of dynamic
SFCs, without the need to a priori set the path of VNFs, thus enabling traffic
steering to be executed by using common solutions (e.g., Open vSwitch, P4-
enabled equipment). Future work includes the investigation on how the current
VNF/SFC descriptors (e.g., TOSCA) have to be adapted to support NSH, and
improve the prototype by supporting novel packet processing frameworks (e.g.,
VPP) and other virtual network technologies (e.g., netmap).

References
1. Bondan, L., dos Santos, C.R.P., Granville, L.Z.: Management requirements for
ClickOS-based network function virtualization. In: 10th International Conference
on Network and Service Management (CNSM) and Workshop, pp. 447–450 (2014).
https://doi.org/10.1109/CNSM.2014.7014210
2. Bu, C., Wang, X., Huang, M., Li, K.: SDNFV-based dynamic network function
deployment: model and mechanism. IEEE Commun. Lett. 22(1), 93–96 (2018)
3. Cisco: Vector packet processing (2018). https://blogs.cisco.com/tag/vector-packet-
processing. Accessed 13 Sept 2018
4. da Cruz Marcuzzo, L., Garcia, V.F., Cunha, V., Corujo, D., Barraca, J.P., Aguiar,
R.L., Schaeffer-Filho, A.E., Granville, L.Z., dos Santos, C.R.: Click-on-OSv: a plat-
form for running click-based middleboxes. In: 2017 IFIP/IEEE Symposium on
Integrated Network and Service Management (IM), pp. 885–886. IEEE (2017)
5. ETSI: Network functions virtualisation - an introduction, benefits, enablers, chal-
lenges & call for action (2012). https://portal.etsi.org/NFV/NFV White Paper.
pdf. Accessed 13 Sept 2018
6. Garcia, V.F., de Freitas Gaiardo, G., da Cruz Marcuzzo, L., Nunes, R.C., dos
Santos, C.R.P.: DeMONS: A DDoS mitigation NFV solution. In: 2018 IEEE 32nd
International Conference on Advanced Information Networking and Applications
(AINA), pp. 769–776. IEEE (2018)
 An NSH-Enabled Architecture for Virtualized Network Function Platforms 387

7. GS E: Network functions virtualisation (NFV); architectural framework

(2014). https://www.etsi.org/deliver/etsi gs/nfv/001 099/002/01.01.01 60/gs
nfv002v010101p.pdf. Accessed 13 Sept 2018
8. Halpern, J.M., Pignataro, C.: Service Function Chaining (SFC) Architecture.
RFC 7665 (2015). https://doi.org/10.17487/RFC7665. https://rfc-editor.org/rfc/
rfc7665.txt
9. Intel: Data plane development kit (2014). http://dpdk.org. Accessed 13 Sept 2018
10. Kohler, E., Morris, R., Chen, B., Jannotti, J., Kaashoek, M.F.: The click modu-
lar router. ACM Trans. Comput. Syst. 18(3), 263–297 (2000). https://doi.org/10.
1145/354871.354874
11. Manco, F., Lupu, C., Schmidt, F., Mendes, J., Kuenzer, S., Sati, S., Yasukata,
K., Raiciu, C., Huici, F.: My VM is lighter (and safer) than your container. In:
Proceedings of the 26th Symposium on Operating Systems Principles, SOSP 2017,
pp. 218–233. ACM (2017). https://doi.org/10.1145/3132747.3132763
12. Martins, J., Ahmed, M., Raiciu, C., Olteanu, V., Honda, M., Bifulco, R., Huici,
F.: ClickOS and the art of network function virtualization. In: Proceedings of
the 11th USENIX Conference on Networked Systems Design and Implementation,
NSDI 2014, CA, USA, pp. 459–473. USENIX Association, Berkeley (2014)
13. Quinn, P., Elzur, U., Pignataro, C.: Network Service Header (NSH). RFC 8300
(2018). https://doi.org/10.17487/RFC8300. https://rfc-editor.org/rfc/rfc8300.txt
14. SWA EG: Virtual network functions architecture (2014). https://www.etsi.org/
deliver/etsi gs/NFV-SWA/001 099/001/01.01.01 60/gs NFV-SWA001v010101p.
pdf. Accessed 13 Sept 2018
15. Zhang, W., Liu, G., Zhang, W., Shah, N., Lopreiato, P., Todeschi, G., Ramakrish-
nan, K., Wood, T.: OpenNetVM: a platform for high performance network service
chains. In: Proceedings of the 2016 Workshop on Hot Topics in Middleboxes and
Network Function Virtualization, HotMIddlebox 2016, NY, USA, pp. 26–31 (2016).
ACM, New York. https://doi.org/10.1145/2940147.2940155
 Personal Book Management Application
on Blockchain

Hiryu Kawaguchi, Yasunao Takano(B) , and Hiroshi Sakuta

Department of Integrated Information Technology,

College of Science and Engineering, Aoyama Gakuin University,
5-10-1 Fuchinobe, Chuo-ku, Sagamihara-shi, Kanagawa 252-5258, Japan
[email protected], {takano,sakuta}@it.aoyama.ac.jp

Abstract. Using a blockchain, an application that guarantees reliabil-

ity, can be realized by managing a distributed ledger of agreements. A
blockchain’s usefulness in applications requiring trust, such as virtual
currency, has been confirmed, and it is being used in actual services.
However, the time taken to add data to the distributed ledger and the
cost of this operation may be too high. As a result, other real applica-
tions for blockchains have not been found yet. In this research, we design
and implement an application for lending and borrowing books on a dis-
tributed ledger. The application is implemented as a distributed appli-
cation on Ethereum, which is a platform used for building blockchain-
based smart contracts. Users of this application can lend and borrow
books through a web application that functions as the user interface of
the application. The usage logs show that the time and fees required for
adding data to the distributed ledger are acceptable.

Keywords: Blockchain · Book management · Web application

1 Introduction
Blockchains [6] are attracting attention from technical and social points of view
because of their reliability, which can handle even very sensitive information such
as virtual currencies [4,6,7]. Such reliability arises from managing a distributed
ledger by forming agreements between every participating nodes with respect to
a data unit called a block. A blockchain has the following advantages:
• no centralized data management;
• immutability;
• Byzantine fault tolerance.
These properties would seem to be useful in many applications at a first glance.
However, it is difficult to design a new application that can take advantage
of the nature of the blockchain [4]. This is because information managed as a
distributed ledger has a cost, and we need to consider it as a real-time property.
Therefore, despite the high degree of attention given to this technology, popular
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 388–396, 2020.
https://doi.org/10.1007/978-3-030-15032-7_33
 Personal Book Management Application on Blockchain 389

applications in real society other than virtual currency are not appearing. From
the social point of view, it is necessary to consider whether there is a privacy
problem regarding the information published as a distributed ledger. In addition,
from the technical point of view, it is necessary to design the application carefully
to ensure the time required for adding data to the distributed ledger and the
cost are suitable for practical use.
Hence, we propose an application on a blockchain that manages books pos-
sessed by an individual. The application manages the lending and borrowing
of books on a distributed ledger. As one example of use, a user first registers
a book he/she owns on the distributed ledger through the web application. In
the web application, all users can see the registered books, and they can also
record in the distributed ledger that they have lent a book. In this study, the
implementation is a distributed application on the Ethereum platform, which
is a platform for building blockchain smart contracts. The implementation was
operated on a test network for a certain period of time, and the usage log for
commissions for the distributed ledger was examined.
The contribution of this work consists of the following two points.
• We designed and implemented an application on a blockchain for the real
book.
• Through experiments, the time and fees required for adding data to the dis-
tributed ledger were measured.
Note that this system assumes that users will hand over the real books
directly and that it will be used in small communities.

2 Applications on Blockchain
A blockchain is a kind of database, in which data is held in a continuously
increasing list of records named blocks. Each block contains a timestamp and a
link to the previous block. Theoretically it is not possible to retroactively change
the data in a blockchain once it has been recorded.
Because of this feature, blockchains can be used for applications such as
contracts, and attention has been focused on automation technology called smart
contracts. For example, the automation of a contract is like the operation of
a vending machine. Only when the two contract conditions “the user inserts
money” and “the user pushes the button for a specific beverage” are satisfied does
the contract consider the interaction “provide a specific drink” automatically.
This series of steps can be expressed as the program of smart contract.
Decentralized applications (Dapps) [5] are applications for automating smart
contracts based on a decentralized authority. Dapps run autonomously on a peer-
to-peer network, execute smart contracts, and record the results in a blockchain.
Distributed execution is performed by computers participating in a peer-to-peer
network for rewards. There are three main issues when implementing a Dapp:
the time taken to add data to the distributed ledger, fees, and privacy. For the
first item, it takes time for a smart contract to be authenticated on a Dapp,
390 H. Kawaguchi et al.

so it is not suitable for applications that require actual real-time processing.

In addition, time variations may occur, so service stability must be considered.
For the second item, because all transactions to the ledger are costly, this must
be considered in the design of the application. For the third item, because the
transactions a user at one address has performed are recorded on the ledger, even
though the ledger is anonymized, it is not possible to hide a series of historical
actions of one user.
Because of these issues, when implementing a Dapp, we must design it
carefully.

3 Proposed System
3.1 Overview

In this study, we propose a Dapp that manages the information of books pos-
sessed by several individuals and the lending and borrowing of these books on a
distributed ledger. For the management of books, the advantage of the blockchain
described in Sect. 1 is that it is not necessary to centrally manage the informa-
tion. As a result, the availability of this system is higher than server–client type
book management systems such as those currently used in libraries. Further-
more, although book lending information is not as sensitive as virtual currency
information, it is necessary to protect the location information of books from
tampering.
The problems of Dapps described in Sect. 2 can be analyzed as follows.

Real-time property. Because the use considered here is accompanied by the actual
handover of books, the real-time requirements are low.
Fees. The cost of book management, which has not been considered so far, will
be paid by individuals.
Privacy. Because the book lending history remains in the ledger, the personal
information of the books a user has read remains on the blockchain.

Regarding the time and fees required for data addition, the contents of the
contract at the time of implementation can be examined when the application
is designed. In addition, these factors can be measured during operation tests.
Finally, it will be necessary to conduct a user study after operating at a certain
scale to determine whether users can tolerate the level of privacy provided by
the application.

3.2 System Structure

Figure 1 shows the overall configuration of the proposed system. Through the
web user interface (WebUI), the user issues transaction instructions to the Dapp,
which manages the location information of books (ISBN, lenders, and borrow-
ers) and user information as a ledger on the blockchain. For the WebUI, the
blockchain can be regarded as a type of database. It is beneficial to save only
 Personal Book Management Application on Blockchain 391

the ISBN within the contract. When lending or borrowing books, because it
is necessary to hand over books as real things, it is necessary to implement a
notification function on the system to convey the requests of other users. In this
system, Slack1 is used because the application is assumed to be used in a small
community in which the books are directly handed over.

Fig. 1. System structure

3.3 Contract Design

For the contract of the transaction of books, we include following three entries:

• the book’s ISBN;

• borrower;
• lender.

Information of borrower is rewritten each time it is borrowed. Because the entry

of a contract represents the relocation of each real book, there can be acceptable
of multiple entries with the same ISBN.
Here, it is important when designing a Dapp on the blockchain to reduce the
cost by saving ISBN of the book in the contract. As mentioned in Sect. 2, because
all transactions to the ledger are costly, an item of data that can pull detailed
information from other services is not included in the contract. For example,
details of the books (author, summary, number of pages, book cover image data)
can be acquired using ISBN; therefore, these items of information can be omitted
without entering the block chain. Because the information in a contract ca be
suppressed, books can be said to be suitable objects for blockchains.
The address and SlackID of the user are stored in the contract and used for
notifications. Because Slack can manage accounts for each workspace, there is
no privacy concern even if the SlackID is published on the ledgers.

1
https://slack.com/.
392 H. Kawaguchi et al.

3.4 Dapps on Ethereum

The proposed Dapp was implemented on Ethereum [8], which is a platform
for building distributed applications and smart contracts. Ethereum is a stan-
dard platform for Dapps, and it has a Turing-complete program execution called
Ethereum Virtual Machine. A smart contract with a high descriptiveness and
readability can be described on the Ethereum platform using the contract-
oriented language Solidity.2 In addition, we used Truffle,3 a framework for devel-
oping smart contracts, for development. Truffle facilitates the implementation of
the connection between the WebUI and the contract, and it becomes possible to
easily deploy the contract to Ropsten, which is the test network of Ethereum,
or the Ethereum main network.

3.5 WebUI
The functions of the WebUI are as follows:
• to select and borrow a book from the bookshelf;
• to add a book;
• to return a borrowed book;
• to connect SlackID and address information.
Figure 2 shows a screenshot of the bookshelf of the WebUI. Books are
arranged on a common bookshelf for all users, and the book title, cover image,
links to book information, and the owner’s address are presented for each book.
These pieces of information are not held in the contract but are stored in an
external database. If a book can be borrowed, the “Borrow” button of the book
can be pressed. If this button cannot be pressed, this indicates that the book
has been lent out. For books that have been borrowed, the “Return” button can
be pressed.
The external database that contains the information about the book uses
the Google Books API. Given the ISBN, the Google Books API returns detailed
book information such as the title and cover image. The use of an external book
database like the Google Books API means that books that do not actually exist
cannot be registered and it is not necessary to input the basic information of
each book manually.
Another advantage of an application on the blockchain is that the interface
can be freely designed. In addition to the WebUI provided by us, as long as a
programmer knows the address of the contract, it is possible to freely create the
user interface using the blockchain as the database.

3.6 Notification Using Slack

Because a user needs to contact the owner of a book to ask to borrow it, we
prepared a notification function on a Slack channel. The proposed system has
the following five notifications.
2
https://solidity.readthedocs.io/.
3
https://truffleframework.com/.
 Personal Book Management Application on Blockchain 393

Fig. 2. WedUI

• SlackID of the person who borrowed or returned a book;

• Book owner’s SlackID;
• Book title;
• Timestamp;
• Fee.
The timing of the notifications is determined by when the book borrowing and
book returning processes are performed on the WebUI. Therefore, the timestamp
is not strictly the time the book was lent but the time at which a request to
borrow a book was entered.
The actual notification screen is shown in Fig. 3. In this study, it is assumed
that if there is a notification from Slack, as shown in the figure, the book is
handed over in real life.

4 Experimental Results
We deployed the Dapp on Ropsten, which is the test network of Ethereum, and
tried lending and borrowing books. We measured the execution time required
for smart contract authentication and the fees required for book borrowing and
lending.

4.1 Execution Time

Starting at the point when the confirmation button on the WebUI was pressed,
the time taken for approval of the transaction from the Slack log was measured.
The average time to approval was 38.3 s and the standard deviation was 23.3 s.
394 H. Kawaguchi et al.

Fig. 3. Notification using Slack

Moreover, the average time from when the button to return the book was
pressed until the approval of the transaction was 41.5 s, and the standard devi-
ation was 34.8 s.

4.2 Fees

Because Ethereum pays the fee in units of “gas”, the gas value was measured for
each operation using a Web application.4 We converted gas units to JPY, and
the average fees required for borrowing and returning books were 3.54 and 1.19
JPY, respectively.
Note that this conversion depends on the Gas Price, which depends on the
usage of the blockchain network.5

4.3 Discussion

Using the experimental results, we examine the issue of realizing a Dapp in the
proposed application.
First, the time required for authentication is 38.3 s when borrowing books
and 41.5 s when returning them. Considering that a user registers this infor-
mation with the WebUI after handing over the real book, there is no practical
problem with these times. Currently, we assume that the application is used in
a community in which books can be delivered directly; therefore, it is reason-
able to assume that there is no need to handle repeated requests within 38.3 s.
Moreover, because the request to deliver books is based on Slack notifications,
a lender may lend the book to the user who requested the book at a later
4
https://check-gas-price.glitch.me.
5
The above price was for 2018/12/04.
 Personal Book Management Application on Blockchain 395

time. Because the system does not operate in real time, the consistency between
the actual location of the book and the borrower on the system does not col-
lapse. Moreover, the standard deviation of borrowing and returning are 23.3 and
34.8 s, respectively, and these values are large. This is due to the current load of
the blockchain network and can be said to be a feature of applications on the
blockchain. Although the time for authentications varies widely, it is within an
acceptable range in absolute time.
The fees for the commission were 3.5 JPY for the rental and 1.19 JPY for
the return. Even assuming that a user has many books, this is an acceptable
amount, considering the cost of managing the location of the books. Further-
more, considering the real management cost of books in libraries and similar
institutions, this cost is sufficiently small.

5 Related Research
Virtual currency is the most effective in applications based on blockchain. Orig-
inally, blockchain was developed as Bitcoin’s background technology [6], and it
was designed to consider contract times and fees. In addition, with respect to
privacy, the transaction information of an account at a certain address is dis-
closed and shared, but anonymity is maintained unless the address and the actual
owner are linked with each other. Considering that transaction information is
shared even for very sensitive information such as currency account information,
it seems that users may not mind releasing information regarding books in this
system. However, a user may need to register a book that may be known to be
held by another user in the system. Even if a user is just borrowing a book, they
should understand that the book lending history is anonymous but accumulated
and released.
As practical examples of a Dapp, many applications such as distributed vir-
tual currency exchanges [9], gift card exchanges [3], and games [1] have been
proposed. These applications enjoy the advantages of blockchains such as anony-
mous and safe electronic data transactions as an extension of virtual currency.
In contrast, this paper proposes a blockchain-based application to enhance the
convenience of lending and borrowing real objects.
In contrast to the practical example above, Bogner et al. [2] proposed an
application that is not operated as a real service but focuses on sharing objects.
Their system assigns a QR code to an arbitrary object that is lent and borrowed
among individuals and manages this by linking it to individuals on a blockchain.
This is the same as the proposed system in that it focuses on the lending and
borrowing of real objects among individuals. However, by restricting the infor-
mation to books, the proposed system can reduce the effort needed for lending
and borrowing, such the effort of acquiring book information from an external
database. In fact, because the information actually placed in a QR code is cur-
rently about 3 MB at most, it is not practical to put additional information other
than the ID of the object in contract. A method using ISBNs like the proposed
system is more suitable for blockchains.
396 H. Kawaguchi et al.

6 Conclusions
In this research, we designed and developed an application on a blockchain that
takes the information about books possessed by a user and manages the lending
and borrowing of the books using a distributed ledger. Using this application as
a case study, we discussed the usefulness in an application involving real objects.
The future tasks for this research are as follows:

1. to conduct a user survey on book lending history privacy;

2. to create an application for lending and borrowing electronic books;
3. to implement payment in virtual currency when books are rented.

For the first item, because the book lending history remains in the ledger,
personal information consisting of the books a user has borrowed remains on
the blockchain. We plan to create questionnaire to determine user tolerances of
privacy. For the second item, if it is possible to create a mechanism by which
books can only be read by one person, this could be applied to electronic books.
Because the locations of all the books are managed by the distributed ledger,
it is sufficient to add a function for opening a book by referring to the dis-
tributed ledger to determine the current reader of the electronic book. For the
third item, the current system considers lending and borrowing books for free,
but by implementing the application using a blockchain, we can also pay the
fees associated with lending and borrowing books. However, legal issues such as
copyright royalties are involved, so this task is not trivial.

References
1. AxiomZen: Cryptokities (2017). https://www.cryptokitties.co/
2. Bogner, A., Chanson, M., Meeuw, A.: A decentralised sharing app running a smart
contract on the ethereum blockchain. In: Proceedings of the 6th International Con-
ference on the Internet of Things, IoT 2016, pp. 177–178 (2016)
3. GyftInc. (2015). https://www.gyft.com/
4. Halaburda, H.: Blockchain revolution without the blockchain? Commun. ACM
61(7), 27–29 (2018)
5. Johnston, D., Onat Yilmaz, S., Kandah, J., Bentenitis, N., Hashemi, F., Gross, R.,
Wilkinson, S., Mason, S.: The general theory of decentralized applications, dapps
(2013). https://github.com/DavidJohnstonCEO/DecentralizedApplications
6. Nakamoto, S.: Bitcoin: a peer-to-peer electronic cash system (2009)
7. Underwood, S.: Blockchain beyond bitcoin. Commun. ACM 59(11), 15–17 (2016)
8. Vitalik, B.: A next-generation smart contract and decentralized application plat-
form. white Paper (2014). https://www.ethereum.org/
9. Warren, W., Bandeali, A.: 0x: an open protocol for decentralized exchange on the
ethereum blockchain (2017). https://0xproject.com/pdfs/0x white paper.pdf
 Performance Evaluation of WMNs
by WMN-PSOSA System Considering
Chi-square and Exponential Client
Distributions

Shinji Sakamoto1(B) , Leonard Barolli2 , and Shusuke Okamoto1

1
Department of Computer and Information Science, Seikei University,
3-3-1 Kichijoji-Kitamachi, Musashino-shi, Tokyo 180-8633, Japan
[email protected], [email protected]
2
Department of Information and Communication Engineering,
Fukuoka Institute of Technology, 3-30-1 Wajiro-Higashi,
Higashi-Ku, Fukuoka 811-0295, Japan
[email protected]

Abstract. Wireless Mesh Networks (WMNs) have many advantages

such as low cost and increased high-speed wireless Internet connectivity,
therefore WMNs are becoming an important networking infrastructure.
In our previous work, we implemented a Particle Swarm Optimization
(PSO) based simulation system for node placement in WMNs, called
WMN-PSO. Also, we implemented a simulation system based on Simu-
lated Annealing (SA) for solving node placement problem in WMNs,
called WMN-SA. Then, we implemented a hybrid simulation system
based on PSO and SA, called WMN-PSOSA. In this paper, we anal-
yse the performance of WMNs by using WMN-PSOSA considering two
types of mesh clients distributions. Simulation results show that a good
performance is achieved for Exponential distribution compared with the
case of Chi-square distribution.

1 Introduction
The wireless networks and devices are becoming increasingly popular and they
provide users access to information and communication anytime and any-
where [2,5,7–9,12,17,22,23,26,27]. Wireless Mesh Networks (WMNs) are gain-
ing a lot of attention because of their low cost nature that makes them attrac-
tive for providing wireless Internet connectivity. A WMN is dynamically self-
organized and self-configured, with the nodes in the network automatically estab-
lishing and maintaining mesh connectivity among them-selves (creating, in effect,
an ad hoc network). This feature brings many advantages to WMNs such as low
up-front cost, easy network maintenance, robustness and reliable service cover-
age [1]. Moreover, such infrastructure can be used to deploy community networks,
metropolitan area networks, municipal and corporative networks, and to support
applications for urban areas, medical, transport and surveillance systems.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 397–406, 2020.
https://doi.org/10.1007/978-3-030-15032-7_34
398 S. Sakamoto et al.

We consider the version of the mesh router nodes placement problem in which
we are given a grid area where to deploy a number of mesh router nodes and
a number of mesh client nodes of fixed positions (of an arbitrary distribution)
in the grid area. The objective is to find a location assignment for the mesh
routers to the cells of the grid area that maximizes the network connectivity
and client coverage. Node placement problems are known to be computationally
hard to solve [10,11,28]. In some previous works, intelligent algorithms have
been recently investigated [3,4,13,15,18–20,24,25].
In [21], we implemented a Particle Swarm Optimization (PSO) based simula-
tion system, called WMN-PSO. Also, we implemented a simulation system based
on Simulated Annealing (SA) for solving node placement problem in WMNs,
called WMN-SA [16,17].
In our previous work, we implemented a hybrid simulation system based on
PSO and SA. We called this system WMN-PSOSA. In this paper, we analyse
the performance of hybrid WMN-PSOSA system considering Chi-square and
Exponential clients distributions.
The rest of the paper is organized as follows. The mesh router nodes place-
ment problem is defined in Sect. 2. We present our designed and implemented
hybrid simulation system in Sect. 3. The simulation results are given in Sect. 4.
Finally, we give conclusions and future work in Sect. 5.

Fig. 1. Relationship among global solution, particle-patterns and mesh routers.

2 Node Placement Problem in WMNs

For this problem, we have a grid area arranged in cells we want to find where to
distribute a number of mesh router nodes and a number of mesh client nodes of
fixed positions (of an arbitrary distribution) in the considered area. The objective
is to find a location assignment for the mesh routers to the area that maximizes
the network connectivity and client coverage. Network connectivity is measured
by Size of Giant Component (SGC) of the resulting WMN graph, while the user
coverage is simply the number of mesh client nodes that fall within the radio
coverage of at least one mesh router node and is measured by Number of Covered
Mesh Clients (NCMC).
 Performance Evaluation of WMNs by WMN-PSOSA System 399

An instance of the problem consists as follows.

• N mesh router nodes, each having its own radio coverage, defining thus a
vector of routers.
• An area W ×H where to distribute N mesh routers. Positions of mesh routers
are not pre-determined and are to be computed.
• M client mesh nodes located in arbitrary points of the considered area, defin-
ing a matrix of clients.

It should be noted that network connectivity and user coverage are among
most important metrics in WMNs and directly affect the network performance.
In this work, we have considered a bi-objective optimization in which we first
maximize the network connectivity of the WMN (through the maximization of
the SGC) and then, the maximization of the NCMC.
In fact, we can formalize an instance of the problem by constructing an
adjacency matrix of the WMN graph, whose nodes are router nodes and client
nodes and whose edges are links between nodes in the mesh network. Each mesh
node in the graph is a triple v = < x, y, r > representing the 2D location point
and r is the radius of the transmission range. There is an arc between two nodes
u and v, if v is within the transmission circular area of u.

3 Proposed and Implemented Simulation System

3.1 WMN-PSOSA Hybrid Simulation System

3.1.1 Particle Swarm Optimization

In Particle Swarm Optimization (PSO) algorithm, a number of simple entities
(the particles) are placed in the search space of some problem or function and
each evaluates the objective function at its current location. The objective func-
tion is often minimized and the exploration of the search space is not through
evolution [14]. However, following a widespread practice of borrowing from the
evolutionary computation field, in this work, we consider the bi-objective func-
tion and fitness function interchangeably. Each particle then determines its move-
ment through the search space by combining some aspect of the history of its
own current and best (best-fitness) locations with those of one or more members
of the swarm, with some random perturbations. The next iteration takes place
after all particles have been moved. Eventually the swarm as a whole, like a flock
of birds collectively foraging for food, is likely to move close to an optimum of
the fitness function.
Each individual in the particle swarm is composed of three D-dimensional
vectors, where D is the dimensionality of the search space. These are the current
position xi , the previous best position pi and the velocity vi .
The particle swarm is more than just a collection of particles. A particle by
itself has almost no power to solve any problem; progress occurs only when the
particles interact. Problem solving is a population-wide phenomenon, emerging
from the individual behaviors of the particles through their interactions. In any
400 S. Sakamoto et al.

case, populations are organized according to some sort of communication struc-

ture or topology, often thought of as a social network. The topology typically
consists of bidirectional edges connecting pairs of particles, so that if j is in i’s
neighborhood, i is also in j’s. Each particle communicates with some other par-
ticles and is affected by the best point found by any member of its topological
neighborhood. This is just the vector pi for that best neighbor, which we will
denote with pg . The potential kinds of population “social networks” are hugely
varied, but in practice certain types have been used more frequently.
In the PSO process, the velocity of each particle is iteratively adjusted so
that the particle stochastically oscillates around pi and pg locations.

3.1.2 Simulated Annealing

Simulated Annealing (SA) algorithm [6] is a generalization of the metropolis
heuristic. Indeed, SA consists of a sequence of executions of metropolis with a
progressive decrement of the temperature starting from a rather high tempera-
ture, where almost any move is accepted, to a low temperature, where the search
resembles Hill Climbing. In fact, it can be seen as a hill-climber with an internal
mechanism to escape local optima. In SA, the solution s is accepted as the new
current solution if δ ≤ 0 holds, where δ = f (s ) − f (s). To allow escaping from
a local optimum, the movements that increase the energy function are accepted
with a decreasing probability exp (−δ/T ) if δ > 0, where T is a parameter called
the “temperature”. The decreasing values of T are controlled by a cooling sched-
ule, which specifies the temperature values at each stage of the algorithm, what
represents an important decision for its application (a typical option is to use a
proportional method, like Tk = γ · Tk−1 ). SA usually gives better results in prac-
tice, but uses to be very slow. The most striking difficulty in applying SA is to
choose and tune its parameters such as initial and final temperature, decrements
of the temperature (cooling schedule), equilibrium and detection.
The acceptability criteria for newly generated solution is based on the defi-
nition of a threshold value (accepting threshold) as follows. We consider a suc-
cession tk such that tk > tk+1 , tk > 0 and tk tends to 0 as k tends to infinity.
Then, for any two solutions si and sj , if f itness(sj ) − f itness(si ) < tk , then
accept solution sj .
For the SA, tk values are taken as accepting threshold but the criterion for
acceptance is probabilistic:

• If f itness(sj ) − f itness(si ) ≤ 0 then sj is accepted.

• If f itness(sj ) − f itness(si ) > 0 then sj is accepted with probability
exp[(f itness(sj ) − f itness(si ))/tk ] (at iteration k the algorithm generates
a random number R ∈ (0, 1) and sj is accepted if R < exp[(f itness(sj ) −
f itness(si ))/tk ]).

In this case, each neighbour of a solution has a positive probability of replac-

ing the current solution. The tk values are chosen in way that solutions with
large increase in the cost of the solutions are less likely to be accepted (but there
is still a positive probability of accepting them).
 Performance Evaluation of WMNs by WMN-PSOSA System 401

3.1.3 WMN-PSOSA System Description

In following, we present the initialization, cooling schedule, particle-pattern, fit-
ness function and client distributions.
Initialization
Our proposed system starts by generating an initial solution randomly, by ad hoc
methods [29]. We decide the velocity of particles by a random process considering
instance, when√the area size is W × H, the velocity is decided
the area size. For √
randomly from − W 2 + H 2 to W 2 + H 2 .
Cooling schedule
In our system, cooling schedule (γ) will be calculated as:
 1.0/T otal iterations
SA Ending temperature
γ= .
SA Starting temperature

Particle-pattern
A particle is a mesh router. A fitness value of a particle-pattern is computed
by combination of mesh routers and mesh clients positions. In other words,
each particle-pattern is a solution as shown is Fig. 1. Therefore, the number of
particle-patterns is a number of solutions.
Fitness function
One of most important thing is to decide the determination of an appropriate
objective function and its encoding. In our case, each particle-pattern has an
own fitness value and compares other particle-pattern’s fitness value in order to
share information of global solution. The fitness function follows a hierarchical
approach in which the main objective is to maximize the SGC in WMN. Thus,
we use α and β weight-coefficients for the fitness function and the fitness function
of this scenario is defined as

Fitness = α × SGC(xij , y ij ) + β × NCMC(xij , y ij ).

Client distributions
Our proposed system can generate a lot of clients distributions. In this paper,
we consider Chi-square and Exponential clients distributions as shown in Fig. 2.

3.2 WMN-PSOSA Web GUI Tool

The Web application follows a standard Client-Server architecture and is imple-

mented using LAMP (Linux + Apache + MySQL + PHP) technology (see
Fig. 3). We show the WMN-PSOSA Web GUI tool in Fig. 4. Remote users
(clients) submit their requests by completing first the parameter setting. The
parameter values to be provided by the user are classified into three groups, as
follows.
402 S. Sakamoto et al.

(a) Chi-square (b) Exponential

Fig. 2. Clients distributions.

Fig. 3. System structure for web interface.

Fig. 4. WMN-PSOSA Web GUI tool.

 Performance Evaluation of WMNs by WMN-PSOSA System 403

• Parameters related to the problem instance: These include parameter values

that determine a problem instance to be solved and consist of number of
router nodes, number of mesh client nodes, client mesh distribution, radio
coverage interval and size of the deployment area.
• Parameters of the resolution method: Each method has its own parameters.
• Execution parameters: These parameters are used for stopping condition of
the resolution methods and include number of iterations and number of inde-
pendent runs. The former is provided as a total number of iterations and
depending on the method is also divided per phase (e.g., number of iterations
in a exploration). The later is used to run the same configuration for the same
problem instance and parameter configuration a certain number of times.

Table 1. Parameter settings.

Parameters Values
Clients distribution Chi-square, exponential
Area size 32.0 × 32.0
Number of mesh routers 16
Number of mesh clients 48
Total iterations 6400
Iteration per phase 32
Number of particle-patterns 9
Radius of a mesh router 2.0
SA starting temperature value 10.0
SA ending temperature value 0.01
Fitness function weight-coefficients (α, β) 0.7, 0.3
Temperature decreasing speed (γ) 0.998921
Replacement method LDIWM

Distribution of Clients=Chi-square distribution Best Solutions Distribution of Clients=Chi-square distribution Best Solutions
Number of Covered Mesh Clients[%]

Average of Best Solutions Average of Best Solutions

Replacement Method=LDIWM Replacement Method=LDIWM
Iteration per Phase=32 Iteration per Phase=32
Giant Component[%]

Total Iterations=6400 Total Iterations=6400

Number of Particle-patterns=9 Number of Particle-patterns=9
100 100

75 75

50 50

25 25

0 50 100 150 200 0 50 100 150 200

Number of Phases Number of Phases
(a) SGC (b) NCMC

Fig. 5. Simulation results of WMN-PSOSA for Chi-square distribution.

404 S. Sakamoto et al.

Distribution of Clients=Exponential distribution Best Solutions Distribution of Clients=Exponential distribution Best Solutions

Number of Covered Mesh Clients[%]

Average of Best Solutions Average of Best Solutions
Replacement Method=LDIWM Replacement Method=LDIWM
Iteration per Phase=32 Iteration per Phase=32
Giant Component[%]
Total Iterations=6400 Total Iterations=6400
Number of Particle-patterns=9 Number of Particle-patterns=9
100 100

75 75

50 50

25 25

0 50 100 150 200 0 50 100 150 200

Number of Phases Number of Phases
(a) SGC (b) NCMC

Fig. 6. Simulation results of WMN-PSOSA for exponential distribution.

4 Simulation Results

In this section, we show simulation results using WMN-PSOSA system. In this

work, we consider Chi-square and Exponential distributions of mesh clients. The
number of mesh routers is considered 16 and the number of mesh clients 48. The
total number of iterations is considered 6400 and the iterations per phase is
considered 32. We consider the number of particle-patterns 9. We conducted
simulations 100 times, in order to avoid the effect of randomness and create a
general view of results. We show the parameter setting for WMN-PSOSA in
Table 1.
We show the simulation results in Figs. 5 and 6. For both client distributions,
the performance for the SGC is almost the same. However, for the NCMC, the
performance is better for Exponential compared with the case of Chi-square
distribution. Therefore, we conclude that the performance for Exponential dis-
tribution is better compared with the case of Chi-square distribution.

5 Conclusions

In this work, we evaluated the performance of a hybrid simulation system based

on PSO and SA (called WMN-PSOSA) considering Chi-square and Exponen-
tial distributions of mesh clients. Simulation results show that the performance
is better for Exponential distribution compared with the case of Chi-square
distribution.
In our future work, we would like to evaluate the performance of the proposed
system for different parameters and scenarios.
 Performance Evaluation of WMNs by WMN-PSOSA System 405

References
1. Akyildiz, I.F., Wang, X., Wang, W.: Wireless mesh networks: a survey. Comput.
Netw. 47(4), 445–487 (2005)
2. Barolli, A., Sakamoto, S., Barolli, L., Takizawa, M.: Performance analysis of simula-
tion system based on particle swarm optimization and distributed genetic algorithm
for WMNs considering different distributions of mesh clients. In: International Con-
ference on Innovative Mobile and Internet Services in Ubiquitous Computing, pp.
32–45. Springer (2018)
3. Barolli, A., Sakamoto, S., Ozera, K., Barolli, L., Kulla, E., Takizawa, M.: Design
and implementation of a hybrid intelligent system based on particle swarm opti-
mization and distributed genetic algorithm. In: International Conference on Emerg-
ing Internetworking, Data & Web Technologies, pp. 79–93. Springer (2018)
4. Girgis, M.R., Mahmoud, T.M., Abdullatif, B.A., Rabie, A.M.: Solving the wireless
mesh network design problem using genetic algorithm and simulated annealing
optimization methods. Int. J. Comput. Appl. 96(11), 1–10 (2014)
5. Goto, K., Sasaki, Y., Hara, T., Nishio, S.: Data gathering using mobile agents for
reducing traffic in dense mobile wireless sensor networks. Mobile Inf. Syst. 9(4),
295–314 (2013)
6. Hwang, C.R.: Simulated annealing: theory and applications. Acta Applicandae
Mathematicae 12(1), 108–111 (1988)
7. Inaba, T., Elmazi, D., Sakamoto, S., Oda, T., Ikeda, M., Barolli, L.: A secure-aware
call admission control scheme for wireless cellular networks using fuzzy logic and
its performance evaluation. J. Mobile Multimed. 11(3&4), 213–222 (2015)
8. Inaba, T., Obukata, R., Sakamoto, S., Oda, T., Ikeda, M., Barolli, L.: Performance
evaluation of a QoS-aware fuzzy-based CAC for LAN access. Int. J. Space-Based
Situated Comput. 6(4), 228–238 (2016)
9. Inaba, T., Sakamoto, S., Oda, T., Ikeda, M., Barolli, L.: A testbed for admission
control in WLAN: a fuzzy approach and its performance evaluation. In: Interna-
tional Conference on Broadband and Wireless Computing, Communication and
Applications, pp. 559–571. Springer (2016)
10. Lim, A., Rodrigues, B., Wang, F., Xu, Z.: k-Center Problems with Minimum Cov-
erage. In: Computing and Combinatorics, pp. 349–359 (2004)
11. Maolin, T., et al.: Gateways placement in backbone wireless mesh networks. Int.
J. Commun. Netw. Syst. Sci. 2(1), 44 (2009)
12. Matsuo, K., Sakamoto, S., Oda, T., Barolli, A., Ikeda, M., Barolli, L.: Performance
analysis of WMNs by WMN-GA simulation system for two WMN architectures
and different TCP congestion-avoidance algorithms and client distributions. Int.
J. Commun. Netw. Distrib. Syst. 20(3), 335–351 (2018)
13. Naka, S., Genji, T., Yura, T., Fukuyama, Y.: A hybrid particle swarm optimization
for distribution state estimation. IEEE Trans. Power Syst. 18(1), 60–68 (2003)
14. Poli, R., Kennedy, J., Blackwell, T.: Particle swarm optimization. Swarm Intell.
1(1), 33–57 (2007)
15. Sakamoto, S., Kulla, E., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: A comparison
study of simulated annealing and genetic algorithm for node placement problem
in wireless mesh networks. J. Mobile Multimed. 9(1–2), 101–110 (2013)
16. Sakamoto, S., Kulla, E., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: A compar-
ison study of hill climbing, simulated annealing and genetic algorithm for node
placement problem in WMNs. J. High Speed Netw. 20(1), 55–66 (2014)
406 S. Sakamoto et al.

17. Sakamoto, S., Kulla, E., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: A simulation
system for WMN based on SA: performance evaluation for different instances and
starting temperature values. Int. J. Space-Based Situated Comput. 4(3–4), 209–216
(2014)
18. Sakamoto, S., Kulla, E., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: Performance
evaluation considering iterations per phase and SA temperature in WMN-SA sys-
tem. Mobile Inf. Syst. 10(3), 321–330 (2014)
19. Sakamoto, S., Lala, A., Oda, T., Kolici, V., Barolli, L., Xhafa, F.: Application
of WMN-SA simulation system for node placement in wireless mesh networks: a
case study for a realistic scenario. Int. J. Mobile Comput. Multimed. Commun.
(IJMCMC) 6(2), 13–21 (2014)
20. Sakamoto, S., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: An integrated simulation
system considering WMN-PSO simulation system and network simulator 3. In:
International Conference on Broadband and Wireless Computing, Communication
and Applications, pp. 187–198. Springer (2016)
21. Sakamoto, S., Oda, T., Ikeda, M., Barolli, L., Xhafa, F.: Implementation and evalu-
ation of a simulation system based on particle swarm optimisation for node place-
ment problem in wireless mesh networks. Int. J. Commun. Netw. Distrib. Syst.
17(1), 1–13 (2016)
22. Sakamoto, S., Obukata, R., Oda, T., Barolli, L., Ikeda, M., Barolli, A.: Performance
analysis of two wireless mesh network architectures by WMN-SA and WMN-TS
simulation systems. J. High Speed Netw. 23(4), 311–322 (2017)
23. Sakamoto, S., Ozera, K., Barolli, A., Ikeda, M., Barolli, L., Takizawa, M.: Imple-
mentation of an intelligent hybrid simulation systems for WMNs based on particle
swarm optimization and simulated annealing: performance evaluation for different
replacement methods. Soft Comput., 1–7 (2017)
24. Sakamoto, S., Ozera, K., Oda, T., Ikeda, M., Barolli, L.: Performance evaluation
of intelligent hybrid systems for node placement in wireless mesh networks: a com-
parison study of WMN-PSOHC and WMN-PSOSA. In: International Conference
on Innovative Mobile and Internet Services in Ubiquitous Computing, pp. 16–26.
Springer (2017)
25. Sakamoto, S., Ozera, K., Oda, T., Ikeda, M., Barolli, L.: Performance evaluation of
WMN-PSOHC and WMN-PSO simulation systems for node placement in wireless
mesh networks: a comparison study. In: International Conference on Emerging
Internetworking, Data & Web Technologies, pp. 64–74. Springer (2017)
26. Sakamoto, S., Ozera, K., Barolli, A., Barolli, L., Kolici, V., Takizawa, M.: Perfor-
mance evaluation of WMN-PSOSA considering four different replacement methods.
In: International Conference on Emerging Internetworking, Data & Web Technolo-
gies, pp. 51–64. Springer (2018)
27. Sakamoto, S., Ozera, K., Ikeda, M., Barolli, L.: Implementation of intelligent hybrid
systems for node placement problem in WMNs considering particle swarm opti-
mization, hill climbing and simulated annealing. Mobile Netw. Appl. 23(1), 27–33
(2018)
28. Wang, J., Xie, B., Cai, K., Agrawal, D.P.: Efficient mesh router placement in wire-
less mesh networks. In: Proceedings of IEEE International Conference on Mobile
Adhoc and Sensor Systems (MASS-2007), pp. 1–9 (2007)
29. Xhafa, F., Sanchez, C., Barolli, L.: Ad hoc and neighborhood search methods for
placement of mesh routers in wireless mesh networks. In: Proceedings of 29th IEEE
International Conference on Distributed Computing Systems Workshops (ICDCS-
2009), pp. 400–405 (2009)
 Realization and Preliminary Evaluation
of MPI Runtime Environment
on Android Cluster

Masahiro Nissato(B) , Hiroki Sugiyama, Kanemitsu Ootsu, Takeshi Ohkawa,

and Takashi Yokota

Graduate School of Engineering, Utsunomiya University,

Yoto 7–1–2, Utsunomiya, Tochigi 321-8282, Japan
[email protected], [email protected]

Abstract. In recent years, performance of mobile computer devices such

as smartphones and tablet devices has been improved, and mobile com-
puter devices are attracted attention as new computational resources
for high performance computation. We are developing a cluster com-
puter system that consists of mobile computer devices running Android
OS. In the preceding researches including ours, super-user authority is
required to install the system files for parallel processing environment
onto the restricted area of file system. Hence, we cannot build the clus-
ter computer system using mobile computer devices without super-user
authority. Furthermore, we clarify the performance of our proposed sys-
tem as compared to our previous system, by using benchmark programs
for MPI parallel processing. The results of performance comparison show
that our proposed system has lower performance than our previous sys-
tem on only one node. However, as the number of nodes increased, the
difference in performance tends to decrease.

1 Introduction

Performance of mobile computer devices such as smartphones and tablets com-

puters has been improved from various aspects such as improved computation
performance by using multi-core CPU and GPU, increased memory and stor-
age capacity, improved network communication speed, etc. Furthermore, these
mobile devices are increasing drastically [1] and we see them everywhere. There-
fore, we can use these mobile devices as new computation resource for high
performance computation.
Busching et al. showed that a cluster computer system can be built by using
smartphones as node computers. In their research, Linux OS is installed on SD
memory card of smartphones [2]. Hinojos et al. also showed a cluster computer
system for distributed processing by using mobile devices. Their system uses
Bluetooth as the communication method [3]. As shown above, mobile devices
are attracting attention as new computational resources for high performance
computing.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 407–418, 2020.
https://doi.org/10.1007/978-3-030-15032-7_35
408 M. Nissato et al.

We are now developing a cluster system, called Android Cluster, aiming

at realizing parallel processing environment using common mobile computer
devices running Android OS. In our system, we use the Message Passing Interface
(MPI) [4] as a parallel processing platform.
In either case [2,5], parallel processing environment is realized by modifying
the file storage area for system files. For that purpose, it is necessary to perform
privileged operations by using super-user authority (called root authority), such
as writing files onto the protected system area. In the case of Android OS, super-
user authority is protected to use and is not permitted to any user on the devices.
If we want to use the super-user authority, we need to unlock the bootloader of
the device and to write a program for acquiring the super-user authority onto
the storage are for system files. However, this is quite dangerous operations, and
if the operation fails, the device possibly becomes unavailable.
In order to realize the cluster computer system using Android devices as
node computer safely, it is necessary to develop the method to build the parallel
processing environment on the Android OS without super-user authority. In this
paper, we build MPI parallel processing platform as Android application that
can operate without super-user authority.
The rest of this paper is as follows. Section 2 describes computing nodes, net-
work configuration and parallel processing platform software of Android Clus-
ters. Section 3 describes how to start MPI parallel processing in the MPI parallel
processing environment with regular user authority. In Sect. 4, we evaluate the
performance of the Android Cluster constructed using our MPI parallel process-
ing platform. Finally, Sect. 5 summarizes this paper.

2 An Overview of Android Cluster

In this section, we explain basic components of our Android Cluster such as

computing nodes, network environment and parallel processing platform.
Figure 1 shows an example of the configuration of our system. In this system,
mobile computer devices running Android OS are used as compute nodes. We
assume that not only smartphones and tablets but also any other computers
running Android OS, such as Raspberry Pi can be incorporated into the cluster
system.
In our cluster, communication facilities such as Wi-Fi and Ethernet provided
for each device are used for communication between compute nodes. Android
devices can communicate with each other via switching hubs and routers. The
devices can perform wireless communication via the Wi-Fi access point (Wi-
Fi AP). Each Android device has a unique IP address. We assume that this
system automatically set an IP address of each Android device based on DHCP.
However, in this research, we set an IP address statically. This is because we
have not implemented DHCP server functions for our cluster.
As parallel processing environment, we adopt the Message Passing Interface
(MPI), which is the standard for parallel distributed processing. This system
uses Open MPI [6] library that is an open-source implementation of MPI. Google
 Realization and Preliminary Evaluation of MPI Runtime Environment 409

provides Android Native Development Kit (NDK) [7,8] for making use of conven-
tional C language program onto Android OS. Android NDK is a cross-compiler
toolchain and we can use it to generate machine language program/library files
to run on Android OS. We used Android NDK to compile/build Open MPI
source code. The C runtime library (so-called bionic libc) [9] on Android OS has
minimal functions that supports to run Android middleware. Therefore, due to
the lack of various functions, we had to port Open MPI for Android OS.

Fig. 1. Example of Android Cluster system

3 MPI Parallel Program Runtime Environment

We explain our MPI parallel program runtime environment (called MPI run-
time environment) that can execute MPI parallel programs without super-user
authority.
Android OS was built a kernel based on the Linux kernel [7]. Therefore,
Android OS realizes its security model for application program by using a part
of Linux protection mechanism such as file permission mechanism, user ID, group
ID, and so on.
In Linux OS, each user has his/her own ID (UID), and various resources
such as file and memory on the computer are protected on a user basis. In
Android OS, this UID is allocated to each Android application program, not
to user. Therefore, each Android application program can basically only access
resources that is owned by the program, and cannot access resources owned by
other programs. In other words, when trying to start another program from
a program, you need to refer to its executable binary file, but it cannot be
referenced, since it is owned by different user (UID). In the case of MPI parallel
program, the executable file of parallel program is launched from the mpirun
410 M. Nissato et al.

(or mpiexec) program, but since it cannot refer to the executable file of parallel
program, parallel execution cannot be started. Only super-user can refer to the
files owned by different users (UIDs), but the Android OS does not provide any
public method for using super-user authority.
This problem can be solved by allowing mpirun and MPI program to be
owned by the same user (UID). For that purpose, we have only to package it as
one Android application program and to install it as one integrated application
program onto Android device. Thus, we can run MPI parallel programs without
super-user authority by the above method.
Figure 2 shows the structure of our MPI runtime environment. This system
consists of Android application written in Java language and Open MPI library
and MPI program written in C/C++ language. We assume the runtime environ-
ment where existing MPI parallel programs (which are usually launched from the
command line shell) run on Android OS. To realize this, we develop an Android
application program, written in Java. The Android application program can
launch machine-native MPI parallel programs by kicking mpirun program.
When building an MPI runtime environment as an Android application, the
following three problems should be solved.

1. File location
2. Creation of MPI parallel process
3. Determination of IP address and user name on each node computer

The first problem is where to place files related to MPI parallel programs.
MPI runtime library and other files necessary for executing the MPI parallel pro-
grams should be placed in the storage area which is accessible from the Android
application.
The second problem is how to create MPI parallel processes. In order to
execute MPI parallel programs, it is necessary to create processes on the local
and remote devices from the Android application.
The third problem is how to determinate the IP addresses of remote devices
and user names on them. When launching MPI parallel processes on the remote
devices via Open MPI library, we use the SSH (Secure SHell) to launch parallel
processes on remote nodes, with public key authentication. It is necessary to
obtain an IP address and account name (i.e. user name that owns MPI applica-
tion program files) on the target node.
In the following sections, we discuss the solutions for these three problems in
detail.

3.1 File Location

Android device has two types storage area accessible from Android application.
One is external storage, the other is internal storage [7].
The external storage refers to removable storage like an SD card, or a built-in
storage treated as a logically external storage.
 Realization and Preliminary Evaluation of MPI Runtime Environment 411

Fig. 2. MPI runtime environment

The external storage is shared by all Android applications and is managed by

using File Allocation Table (FAT) [10] file system. FAT file system is originated
from MS-DOS, it does not support permission management of Linux OS. It is
impossible to grant execution authority to files placed in external storage, and
Android OS handles all files on external storage as non-executable file. In other
words, if we place executable files and library files on external storage, they
cannot be executed. Thus, we cannot use external storage as the storage area
for placing executable program files and library files.
The internal storage is used for storing the system files of Android OS and
its application program files. The internal storage is managed by using ext [10]
file system. The ext file system supports the permission management mechanism
of Linux OS.
Each Android application program file is placed in its own dedicated direc-
tory, and the adequate file permission is added to each file. For example, the
executable program files and library files have the access permission of read-
able and executable. Therefore, we decide to place the MPI executable parallel
program files and its related files such as MPI runtime libraries in the internal
storage.
All Android applications are managed by the Android OS using a unique
identification number called application ID. When an application ID is app id,
we can use ‘/data/data/app id/files’ directory as dedicated directory. We use
this directory for storing executable files and libraries.
In our system, the MPI parallel program is started from our Android appli-
cation written in Java. Since our Android application should be able to access
the MPI program files including mpirun/mpiexec program without super-user
authority, they should be owned by the same user (UID). Therefore, we place
them in the same directory with the same user ID. For this purpose, we package
the MPI program files into our Android application. By this, when install the
application, the MPI program files are stored into the same dedicated directory
412 M. Nissato et al.

for Android application and owned by the same user. Thus, our Android appli-
cation can access the MPI program files without super-user authority.

3.2 Process Creation

In our system, we assume that the MPI programs are stared to execute from the
Android application program, by using mpirun/mpiexec command.
Figure 3 shows how to execute the MPI program in our system.
First, our Android application program starts to execute the mpirun pro-
gram, the program launcher of MPI parallel program. Our application program
creates a process for mpirun program and passes the information about parallel
execution to the mpirun. Then, the mpirun program starts to execute the MPI
parallel program.
The mpirun program creates the processes on the local and remote nodes.
When creating the processes on remote nodes, the mpirun first creates a process
for orted (that is a management process for MPI parallel processes on remote
node), and then the orted creates processes for MPI parallel program.
In the following sections, we discuss how to create local and remote processes
from the Android application program.

Fig. 3. MPI parallel processing on Android application

3.2.1 Local Process Creation

Android application program can create processes by the following two alterna-
tive methods.

1. fork system call through JNI

2. ProcessBuilder class of Java API framework
 Realization and Preliminary Evaluation of MPI Runtime Environment 413

The first method is to create processes by using fork system call through
Java Native Interface (JNI).
JNI is a mechanism in the Java platform for linking machine native program
code with Android application program that is usually in the form of bytecode.
JNI allows the Java application program to access the program functions in the
form of machine native binary code. By using JNI, we can use the OS system call
from Android application program. That is, through JNI, Android application
program written in Java can invoke the program function that creates processes
by the fork system call.
The second method is to create processes by using ProcessBuilder class of
Java API framework. ProcessBuilder class of Java API manages OS processes,
and we can create multiple processes by using it. The second method is easier to
use than the first one, since we can describe the process creation directly in Java
source code of Android application program. Therefore, we decide to implement
the process creation by the second method.

3.2.2 Remote Process Creation

In the Open MPI environment, creating process on remote node is accompanied
by remote login to the target node computer. In order to login the remote node,
its IP address and the user name on the target node are required. The method
to acquire these information are discussed in the next section.
In our system, we use OpenSSH [11] for the process of remote login, since it
is commonly used and secure communication between nodes is realized.
In OpenSSH, in order to realize remote login, a daemon process for accepting
connections from other devices which is called sshd is required to running on each
node device in cluster. In this system, our Android application starts the process
of sshd by the method described in the Sect. 3.2.1. Therefore, the files related to
OpenSSH are packaged into our Android application program and installed into
the same directory with our application program. Furthermore, SSH daemon
‘sshd’ runs as the same user ID of application program.
Usually, SSH daemon uses the port number 22 for accepting connection from
other node, however, this is permitted to use only for super-user. Therefore, we
modify the port number 22 to the one permitted to regular user authority.

3.3 Determination of IP Address and User Name

Like other related researches, in our previous Android Cluster [5], creating paral-
lel processes on remote nodes required super-user authority. On the other hand,
our new system does not require super-user authority.
In our system, when the MPI parallel processes are started on remote node,
remote login is performed by SSH with the user name on the target IP address.
In Android OS, a unique user ID and name are assigned to each application by
OS, when the application is installed on device.
The problem is that different user ID/name might be assigned on each node,
even if the same application. In order to solve this problem, we implement the
414 M. Nissato et al.

mechanism to determine the IP address of target node and the user name on it
for the use of remote login.
We assume that the number of node computers might change dynamically,
since our Android Cluster consists of mobile computer devices. Some node might
leave from the cluster and some other node might join to the cluster. Therefore,
it is necessary to manage the configuration of cluster system, such as the IP
addresses of cluster member node and the user names on them.
In our system, in order to solve this problem, we use the mapping tables
that maintains the node devices and user names in the cluster. To manage the
mapping table, we have implemented the daemon program that checks the IP
addresses of nodes and user names in the cluster and update the mapping tables.
Open MPI uses hostfile that records IP address of each node in order to
specify the target nodes that join in parallel processing. We use the hostfile as
the mapping table. In our previous Android Clusters [5], we developed a daemon
program that automatically creates and maintains hostfile in order to keep the
list of currently available node devices up to date, even when the configuration
of cluster dynamically changes.
In our MPI runtime environment, we add new functions of the daemon pro-
gram which collecting IP addresses and login user names of each node and auto-
matically updating. Furthermore, we develop the nodeinfo as mapping table for
IP addresses and login user names. The daemon program automatically collect
the IP addresses and user names of node devices in the cluster. Then, The dae-
mon program update two mapping tables. One is the hostfile, the other is the
nodeinfo.
Furthermore, we have implemented the wrapper script for process creation.
This wrapper script reads the nodeinfo and runs the SSH command with appro-
priate IP address of the target node and the user name on it.
Our Android application uses this wrapper script instead of SSH command
for the remote process creation. This wrapper script is invoked from mpirun
program that is invoked from our launcher application. The mpirun passes the
information of IP address of target remote node, but does not pass the user name
on it. This is because Open MPI assumes that the same user account exists on
every node in the cluster system. However, in our cluster system, the user name
might be different for each node, as described in the former section. Our wrapper
script refers to the nodeinfo, searches the user name on the specified node, and
invokes the SSH command with IP address and its user name.
This wrapper script is a shell script for mksh (a compatible shell of Korn
Shell) [12]. The mksh is the default shell program in Android OS. It is packaged
in our Android application, and installed in the same directory with the same
user ID. To make the mpirun program invoke this wrapper script instead of SSH,
our application executes the mpirun with ‘mca plm rsh agent’ option, by using
ProcessBuilder class.
 Realization and Preliminary Evaluation of MPI Runtime Environment 415

4 Evaluation Results
In our MPI runtime environment, parallel process is started from the Android
virtual machine, there is a possibility that overhead of parallel processing may
be generated. The overhead caused by starting MPI parallel program from the
Android application might affect the performance of parallel processing.
In this section, in order to measure the overhead of our MPI runtime envi-
ronment, we compare the performance of our new MPI runtime environment
with that of our previous one in our previous MPI runtime environment which
requires super-user authority [5].
We build the cluster system that consists of four Huawei MediaPad M3 for
evaluation. Table 1 shows the specification of the device. It is a tablet computer
with 8-core CPU, running Android OS 7. It can communicate with other devices
by using 5 GHz Wi-Fi (IEEE 802.11ac). As the access point of Wi-Fi, we use
Aterm WG2600HP2 manufactured by NEC Corporation.
For the evaluation, we use the qn24b benchmark [13] and the Himeno bench-
mark [14].
The qn24b benchmark is a solution program of the N-queen problem which
is one of combinatorial optimization problems. We use the qn24b benchmark
parallelized using MPI and execute it with problem size N set to 17.
The Himeno benchmark is a benchmark program that measures the process-
ing speed of the main loop of the Poisson equation solver by Jacobi’s iterative
method. The Himeno benchmark is said to be the benchmark whose computa-
tional performance is highly affected by the memory access performance bench-
mark. In this evaluation, the size of a matrix used in the Himeno benchmark
is (i, j, k) = (257, 257, 513). The Himeno benchmark automatically determines
the number of iterations of the main loop so that the execution time of the
benchmark is 60 s, but we fix the loop iteration count to 100 in order to fix the
calculation amount, in order to fix the conditions of each measurement.

Table 1. Specification of Android Cluster node

Device Huawei MediaPad M3

CPU HiSilicon Kirin 950
8 cores (1.81 GHz × 4, 2.3 GHz × 4)
Memory LPDDR4 SDRAM 4 GB
OS Android 7.0
Network IEEE 802.11 a/b/g/n/ac

Figure 4 shows the execution time of qn24b benchmark. In the figure, Pro-
posal indicates the execution time of our proposed system, and Conv. indicates
that of our previous system. The horizontal axis is the number of nodes and
process. For example, ‘1(8)’ refers to 1 node with 8 processes.
416 M. Nissato et al.

When comparing the execution time of Conv. and Proposal, the difference in
execution time is the largest when it is 1 node, and the execution time of Conv.
is about 20% less execution time of Proposal. Furthermore, a difference between
the execution time is shortest when it is 3 nodes, the execution time of Conv. is
about 6% less than the execution time of Proposal.
Figure 5 shows the computation FLOPS performance of Himeno benchmark.
The difference between the MFLOPS value is the largest at one node, and the
MFLOPS value of Proposal is about 12% less than MFLOPS value of Conv. The
difference between MFLOPS values at two nodes is about 3%, and at 3 nodes
and 4 nodes, the difference between MFLOPS values is about 0.9%, so there are
no significant differences between Conv. and Proposal.

Fig. 4. Execution time of qn24b benchmark

Fig. 5. MFLOPS performance of himeno benchmark

 Realization and Preliminary Evaluation of MPI Runtime Environment 417

The result of qn24b shows that our proposed system has lower performance
than our previous one in all cases of the number of nodes and processes. However,
as the number of nodes increased, the difference in performance tends to decrease.
On the other hand, the result of Himeno benchmark shows that our proposed
system has almost same performance with our previous one in the case of 3 and
4 nodes. In the case of 1 node and 2 nodes, the performance of our previous is
lower than that of our previous.
By the measurement results, we found that, although our proposed method
realizes MPI parallel processing without super-user authority, it is necessary to
improve performance.

5 Conclusion

In this paper, we realized a parallel processing environment using Open MPI

on Android OS. In our system, MPI parallel programs are launched from user-
level Android application to execute on Android OS. Therefore, any super-user
authority is not required.
Furthermore, we evaluated the performance of parallel processing by using
our system, comparing with our preceding system that requires the super-user
authority.
As a result of performance evaluation, we confirm that the difference of execu-
tion result between our system and our preceding system becomes the maximum
at qn24b benchmark executed on one node. The execution time of qn24b bench-
mark on our system is at most about 20% slower than the execution time of
our preceding system. In our system, we run our Android application on each
node of the Android Cluster, when running benchmarks. We think that due to
the Android application consumed the CPU resources of each node, the execu-
tion time become slower than the preceding system. As the result of Himeno
benchmark, we confirm that MFLOPS value of our system lower than that of
preceding system about 12%, when the benchmark executed on 1 node. How-
ever, there were no significant difference in the MFLOPS value in the evaluation
results of more than 2 nodes. The reason for this is that the influence of the con-
sumption of CPU resources by Android application is small, due to the Himeno
benchmark is a benchmark which the memory bandwidth of the systems greatly
affects the execution result. Detailed analysis of this evaluation result is our
future task.

Acknowledgements. This work was supported by JSPS KAKENHI Grant Numbers

16K00068, 17K00072.
418 M. Nissato et al.

References
1. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2016–
2021 White Paper. https://www.cisco.com/c/en/us/solutions/collateral/service-
provider/visual-networking-index-vni/mobile-white-paper-c11-520862.pdf
2. Büsching, F., Schildt, S., Wolf, L.: DroidCluster: towards smartphone cluster
computing—the streets are paved with potential computer clusters. In: Distributed
Computing Systems Workshops (ICDCSW), pp. 114–117 (2012)
3. Hinojos, G., Tade, C., Park, S., Shires, D., Bruno, D.: BlueHoc: Bluetooth Ad-Hoc
network Android distributed computing. In: International Conference on Parallel
and Distributed Processing Techniques and Applications (PDPTA), pp. 468–473
(2013)
4. MPI Forum. https://www.mpi-forum.org. Accessed 9 Nov 2018
5. Sawada, Y., Arai, Y., Ootsu, K., Yokota, T., Ohkawa, T.: Performance of Android
cluster system allowing dynamic node reconfiguration. Wirel. Pers. Commun.
93(4), 1067–1087 (2017)
6. Gabriel, E., et al.: Open MPI: goals, concept, and design of a next generation MPI
implementation. In: Proceedings of the 11th European PVM/MPI Users’ Group
Meeting, pp. 97–104 (2004)
7. Android Developers. https://developer.android.com/. Accessed 13 Nov 2018
8. Qian, X., Zhu, G., Li, X.F.: Comparison and analysis of the three programming
models in Google Android. In: First Asia-Pacific Programming Languages and
Compilers, Workshop (APPLC) in Conjunction with PLDI 2012, pp. 1–9 (2012)
9. Git repositories on android. https://android.googlesource.com/platform/bionic.
Accessed 13 Nov 2018
10. Bovet, D.P., Cesati, M.: Understanding the Linux Kernel, 3rd edn. O’Reilly, New-
ton (2005)
11. OpenSSH. https://www.openssh.com/. Accessed 13 Nov 2018
12. MirOS: mksh. http://www.mirbsd.org/mksh.htm. Accessed 13 Nov 2018
13. Kise, K., et al.: Solving the 24-queens problem using MPI on a PC cluster. Technical
report UEC-IS-2004-6, Graduate School of Information Systems, The University
of Electro-Communications, June 2004
14. The himeno benchmark. http://accc.riken.jp/supercom/documents/himenobmt/.
Accessed 13 Nov 2018
 Accelerating the Digital Transformation
of Business and Society Through Composite
Business Ecosystems

Shridhar Choudhary1(&), Ian Thomas2, Mehdi Bahrami3,

and Motoshi Sumioka1
1
Fujitsu Laboratories Ltd., Kawasaki, Japan
{s.choudhary,sumioka.motoshi}@jp.fujitsu.com
2
Fujitsu RunMyProcess, 3 rue de Gramont, Paris, France
[email protected]
3
Fujitsu Laboratories of America, Inc., Sunnyvale, CA, USA
[email protected]

Abstract. We believe that the increasing availability of web-connected busi-

ness and technology services is rapidly leading us toward an era of ‘composable
businesses’ in which people will be able to create new value by simply linking
together existing assets and resources. Given the current economic disruption
caused by the rapidly decreasing lifespans of traditional companies, a paradigm
shift on this scale could have profound consequences for the way in which
economies and societies function. We therefore believe it is important to ensure
that this new environment is open, supports mass participation and enables
everyone to contribute towards - and share in the benefits of – new economic
networks. In this paper, we therefore propose a set of technologies and practices
to empower distributed groups of people to co-create, implement and operate
new web-scale business models as “composite business ecosystems”.

Keywords: Digital transformation
Ecosystems
Business modelling

1 Introduction

The rapidly accelerating pace of change caused by globalization and the widespread
use of digital technologies is having a significant effect on the way in which companies
and governments create economic and social value. Over the past 60 years the average
lifespan of a Fortune 500 company has dropped from around 60 years in 1958 to less
than 20 in 2012 [1] and is forecast to fall to 12 years or less by 2027 [2]. In parallel we
have seen accelerating failures, with 52% of Fortune 500 companies either going
bankrupt, being acquired, or ceasing to exist during the last 15 years due to digital
disruption [3]. Given that large companies have traditionally provided significant and
long term social benefit in terms of employment, innovation and taxation these
apparent trends towards shorter lifespan and higher failure rates are a potentially
serious challenge for the future.
During this same period, however, we have also seen a significant increase in the
numbers of high value startups being created by small, entrepreneurial teams. In fact,
© Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 419–430, 2020.
https://doi.org/10.1007/978-3-030-15032-7_36
420 S. Choudhary et al.

the time taken for new ventures to reach a valuation of $1 billion appears to be
decreasing significantly - from about 20 years for a traditional Fortune 500 company to
as little as 18 months for a new digital startup [4]. What do these new companies do
differently that could explain these differences in outcome?
We believe that increasing digitization is creating opportunities for individuals and
companies to create value in powerful new ways. By using connectivity to interact with
end-users directly, digital organizations can provide anywhere, anytime services to
global audiences while simultaneously making them increasingly personal through
sophisticated analysis of individual behaviors [5]. By externalizing capabilities and
resources – through APIs, cloud services, peer business models and crowdsourcing –
they are able to build flexible, network-based business models that can scale expo-
nentially [4]. And by populating this network with small, self-directed teams in place of
traditional hierarchies they are able to move at unprecedented speed [6].
Today the technologies and practices used by these new organizations – e.g.
business model experimentation, agile development, data-driven decisions, fail fast,
networked teams and business-as-software - are largely emergent, unstructured and ad-
hoc. Furthermore they are highly reliant on scarce, highly skilled and expensive talent
with deep technology expertise. This makes access to the necessary practices difficult
for both traditional organizations – whose hierarchical operating models were built for
a very different era – and society as a whole – since the majority of people have
valuable domain expertise but lack advanced technology skills. Unfortunately this
creates the potential for single networks to capture unprecedented economic value
while employing a fraction of the resources of their industrial age predecessors –
potentially leading to imbalances of power, concentrations of wealth and economic
disenfranchisement.
In this paper we want to discuss opportunities to industrialize and democratize the
emerging practices of digital value creation through composite business ecosystems
[28] that support a more equitable and participatory economy. Our goal is to explore
the creation of a supportive, automated environment in which distributed teams of
people can work together to build and deploy new business models by composing ideas
and assets before taking a fair share of the resulting value (Fig. 1). In this way we hope
to first empower existing organizations to accelerate their transformation - by simpli-
fying and ‘packaging’ the necessary practices – before ultimately empowering anyone
to capture, share and monetize their skills, ideas or assets within the new digital
economy. In order to explore these ideas further the rest of this paper will consider:
• the major components we have developed to realize a first practical implementation
of this concept;
• the technologies, standards and practices that we have incorporated into each
component;
• an evaluation of our initial results in testing our concept; and
• a discussion of intended future work.
 Accelerating the Digital Transformation of Business 421

Fig. 1. Composite business platform

2 Major Components of Our Composite Business Platform

Our ultimate goal is to make the implementation of new business models as straight-
forward as sketching out ideas, finding and connecting resources, building any nec-
essary new services and pushing a button to make it live. In such an environment
companies would be able to easily and continuously test a wide range of ideas on an
almost daily basis, connecting and reconnecting value chains in different ways or
adding new capabilities using low code approaches and modelling languages such as
BPMN [8]. Furthermore, the reach of mobile devices in combination with the indus-
trialization of execution infrastructures – due to highly scalable clouds – means that
even simple ideas would have the potential to become global scale businesses in a very
short time.
While such an environment would have seemed unthinkable just a few short years
ago, we believe that accelerating industrialization of technology – coupled with a
movement towards smaller, more agile teams - is making these ideas increasingly
tractable. Underlying infrastructure is cheap, abundant and easily accessible. Teams
can quickly build micro services and make them available via application programming
interfaces (APIs). Modelling and low code platforms are making digital service
development accessible to ever wider groups of people. And in aggregate the resulting
APIs – across on premise, cloud and even IoT - are creating a global ecosystem of
concrete digital services which can form the building blocks necessary to model
complex digital value chains.
Our composite business platform therefore makes use of each of these advances to
focus on three major components, each of which builds on the one below:
1. Ecosystem Building: Creating a base ecosystem of already available assets and
skills that people can use to accelerate the scalable implementation of their idea;
422 S. Choudhary et al.

2. Fast Implementation and Feedback: Linking the ecosystem to a fast, low code
platform that distributed teams can use for building and connecting assets without
the need for deep technology skills; and
3. Collaborative Business Design: Wrapping both the ecosystem and the platform in a
collaborative business design environment that enables the assisted modelling,
testing, deployment and optimization of value chains.
These components can be seen in Fig. 2.

Fig. 2. Major components

We shall consider each in turn.

2.1 Ecosystem Building

Today there are billions of people, services and devices connected to the global net-
work – and this number is growing rapidly as the Internet of Things accelerates [9].
Today these resources are isolated and difficult to access, however, with any APIs, they
offer fragmented across different repositories and information silos. Equally impor-
tantly, while standards such as the open API specification [10] and the constrained
application protocol [11] can help facilitate technical interoperability, we observe that
the vast majority of metadata necessary to enable real world usage of APIs is still stored
in developer portals optimized for human consumption.
While such fragmentation makes services difficult to find and use we also believe
that it is unrealistic to build ‘one-repository-to-rule-them-all’ by enforcing order onto
the fundamentally chaotic and Darwinian nature of the Web. As a result, we believe
that any attempt to build a useful business ecosystem must be able to discover,
aggregate and organize data from many different locations in much the same way as a
search engine. As with crawling the Web, the primary tasks of gathering, understanding
 Accelerating the Digital Transformation of Business 423

and performing initial evaluation are too complex and time consuming to be imple-
mented via human analysis and curation; we therefore believe that future service
ecosystems must be primarily powered by automated discovery, categorization and
validation.
To address this need we have implemented a suite of technologies for crawling
distributed API repositories, inferring API usage information, generating and testing
usage templates and providing recommendations to consumers based on their intent
[12]. Other research has also looked to address this topic – for example [13] proposed a
method to extract API information from Github data, while [14, 15] used deep learning
across small API sets – but our technologies go beyond these examples by utilizing
deep learning to extract information from large scale API documentation.
By using the information we collect we are able to build a meta-repository of useful
data – such as metadata, endpoints, parameters and security requirements – before
performing automated validation and publishing of APIs for use. More specifically our
ecosystem building technologies can:
• Extract large volumes of information from API repositories: By using natural
language processing we are able to discover and transform API documentation into
structured metadata and knowledge graphs;
• Perform basic validation of API behavior: By utilizing the discovered information
we can automatically test and confirm the expected behavior of APIs, providing a
preliminary validation of their function; and
• Recommend one or more APIs to fulfil a particular purpose: By comparing user
input with the generated metadata we are able to filter APIs and predict which may
be suitable for the user’s purpose.
This suite of “API Learning” technologies forms the foundation of our ecosystem
building component, generating a repository of Open API Specifications and associated
metadata that can be used in the creation of higher order systems.

2.2 Fast Implementation and Feedback

Having a comprehensive ecosystem of services is a great start - but it is of little use if
we do not also provide the tools necessary to create value by composing and enriching
it. The way in which we approach this challenge can have significant implications,
however.
Effectively we will not be able to meet our goals for speed, agility and wide
participation if we continue to force people to painstakingly knit resources together
using low level coding and operational techniques. Even if we were to ignore our
participatory goals and enforce a high level of technical skill, there have been estimates
suggesting that by 2020 there will simply not be enough skilled developers to fulfil the
necessary demand [16].
As a result the second major component of our platform – which represents the core
implementation and runtime engine of our environment - is a low code platform that
complements our ecosystem by providing a modelling environment within which
small, agile teams can build, integrate and deploy a range of different kinds of digital
components and orchestrations as independent micro services [17]. This capability
424 S. Choudhary et al.

supports our goal in two ways – firstly by providing a concrete orchestration envi-
ronment for the execution of digital value chains and secondly by reducing the barriers
to building and publishing services to enhance the ecosystem. To fulfil this need we
have used our RunMyProcess platform as a foundation – a cloud-native, multi-tenant
environment that enables people to quickly build applications using drag and drop
tooling, process orchestration and a wide library of standard connectors for on premise,
cloud and IoT based resources [18].
Historically the pre-built integration connectors offered by RunMyProcess – over a
thousand in fact – were one of our users’ most highly valued features. But as we
attempt to encompass new digital ecosystems it becomes increasingly uneconomic to
manually build, test and evolve connectors over time. While this manual approach was
acceptable in a closed world of static enterprise architectures with a bounded set of
popular systems this clearly does not work for a more global, dynamic and business-
oriented ecosystem that is in a constant state of flux and evolution.
To address this issue we have evolved the RunMyProcess connector library to
become a window onto the meta-repository created by our API learning technologies,
using the output of API learning to automatically generate the equivalent connectors in
RunMyProcess. This provides users with a fully seamless way to compose new value
by linking their applications and processes to already available services within the
automated ecosystem - and it further creates an interesting future opportunity for highly
targeted and context specific service recommendations during modelling.
Together the ecosystem and low code platform are intended to provide the foun-
dation of our fast implementation and feedback component. Together they create a
workable foundation for generating and leveraging an ecosystem of distributed
resources.

2.3 Collaborative Business Design

As part of the accelerating progress of digitization, information technology has moved
from being a tool for internal efficiency to a whole new creative and disruptive medium
of its own. While tools for implementing enterprise efficiencies are commonplace, tools
for co-creating transformational value with others continue to be ersatz and ad hoc. As
a result the final component of our platform aims to give distributed groups of people a
supportive environment to co-create new business models that leverage the ecosystem
and low code foundations we have already discussed.
In considering the kind of business modelling support we need to provide we
looked at leading digital pioneers such as Amazon, Netflix and LinkedIn. In doing so
we observe that they operate extremely distributed, loosely coupled organizational
models consisting of networks of micro services operated by independent teams with
no central governance or control [19–21]. These models explicitly encourage the
emergence of small, entrepreneurial teams with a high degree of customer focus and an
organizational structure optimized for speed, flexibility and minimal cross-team com-
munication and coordination.
These organic operating models appear somewhat antithetical to classical enterprise
architecture practices. New digital organizations strive to minimize enterprise-wide
communication and coordination costs via networks of small teams, whereas traditional
 Accelerating the Digital Transformation of Business 425

enterprise architecture practices strive to enable enterprise-wide communication and

coordination via a complete model of the way in which the enterprise should operate.
As explained in [25] this latter approach worked tolerably well within the traditional
closed systems of simpler, slower moving hierarchical organizations but is not fit for
purpose in the open, participatory and fast evolving ecosystems we wish to facilitate.
As a result our goal is to enable distributed, possibly unrelated, groups of people to
co-create value in ways that exhibit the core characteristics of these new digital
organizations, namely: independent teams, offering useful business capabilities used by
many dynamic value chains optimized for the delivery of defined user needs. This
trend towards user-centric co-creation and value oriented methods for understanding
and constructing business models is also considered in [7].
To realize these needs we have combined a highly flexible existing collaboration
environment with an innovative business mapping technique called Wardley Mapping
[26, 27].
Our collaboration environment was created during research to help innovators –
such as entrepreneurs – overcome the difficulties of developing new ideas. It does this
by giving them an environment in which to capture their idea, find and engage potential
collaborators and work with them in a shared environment towards a common goal
(Fig. 3).

Fig. 3. Screenshot of an example project

But in order to turn these ideas into working businesses we need to be able to move
beyond ideation and extend our environment into business model design and strategy.
One very promising technique that covers both of these areas is Wardley Mapping [26,
27]. In this technique people focus on first understanding user needs before mapping
out the value chain of components necessary to realize them. A highly simplified
Wardley Map for delivering a cup of coffee is shown in Fig. 4.
426 S. Choudhary et al.

Fig. 4. An example Wardley map

Wardley maps exhibit several interesting characteristics when considered against

the requirements we outlined for co-creating new ecosystem-based business models;
firstly they are visual and allow development of shared understanding; secondly they
are anchored with user need and so keep people focused on value; thirdly they visualize
the necessary value chain, using the y axis to demonstrate how visible components are
to the user (visible to invisible components); fourthly they use the x axis to estimate
component maturity and enable build-buy-use decisions (genesis to utility); and finally
each capability on the map can itself be defined by a map – fitting neatly into the
concept of independent services which can be composed into higher order value.
In our implementation we have created a lightly customized version of Wardley
maps which standardizes the nature of value chain components. Today Wardley
mapping leaves the nature of the components shown in a value chain totally open to the
person creating the map, making it potentially difficult to compare components across
maps and implement recommendations for reuse. To facilitate the bottom up growth of
a comparable ecosystem of services, however, we have explicitly adopted business
capabilities for modelling the components within the value chain. Business capabilities
are defined as an organizations ability to perform a certain activity from a functional
perspective [22] and have been a useful construct for aligning business stakeholders
within technology in traditional enterprise architecture [23, 24]. As a result we believe
that business capabilities can offer a consistency of approach that will simplify the
ecosystem and future work on automated capability matching and recommendation.
Finally we believe that value chains of business capabilities also close the gap
between concepts and implementation, enabling us to more easily map business
functions to concrete APIs for implementation by independent teams (Fig. 5). To date
we have carried out simple experiments that interchange capability definitions in the
business modelling environment with APIs for implementation in our platform; this
continuity from business modelling to implementation will be a major area for future
work.
 Accelerating the Digital Transformation of Business 427

Fig. 5. Mapping from ideation to implementation via capabilities

3 Results

In this section we present individual results for each component of the platform
together with some early indications of the usefulness of the end-to-end concept.

3.1 Ecosystem Building

To evaluate the potential of our ecosystem building technologies we used API learning
to gather information from a range of publically available developer portals such as
programmable web [29], API Guru [30] and API harmony [31]. We evaluated the
correctness of the extracted information both manually and automatically through a
comparison of the original HTML documentation and the generated JSON file [12].
Our evaluation results showed that we were able to correctly extract 86.75% of API
endpoints and 81.29% of API attributes. We were also able to perform automated API
validation by calling each API endpoint to check the response code, resulting in 1,453
active (76%) and 470 inactive (24%) APIs. The active APIs were published into our
catalogue for downstream use. We believe this represents an acceptable start for our
ecosystem building and plan to continue refining our approach to increase accuracy.

3.2 Fast Implementation and Feedback

To maximize the value of our ecosystem we wanted to make it seamlessly available to
users at point of need, i.e. when building systems within our platform. To do this we
were able to use the metadata collected by our ecosystem building technologies to
significantly increase the number of pre-configured connectors available to users. By
processing OAS API specifications from 992 validated APIs we were able to generate
17,795 connectors within the RunMyProcess environment, harvested from multiple
sources and representing a huge increase on the 2,500 we were able to maintain
manually. While this larger library of connectors opens up significant opportunities for
428 S. Choudhary et al.

users to orchestrate external resources in general, we believe that the ability to use these
resources directly within the low code environment maximizes their utility.

3.3 Collaborative Business Design

In a recent event organized by the One Japan1 organization we were able to collect
direct feedback on our collaboration environment from around 200 young Japanese
employees whose aim is to create disruptive innovation. Through this engagement we
were able to gather positive feedback on the current ideation features - like task
management, file sharing and people recommendations – which were considered to
provide useful support during the early stages of idea development. At the same time
we also received important input for over 100 potential new features, such as real-time
editing, oauth2 authentication and better information filtering.

3.4 Overall Approach

We presented our end-to-end concept and technologies at Fujitsu Forum Munich in
November 20182, using an early stage prototype to gather input. As part of this process
we interviewed more than 20 CEOs and CIOs from 8 different countries, looking for
feedback on whether their organizations suffered from difficulties implementing more
disruptive innovation. All of them agreed with the need for faster ways to test and
implement business ideas and were impressed with our concept due to its scope from
business design to implementation; 6 of them went further and said that they have an
active need to implement the kind of digital ecosystems we were demonstrating while
the remaining 14 wanted to explore the idea more with colleagues.

4 Conclusions and Future Work

In this paper we have attempted to summarize our initial work towards a more col-
laborative, flexible and democratic platform for realizing value from new digital
ecosystems. To do this we have shown the major components we are working on to
realize this environment and discussed our early research and implementation work.
Through this work we have concluded that the concept we describe is both feasible and
– based on feedback – potentially of great value for both entrepreneurs and large
enterprises.
Our work to date has mainly focused on the individual components, however, and
there is much still to do to improve these and work on the integration necessary to
create an end-to-end environment for turning ideas into running businesses. Some of
the key future work we are focused on includes:

1
https://onejapan.jp/.
2
http://www.fujitsu.com/global/microsite/forum-2018/.
 Accelerating the Digital Transformation of Business 429

• Accuracy and usability: Improving the accuracy of our ecosystem building tech-
nologies while extending their range into additional areas - e.g. serverless reposi-
tories, IoT management hubs, on premise legacy systems – and adding more
context via a semantic layer that helps people make sense of the ecosystem.
• Integration and automation: Continually increasing the level of integration,
automation and recommendation across components to enable truly end-to-end
progress from business model through to implementation and operations with
minimum skill requirements.
• Rapid feedback loops: Using the data captured throughout the process to implement
automated feedback loops for business model optimization post deployment – using
predictions and recommendations to enable continual alignment between intentions
and observed operational behavior.
• Reward mechanisms: Exploring the use of cryptocurrencies and other financial
tools to ensure that participants within the distributed ecosystem are fairly rewarded
for their contributions.
• AI-assisted implementation: Use of intelligent agents and conversational pro-
gramming interfaces which allow users to focus on strategy while maximizing the
delegation of analysis and implementation work to our environment.

References
1. Klerk, E., et al.: Disruptive forces in Europe: a primer credit suisse research note (2017)
2. Anthony, S., et al.: Corporate longevity forecast: create destruction is accelerating (2018).
https://www.innosight.com/insight/creative-destruction/. Accessed 10 Dec 2018
3. Constellation Research: Sneak Peeks from Constellation’s Futurist Framework and 2014
Outlook on Digital Disruption. https://www.constellationr.com/blog-news/research-
summary-sneak-peeks-constellations-futurist-framework-and-2014-outlook-digital. Acces-
sed 10 Dec 2018
4. Ismail, S.: Exponential Organizations: Why New Organizations Are Ten Times Better,
Faster, and Cheaper than Yours (and What to do About it). Diversion Books, New York
(2014)
5. Wired: This is how Netflix’s top-secret recommendation system works. https://www.wired.
co.uk/article/how-do-netflixs-algorithms-work-machine-learning-helps-to-predict-what-
viewers-will-like. Accessed 08 Dec 2018
6. Hoffman, A.: Amazon delivers a secret recipe for agility, adaptability, and dominance.
https://www.inc.com/quora/amazon-delivers-a-secret-recipe-for-agility-adaptability-
dominance.html. Accessed 06 Dec 2018
7. Ramljak, D.: Business models and value oriented service design elements in ecosystem
architecture. In: 25th International Conference on Software, Telecommunications and
Computer Networks (SoftCOM), pp. 1–6. Split (2017)
8. Allweyer, T.: BPMN 2.0: Introduction to the Standard for Business Process Modeling.
Books on Demand, Norderstedt (2016)
9. Statista: Internet of Things (IoT) connected devices installed base worldwide from 2015 to
2025 (in billions). https://www.statista.com/statistics/471264/iot-number-of-connected-
devices-worldwide/. Accessed 09 Dec 2018
430 S. Choudhary et al.

10. Linux Foundation: The open API specification. https://github.com/OAI/OpenAPI-

Specification. Accessed 05 Dec 2018
11. Shelby, Z., et al.: RFC 7252: Constrained application protocol (CoAP). IETF request for
comments (2014)
12. Bahrami, M., Park, J., Liu, L., Chen, W.: API learning: applying machine learning to
manage the rise of API economy. In: Companion Proceedings of the Web Conference,
Geneva, pp. 151–154 (2018)
13. Wittern, E., Muthusamy, V., Laredo, J., Vukovic, M., Slominski, A., Rajagopalan, S.,
Jamjoom, H., Natarajan, A.: API harmony: graph-based search and selection of apis in the
cloud. IBM J. Res. Dev. 60(2–3), 12:1–12:11 (2016)
14. Li, D., Salonidis, T., Desai, N.V., Chuah, M.C.: DeepCham: collaborative edge-mediated
adaptive deep learning for mobile object recognition. In: IEEE/ACM Symposium on Edge
Computing (SEC), Washington, D.C., pp. 64–76 (2016)
15. Li, D., Wang, X., Kong, D.: DeepRebirth: accelerating deep neural network execution on
mobile devices. In: 32nd AAAI Conference on Artificial Intelligence, New Orleans,
pp. 2322–2330 (2018)
16. Code.org: 1,000,000 more jobs than students by 2020. https://web.archive.org/web/
20180122182035/https://code.org/images/cs-stats/more-jobs-than-students.png. Accessed
07 Dec 2018
17. Lewis, J., Fowler, M.: Microservices resource guide. https://www.martinfowler.com/
microservices/. Accessed 07 Dec 2018
18. RunMyProcess. https://www.runmyprocess.com. Accessed 10 Dec 2018
19. Gigaom: The biggest thing amazon got right: the platform. https://gigaom.com/2011/10/12/
419-the-biggest-thing-amazon-got-right-the-platform. Accessed 11 Dec 2018
20. Ngnix: Adopting microservices at netflix: lessons for architectural design. https://www.
nginx.com/blog/microservices-at-netflix-architectural-best-practices/
21. InfoQ: From a monolith to microservices + REST: the evolution of linkedin’s service
architecture. https://www.infoq.com/presentations/linkedin-microservices-urn. Accessed 11
Dec 2018
22. Freitag, A., Matthes, F., Schulz, C., Nowobilska, A.: A method for business capability
dependency analysis. In: International Conference on IT-Enabled Innovation in Enterprise
(ICITIE), Sofia (2011)
23. Bondel, G., Faber, A., Matthes, F.: Reporting from the implementation of a business
capability map as business-IT alignment tool. In: 22nd International Enterprise Distributed
Object Computing Workshop (EDOCW), Stockholm, pp. 125–134 (2018)
24. The Open Group: The TOGAF standard, version 9.2. (2018)
25. Zimmermann, A., Schmidt, R., Sandkuhl, K., Jugel, D., Bogner, J., Möhring, M.: Evolution
of enterprise architecture for digital transformation. In: 22nd International Enterprise
Distributed Object Computing Workshop (EDOCW), Stockholm, pp. 87–96 (2018)
26. Wardley, S.: Anticipating the future: an introduction to value chain mapping. http://www.
youtube.com/watch?v=NnFeIt-uaEc. Accessed 08 Dec 2018
27. Wardley, S.: Situation normal, everything must change. http://www.youtube.com/watch?v=
Ty6pOVEc3bA. Accessed 08 Dec 2018
28. Thomas, I., Gaide, S., Hug, M.: Composite business ecosystems for the web of everything:
using cloud platforms for web convergence. In: 7th International Conference on Innovative
Mobile and Internet Services in Ubiquitous Computing (IMIS), Taichung, pp. 535–540
(2013)
29. Programmable Web. https://www.programmableweb.com/. Accessed 10 Dec 2018
30. APIs.guru. https://apis.guru/. Accessed 10 Dec 2018
31. API Harmony. https://apiharmony-open.mybluemix.net/public. Accessed 10 Dec 2018
 An IP Multimedia Subsystem Service
Discovery and Exposure Approach
Based on Opinion Mining by Exploiting
Twitter Trending Topics

Armielle Noulapeu Ngaffo(&), Walid El Ayeb, and Zièd Choukair

Mediatron Lab, Higher School of Communications of Tunis, Tunis, Tunisia

{armielle.ngaffo,walid.elayeb,
zied.choukair}@supcom.tn

Abstract. Being one of the most solicited content (opinions) sharing platforms,
Twitter is a granary of information serving as a base for our service
discovery/exposure approach proposed in this paper. The growth of data caused
by the internet growth has led to the birth of a growing number of services,
making the task difficult for telecommunication operators in competition. In this
paper, we propose a dual service discovery/exposure approach to reduce the gap
between offered services and subscribers’ needs in an IMS context. This
approach is based on opinion mining related to Twitter trending topics in order
to estimate the sensitivity of the target user to a service or another. Compared to
both the classic approach and the collaborative service discovery/exposure
approach, our results show an improvement in the accuracy and error of the
service targeting at the target user’s starting phase on the operator’s network.

1 Introduction

IP Multimedia Subsystem (IMS) is a platform designed primarily for seamless access to

a rich and varied range of multimedia services [1]. This being said, the classic service
discovery/exposure approach within IMS has shown its limits since the operator’s
network is the only source of information to estimate the target user’s sensitivity to a
service or another. Indeed, the classic service discovery approach presents poor per-
formances [2], mostly when the user is a beginner on operator’s network since it does
not hold any log about his use of service, no data on his consumption habits in terms of
service.
Moreover, social networks are nowadays a very solicited way by Internet users who
come there to share their points of view concerning current topics, to share their
emotions, their feelings, their experiences about purchased products or consumed
services. By hyperactivity of Internet users, social networks are positioned as true
granaries of information, thus constituting the opportunity to adjust the service offering
by further reducing the gap between offered and needed services. In order to overcome
the limitations of the classic service discovery/exposure approach within IMS, we
address in this paper a proposition of an opinion mining-based service
discovery/exposure approach. Indeed, one of the major difficulties negatively
© Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 431–445, 2020.
https://doi.org/10.1007/978-3-030-15032-7_37
432 A. N. Ngaffo et al.

impacting the performance of existing service discovery/exposure approaches is the

lack of information about the target user’s consumption habits of the service, defeating
information about the user’s sensitivity relatively to a service or another. It should be
remembered that the main purpose of service intermediation is to return to the service
requestor the results in terms of relevant services aligned on his needs. For its popu-
larity [3–7], Twitter social network serves us as a data carrier by its Trending Topics
(TTs) that we explore to estimate the sensitivity of the user about a given service.
Our approach has the main advantage of offering relevant services when the service
requestor is a beginner on the operator’s network. Indeed, our approach mines massive
data on Twitter social network to compensate for the information deficit of the oper-
ator’s network. This improves the accuracy and error of service targeting under these
conditions.
In the rest of this paper, Sect. 2 introduces IMS basic architecture, IMS functional
entities, and existing service discovery/exposure approaches. Section 3 presents the
model of our approach and at the end Sect. 4 presents our experiments and the results
obtained.

2 State of Art

2.1 Presentation of IMS-Related Entities

IP Multimedia Subsystem is a platform that allows subscribers to access a rich and
varied range of multimedia services regardless of the used access technology (2G, 3G,
etc.). According to the technical specifications [1] of IMS, there are two types of nodes.
Namely the Call Session Control Functions (CSCF) nodes and the Subscriber Server
nodes (HSS: Home Subscriber Server) whose main function is to assist CSCFs. Fig-
ure 1 shows the basic IMS architecture in which the IMS layer houses the entities
hosting subscriber profile information, log of service usage, and service description.
This is the case of HSS which contains information on subscriber profiles; service
profiles hosted by the Applications/Services layer and based on the Subscription Profile
Repository (SPR) to inquire about subscriber subscription information. SLF (Sub-
scriber Location Function) makes possible the matching (when in the network there are
several HSSs) between a subscriber and the HSS in which his profile is hosted. All
these entities are involved in the classic service discovery/exposure approach through
the verification of conditions constituting the iFC (Initial Filter Criteria) for addressing
the service request to the appropriate application server.
The quantity and quality of information about the subscriber’s behavior, prefer-
ences, needs, service consumption patterns etc. are crucial to the relevance of the
results in the service discovery/exposure process. Although all IMS entities host a
significant amount of data on the subscriber which allows the classic approach of
service discovery/exposure to excel in these conditions, when the subscriber starts on
the operator’s network in the absence of data relating to his needs, his preferences, his
log of service usage, the classic approach presents poor performance in terms of service
targeting accuracy. Our approach overcomes this limitation by sourcing massive data
 An IP Multimedia Subsystem Service Discovery 433

(using opinion mining/sentiment analysis) from social networks specifically on Twitter

to learn more about the subscriber’s sensitivity to a given service.

Fig. 1. The basic IMS architecture [1]

In the next subsection we will look at existing service discovery/exposure

approaches to highlight the contribution of our approach to existing service discovery/
exposure models.

2.2 Existing Service Discovery/Exposure Approaches

Service Discovery/Exposure Based on Keyword Clustering, the Subscriber Profile

and the Subscriber’s Logs of Use of Service. [8–10] propose an approach of service
discovery based on keyword clustering and user profile. El Ayeb et al. in [2] propose a
service discovery/exposure approach based on the subscriber profile and service
description from which a similarity matrix is developed. Authors apply to this
subscriber-service similarity matrix, the Hungarian algorithm to determine the most
optimal subscriber-service assignment. However, this approach, which is strongly
focused on subscriber data held by the operator’s network, displays poor performance
during the subscriber’s registration phase, meaning when the service requestor starts on
the operator’s network. The poor performance in that case is due to the fact that the
operator’s network does not have enough information about the subscriber at that
moment to return relevant results aligned on his needs, preferences and requirements.
Service Discovery/Exposure Based on Opinion Mining. Bhor et al. in [3] propose a
service exposure approach based on the exploration of user data collected on Facebook,
Twitter and Instagram platforms in order to classify users according to their preferences
to adjust the service proposition. Authors in [11] offer a similar approach by collecting
tweets associated with a TV program left by viewers on Twitter in order to capture
434 A. N. Ngaffo et al.

viewers’ feelings and appreciation of the TV program in question. In [5], authors

collect the tweets relating to the provision of the internet service by TELKOM1 internet
access provider to estimate the customer churn rate. In [12] and [13] authors collect
data and feedbacks left on Amazon2 and Flipkart3 to respectively rate products and
estimate iPhone sales. Moreover in [14] the authors collect the data on Twitter and
analyze them in order to probe the expectations of the users regarding the opening of a
new Pizzeria.
Previous approaches propose service discovery models for some of them and ser-
vice exposure models for others, but no dual opinion-mining based service
discovery/exposure model. Moreover, the existing approaches are strongly focused on
the activity (comments, shares, likes …) of the target user and thus unable to display
good performance when the activity of the target user on the social networks is not
sufficiently intense. In addition, previous approaches adjust the provision of a single
service. What motivates us to propose a dual model of service discovery/exposure
based on opinion mining to reduce the gap between the needs of the target user and the
multitude of existing services in the IMS context. For that, our approach is based on the
opinion mining of the target user or his friends’ feelings (meaning sentiment analysis of
the target user’s opinion influencers whether the activity of the target user on the social
networks is not sufficiently intense) with regard to Twitter Trending topics in order to
estimate the sensitivity of the target user for a given service.

3 Our Opinion Mining-Based Service Discovery/Exposure

Approach

3.1 Presentation of Our Opinion Mining-Based Service

Discovery/Exposure Model
This subsection presents our dual service discovery/exposure model. Indeed, Fig. 2
presents our model based on the opinion mining of Twitter users regarding Trending
Topics (TTs). The goal of our approach is to estimate the sensitivity of a user to a given
service through his point of view concerning trending topics (for which he gave his
opinion) on Twitter social network. Moreover, whether the target user does not display
hyperactivity regarding these current topics, then we explore the point of view of his
opinion influencers in order to get as close as possible to that of the targeted user. The
sought sensitivity of the target user to a given service is thus estimated by a transpo-
sition of his sensitivity (or that of his opinion influencers) on trending topics for which
he gave his opinion. Opinion influencers of the target user are those users (Twitter
friends of the target user) with whom the target user shares similar opinions, similar
preferences and therefore similar sensitivities to the same services. Then these influ-
encers of the target user opinion allow us to estimate implicitly the sensitivity of the

1
www.telkom.co.za/.
2
www.amazon.fr/.
3
www.flipkart.com/.
 An IP Multimedia Subsystem Service Discovery 435

latter to a given service when his activity on social networks is poor. This sensitivity
expressed through a service rating score (a value assigned to a service) is the degree of
concordance of the service in question with the needs and preferences of the target user.
Hypothesis 1. The target user has a Twitter account with a Twitter friends’ network
and displays a rich activity with his friends.
Hypothesis 2. The target user’s friends display a rich Twitter activity with Twitter
trending topics.

Fig. 2. Our proposed opinion mining-based service discovery/exposure model

As illustrated in Fig. 2, our opinion mining-based service discovery/exposure

model starts with the determination of the top-K opinion influencers of the target user.
This is the subject of the next subsection.

3.2 Our Estimating Process of Target User’s Top-K Opinion Influencers

This subsection presents the process of opinion influencers computing. Indeed, Fig. 3
presents our calculating process of opinion influencers of the target user. Estimating the
opinion influence within Twitter is expressed through criteria such as positive retweets
issued by the target user on the posts of his friends (who are potential opinion influ-
encers of the target user), tweets marked as target user favorites on the posts of his
friends, and in the end the following relationships that the target user has with his
friends. Indeed in [15], the influence exerted by a followee (a friend) on the follower
(the target user) is manifest through the analysis of the target user retweets, favorites,
mentions, the number of followers, the number of followees but without any precision
on the negative or positive nature of the influence. In our process, we rather consider
the opinion influence, to mean the positive influence exerted by the target user’s
Twitter friends on the latter.
436 A. N. Ngaffo et al.

Fig. 3. Our estimating process of opinion influence

The estimating method of the opinion influence is built on a data pre-processing

step that involves data extraction and recovery of only positive retweets of the target
user. In this step, we perform a sentiment analysis using an opinion word lexicon in
which each word is associated to a score expressing the intensity and the positive,
negative or neutral meaning of the word. Then the opinion score of a retweet is
calculated as follows:
Xpositivewordtotal Xnegativewordtotal
w¼1
scorew  wordw  v¼1
scorev  wordv : ð1Þ

Our approach heavily relies on the exploitation of positive retweets and target user
favorites from which we estimate:
• The probability the target user retweets a tweet from his Twitter friend
• The probability the target user marks as favorites, a tweet of his Twitter friend
• The opinion influence score of the target user’s friend using our EPR (Enhanced
PageRank) algorithm.

3.3 Estimation of Opinion Influence Probability and the Favorite Tweet

Marking Probability

Mathematical Model. The unidirectional and asymmetric nature of Twitter relation-

ships [7] leads us to model friendships on Twitter using G = (U, E) a directed graph.
Where U is the aggregate of nodes meaning the Twitter users, and E represents the
aggregate of directed arcs meaning the “following relationships” between users. Let eij
be the directed arc expressing the relationship in which ui the target user “follows” uj
with ui ; uj 2 U. Figure 4 presents the relationship between ui and uj where the tuple
(hij ; Xij ; xij , yij ) weighting eij 2 E. hij and Xij as parameters to estimate, are severally the
probability the user ui retweets a tweet of uj and the probability the user ui marks as
favorite a tweet of uj. Let xij be the count of positive retweets made by ui on the tweets
of uj ; and yij the count of favorite tweets of ui related to the tweets of uj .
Hypothesis 3. Our study is led assuming the behavior of Twitter users over the period
of observation is constant.
 An IP Multimedia Subsystem Service Discovery 437

Hypothesis 4. We consider that hij and Xij are both uniform probability distributions.

Fig. 4. Our model of Twitter relationship

The Problem Statement. Let ui be the target user having N Twitter Friends. With:
• h = (hi1 ; . . .; hij ; . . .; hiN ) representing the tuple of probabilities of positive retweets
that ui made severally on the posts of u1 ; . . .; uj ; . . .; uN users. In order to shorten the
scripture, we will write h = (h1 ; . . .; hj ; . . .; hN ).
• X = (Xi1 ; . . .; Xij ; . . .; XiN ) the tuple of probabilities of favorite tweets that ui indi-
cated severally on the posts of u1 ; . . .; uj ; . . .; uN users. Afterwards, we will write
X = (X1 ; . . .; Xj ; . . .; XN ).
• x = (xi1 ; . . .; xij ; . . .; xiN ) the tuple of counts of positive retweets that ui made sev-
erally on the posts of u1 ; . . .; uj ; . . .; uN users. Thereafter we will write
x = (x1 ; . . .; xj ; . . .; xN ).
• y = (yi1 ; . . .; yij ; . . .; yiN ) the tuple of counts of favorite tweets indicated by ui sev-
erally on the posts of u1 ; . . .; uj ; . . .; uN users. In order to shorten the scripture, we
will write y = (y1 ; . . .; yj ; . . .; yN ).
We perform MLE (Maximum Likehood Estimation) assuming that the target user
performs positive retweets and/or favorite tweet markings on the postings of his Twitter
friends with an equiprobable manner, and to mean with equal chances. We proceed to
the estimation of probabilities hij et Xij by using the Maximum Likehood method [16]
and obtain ^hijMLE and X
^ ijMLE estimated probabilities as follows:

^hijMLE ¼ P xj ^ ijMLE ¼ P yj
and X : ð2Þ
N N
j¼1 xj j¼1 yj

Now we can deduce the opinion influence score of the target user’s Twitter friends.

3.4 Opinion Influence Score Computing Using Our Enhanced PageRank

Algorithm

Opinion Influencing Power of a Target User’s Twitter Friend. The standard PR

(PageRank algorithm) makes it possible to evaluate the influencing power of a Twitter
user essentially being based on his “following relationships” [17]. In our case, we
rather estimate the opinion influencing power of a Twitter user by adding to the
standard PR, the contribution of involvement of both positive retweets and favorite
tweet markings. Figure 5 presents the principle of our Enhanced PageRank algorithm
that we perform to estimate the opinion influencing power of each Twitter friend of the
target user. Let u1 ; . . .; uj ; . . .; uN be the users illustrated by the nodes. The size of node
is proportional to the opinion influencing power (of each user) written s1 ; . . .; sj ; . . .; sN :
438 A. N. Ngaffo et al.

Let Pji ¼ ððpr Þji ; ðpF Þji ; ðpf Þji ÞT be the tuple of probabilities corresponding to the influence
uj exerted on ui the target user following three dimensions such as:

• ðpr Þji : represents the aforementioned probability ^hijMLE

• ðpF Þji : represents the aforementioned probability X ^ ijMLE
j
• ðpf Þi : represents the probability the user uj is followed by the target user ui . It is
equal to Count of 1friends i
ðÞ

Notation. ERP(uj Þ ¼ sj is the power of opinion influence of uj . As stated by our

Enhanced PageRank algorithm, the power of opinion influence S of a given user u is
defined as below:

ð3Þ

Where Mr , MF , Mf are transition matrices:

 
• Mr ¼ ^hijMLE : the matrix of probabilities associated to positive retweets
NN
amongTwitterusers
• MF ¼ X ^ ijMLE : the matrix of probabilities associated to favorite tweets among
NN
Twitterusers 
• Mf ¼ Count of 1friends i : the matrix of probabilities associated to the
ð Þ NN
“following relationships” among users
• ɣ, b, k coefficients are positive weights such that ɣ + b + k = 1
• q means the damping factor [18] employed in the standard PR.

Fig. 5. Our enhanced PageRank algorithm illustration

 An IP Multimedia Subsystem Service Discovery 439

Opinion Influence Score the User uj Exerted on ui . We express the opinion influ-
ence exerted by uj on ui , as the result of the combination of several kinds of influence
according to the three following dimensions:
• The favorite tweet marked by ui on the uj tweet with an influence defined by the
inflðj!iÞF score as below:

^ ijMLE  sj :
inflðj!iÞF ¼ ðpF Þji  sj ¼ X ð4Þ

• The “following relationship” in which the target user ui follows uj with an influence
defined by the inflðj!iÞf score as below:

1
inflðj!iÞf ¼ ðpf Þji  sj ¼  sj : ð5Þ
Count of friends ðiÞ

• Positive retweets of ui on the uj tweet with an influence defined by the score

inflðj!iÞr as below:

inflðj!iÞr ¼ ðpr Þji  sj ¼ ^

hijMLE  sj : ð6Þ

The several kinds of opinion influence the user uj exerts on ui are included in the
score vector denoted as inflj!i ¼ ðinflðj!iÞr ; inflðj!iÞF ; inflðj!iÞf ÞT and expressed as
below:

inflj!i ¼ Pji :sj : ð7Þ

• Pji ¼ ððpr Þji ; ðpF Þji ; ðpf Þji ÞT is the probability vector of positive retweets, of favorite
tweet markings, and of following relationships
• sj is the power of opinion influence of uj
• The score of opinion influence is expressed by the norm of the vector inflj!i
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
ScoreOpinionInfluence ¼ infl2ðj!iÞr þ infl2ðj!iÞF þ infl2ðj!iÞf : ð8Þ

Once the target user’s opinion influencers determined, the sought sensitivity of the
target user to a given service is defined as a transposition of the target user’s sensitivity
to TTs (Trending Topics) for which it is expressed his opinion (or that of his opinion
influencers). Indeed, the sensitivity transposition process is made thanks to a services-
to-TTs mapping (each service offered corresponds to at least one TT). As a result, the
sensitivity of the target user to a given service is the average of the opinion scores
(related to the TT matching the service in question) obtained at the outputs of the
sentiment analysis of each Target user’s opinion influencers. In the next subsection we
440 A. N. Ngaffo et al.

perform a classification of TTs and services in order to be able to do mapping of

services-to-TTs since these are services to be discovered/exposed.

3.5 Twitter Trending Topic and Service Classification

The eighteen category labels listed in [6] are used to store each TT. These are: Art &
design; Books; Business; Charity & deals; Fashion; Food & drink; Health; Holidays &
dates; Humor; Music; Politics; Religion; Science; Sports; Technology; TV & Movies;
Others news and Other. This classification of each TT makes it possible to have an idea
of the type of services in adequacy with each TT. In other words, the category (in
Fig. 6(a)) to which a TT belongs indicates the type of offered service to the target user.
For it to work, a service classification in category is also needed in order to match a TT
to equivalent services. Figure 6(b) shows the mapping between TTs and offered
services.

Fig. 6. Service-Twitter trending topic mapping

The algorithm from Network-based modeling in [6] is that of the TT classification

for which we opted because of its performance. The classification approach of TT
based on Network-based modeling consists for a given TT to determine 5 similar
(manually labelled) TTs (similarity of TTs according to the User Similarity Model [6]).
Then the membership classes of the 5 similar TTs are used to predict that of the given
TT. Identically, a classification of the services previously aggregated into clusters
(clustering as data reduction technique) is also performed to facilitate the services-TTs
mapping process. The purpose of the mapping between services and TTs is to make
sure that at the end of the opinion mining process (consisting of the analysis of the
target user’s opinion influencers’ sentiments relatively to a given TT), we can match
the opinion score related to TT, to a set of services eligible to be discovered/exposed to
the end user.

3.6 Our OMSDEA Presentation (Opinion Mining-Based Service

Discovery/Exposure Architecture)
Figure 7 illustrates the architecture of the Ip Multimedia Subsystem extended by our
service discovery/exposure approach based on opinion mining. In Fig. 7, the “opinion
mining module” is fed by the sentiment analysis of the opinion influencers of the target
user about TTs related to the current position of the target user. And the Opinion
 An IP Multimedia Subsystem Service Discovery 441

module feeds the User-Service repository, the User profile extended repository, the
Service profile extended Repository which contains respectively the service rating
scores expressing the users’ sensitivity to the services, the users’ profiles and service
description. They are all involved in the estimation of the target user sensitivity to a
givens service.

Fig. 7. Our OMSDEA

4 Experiments and Results

4.1 Data Summary

We gathered a dataset of 1036 labelled trending topics [4] and 9,673,959 IDs of tweets
with 8624 users. The datasets contained 4,164,570 retweets and 1,874,630 following
relationships. Thanks to Hydrator tool, we have reconstructed the tweets whose IDs
were collected. These tweets have been stored in a JSON file. We used AFINN [19] as
opinion word lexicon, that has the specificity of being accurate concerning microblogs
by the consideration of slangs, abbreviations used on microblogs. We established the
polarity of the tweets (positive or negative) using AFINN-165 version containing 3382
opinion words meaning 905 more than the original version AFINN-111. We consid-
ered only active users to increase the accuracy of results. Figure 8 illustrates our
simulation environment.
442 A. N. Ngaffo et al.

Fig. 8. IMS simulation environment

4.2 Presentation of Our Results

Top-K Opinion Influencers Results. Figure 9 presents the results obtained con-
cerning the determination of opinion influencers of the user named ‘_heymike’ after
execution of our EPR algorithm. This is the list of the top 15 opinion influencers of the
user named ‘_heymike’. This list is obtained after ranking opinion influence scores of
the 224 Twitters friends of the user ‘_heymike’ in descending order (we chose to
partially hide the concerned user IDs in order to preserve their anonymity).

Fig. 9. The 15 first opinion influencers of the user named ‘_heymike’

Targeting Accuracy and Targeting Error. Figures 10 and 11 respectively present

trends of the Targeting Accuracy and Targeting Error according to the Density User-
service matrix which is the matrix of all the rating scores obtained by each service
relatively to each user. We emphasize the fact that the amount of information related to
the service rating scores is crucial for the service discovery/exposure. For this purpose,
we distinguish two phases that reflect the lifetime of the target user in the operator’s
network. We compare our approach to the classic approach [2] and the collaborative
[20] service discovery/exposure approach. The learning phase represents the start-up
period of the target user in the operator’s network and the exploitation phase represents
the period during which the amount of information about service rating score is
increased. The learning phase is the one that is about improving performance through
the use of our approach. Figure 10 illustrate an improvement in the accuracy of service
 An IP Multimedia Subsystem Service Discovery 443

targeting, while the classic approach and the collaborative approach show poor per-
formance during the learning phase due to the insufficiency of information concerning
the target user.

Fig. 10. Evolution of targeting accuracy

Moreover, Fig. 11 shows an improvement by a reduction in the service targeting

error due to the use of the opinion mining of the target user’s Twitter data to com-
pensate for the information deficit that concerns the operator’s network.

Fig. 11. Evolution of targeting error

From Figs. 10 and 11 above, we observe the classic approach and the collaborative
approach fail during the learning phase because they are fed with poor information
(relative to the target user) only by the operator’s network, while our approach shows
better performances as it compensates for the information deficit by crawling the data
generated by the target user on Twitter social network.
444 A. N. Ngaffo et al.

5 Conclusion

In this paper we have proposed a service discovery/exposure approach based on

opining mining of user data related to Twitter trending topics. The proposed approach
aims to improve the accuracy of service targeting to compensate for the deficit of
information (directly related to the targeted subscriber) held by the operator’s network.
This situation is perceived when the subscriber starts on the operator’s network. By
mining the massive data accumulated on the Twitter social network, our approach
compensates for the failure of the classic approach and the collaborative service
discovery/exposure approach. To achieve this, we estimate the target user’s sensitivity
to a given service through the opinion mining of his Twitter friends regarding TTs.
Indeed, in the absence of TT-related tweets from the target user, our approach is based
on his opinion influencers. That is to say the Twitter friends of the target user who serve
as mirror of opinion for the target user. In other words, they are users with whom the
target user most often shares the same point of view, the same opinion to about topics.
Once the target user’s opinion influencers determined, the sought sensitivity of the
target user to a given service is defined as a transposition of the target user’s sensitivity
to TTs for which it is expressed his opinion (or that of his opinion influencers). Indeed,
the sensitivity transposition process is performed thanks to a services-to-TTs mapping.
As a result, the sensitivity of the target user to a given service is the average of opinion
scores (related to the TT matching the service in question) obtained at the outputs of the
sentiment analysis of each Target user’s opinion influencers. At the end, we proposed
an IMS architecture extended by our approach. The results of our experiments show
that our approach improves the accuracy of service targeting and service targeting error
compared to the classic approach and the collaborative service discovery/exposure
approach. Therefore, our approach is consistent with our previous work [20] as an
alternative to ensure the relevance of the results in the learning phase meaning when the
subscriber starts on the operator’s network. We plan to improve our work by studying
the contribution of the inclusion of others influence indicators such as user’s mentions,
the credibility of Twitter accounts (“verified” or not). Moreover, we plan to extend
mining of the target user’s data to other content sharing platforms such as Facebook,
GooglePlus, YouTube etc.

References
1. 3rd Generation Partnership Project: Technical Specification Group Services and System
Aspects; IP Multimedia Subsystem (IMS); Stage 2 (Release 15) (2017)
2. El Ayeb, W., Choukair Z.: New joint dual mobile service discovery-exposure model using
bipartite graph. In: 2015 5th International Conference on Communications and Networking
(COMNET), pp. 1–8 (2015)
3. Bhor, H.N., Koul, T., Malviya, R., Mundra, K.: Digital media marketing using trend analysis
on social media. In: 2018 2nd International Conference on Inventive Systems and Control
(ICISC), pp. 1398–1400 (2018)
4. Zubiaga, A., Spina, D., Martínez, R., Fresno, V.: Real-time classification of Twitter trends.
J. Assoc. Inf. Sci. Technol. 66(3), 462–473 (2015)
 An IP Multimedia Subsystem Service Discovery 445

5. Napitu, F., Bijaksana M.A., Trisetyarso, A., Heryadi, Y.: Twitter opinion mining predicts
broadband internet’s customer churn rate, 141 p. (2017)
6. Lee, K., Palsetia, D., Narayanan, R., Patwary, M.M.A., Agrawal, A., Choudhary, A.: Twitter
trending topic classification. In: 2011 IEEE 11th International Conference on Data Mining
Workshops. p. 251–258 (2011)
7. Deng, Q., Dai, Y.: How your friends influence you: quantifying pairwise influences on
Twitter. In: 2012 International Conference on Cloud and Service Computing, pp. 185–192
(2012)
8. Kalaï, A., Zayani, C.A., Amous, I.: User’s social profile – based web services discovery. In:
2015 IEEE 8th International Conference on Service-Oriented Computing and Applications
(SOCA). pp. 2–9 (2015)
9. La Torre, G., Monteleone, S., Cavallo, M., D’Amico, V., Catania, V.: A context-aware
solution to improve web service discovery and user-service interaction. In: 2016
International IEEE Conferences on Ubiquitous Intelligence and Computing, Advanced
and Trusted Computing, Scalable Computing and Communications, Cloud and Big Data
Computing, Internet of People, and Smart World Congress (UIC/ATC/ScalCom/CBDCom/
IoP/SmartWorld), pp. 180–187 (2016)
10. Ma, J., Zhang, Y.: Efficiently finding web services using a clustering semantic approach. In:
CSSSIA 2008 Proceedings of the 2008 International Workshop on Context Enabled Source
and Service Selection, Integration and Adaptation: Organized with the 17th International
World Wide Web Conference (WWW 2008), Article no. 5, 22 avr 2008
11. Tiara, Sabariah, M., Effendy, V.: Sentiment analysis on Twitter using the combination of
lexicon-based and support vector machine for assessing the performance of a television
program, 386 p. (2015)
12. Kiran, M.V.K., Vinodhini, R.E., Archanaa, R., Vimalkumar, K.: User specific product
recommendation and rating system by performing sentiment analysis on product reviews. In:
2017 4th International Conference on Advanced Computing and Communication Systems
(ICACCS), pp. 1–5 (2017)
13. Lassen, N., Madsen, R., Vatrapu, R.: Predicting iPhone sales from iPhone tweets, vol. 2014,
81 p. (2014)
14. Rathore, A., Ilavarasan, P.V.: Social media analytics for new product development: case of a
pizza, 213 p. (2017)
15. Mei, Y., Zhong, Y., Yang, J.: Finding and analyzing principal features for measuring user
influence on Twitter. In: 2015 IEEE First International Conference on Big Data Computing
Service and Applications, pp. 478–486 (2015)
16. Huang, J.: Maximum likelihood estimation of dirichlet distribution parameters (2005)
17. Cha, M., Haddadi, H., Benevenuto, F., Gummadi, P.K.: Measuring user influence in Twitter:
the million follower fallacy. In: ICWSM (2010)
18. Fu, H.-H., Lin, D., Tsai, H.-T.: Damping factor in google page ranking: research articles,
vol. 22, 431 p. (2006)
19. Nielsen, F.Å.: A new ANEW: evaluation of a word list for sentiment analysis in microblogs.
CoRR [Internet] (2011). http://arxiv.org/abs/1103.2903
20. Ngaffo, A.N., El Ayeb, W., Choukair, Z.: IMS predictive collaborative service discovery
approach. In: 2018 7th International Conference on Communications and Networking
(COMNET), Tunisia, Hammamet, pp. 1–8 (2018)
 A Hybrid Cross-Layer Protocol
for Wireless Sensor Networks Based
on Backbone

Piercarlo Fermino Soares(B) , João Carlos Giacomin, and Tales Heimfarth

UFLA, Universidade Federal de Lavras, Lavras, Brazil

[email protected], {giacomin,tales}@dcc.ufla.br

Abstract. This paper presents BWise, a semi-synchronous cross-layer

protocol for wireless sensor networks. BWise targets to mitigate the
latency caused by sleep-delay in duty-cycled WSNs. BWise combines an
asynchronous MAC protocol with a geographic routing. A synchronous
part of the network is used to route packets with reduced latency and
low energy consumption. A small number of nodes (<5%) compose this
fast route. No overhead occurs since time synchronization is piggybacked
in ACK message. BWise protocol was tested in simulation against X-
MAC and WiseMAC, two classic protocols of literature. Both of them
were combined with a greedy geographic routing protocol. For the tested
scenarios, our protocol outperforms the other two both in latency and
energy.

1 Introduction
Wireless sensor networks (WSN) are Ad Hoc networks composed by a large
number of low power devices denominated sensor nodes. These devices counts
with five main components: a microprocessor, a small integrated radio, a memory
unit, some sensors and an energy unit [1].
Energy is a scarce resource in WSNs, since sensor nodes are powered by
small batteries and maintenance is not feasible in many cases. Since radio range
is limited, data are transmitted from a source node to a destination by mul-
tihop, with intermediate nodes relaying the message using store and forward
technique [2]. Control protocols are commonly divided in five layers defined as
Application, Transport, Network, MAC (medium access control) and Physical
[10]. Low energy consumption is the main goal in protocols development for all
these layers.
In order to save energy, sensor nodes employ duty-cycle technique, putting
the radio down most of the time (>90%). Periodically the radios are turned on
to perform communication (if necessary), then the radios are turned off again.
The more common duty-cycled MAC protocols can be divided in asyn-
chronous, as B-MAC [11], WiseMAC [5] and X-MAC [3], and those with common
active periods, as S-MAC [14], TMAC [12] and D-MAC [9]. WiseMAC is one of
the most energy efficient MAC protocols in literature [5].
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 446–457, 2020.
https://doi.org/10.1007/978-3-030-15032-7_38
 A Hybrid Cross-Layer Protocol for WSN Based on Backbone 447

Duty-cycle is efficient to save energy in WSNs, but a drawback arises from its
use. A sender node with a message to be sent to a receiver, into its radio range,
must wait until the neighbor enters its active period and turns its radio on. This
phenomenon is denominated sleep-delay, and is the main cause of end-to-end
latency.
Many research works in WSN are dedicated to overcome the end-to-end trans-
mission latency while conserving energy. The cross-layer approach has been used
to mitigate latency in asynchronous duty-cycled networks. In this approach,
decisions about the way each layer will behave are not isolated taken, though
considering information coming from other layers [13].
In this article, a cross-layer protocol denominated BWise is proposed. Cross-
layer nature arises from the interactions between a network layer and an asyn-
chronous MAC layer. Sensor nodes are divided into two classes, the common
node class and the special node class. Common nodes use WiseMAC protocol
without synchronization in MAC layer and a greedy geographic routing protocol
in Network layer. A small number of nodes (<5%) compose the special class.
These nodes form an one direction route, denominated backbone, a highway to
transmit messages. The nodes in backbone use a modified WiseMAC protocol
in MAC layer staggering their wake up times. In this mode of operation, a very
small latency is achieved. No additional energy is needed to synchronize nodes
in backbone, since the time information is piggybacked in ACK messages.
The rest of the paper is organized as follows. Section 2 presents Related works,
Sect. 3 presents the BWise protocol, Sect. 4 the Experimental results, Sect. 5
presents Conclusion.

2 Related Work

MAC protocols are classified into three groups: scheduled protocols, protocols
with common active periods and asynchronous protocols [4].
S-MAC [14] is the first representative protocol of common active period class.
A sync message is sent at the beginning of communication to synchronize neigh-
bor nodes. S-MAC is a duty-cycled protocol with increased latency end-to-end
caused by sleep-delay. D-MAC [9] is another protocol inspired in S-MAC. It stag-
gers the wake-up times of nodes in the direction of sink node. Latency improve-
ment is achieved since the main direction of communications is network-to-sink.
A tree structure from sink is created, and every level in tree is leveled to its
composing nodes to wake-up after previous levels.
In duty-cycled WSNs, asynchronous protocols achieve best energy perfor-
mances [15]. These protocols periodically probe the communication channel in
order to detect transmissions. A node with a message to be sent starts transmit-
ting a large preamble, or a sequence of small preambles, in order to advise the
target node of its intention in establish communication. Sensor nodes uses the
same duty-cycle but there is no synchronization, the wake-up times of the nodes
are independent from each other.
One of the first protocols with this approach was the WiseMAC [5]. Sensor
nodes maintain a local table with the schedule of neighboring nodes [5]. Sender
448 P. F. Soares et al.

node use this table to wake-up close to its neighbor, minimizing the number of
preambles used. This protocol also suffers from sleep-delay.
B-MAC [11] is one of the earliest asynchronous protocol. A sender node trans-
mits a long preamble to keep its neighbors awaken in order to receive the real
message at the end. Differently form B-MAC, a sender node with X-MAC [3]
transmits a sequence of short preambles in order to achieve rendezvous. Pream-
bles are interleaved with listening periods, when sender node waits for an eACK
from intended receiver. When the receiver wakes up and decodes the preamble, it
sends back an eACK, a short message asking sender node to send data. X-MAC
improves latency observed in B-MAC, since the average time taken to establish
communication is divided by two. Besides that, X-MAC avoids spend energy in
overhearing.
EX-MAC [8], is an extension of X-MAC. It uses a staggering mechanism
similar to DMAC. EX-MAC was conceived for WSNs targeted to event detection,
with the majority of traffic occurring in burst towards sink node. A channel
reservation is used to guarantee network staggering and reduce latency. The
first message sent in a event occurrence is transmitted slowly to sink node as
with X-MAC protocol. Each node in the path of sink informs in preamble packet
the next time it is intend to wake up. Then each receiver node adjust its wake
up time according to previous node in the path. The others data messages are
quickly delivered to sink.
GB-MAC [6] is a cross-layer protocol dealing with MAC and Routing func-
tions. A MAC layer based on X-MAC changes information with a routing layer
where a geographic protocol is implemented. This protocol uses a backbone to
create a high speed path to message traffic. The nodes in backbone take a sleep
time shorter than that of common nodes in the network. The delivery time of
messages to the sink is reduced at the expense of a small increase in energy
consumption.
This work presents a new cross-layer protocol composed of an asynchronous
MAC protocol based on WiseMAC and a greedy geographic routing protocol. A
backbone is used in the same way as GB-MAC in order to achieve small latency,
but no additional energy is needed. The reduction in latency is achieved by
staggering the nodes in backbone, like in EX-MAC.
Unlike other protocols that stagger the nodes, data can travel from any net-
work position to any destination, with the backbone as a generic path for any
node and does not relying on sink.

3 The BWise Protocol

BWise is a Cross-layer protocol for Wireless Sensor Networks. Its main goal is
to reduce the end-to-end latency common in duty-cycled WSNs. BWise employs
duty-cycle to save energy, like other MAC protocols for WSNs, maintaining
sensor nodes in low energy state the most of time (>90%). All the nodes wake
up periodically with the same interval but with no synchronization.
The main strategy to reduce end-to-end latency is the use of a backbone, a
fast traffic way inside the network. There is no hardware difference between the
 A Hybrid Cross-Layer Protocol for WSN Based on Backbone 449

backbone nodes and the common nodes. Any node in the network can be elected
as backbone node. In order to achieve fast traffic, backbone nodes are staggered
in one direction, so that a node enters the awaken mode immediately before the
previous node in the path starts to proceed communication. Common nodes are
asynchronous, waking up in their own wake-up time and behaving like regular
WiseMAC.
Figure 1 shows an example of network with backbone nodes represented by
brown circles and common nodes in blue.

Fig. 1. Example of BWise backbone Fig. 2. BWise route choice

Before starting a transmission, source node (S) calculates if the use of back-
bone is advantageous in routing the message to destination (D).
The cross-layer nature of BWise is due to interaction of network (NWT) and
medium access control (MAC) layers in backbone nodes. The information of the
kind of node (common or backbone) comes from NWT. This information is used
by MAC to determine its behavior. The network layer uses this information to
determine each entry and exit point relative to originator and destination of
every new message. In this situation the MAC layer uses the entry point and
exit point determined in network layer, accessing this information in the layer,
to change behavior of wake-up of entry point and exit point.
The network layer uses unicast approach and switch between routing pack-
ets through backbone and routing using geo-routing. Before starting transmis-
sion, source node (S) executes BWise routing calculations, using its own geo-
graphic position and the position of the destination (D). Firstly, S calculates the
euclidean distance between itself and destination (D). Then it calculates its own
distance to every node of backbone and gets the minimum value. After that, it
calculates the distances between destination and all backbone nodes. After all,
it compares the results of direct path, from S to D without backbone, and the
distance composed by th sum of two minimum distances relative to backbone.
Distance information is used as an estimation of end-to-end latency, considering
that it is proportional to the number of hops used to route the message from S to
D. If backbone distance is lower than direct distance, BWise routes the message
via backbone, otherwise direct path will be used. An important consideration
is that hops inside backbone are not taken in account to estimate end-to-end
450 P. F. Soares et al.

latency, as soon as one hop latency inside backbone is much lower than hop
latency between common nodes. When backbone takes part in the route, the
backbone node closer to S is denominated entry point and that closer to D is
the exit point.
Figure 2 presents a message sent from source (green circle) to destination
(red circle). Blue circles are regular nodes and brown are backbone nodes. The
distances between routing nodes are measured relatively to the radio range (R).
One can see that both paths would use the same number of hops (4), but the
hight hand path is selected since one hop inside backbone shortens the end-to-end
latency. The MAC layer is asynchronous and uses duty-cycle to reduce energy
consumption. All the nodes keeps schedules of neighbors to start communication
just before the next hop node wakes up. A backbone node sets its wake up time
according previous node in the path. In this way, any node in backbone wakes
up just before the up stream node starts to establish communication.
Figure 3 presents the relay of a packet using the backbone as part of the
route. N 1 and N 2 are regular nodes and B1, B2, B3 are backbone nodes.

Fig. 3. Transmission of packet using three backbone nodes. N1 and N2 are common
nodes. B1 is the entry node of the backbone and B3 is the exit node.

Entry point node (B1) in backbone takes a long time in receiving the message
from the regular node N 1, so that it can not transmit the message immediately
after reception. This node waits until the next node in backbone path be ready to
receive (B2). At the exit point (B3), when communicating to a regular node N 2,
it behaves like a common node, waiting to start communication before the next
(common) node in the route wakes up. In this case it sends a longer preamble
sequence since it does not have the perfect schedule of N 2.
Nodes that are not part of backbone operate as regular WiseMAC, main-
taining their own sleep/awaken cycles. If a message have to be transmitted, the
sender node waits until the wake up time of next hop node to start communica-
tion process.
Figure 4 presents behavior of a common node (not backbone) using a long
preamble sequence when none information of neighbor is known.
Figure 5 presents behavior common node using a short preamble sequence
when information about neighbor was exchanged, due to previous transmission.
In Fig. 4 a long time passed since the last communication between Tx and
Rx nodes. Then, the knowledge about Rx wake up time is not reliable in Tx. A
 A Hybrid Cross-Layer Protocol for WSN Based on Backbone 451

Fig. 4. Transmission packets in BWise Fig. 5. Transmission packets in BWise

with long preamble sequence with short preamble sequence

long preamble sequence must be used to compensate the uncertainty about Rx

on time. In Fig. 5 this uncertainty is smaller than in Fig. 4 because there was a
short interval since last communication. This uncertainty about neighbor nodes
on times is due to clock drift.
Clock drift is a physical phenomenon of crystal oscillators that counts the
passing time in sensor nodes. In order to mitigates the problems with time uncer-
tainty between neighbor nodes, time information is piggybacked in all ACK mes-
sages every communications. In order to compensate clock drift, Tx starts com-
munication process sending a sequence of small preambles informing its intention
in sending a message. This process is initiated far earlier than Rx wake up time.
When Rx turns its radio on and receives a preamble, it waits for a data message
at the end. After receiving data message, Rx sends back an ACK containing its
next wake up time.

4 Experimental Results
In this section we present the results of our simulation tests make to appraise
the performance of our protocol.

4.1 Simulation Setup

The protocol was implemented in the Grubix simulator, which is an event-
oriented simulator written in Java, based on the ShoX simulator [7]. In this
work MAC and Network layers have been programmed.
To evaluate the protocols behavior, a network of 2000 fixed nodes randomly
positioned in a 500 × 500 m area was considered. A radio consumption of 60 mW
was set to transmit, receive and idle modes.
An unique disc propagation model of 40 m radio range was considered. The
bit rate was 250 kbps. The backbone was composed of 20 static nodes disposed
in a 700 m long line. The size of packets was set to: 256 bits for preamble, 512
bits for data, 128 bits for ACK. Carrier sense duration was 0.002048 s. Clock-
drift was set in ±20 ppm range, it is a representative value for commercial sensor
nodes.
The topology of networks is known and the backbone nodes are static and
always located in the same network position. For the BWise protocol, an initial
transmission was made to perform the primary synchronization in the backbone.
This is not account in the results. After that the synchronization is maintained
by the data traffic through backbone nodes.
452 P. F. Soares et al.

Before starting transmission, a sender node performs a 2 ms carrier sense,

in order to detect if the communication channel is free. If a transmission is
detected, a random back off is made. When a free channel is detected, sender
node starts to send a sequence of small preambles, each one with 1 ms duration.
If a reliable information about active time of receiver is not available to sender
node, a long sequence of 98 preambles is sent. This sequence can be reduced to
2 small preambles when sender node is fully aware of receiver schedule. When a
long time separate two events of communication, the sequence of preambles must
be enlarged to compensate clock-drift. After sending the preamble sequence,
sender node transmits data message.
Inside backbone, a high message flux is expected, then just short pream-
ble sequences are needed. Only if too much time pass between transmissions,
backbone nodes need to compensate clock-drift.
The backbone was not especially determined. A simple backbone was stati-
cally created and used. In a future work, construction methods of backbone, with
considerations like node neighbors number, energy availability and other inter-
esting characteristics should be employed, which will probably result in better
outcomes.
Two evaluation scenarios were used to test the protocol produced, every sce-
nario consisted by transmissions of 10, 20 and 30 packets. In both scenarios
collisions did not occur, since new packets were start sending only when previ-
ous packet reached destination. A minimum distance of 350 m always happen
between any node pair sorted. After executions, data about energy consumption
and latency was gathered to analysis.
In our tests, we compared regular WiseMAC using geographic routing, X-
MAC using geographic routing and BWise, all of three using unicast to send
messages. In order to achieve statistic relevant results, each scenario was executed
30 times with respect to a confidence level for confidence interval set to γ =
1 − α, α = 0.05.
To simulate the protocol, the adopted configurations were: radio range of
fixed 40 m, power of 60 mW, a network density of 2000 static nodes in an area of
500 × 500 m, with random node positioning. The backbone used was static com-
posed of 20 nodes that connected a maximum distance of 707 m in the network.
The topology of networks is known and the backbone is always located in
the same position in them. For the BWise protocol, an initial transmission was
made to perform the primary synchronization in the backbone. This is not taken
in account in the results. After that the synchronization is maintained by the
data traffic through backbone.

4.2 First Scenario

The first consisted of the 10, 20 and 30 nodes within two separate areas of
the network that were connected through the backbone. These nodes formed
sender/receiver pairs and were randomly selected in respective areas, without
any repetition of sender/receiver pair. Senders are selected in order to respected
the direction of the backbone.
 A Hybrid Cross-Layer Protocol for WSN Based on Backbone 453

Fig. 6. Total energy consumption in Fig. 7. Average energy consumption

WiseMAC and BWise per transmission in WiseMAC and
BWise

This scenario was used to compare the WiseMAC and the protocol proposed
in this work. The total energy consumption, the energy consumption per trans-
mission and the transmission latency were evaluated. X-MAC was not used in
this scenario because it’s energy consumption per transmission, in average, is
constant.
For the first scenario, Fig. 6 represents the energy consumption of both pro-
tocols tested. BWise presented a lower consumption than WiseMAC, in this
scenario. WiseMAC depends on the number of paths known to reduce energy
consumption, so with more transmissions in the network, the rate of grow is
reduced.
The average energy consuption per transmission in both WiseMAC and
BWise is reduced with the increasing number of sent packets, as presented in
Fig. 7. This can be explained by the fact that known paths are used to reduced
preambles.
Regarding latency, Fig. 8 shows the decrease in latency of the BWise protocol
compared to WiseMAC. Latency remains constant for any number of transmis-
sions in this scenario. The latency in the BWise protocol is reduced because the
backbone is synchronized and the data travels directly through it without the
need for long waits of time. WiseMAC suffers from sleep delay since nodes do
not bother to transmit data quickly.

Fig. 8. Average latency in WiseMAC and BWise (1st scenario).

454 P. F. Soares et al.

4.3 Second Scenario

The second scenario maintained the choice of nodes within these two areas,
but added a pattern of repetition of previously chosen nodes. This is because
WiseMAC is ideally functioning energetically when it uses known nodes or paths,
thus reducing the number of preamble packets. In this second scenario the X-
MAC protocol was added for comparison. Transmission pattern in this scenario
is show in Table 1.

Table 1. Transmission pattern in second scenario

Packets Pattern
10 2, 2, 3, 1, 1, 1
20 2, 2, 3, 1, 1, 1, 2, 2, 3, 1, 1, 1
30 2, 2, 3, 1, 1, 1, 2, 2, 3, 1, 1, 1, 2, 2, 3, 1, 1, 1

The first sender receiver pair send two packets, then the second sender
receiver pair send two packets and third pair send three packets. After that
every pair send only one packet, in previous order, completing all 10 packets.
In 20 and 30 packet cases, this pattern repeats, but after every three pairs of
sender/receiver nodes that sent 10 packets, a new group of three pairs did that.
This way in 20 packets transmissions exists 6 node pairs and 30, 9 node pairs.
For the second scenario, Fig. 9 represents the total energy consumption of
WiseMAC, BWise and X-MAC.
BWise proposed here intends to reduce end-to-end latency caused by sleep-
delay. When compared to other two well known asynchronous MAC protocols of
literature, BWise presented better performance in latency as well as in energy
consumption. However, it must be emphasized that the use of a backbone in
BWise can reduce the network lifetime, since the backbone nodes may have their
energy depleted faster than the others in the network. In order to overcome this
problem, an advanced version of BWise is planed, in which the backbone is
periodicaly reorganized. The remaining energy is one of the factors considered
to elect a node to be part of backbone in the new BWise.
It is possible to verify that the X-MAC is more economic at the beginning of
operation, when compared to WiseMAC. X-MAC interrupts a preamble sequence
when it receives a CTS, so it does the interrupt and initiation of the DATA
transmission. In WiseMAC this does not happen because it does not contain
any CTS packets, it sends preambles and when the sequence ends, the DATA
is sent. For the BWise protocol, the initial consumption is very low, since the
backbone nodes only use the minimum number of preambles because they are
synchronized. It is not necessary to use long preambles to signal DATA sending
due to this synchronization. But in BWise protocol nodes outside backbone use
long preamble for the first transmission, but the number of transmissions in this
mode is low when compared to the WiseMAC. When the transmission amount
 A Hybrid Cross-Layer Protocol for WSN Based on Backbone 455

Fig. 9. Total energy consumption in Fig. 10. Average energy consumption

WiseMAC, X-MAC and BWise in WiseMAC, X-MAC and BWise

increases, the WiseMAC has an per transmission energy consumption always

at the same level, it happens because in this scenario, repetitions of nodes are
guaranteed. So it always uses the same amount of energy but as more data
travels, total consumption increases. For BWise, the increase in consumption
follows the amount of data, because there is a very large flow of data throughout
the backbone, the nodes that compose it perform much more transmissions than
the other nodes. The X-MAC, in average, uses the same number of preambles for
communication. The energy consumption in X-MAC tends to increase as more
transmissions happen in the network. This growth is correlated with the number
of transmissions and independent to the nodes repetition.
Regarding average energy consumption, Fig. 10, BWise holds the best energy
consumption. This happens because even when no paths outside backbone are
known, only a few hops happen outside backbone, then backbone low consump-
tion reduces energy consumption per transmission.
XMAC does not considerate any means of reducing preamble sequence, in
every new transmissions, even between known nodes, it starts preamble sequence
always from beginning as if does not know any neighbour.
WiseMAC need a known path to reduce energy consumption, with this, as
number of transmissions grows, WiseMAC begin saving more energy, as reuses
previous paths and nodes to relay messages. However, our tests, it never over-
come BWise, because a larger number of hops happen with longer preamble
sequence, before sufficient paths are known to use only short preamble sequence.
Figure 11 represents the latency for scenario 2 with guaranteed node repeti-
tion. All three protocols have the average stable latency behavior. WiseMAC has
a higher latency between the three protocols and its latency tends to stabilize
as the repetitions happen and the use of long preambles decreases. BWise pro-
tocol had the lowest latency among the three. This is justified by the backbone
flow that compensates the sleep delay problem present in both WiseMAC and
XMAC.
456 P. F. Soares et al.

Fig. 11. Average latency in WiseMAC, X-MAC and BWise (2nd scenario).

5 Conclusion
This paper presents an asynchronous cross-layer protocol with node staggering
and local synchronization. The protocol was able to reduce the sleep delay that
occurs in duty-cycled protocols and obtained a better result when compared to
WiseMAC and X-MAC, two of good performance asynchronous MAC protocols
of literature.
Sleep-delay is a common problem in protocols that use duty-cycles as a mech-
anism to reduce energy consumption. BWise proposed here intends to reduce
end-to-end latency caused by sleep-delay. When compared to other two well
known asynchronous MAC protocols of literature, BWise presented better per-
formance in latency as soon as in energy consumption. However, it must be
emphasized that the use of a backbone in BWise, can reduce the network life-
time, since the backbone nodes may have its energy depleted faster than the
others in the network. In order to overcome this problem, an advanced version
of BWise is currently developed, in which the backbone is periodically reorga-
nized. The remaining energy is one of the factors considered to elect a node to
be part of backbone in the new BWise.

Acknowledgements. This study was financed in part by the Coordenação de Aper-

feiçoamento de Pessoal de Nı́vel Superior - Brasil (CAPES) - Finance Code 001. Also,
the present work was realized with support of Fundação de Amparo à Pesquisa de
Minas Gerais (FAPEMIG).

References
1. Akyildiz, I.F., Su, W., Sankarasubramaniam, Y., Cayirci, E.: Wireless sensor net-
works: a survey. Comput. Netw. 38(4), 393–422 (2002)
2. Avvenuti, M., Corsini, P., Masci, P., Vecchio, A.: Energy-efficient reception of large
preambles in mac protocols for wireless sensor networks. Electron. Lett. 43(5), 59–
60 (2007)
3. Buettner, M., Yee, G.V., Anderson, E., Han, R.: X-MAC: a short preamble MAC
protocol for duty-cycled wireless sensor networks. In: Proceedings of the 4th Inter-
national Conference on Embedded Networked Sensor Systems, SenSys 2006, pp.
307–320. ACM, New York (2006)
 A Hybrid Cross-Layer Protocol for WSN Based on Backbone 457

4. Cano, C., Bellalta, B., Sfairopoulou, A., Oliver, M.: Low energy operation in wsns:
a survey of preamble sampling mac protocols. Comput. Netw. 55(15), 3351–3363
(2011)
5. El-Hoiydi, A., Decotignie, J.-D.: WiseMAC: an ultra low power MAC protocol for
multi-hop wireless sensor networks. In: Proceedings of the 1st International Work-
shop on Algorithmic Aspects of Wireless Sensor Networks (ALGOSENSORS), pp.
18–31. Springer, Berlin (2004)
6. Heimfarth, T., Giacomin, J.C., Fassio, A.V., de Araujo, J.P.: GB-MAC: a backbone
based low latency protocol for WSNs. In: Proceedings of the 29th Annual ACM
Symposium on Applied Computing, SAC 2014, pp. 692–698. ACM, New York
(2014)
7. Lessmann, J., Heimfarth, T., Janacik, P.: ShoX: an easy to use simulation platform
for wireless networks. In: Tenth International Conference on Computer Modeling
and Simulation, UKSIM 2008, pp. 410–415, April 2008
8. Li, Y.-X., Shi, H.-S., Zhang, S.-P.: An energy-efficient MAC protocol for wireless
sensor network. In: 2010 3rd International Conference on Advanced Computer
Theory and Engineering (ICACTE), vol. 4, pp. V4-619–V4-623, August 2010
9. Lu, G., Krishnamachari, B., Raghavendra, C.S.: An adaptive energy-efficient and
low-latency MAC for data gathering in wireless sensor networks. In: 2004 Proceed-
ings of the 18th International Parallel and Distributed Processing Symposium, p.
224, April 2004
10. Mendes, L.D., Rodrigues, J.J.: A survey on cross-layer solutions for wireless sensor
networks. J. Netw. Comput. Appl. 34(2), 523–534 (2011)
11. Polastre, J., Hill, J., Culler, D.: Versatile low power media access for wireless
sensor networks. In: Proceedings of the 2nd International Conference on Embedded
Networked Sensor Systems, SenSys 2004, pp. 95–107. ACM, New York (2004)
12. van Dam, T., Langendoen, K.: An adaptive energy-efficient MAC protocol for
wireless sensor networks. In: Proceedings of the 1st International Conference on
Embedded Networked Sensor Systems, SenSys 2003, pp. 171–180. ACM, New York
(2003)
13. Vuran, M.C., Akyildiz, I.F.: XLP: a cross-layer protocol for efficient communication
in wireless sensor networks. IEEE Trans. Mob. Comput. 9(11), 1578–1591 (2010)
14. Ye, W., Heidemann, J., Estrin, D.: An energy-efficient MAC protocol for wireless
sensor networks. In: Proceedings of the Twenty-First Annual Joint Conference of
the IEEE Computer and Communications Societies, vol. 3, pp. 1567–1576 (2002)
15. Ye, W., Silva, F., Heidemann, J.: Ultra-low duty cycle MAC with scheduled chan-
nel polling. In: Proceedings of the 4th International Conference on Embedded
Networked Sensor Systems, SenSys 2006, New York, NY, USA, pp. 321–334. ACM
(2006)
 Implementing Lightweight IoT-IDS
on Raspberry Pi Using Correlation-Based
Feature Selection and Its Performance
Evaluation

Yan Naung Soe1,2(&), Yaokai Feng2, Paulus Insap Santosa1,

Rudy Hartanto1, and Kouichi Sakurai2
1
Universitas Gadjah Mada, Yogyakarta, Indonesia
[email protected], {insap,rudy}@ugm.ac.id
2
Kyushu University, Fukuoka, Japan
[email protected], [email protected]

Abstract. The application of many IoT devices is making our world more
convenient and efficient. However, it also makes a large number of cyber-attacks
possible because most IoT devices have very limited resources and cannot
perform ordinary intrusion detection systems. How to implement efficient and
lightweight IDS in IoT environments is a critically important and challenging
task. Several detection systems have been implemented on Raspberry Pi, but
most of them are signature-based and only allow limited rules. In this study, a
lightweight IDS based on machine learning is implemented on a Raspberry Pi.
To make the system lightweight, a correlation-based feature selection algorithm
is applied to significantly reduce the number of features and a lightweight
classifier is utilized. The performance of our system is examined in detail and
the experimental result indicates that our system is lightweight and has a much
higher detection speed with almost no sacrifice of detection accuracy.

1 Introduction

1.1 Background and Motivation

Nowadays, it is said that the IoT (Internet of Things) era has been coming up and our
world will become more convenient and more efficient. The network-connected devices
will become around 27 billion by 2021 [1]. As the Internet of Things (IoT) continues to
develop in many domains and applications, the cyber-attacks will become more pop-
ular because the most of IoT devices have very-limited resources (e.g. memory and
computation power) and thus they cannot carry out complicated intrusion detection
systems [1]. For example, the number of DDoS attacks grew 172% in 2017 [1].
According to the Kaspersky lab’s report [10], the number of IoT malwares in 2017 has
increased by up to seven times compared with the number in 2013 and many of them
are very dangerous. For example, the peak of attack traffic of a cyber-attack launched
by a huge number of Mirai-infected IoT devices reached up to 620 Gbps.

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 458–469, 2020.
https://doi.org/10.1007/978-3-030-15032-7_39
 Implementing Lightweight IoT-IDS on Raspberry Pi 459

Generally, there are mainly two kinds of IDS (Intrusion Detection System):
anomaly and misused approaches. Most of public IDSs are misuse based approaches
which is based upon predefined signatures and such detectors cannot effectively deal
with unknown attacks or new variants of the known attacks [14]. On the other hand, the
anomaly detector identifies what deviate from normal behavior. Although such
detectors can detect both known and unknown attacks, they may generate too many
false alarms [6, 19], which greatly hinder their practical application.
Due to the resources constraint of IoT devices, the IDS for IoT environment must
be very lightweight. In the most of the existing detection systems in IoT environments,
the signature-based methods are used and only a limited number of rules are config-
ured. In this study, we implement a lightweight IDS system for IoT environments. We
use the CFS (Correlation-based Feature Selection) algorithm to significantly decrease
the number of features. And, a lightweight classifier, J48 (a decision tree algorithm), is
used. Our system is lightweight enough to be implemented on Raspberry Pi system and
its detection performance is examined.

1.2 Challenging Issues

Since the traditional security countermeasures cannot be efficiently transplanted to IoT
environments due to the limited resources in IoT devices, the detection systems in IoT
environments must be very lightweight, which means that not much storage and much
computation power are needed for such IDS systems. Especially, the number of the
features used for detection should not be large and the classifier also must be simple.
Another challenge is how to find suitable datasets for training and testing the
detection system. Many researches still use KDDCUP 99 datasets [22] or its variant
version KDD NSL [4]. Obviously, such datasets are too outdated. For example, they
have no or not enough modern attacks and the distribution of the data in networks has
also changed much.

1.3 Our Contributions

In this paper, our main contributions are as follows.
(1) A lightweight IDS for IoT environments is proposed and implemented on
Raspberry Pi. In our detection system, the number of features used for detection is
greatly decreased using CFS algorithm, and J48 is used as a lightweight classifier.
(2) The detection performance (processing time and detection rates) of our proposal is
examined in detail using the UNSW-NB15 dataset [17, 18]. Performance com-
parison is made between before and after using the CFS algorithm. That is, the
effectiveness of the CFS algorithm for IDS in IoT environments is investigated.
The experiment result indicates that using CFS to select features is very helpful
for implementing a lightweight IDS. Specifically, it can greatly reduce the pro-
cessing time almost without sacrifice of detection rates. More importantly, without
the help of CFS algorithm, Raspberry Pi cannot deal with the whole dataset used
in our experiment. However, our proposal can handle the entire dataset in the
same experimental environment.
460 Y. N. Soe et al.

(3) The detection performance in the case of Naïve Bayes algorithm being used as the
lightweight classifier is also investigated. We found that if Naïve Bayes algorithm
is used, the detection rates become not stable for different kinds of attacks,
although the processing time can also be reduced much. This indicates that Naïve
Bayes algorithm is not suitable for our proposed system.

2 Related Works

Most of the public IDS are based on pattern matching and are static approaches.
Attacks are becoming sophisticated and attackers can circumvent rules-based detection
techniques. Thus, machine learning based approaches are thought to be promising.
In recent years, as more and more IoT devices are actually deployed, IDS in IoT
environments has attracted attentions from many researchers and developers. The
researches [3, 8] addressed specific types of threats targeting specific devices. The work
[3] proposed a detection system for sinkhole which is the attack on routing devices.
The detection rate for sinkhole attacks is up to 92% and 72% on fixed and mobile
scenario, respectively. The work [8] tried to prevent the three different levels of battery
exhaustion attack on BLE based mesh network.
Like our study, the researches [7, 15, 20, 21, 23, 25] proposed IDS solutions also
using Raspberry Pi. The automata based IDS [7] was implemented by constructing the
previous traffic natures and matching the present traffic for detecting three types of
attacks. The experimental results in the works [15, 20] indicated that public IDS
systems, snort and bro, can be run on Raspberry Pi device. The work [15] observed that
when such traditional IDS are used on Raspberry Pi, the rules must be limited and
otherwise, the Raspberry Pi system will crash. The work [21] implemented a real-time
DDoS detection architecture using Complex Event Processing (CEP) algorithm. CEP is
an emerging and promising technology that allows real-time analysis of continuous
data-steams. The work [23] proposed an IoT router for verifying the DNS traffic
generated from IoT devices. An IDS/IPS solution for RFID using Raspberry Pi is
implemented in the work [25]. Although all the above researches claimed a detection
system is implemented and some attacks are successfully detected, obviously how to
make the detection system itself lightweight is critical and unavoidable if we want to
implement an actually efficient detection system in IoT environments.

3 Our Proposal

3.1 General Flow

The general flow of our proposal is shown in Fig. 1. At first, many possible features are
defined. Then, CFS algorithm is used on the training data to select the really necessary
features. In this phase, the number of features can be decreased greatly, which can
significantly decrease the necessary computation. The features selected by CFS are
used to train the classifier. As mentioned above, J48 is used as the classifier because it
is simple and lightweight. In our experiment, seven features are selected by CFS from
 Implementing Lightweight IoT-IDS on Raspberry Pi 461

the original 49 features. In the test phase, the selected features are extracted from the
test data and then the classification is decided by the classifier. In this study, Raspberry
Pi 3 is used as the experiment environment to implement this lightweight system and its
performance is examined in detail.

Training data Feature Definition

(Many possible features are defined)

Feature Selection
(CFS algorithm is used)

Feature Lightweight Classifier

Extraction (J48 algorithm)

Test data Performance Evaluation

Fig. 1. Proposed IDS architecture which is using J48 (a decision tree) algorithm for
classification and the CFS algorithm for feature selection.

3.2 Correlation-Based Feature Selection (CFS)

Feature selection methods can be used to select the relevant features which may be
most related with desired results. And, they should not condense the accuracy of a
predictive model [13]. There are mainly three types of feature selection methods such
as filter-based, wrapper-based and embedded-based approaches [24]. Every approach
has its own weaknesses. For example, finding the suitable learning is difficult in filter-
based approaches. Although the filter-based approach is computationally light, the
accuracy may be reduced due to the fact that the nature of the underlying algorithm is
ignored. The huge number of computing will be required in the wrapper-based
approaches because of the combination process of feature selection and learning pro-
cess. The embedded-based approach is to incorporate the selection process with the
training part.
To implement a lightweight detection system, a lightweight feature selection pro-
cess is necessary. Correlation-based feature selection (CFS) can be used in the filter-
based approach and it is a simple algorithm evaluating the corresponding relations
between the outputs and correlated input features [9]. This algorithm claims that feature
selection for classification in machine learning can be achieved on the basis of cor-
relation between features. Empirical evidence shows that, along with irrelevant fea-
tures, redundant information also should be eliminated [11, 12]. A feature is redundant
if one or more of the other features are highly correlated with it. The CFS algorithm is
462 Y. N. Soe et al.

based on the fact that a good feature subset is one that contains features highly cor-
related with the class and uncorrelated with each other. Irrelevant or redundant features
should be ignored also because they may raise the computation process and even
worsen the detection accuracy. Equation (1) gives the merit of a feature subset and the
feature subset having the highest merit will be the result of feature selection. In Eq. (1),
k is the number of features in the current subset, rcf is the average value of all feature-
class correlations and rff is the average value of all feature-feature correlations. CFS
can support most of the attribute types such as binary, date, nominal, empty (missing)
values and unary attributes.

krcf
MSk ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð1Þ
k þ kðk  1Þrff

3.3 J48
J48 (C4.5) is one of decision tree generation algorithms developed by Ross Quinlan
[2]. It is the descendant of the ID3 algorithm and can be used as a statistical classifier. It
is a tree-like structure which consist root node and leaf nodes are derived from it. The
leaf nodes may represent classes or class attributes. It is constructed by using infor-
mation gain and entropy criteria. And then, it generates the rules for the target out-
comes. It can handle both continuous and discrete features. By using the pruning
techniques, the overfitting problem can be solved. It can also be used on training data
having incomplete data and different weighted features. J48 is used in many researches
and actual systems.

4 Experiments

Raspberry Pi 3 Model B is the experiment platform to implement our lightweight IDS

and the open source Weka [16] is used to examine the performance. The dataset
UNSW-NB 15 [17, 18] is used as training data and test data. As mentioned before, J48
is used for classification. The effectiveness of CFS algorithm is examined, including its
influence on processing time and detection rates. We also try to use Naïve Bayes
algorithm to replace J48 and see which is better for our system.

4.1 Dataset
UNSW-NB15 was created in the Cyber Range Lab of the Australian Centre for Cyber
Security (ACCS) [17, 18]. Totally, 175,341 instances exist in this dataset including
nine different kinds of attacks as shown in Table 1.
 Implementing Lightweight IoT-IDS on Raspberry Pi 463

Table 1. Nine kinds of attacks in UNSW-NB15 [18].

Attack type Description
Fuzzers Attempting to cause a program or network suspended by feeding it the
randomly generated data
Analysis It contains different attacks of port scan, spam and html files penetrations
Backdoors A technique in which a system security mechanism is bypassed stealthily
to access a computer or its data
DoS A malicious attempt to make a server or a network resource unavailable to
users, usually by temporarily interrupting or suspending the services of a
host connected to the Internet
Exploits The attacker knows of a security problem within an operating system or a
piece of software and leverages that knowledge by exploiting the
vulnerability
Generic A technique works against all block-ciphers (with a given block and key
size), without consideration about the structure of the block-cipher
Reconnaissance Contains all Strikes that can simulate attacks that gather information
Shellcode A small piece of code used as the payload in the exploitation of software
vulnerability
Worms Attacker replicates itself in order to spread to other computers. Often, it
uses a computer network to spread itself, relying on security failures on the
target computer to access it

4.2 Features Selected by CFS Algorithm

49 features are recommended in the original datasets [17, 18]. In our experiment, seven
of them, shown in Table 2, are selected by CFS algorithm.

Table 2. Selected features from UNSW-NB 15 by CFS.

Feature name Type Description
service Nominal Such as http, ftp, smtp, ssh, dns and ftp-data
sbytes Integer Source to destination bytes
sttl Integer Source to destination time to live
dttl Integer Destination to source time to live
sload Float Source bits per second
ct_dst_sport_ltm Integer No. of rows of the same dstip and the sport in 100 rows
ct_srv_dst Integer No. of rows of the same service and dstip in 100 rows
464 Y. N. Soe et al.

4.3 Performance Evaluation

The performance (processing time and detection accuracy) of our system for detecting
each of the different nine kinds of attacks is examined. The investigated detection
accuracy includes: true positive rate (TPR), false positive rate (FPR), precision, recall
and F-measure.
In our experiment, we found that, in the case of all the original 49 features are used,
our Raspberry Pi could only handle about 80% at the most of the whole dataset. That is,
the system crashed when more instances were used. Thus, in our experiment, 140,273
(not the all) instances are used, which almost reached the up-limitation of our system
when all the 49 features are used. Of the 140,273 instances, 92,580 instances (about
two thirds of the total) are used for training and 47,693 instances (about one third of the
total) for test. Anyway, after the CFS algorithm is used to decrease the number of
features to seven, our Raspberry Pi can handle the whole dataset.
We also tried Naïve Bayes classifier replacing J48 to see which is more suitable for
our system.

4.3.1 Processing Time

According to our experiment, the processing time for each kind of attacks was
decreased greatly with the help of the CFS algorithm. Because of the space limitation,
only the total processing times for all the nine kinds of attacks are shown in Fig. 2.
Again, J48 algorithm is used as the classifier in our system.

(a) Training Time (b) Testing Time

Fig. 2. The processing time comparison between using CFS and not using CFS in the case of
J48 classifier.
 Implementing Lightweight IoT-IDS on Raspberry Pi 465

4.3.2 Detection Accuracy

The investigation result on detection rates is shown in Fig. 3.

(a)TPR (True Positive Rate) (b) FPR (False Positive Rate)

(c) Precision (d) Recall

(e) F-Measure

Fig. 3. The detection rate comparison between using CFS and not using CFS in the case of J48
classifier.
466 Y. N. Soe et al.

From Figs. 2 and 3, we can observe that, if the CFS algorithm is used for feature
selection, the processing time has been reduced greatly for both training and testing,
without clear sacrifice of detection rates.

4.3.3 Observations
From our experimental result, we can observe:
(1) Using the CFS algorithm greatly made our system lightweight. In particular, after
the number of features was significantly decreased, the processing time for
training and that for testing are reduced greatly (see Fig. 2). More importantly, if
the original 49 features are used, only about 80% at the most of all the instances in
the entire dataset could be handled by our Raspberry Pi. However, our lightweight
IDS system can handle all the instances of the dataset.
(2) Our detection system had similar detection accuracy (see Fig. 3). That is, our
system successfully accomplished lightweight with almost no sacrifice of detec-
tion accuracy.
(3) Our lightweight detection system even has better detection accuracy in some
cases. For example, the detection accuracy for “worms” in Fig. 3. This means that
irrelevant or redundant features may not only raise the computation and storage
cost and even deteriorate the detection accuracy, which has also been said by
some other researches [5].

4.3.4 How About Using Naïve Bayes Classifier

Besides J48 algorithm, the Naïve Bayes algorithm is also a well-known lightweight
classifier. Thus, we also investigate the detection performance of our system using the
Naïve Bayes algorithm instead of J48 algorithm. The processing times for both training
and testing are shown in Fig. 4 and the detection accuracy is in Fig. 5.

(a) Training Time (b) Testing Time

Fig. 4. The processing time comparison between using CFS and not using CFS in the case of
the Naïve Bayes classifier.
 Implementing Lightweight IoT-IDS on Raspberry Pi 467

(a) TPR (True Positive Rate) (b) FPR (False Positive Rate)

Fig. 5. The detection rate comparison between using CFS and not using CFS in the case of the
Naïve Bayes classifier.

From Figs. 4 and 5, we can know that, in the case of Naïve Bayes classifier, the
processing time is also reduced greatly. However, the detection rates for the attacks of
Analysis, Backdoor, DoS and Exploits have decreased clearly. This is seemingly
because of the distribution of the attacks. The detailed reason will be investigated in
detail in our future work.

5 Conclusion and Future Work

As the spreading of IoT applications, the cyber-attacks are targeting IoT devices and
have become a big challenging problem. This is mainly because most of the IoT
devices have very limited resources, very low computation power and a small size of
memory. Thus, most of IoT devices cannot install and carry out the traditional intrusion
detection systems that are designed for the conventional network systems. Therefore,
how to design and implement lightweight intrusion detection systems for IoT envi-
ronments has been an urgent issue. Machine learning based intrusion detection tech-
nologies have been paid much attention by many researchers and developers, although
signature-based approaches are still the main stream in actual applications. In this
study, based on machine learning technology, we implemented a lightweight system
for detecting cyber-attacks in IoT environments. In our system, we use a lightweight
and efficient feature selection algorithm, CFS, to reduce the number of features greatly
and efficiently, and a well-known version of decision tree, J48 algorithm, to accomplish
the classification. Our system was implemented on a Raspberry Pi system and its
performance was examined using UNSW-NB15 dataset. According to our experiment,
our system can handle all the instances in this dataset and has a much higher speed for
training and having almost no sacrifice of detection accuracy. By the way, if the step of
feature selection was not taken, the system can only deal with about 80% at the most of
all the instances in the dataset.
Also, we investigated what happen if another well-known classifier, the Naïve
Bayes algorithm, is used as the classifier. We found that the detection accuracy has
468 Y. N. Soe et al.

been deteriorated for some cases, although the processing speed was improved much
for all the cases. The reason for this will be investigated in detail in our future work.
In the future work, also we will implement a multi-threading architecture for our
system to accomplish faster detection because Pi 3 device has four cores CPU and it
can support multithreading, and we will also try other feature selection algorithms to
achieve more accurate detection results.

Acknowledgments. The authors are grateful for the financial support provided by AUN/SEED-
Net Project (JICA). This research is also partially supported by Strategic International Research
Cooperative Program, Japan Science and Technology Agency (JST), JSPS KAKENHI Grant
Numbers JP17K00187 and JP16K00132.

References
1. Andra, U.: Network Security in the Age of Hyperconnectivity_ Pervasive, Proactive, and
Persistent Protection is Essential to Thwart Cyberattacks. https://blogs.cisco.com/sp/
network-security-in-the-age-of-hyperconnectivity-pervasive-proactive-and-persistent-
protection-is-essential-to-thwart-cyberattacks
2. Ashari, A., Paryudi, I., Min, A.: Performance comparison between Naïve Bayes, decision
tree and k-nearest neighbor in searching alternative design in an energy simulation tool. Int.
J. Adv. Comput. Sci. Appl. 4, 33–39 (2013)
3. Cervantes, C., Poplade, D., Nogueira, M., Santos, A.: Detection of sinkhole attacks for
supporting secure routing on 6LoWPAN for internet of things. In: Proceedings of the 2015
IFIP/IEEE International Symposium on Integrated Network Management, IM 2015,
pp. 606–611 (2015)
4. Dhanabal, L., Shantharajah, S.P.: A study on NSL-KDD dataset for intrusion detection
system based on classification algorithms. Int. J. Adv. Res. Comput. Commun. Eng. 4, 446–
452 (2015)
5. Feng, Y., Akiyama, H., Lu, L., Sakurai, K.: Feature selection for machine learning-based
early detection of distributed cyber attacks. In: The IEEE Cyber Science and Technology
Congress (CyberSciTech), CyberSciTech 2018, pp. 173–180 (2018)
6. Feng, Y., Hori, Y., Sakurai, K., Takeuchi, J.: A behavior-based method for detecting
distributed scan attacks in darknets. J. Inf. Process. (JIP) 21(3), 527–538 (2013)
7. Fu, Y., Yan, Z., Cao, J., Koné, O., Cao, X.: An automata based intrusion detection method
for internet of things. Mob. Inf. Syst. 2017, 6–10 (2017)
8. Guo, Z., Harris, I.G., Jiang, Y., Tsaur, L.F.: An efficient approach to prevent battery
exhaustion attack on BLE-based mesh networks. In: 2017 International Conference on
Computing, Networking and Communications, ICNC 2017, pp. 1–5 (2017)
9. Hall, M.A.: Correlation-Based Feature Selection for Machine Learning. Department of
Computer Science, The University of Waikato (1999)
10. Kaspersky Lab: Amount of Malware Targeting Smart Devices more than Doubled in 2017
(2017). https://www.kaspersky.com/about/press-releases/2017_amount-of-malware
11. Kohavi, R., John, G.: Wrappers for feature subset selection. Artif. Intell. Spec. Issue
Relevance 97(1–2), 273–324 (1996)
12. Kohavi, R., Sommerfield, D.: Feature subset selection using the wrapper method: overfitting
and dynamic search space topology. In: The First International Conference on Knowledge
Discovery and Data Mining (1995)
 Implementing Lightweight IoT-IDS on Raspberry Pi 469

13. Kuhn, M., Johnson, K.: An introduction to feature selection. In: Applied Predictive
Modeling, pp. 487–519 (2013)
14. Kumar, R., Sharma, D.: HyINT: signature-anomaly intrusion detection system. In: 2018 9th
International Conference on Computing, Communication and Networking Technologies
(ICCCNT), pp. 1–7 (2018)
15. Kyaw, A.K., Chen, Y., Joseph, J.: Pi-IDS: evaluation of open-source intrusion detection
systems on Raspberry Pi 2. In: 2015 2nd International Conference on Information Security
and Cyber Forensics, InfoSec 2015, pp. 165–170 (2016)
16. Witten, I.H., Frank, E.: Data mining: practical machine learning tools and techniques. In:
Library of Congress Cataloging-in-Publication Data (2005)
17. Moustafa, N., Slay, J.: The significant features of the UNSW-NB15 and the KDD99 data sets
for network intrusion detection systems. In: Proceedings - 2015 4th International Workshop
on Building Analysis Datasets and Gathering Experience Returns for Security, BADGERS
2015, pp. 25–31 (2017)
18. Moustafa, M., Slay, J.: UNSW-NB15: a comprehensive data set for network intrusion
detection systems (UNSW-NB15 network data set). In: Proceedings of 2015 Military
Communications and Information Systems Conference, MilCIS 2015 (2015)
19. Pacheco, J., Zhu, X., Badr, Y., Hariri, S.: Enabling risk management for smart infrastructures
with an anomaly behavior analysis intrusion detection system. In: Proceedings - 2017 IEEE
2nd International Workshops on Foundations and Applications of Self* Systems, FAS*W
2017, pp. 324–328 (2017)
20. Sforzin, A., Marmol, A.G., Conti, M., Bohli, J.M.: RPiDS: Raspberry Pi IDS - a fruitful
intrusion detection system for IoT. In: 2016 Intl IEEE Conferences on Ubiquitous
Intelligence & Computing, Advanced and Trusted Computing, Scalable Computing and
Communications, Cloud and Big Data Computing, Internet of People, and Smart World
Congress, pp. 440–448 (2016)
21. Cardoso, D.S., Lopes, R.F., Teles, A.S., Magalhaes, F.B.V.: Real-time DDoS detection
based on complex event processing for IoT. In: Proceedings - ACM/IEEE International
Conference on Internet of Things Design and Implementation, IoTDI 2018, pp. 273–274
(2018)
22. Bay, S.D., Kibler, D., Pazzani, M.J., Smyth, P.: The UCI KDD archive of large data sets for
data mining research and experimentation. SIGKDD Explor. 2, 81 (2000)
23. Sperling, T.L., Caldas Filho, F.L., Sousa, R.T., Martins, L.M., Rocha, R.L.: Tracking
intruders in IoT networks by means of DNS traffic analysis. In: 2017 Workshop on
Communication Networks and Power Systems (WCNPS), pp. 1–4 (2017)
24. Zena, M.H., Gillies, D.F.: A review of feature selection and feature extraction methods
applied on microarray data. Adv. Bioinform. 2015, 198363, 1–13 (2015)
25. Zitta, T., Neruda, M., Vojtech, L.: The security of RFID readers with IDS/IPS solution using
Raspberry Pi’. In: 2017 18th International Carpathian Control Conference, ICCC 2017,
pp. 316–320 (2017)
 Quantifying the Limitations
of Learning-Assisted Grammar-Based
Fuzzing

Yuma Jitsunari, Yoshitaka Arahori(B) , and Katsuhiko Gondow

Tokyo Institute of Technology, Tokyo, Japan

[email protected]

Abstract. Grammar-based fuzzing is effective at finding the vulnerabil-

ities of various input-parsing programs which take as inputs complex data
conforming to a certain grammar. Traditional grammar-based fuzzing
techniques require a manually-generated grammar for valid test input
generation. However, writing an input grammar by hand has two major
drawbacks: (1) it is costly and error-prone and (2) it has no capability
to generate interesting inputs which induce high test-coverage (for find-
ing many vulnerabilities). To address these problems, a state-of-the-art
technique, Learn&Fuzz, automatically generates an input grammar via
deep neural network-based statistical learning. Even Learn&Fuzz, how-
ever, has significant limitations; especially, it cannot successfully gener-
ate a (long) sequence of instructions (consisting of opcode plus zero or
more operands), which contribute to high test-coverage of instruction-
interpreting code. In this paper, we focus on and quantify the limitations
of the current learning-assisted grammar-based fuzzing, i.e, how ineffec-
tive it is at generating instruction sequences triggering high test coverage.
Through our experiments using a re-implementation of Learn&Fuzz and
real instruction-interpreting code, we measure the test-coverage of the
target code when tested by Learn&Fuzz. Our experimental results show
the coverage is surprisingly low, and the analysis of the results open
up new research directions to enhance learning-assisted grammar-based
fuzzing.

Keywords: Grammar-based fuzzing · Grammar learning

1 Introduction

1.1 What Is Fuzzing?

Definition of Fuzzing: Input fuzzing (or fuzzing) is an automated software-

testing technique for finding security vulnerabilities in input-parsing code. To
find vulnerabilities, it tests the target input-parsing code by repeatedly feeding
synthetic inputs. The process of synthesizing inputs is aimed at guiding the

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 470–484, 2020.
https://doi.org/10.1007/978-3-030-15032-7_40
 Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing 471

target input-parsing code towards certain execution states which trigger security
bugs or vulnerabilities.
Input fuzzing techniques can be classified into two categories, in terms of
how test inputs are synthesized: mutation-based and generation-based fuzzing.
Mutation-based fuzzing synthesizes test inputs by modifying existing ones. Using
randomization or heuristics, it mutates existing test inputs to synthesize new
ones, which lead to the exploration of those execution states (or security vul-
nerabilities) not found by existing test inputs. Mutation-based fuzzing has an
advantage that it does not require the expert knowledge about the specifica-
tion of valid test inputs. In contrast, generation-based fuzzing synthesizes new
test inputs based on a specification of valid inputs. Unlike mutation, generation-
based fuzzing can always synthesize well-formed test inputs, though it requires
the (manually-written) input specification from which new inputs are generated.
Goal of Fuzzing: The goal of fuzzing is to find security bugs or vulnerabilities
in the target program under test by repeatedly feeding those test inputs that will
guide the target towards unexplored execution states triggering vulnerabilities.
Existing fuzzers [9] have been shown to be effective at finding vulnerabilities such
as buffer overflows, use-after-frees, cross site scripting (XSS), and SQL injection
(SQLI). A recently proposed fuzzer (IoTFuzzer [1]) focuses on the detection of
security bugs in IoT applications.
Representative Approaches for Fuzzing: There are three major approaches
for implementing a fuzzing tool: black-box, white-box, and grammar-based
fuzzing We briefly describe each of them below.

1. Black-Box Fuzzing. Black-box fuzzing [13] is a fuzzing technique, which

randomly mutates well-formed inputs, often at a byte granularity. It does
not require the users of having any knowledge about the internal structure of
the target code under testing. Black-box fuzzing is thus known to be easy to
apply to many applications. However, mutated inputs are not guaranteed to
be well-formed, often resulting in very low code coverage.
2. White-Box Fuzzing. In contrast with black-box approach, white-box
fuzzing mutates existing test inputs to get new ones, considering the inter-
nal structure of the application under test. Typical white-box fuzzers, such as
SAGE [4], employ symbolic execution to collect path constraints, which reflect
the internal structure of the target code, and then use a constraint solver to
generate new test inputs. The generated test inputs are mostly well-formed,
and are promising at guiding the target application towards unexplored exe-
cution paths, thus increasing code coverage.
3. Grammar-based Fuzzing. Grammar-based fuzzing [12,13] targets appli-
cations that take complex structured inputs, such as parsers. It utilizes a
grammar, which specifies the valid format of the inputs, in order to gener-
ate new test inputs. Grammar-based fuzzing allows for the generation of test
inputs at token granularity, rather than at byte granularity, and the generated
inputs are guaranteed to be well-formed.
472 Y. Jitsunari et al.

1.2 Problems with Grammar-Based Fuzzing

Traditional grammar-based fuzzing suffers from two major problems: (1) its
application cost is prohibitively high and (2) the code coverage achieved by
it is low. The first problem with grammar-based fuzzing is its application cost.
Most of the existing grammar-based fuzzers require the users of describing the
input grammar manually. Moreover, the expert knowledge is mandatory for the
correct description of the input grammar. These requirements make the applica-
tion cost of grammar-based fuzzing prohibitively high, hindering its wide spread
use in practice. The second problem with grammar-based fuzzing is the low code
coverage. Though the new inputs generated by grammar-based fuzzing are guar-
anteed to be well-formed even for a complex input structure, such inputs give
no guarantee that they can guide the target code towards unexplored execution
paths. The space of well-formed inputs conforming to a complex input grammar
is vast, and in general, only a relatively-small subset of the space can contribute
to the high code coverage. However, grammar-based fuzzing alone cannot sup-
port the generation of such test inputs that induce high code coverage.

1.3 Learn&Fuzz: A New Grammar-Based Fuzzing

To overcome the first problem with grammar-based fuzzing (i.e., its high appli-
cation cost), Godefroid et al. propose Learn&Fuzz [3], a machine learning-based
technique for automating the generation of inputs conforming to a structured
grammar. Using a large amount of example structured-inputs (i.e., PDF objects),
Learn&Fuzz trains a recurrent neural network to learn a generative model, which
can automatically generate new test inputs that are statistically well-formed
with respect to the structured input grammar. Unlike traditional grammar-based
fuzzing, Learn&Fuzz automates the input-grammar synthesis to relieve the users
of manually describing the complex input grammar. This reduces the application
cost of Learn&Fuzz, which can be viewed as having solved, to some extent, one
of the major two problems with grammar-based fuzzing.

1.4 Problems with Learn&Fuzz

Though it is promising for easily fuzz-testing applications with a structured input

grammar, Learn&Fuzz still suffers from two major problems, described below.

1. Generating an Instruction Sequence. Learn&Fuzz cannot automatically

generate an instruction sequence, a more complex form of structured inputs.
Although it succeeded in learning a simple input grammar, which specifies
the text format of PDF objects, Learn&Fuzz cannot learn a generative model
for generating an instruction stream, which constitutes an important binary-
format part of the entire PDF file. In general, Learn&Fuzz is not able to
automatically generate an instruction sequence, which frequently appears in
structured inputs, not limited to PDF files.
 Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing 473

2. Achieving High Code Coverage. Despite its ability to learn a structured

input grammar, Learn&Fuzz cannot achieve high code coverage, especially
when applied to applications with structured inputs that include an instruc-
tion sequence.

1.5 Contributions
Towards the goal of establishing learning-assisted grammar-based fuzzing that is
both easy-to-use and achieves high code coverage by addressing open problems
with Learn&Fuzz, this paper makes, as a first step, the following key contribu-
tions.

1. Quantifying the Limitations of Learn&Fuzz. We quantify the limi-

tations of Learn&Fuzz, a state-of-the-art learning-assisted grammar-based
fuzzing. For this purpose, we reimplement the core of the Learn&Fuzz sys-
tem to measure the code coverage when we apply it to an instruction-stream
parser embedded in PDFium, a PDF renderer in the Chrome browser.
2. Defining Future Directions of Grammar-based Fuzzing. Through the
analysis of the measurement of the Learn&Fuzz limitations, we define the
future directions of research in learning-based grammar-based fuzzing. Espe-
cially, we point out the significance of learning and generating an instruction
sequence, which is crafted to guide the fuzzer towards unexplored parser-code
paths for achieving higher coverage.

2 Grammar-Based Fuzzing and Its Problems

2.1 Grammar-Based Fuzzing
Neither black-box nor white-box fuzzing can achieve high code coverage, when
applied to programs, such as parsers, which take structured inputs conforming to
a grammar. The reason is that both fuzzing techniques cannot always generate
well-formed (or valid) structured inputs, which pass lexical or syntax checking
by the parser. Both black-box and white-box fuzzers mutate existing test inputs
in a fine-grained manner, typically at byte granularity. Such fine-grained input
mutation does not consider the input grammar. As a result, new test inputs
obtained are likely to violate the input grammar or be ill-formed. Ill-formed
(or invalid) inputs are quickly rejected by the parser upfront, and therefore
such inputs are not able to guide the target code under fuzzing towards deep
execution paths, which trigger security vulnerabilities. Thus, both black-box and
white-box fuzzers face the low code-coverage, when applied to structured input-
parsing code.
In contrast, grammar-based fuzzing is a promising technique for achieving
high code coverage of structured input parsers. Grammar-based fuzzing mutates
existing test inputs in a coarse-grained manner, namely at token granularity.
Further, it constrains the way to mutate inputs so that new inputs obtained can
conform to an input grammar. New test inputs generated by grammar-based
474 Y. Jitsunari et al.

fuzzing are thus guaranteed to be well-formed (or valid). Unlike ill-formed ones,
well-formed inputs pass the lexical or syntactic checking by the parser, and
therefore have an opportunity to guide the target code under test towards deep
execution states, which may involve security bugs. Grammar-based fuzzing can
thus achieve higher code coverage of structured input parsing code than typical
black-box or white-box fuzzing.

2.2 Motivating Example of Structured Inputs

As an motivating example of structured inputs conforming to a complex

grammar, we use PDFs throughout this paper, which is consistent with the
Learn&Fuzz paper [3].

Organization of a PDF File. A PDF file consists mainly of three elements.

The first element is a PDF body, which is composed of multiple PDF objects.
Each PDF object contains the sub-contents of a PDF document. In a sense, a
PDF object corresponds to a variable in a program, because it can takes on
one of eight types: boolean, number, string, name, array, dictionary, stream, and
null. The declaration of a PDF object starts with the string obj and ends with
endobj. PDF objects may contain stream objects. Stream objects consist of an
instruction sequence, which instructs the PDF rendering engine how to display
characters and images within a PDF document.
The second element of a PDF file is a cross-reference table. Each table holds
metadata, such as the location of each PDF object within the PDF document
and whether each PDF object is in use or not. The location of each PDF object
is specified as the starting address of the object, and the address is in byte offset
from the beginning of the PDF document.
The third element is a trailer. Each trailer holds metadata for the PDF file,
such as the location of each cross-reference table, the size of the entire PDF file,
and the number of PDF objects within the PDF file.

Stream Object. Stream objects are a variant of a PDF object. A stream object
contains data bytes, such as image files and an instruction sequence (or a page
stream) to instruct the PDF renderer how to display images and characters in the
PDF document. Typically, a stream object is embedded in the PDF document
in a compressed binary format. Similar to a PDF object, the declaration of a
stream object starts with the string stream and ends with endstream. Note
that Learn&Fuzz cannot deal with stream objects by design: it excludes stream
objects from a corpus of PDF objects for learning.

Page Stream. A page stream is a part of a stream object. Each page stream
consists of a sequence of instructions, which instruct the PDF rendering engine
the display of images and characters in the PDF document. Each instruction
is described in postfix notation, i.e., it starts with operands and ends with an
 Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing 475

operator. The notation of page-stream instructions is close to that of PostScript

or FORTH.

2.3 Problems with Grammar-Based Fuzzing

Despite of its effectiveness at fuzzing programs with structured inputs, grammar-

based fuzzing suffers from the following two major problems.

1. High Application Cost. Most of grammar-based fuzzers require the users

to write a complex input grammar by hand. This task requires an expert
knowledge of the grammar, takes much time, and is quite difficult to complete
correctly. Thus, the cost of applying grammar-based fuzzing is typically high.
2. Low Code Coverage. Test inputs conforming to an input grammar cannot
necessarily achieve high code coverage. Rather, in general, only a small subset
of the well-formed input space contributes to increasing code coverage. In
other words, those inputs which are generated merely by the grammar tend
to result in low code coverage.

3 Learn&Fuzz

To address the two problems inherent to grammar-based fuzzing (Sect. 2.3),

Godefroid et al. proposed Learn&Fuzz [3]. Learn&Fuzz makes the following two
significant contributions.

1. Lowering Application Cost. Using a statistical machine learning tech-

nique, Learn&Fuzz automates the synthesis of a structured input grammar.
This relieves the users from writing a complex input grammar by hand, thus
lowering the cost of applying grammar-based fuzzing. Based on a large corpus
of well-formed PDF objects, Learn&Fuzz trains a recurrent neural network
(char-rnn [7,14]) to learn a generative model for automatically generating
new PDF objects. The training process is done in an end-to-end manner, and
therefore the application cost of Learn&Fuzz is low enough.
2. Increasing Code Coverage. In addition to automating the input-grammar
synthesis, Learn&Fuzz makes an effort to increase code coverage. By modi-
fying (or fuzzing) a newly generated PDF object in a probabilistically con-
trolled manner, Learn&Fuzz can steer the target code towards unexplored
error-handling code. This contributes to increasing the overall code coverage.

3.1 Overview of the Learn&Fuzz System

Learning a Generative Model for PDF Objects. Given a large corpus of

PDF objects, Learn&Fuzz trains LSTM [8], a variant of RNN, to learn a gen-
erative model for automatically generating new PDF objects. Unlike traditional
RNNs, an LSTM is able to predict the next character based on the previous
long sequence of characters rather than the last one character, because it avoids
476 Y. Jitsunari et al.

the vanishing gradient problem, which traditional RNNs suffer from. Exploiting
the ability of LSTM to learn and infer long sequences of characters, Learn&Fuzz
has succeeded in learning a generative model of PDF objects, which consist of a
long sequence of characters.
To train an LSTM model for a generative model of new PDF objects,
Learn&Fuzz utilizes a large corpus of PDF objects, regarding each PDF object
as a sequence of characters. As the first step of the training process, Learn&Fuzz
concatenates all of the PDF object files si in the corpus into one, which gener-
ates a very long sequence of characters S = s1 + ... + sn . As the second step,
it divides the very long sequence into multiple fixed-size sequences, which are
used as training input sequences. In Learn&Fuzz, each split sequence is set to
have fixed size d = 100 characters. As the third step, it prepares training output
sequences by shifting each input sequence by one character. More formally, if we
denote the subsequence of S between indices k and l as S[k : l], the i-th training
input sequence ti is expressed as S[i ∗ d : (i + 1) ∗ d] and the i-th training output
sequence to as S[i ∗ d + 1 : (i + 1) ∗ d + 1]. As the final step, Learn&Fuzz trains
an LSTM to learn a generative model, by giving so constructed input-output
training sequences to the LSTM. This training is performed end-to-end, thus
reducing the application cost of the Learn&Fuzz system.
The learnt generative model can not only predict the next character c, given
an input sequence seq, but also return its probability p(c). Namely, the LSTM
model can learn a probability distribution D(seq, θ) over a sequence seq of char-
acters, which can return p(c|seq), the probability of the next character c when
given an input sequence seq of characters.

The SampleFuzz Algorithm for Better Code Coverage. Using the learnt
LSTM model, Learn&Fuzz generates new PDF objects for fuzzing. The detailed
algorithm SampleFuzz for this generation process is shown below.
The SampleFuzz Algorithm[3]
1 s e q := ” o b j ”
2 w h i l e not s e q . en d swith (” en d ob j ”) do
3 c , p ( c ) := sample ( D ( seq , θ ) )
4 pf uzz := random ( 0 , 1 )
5 i f pf uzz > tf uzz ∧ p ( c ) > pt then
6 c := argmin c {p ( c ’ ) | D ( seq , θ ) }
7 end i f
8 s e q := s e q + c
9 i f l e n ( s e q ) > MAXLEN then
10 s e q := ” o b j ”
11 end i f
12 end w h i l e
13 return seq

The two key ideas of the SampleFuzz algorithm are: (1) to sample the learnt
distribution D(seq, θ) to get the next character c and its probability p(c), given
 Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing 477

an input sequence seq (line 3), and (2) to inject anomaly into the PDF objects
generated in a probabilistically-controlled manner (lines 4 through 8) so that it
can cover more unexplored code such as error-handling code.
The SampleFuzz algorithm starts to generate a new PDF object, by setting
the initial sequence seq as “obj” (line 1). It repeats the generation of the next
character c until the generated output sequence seq ends with “endobj” (line 2).
In each iteration of the while loop, it samples the learnt probability distribution
D(seq, θ) to get the next character c and its probability p(c), given the current
input sequence seq (line 3). Then it gets a random value pf uzz between 0 and
1, which controls the probability for injecting an anomaly into the generated
sequence (line 4). At line 5, pf uzz and p(c) are compared with tf uzz and pt ,
respectively, where tf uzz and pt are user-given thresholds for controlling the
probability of the anomaly injection. Namely, if pf uzz and p(c) are greater than
tf uzz and pt , respectively, then the next character c (predicted by the learnt
model to have the highest probability) is replaced with another character c ,
which has the lowest probability p(c ) in the learnt model. This replacement
implements a probabilistically-controlled anomaly injection. The main idea of the
SampleFuzz algorithm is thus to probabilistically inject rare-observed characters,
as an anomaly, into the generated PDF object, such that the anomaly injection
takes place only in positions where the learnt LSTM model predicts the next
character with higher probability.

3.2 Evaluation of Learn&Fuzz in the Original Paper

In the original paper [3], the effectiveness of Learn&Fuzz was measured in terms
of two metrics: coverage and pass rate.

1. Coverage. Coverage means the number of unique instructions executed dur-

ing the fuzzing of the target application. In general, the more the instruction
coverage, the more chances security vulnerabilities are found. The original
paper [3] measured the coverage when Learn&Fuzz is applied to the fuzzing
of the PDF parser of Microsoft Edge browser. However, such coverage metric
says nothing about what percentage of the entire instructions were actually
executed during fuzzing.
2. Pass rate. Pass rate is the rate of actually accepted inputs among the gener-
ated ones. A high pass rate indicates the high ability of the generative model
to generate well-formed inputs.

In contrast with the original paper of Learn&Fuzz, we measure, in this paper,

the percentage of actually executed instructions/branches (during fuzzing)
among all of instructions/branches. The reason is that we believe such cover-
age metric is suitable for our purpose: to quantify the limitations of the original
Learn&Fuzz approach. Because of our purpose, we do not measure the pass rate
throughout the experiments in this paper.
478 Y. Jitsunari et al.

3.3 Open Problems with Learn&Fuzz

Learn&Fuzz is promising for fuzzing applications, which take as inputs com-

plex data conforming to a structured grammar. However, it holds two major
problems. The purpose of this paper is to measure how significant the problems
are by quantifying the limitations of Learn&Fuzz, and to figure out the main
causes of the current limitations so as to open up new research directions of
learning-assisted grammar-based fuzzing. We identify two open problems with
Learn&Fuzz as follows.

1. Generating an Instruction Sequence. In general, Learn&Fuzz is not

able to learn a generative model for instruction sequences. Unfortunately,
an important binary-format part of an entire PDF file consists of instruction
sequences, such as page streams in a stream object. The original paper of
Learn&Fuzz claims that black-box fuzzing is more effective, or enough, for
fuzzing the binary-format part of the PDF file than grammar-based fuzzing.
However, we experimentally show that this claim cannot hold for instruction
sequences in PDF files and that learning a generative model for instruction
sequences is significant for achieving higher code coverage of the PDF parser,
which deals with not only text-format PDF objects but also binary-format
stream objects.
2. Achieving High Code Coverage. Learn&Fuzz fails to cover enough parser
code for binary-format stream objects, which include instruction sequences.
The original paper of Learn&Fuzz measures and reports only the number of
unique statements (or instructions) in the parser code for text-format PDF
objects, which are actually covered by fuzz-testing. Based on their measure-
ment, the paper claims that Learn&Fuzz can achieve higher coverage of parser
code for text-format PDF objects than existing fuzzing techniques. However,
we arguably claim that such measurement is not enough and that Learn&Fuzz
can only achieve low code coverage of the parser for binary-format stream
objects, namely, for the instruction-sequence parser. In contrast with the mea-
surement by the original paper, this paper quantifies code coverage in terms
of the percentage of instructions/functions (executed during fuzzing) among
all of the instructions/functions in the parser code. Our measurement is more
suitable for pointing out how low the code coverage achieved by Learn&Fuzz
is, especially for instruction-sequence parser code.

4 Experiments

4.1 Purpose of Experiments

The purpose of experiments in this section is to quantify the two open problems
with Learn&Fuzz described in Sect. 3.3. For this purpose, we use our reimple-
mentation of the Learn&Fuzz system to perform experiments to measure its
limitations in terms of the following two metrics.
 Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing 479

1. Capability to Generate Instruction Sequences. We investigate the

capability of Learn&Fuzz to generate instruction sequences in the page
streams of stream PDF objects.
2. Code Coverage. We quantify Learn&Fuzz’s coverage of the parser code for
instruction sequences in the page streams.
Unlike the original paper of Learn&Fuzz, our coverage metric is the per-
centage of instructions/functions (executed during fuzzing) among all of the
instructions/functions of the tested parser code. We believe such metric is more
suitable for pointing out how low the coverage by Learn&Fuzz is and to what
degree the capability of Learn&Fuzz to generate instruction sequences is limited.

Table 1. Experimental environment

Hardware/software Spec/version
OS ubuntu16.04 LTS
CPU Intel core i7-6700
GPU GeForce GTX980
RAM 16 GB
Cuda V8.0.61
Tensorflow gpu 1.4.0
Keras 2.1.3

4.2 Methodology
Experimental Setup. Since the source code of the Learn&Fuzz system is not
open, we have implemented our original prototype of Learn&Fuzz to perform
our experiments. As the target application for fuzzing, we use PDFium [6], an
open-source PDF rendering engine of the Google Chrome browser [5]. Of course,
PDFium includes a PDF parser that deals with instruction sequences in page
streams of stream PDF objects. PDFium can be built by the Clang compiler [10]
of LLVM [11]. At compile-time, the Clang compiler inserts coverage-measuring
code into PDFium, which collects the information of coverage metric at run-time.
Other hardware/software setup for our experiments is shown in Table 1.

Training and Learning Methods. For our experiments, we followed the three
steps below to prepare new instruction sequences in page streams of stream PDF
objects, which were given to the PDFium to measure our coverage metric.
1. Collecting a Training Dataset. As a training dataset for our Learn&Fuzz
system, we collected 261 PDF files (which contain 10988 page streams in total)
from the Internet. After the original paper of Learn&Fuzz, we use a host PDF
file (hostPDF, in short) to host new page streams that our Learn&Fuzz system
generates from the learnt model. Our coverage measurement is done by giving
hostPDFs embedded with new page streams to the PDFium.
480 Y. Jitsunari et al.

2. Training an LSTM. After the original paper of Learn&Fuzz, we use an

LSTM [8] model with two hidden layers. Each layer is composed of 128 nodes
(for 128 hidden states). Following the same procedure described in Sect. 3.1,
we train the LSTM to learn a generative model for new page streams (i.e.
instruction sequences) of stream PDF objects.
3. Generating New Instruction Sequences in Page Streams. Using the
learnt model, we generate new 1000 page streams, which contain new instruc-
tion sequences. We incorporate the generated page streams into a hostPDF
to obtain new 1000 PDF files, which are given to the PDFium for coverage
measurement.

4.3 Results

We show the results of measuring coverage by each fuzzing technique in Table 2.

The row “Baseline” indicates the line and function coverage of PDFium when
it is given 1000 PDF files, which are generated from a hostPDF by embed-
ding it with 1000 page streams randomly selected from 10988 page streams
collected from the Internet. These 1000 page streams were decoded into text-
format before embedded into the hostPDF. The row “Baseline+Black-box” indi-
cates the coverage as with “baseline” except that the page streams, which
are decoded and embedded into hostPDF, are directly modified by a black-
box fuzzing technique. After the original paper of Learn&Fuzz, we use a fuzz-
factor of 100. This means that the length of the page stream divided by 100
is the average number of bytes to be modified in that page stream. The row
“Baseline(Flate/LZW/RL encode)+Black-box” indicates the coverage as with
“baseline” except that the page streams are decoded by Flate/LZW/RL before
embedded into the hostPDF. The row “Learn&Fuzz” indicates the coverage of
PDFium that Learn&Fuzz achieves by giving PDFium 1000 PDF files, which are
generated from a hostPDF by embedding it with 1000 page streams generated
by the procedure described in the previous section. The table shows coverage
measurements as the average of 10 test runs.

Table 2. Coverage comparison.

Fuzzing technique Line Function

Baseline 10240.4 (12.2%) 1349.7 (20.3%)
Baseline+Black-box 10678.3 (12.6%) 1368.2 (20.6%)
Baseline(Flate encode)+Black-box 5130.9 (6.0%) 849.5 (12.7%)
Baseline(LZW encode)+Black-box 4143.0 (4.8%) 754.0 (11.3%)
Baseline(RL encode)+Black-box 4615.9 (5.3%) 780.2 (11.7%)
Learn&Fuzz 9902.7 (11.7%) 1317.9 (19.8%)
 Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing 481

4.4 Analysis of Results

Overall, the coverage by each fuzzing technique including Learn&Fuzz is sur-

prisingly low, although the original paper of Learn&Fuzz claims that black-
box fuzzing is more effective, or enough, for fuzzing the binary-format part of
the PDF file (including PDF streams) than grammar-based fuzzing. However,
the results of our coverage measurements show that this claim cannot hold for
instruction sequences in PDF files.

Table 3. Average length and lines of 340707 objects and 10974 page streams.

Length Lines
Object 112.0 3.3
Page stream 14097.7 772.1

The most notable point is that even Learn&Fuzz cannot achieve high coverage
of PDFium, when applied to the fuzzing of instruction sequences of page streams
in stream PDF objects. We arguably analyze the main reasons for such low
coverage are: (1) failure to generate valid instruction opcode and operands and
(2) failure to generate interesting instruction sequences that contribute to high
code coverage. For instance, the following fragment shows that Learn&Fuzz failed
to generate valid instruction opcode (note that instructions in page streams are
in postfix notation). The correct opcode in line 1 is T* or W* rather than *.
Moreover, the correct opcode in line 2 is TJ instead of J.

Example of failure to generate valid opcode.

1 0 134.132 99.338 ∗
2 [ (W) 34 ( o ) 1 ( r ) −301 ( l ) −479 ( d ) ] J

Another fragment shows that Learn&Fuzz failed to generate valid instruc-

tion opcodes. Though it succeeded in generating valid instruction opcode (TJ),
Learn&Fuzz could not generate valid type and number of operands correspond-
ing to the opcode:

Example of failure to generate valid operands.

1 [ ( H e l l o world 5 . 7 H e l l o new world ] TJ
2 [ ( H e l l o world 1 0 . 0 ) ( H e l l o ] TJ

Moreover, the following fragment is an example indicating that Learn&Fuzz

failed to generate interesting instruction sequences that contribute to high cov-
erage. In the example, Learn&Fuzz succeeded in generating valid opcode and
the corresponding valid operands, but repeated to generate instructions with
the same opcode (l), which do not contribute to achieving high coverage:
482 Y. Jitsunari et al.

Example of failure to generate interesting instruction sequences.

1 100 Tw
2 0 Tc
3 0 −100 Td
4 13.22 6531.66 l
5 604.41 780.435 l ..
6 .
7 Repetition of instructions. with the same operand.
8 ..
9 5325.96 792.53 l
10 28.8 5365.22 l
11 6332.48 6465.56 l

5 Discussion
In this section, we discuss and define new research directions of learning-
assisted grammar-based fuzzing, based on the analysis of the experimental results
in Sect. 4.4. More specifically, we discuss, in more detail, the limitations of
Learn&Fuzz and future approaches for extending it to achieve higher code cov-
erage of the parser of instruction sequences such as page stream in stream PDF
objects.

5.1 Generating Valid Instruction Sequences

The first limitation of Learn&Fuzz is that it often fails to generate sequences
of instructions with valid opcode and valid type and number of operands. This
is because the SampleFuzz algorithm predicts and fuzzes (or modifies) the next
character rather than the next token. As a result, it often generates ill-formed
opcode and/or invalid type and number of operands. When given an invalid
instruction (which holds an invalid opcode and/or invalid operands), the PDF
parser tends to move to error-handling code in the parser upfront and therefore
fails to execute unexplored code deep in the parser backend.
A partial solution of addressing this problem would be to use a more advanced
generative model [2], which can be trained and learned both at the character-
level and token-level. Based on this model, it is possible to train and learn a
generative model for instruction sequences, where token-level training/learning
is applied to instruction opcode and character-level training/learning is applied
to instruction operands. Using this model (or similar advanced one), we expect
that we can avoid generating instructions with invalid opcode, which steers the
target parser towards error-handling code in the parser upfront.
However, even if we use more advanced generative model such as the above
one, it will still be an open problem to generate valid type and number of instruc-
tion operands. Thus, establishing training and learning methods for generating
valid operands as well as valid opcode is a significant open problem for advancing
the current learning-assisted grammar-based fuzzing.
 Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing 483

5.2 Generating Instruction Sequences Inducing High Coverage

The second limitation of Learn&Fuzz is that it often fails to generate inter-

esting instruction sequences that contribute to achieving high code coverage
of the parser, even if it succeeds in generating sequences of instructions with
valid opcode and operands. We expect that one of the main reasons for this
problem is that instruction sequences (in page streams of stream PDF objects)
are too long to generate using the generative model of Learn&Fuzz. As Table 3
shows, the average length of PDF objects is around 112 characters, whereas
the average length of page streams is about 14097 characters. We believe that
the gap between the lengths of PDF objects and page streams makes it quite
hard for Learn&Fuzz’s generative model for PDF objects to generate instruction
sequences (in page streams) inducing high code coverage of the PDF parser.
Thus, establishing a technique for generating long instruction sequences that
trigger high code coverage of the parser is also an important open problem with
the current learning-assisted grammar based fuzzing.

6 Conclusions

In this paper, we have quantified the limitations of Learn&Fuzz, state-of-the-art

learning-assisted grammar-based fuzzing. Through our experiments with a reim-
plementation of Learn&Fuzz, we figured out that Learn&Fuzz fails to achieve
enough code coverage, when applied to the fuzzing of a real parser that takes as
inputs instruction sequences such as page streams of stream PDF objects. The
analysis of our experimental results implies that the main reasons for such low-
coverage problem with Learn&Fuzz are two-folds: (1) it often fails to generate
valid opcode and operands of instructions and (2) it is not able to generate long
instruction sequences that steer Learn&Fuzz towards many unexplored code and
induce high code coverage. We thus define future directions of learning-assisted
grammar-based fuzzing as solving these two significant problems.

References
1. Chen, J., Diao, W., Zhao, Q., Zuo, C., Lin, Z., Wang, X.F., Lau, W.C., Sun, M.,
Yang, R., Zhang, K.: IoTFuzzer: discovering memory corruptions in IoT through
app-based fuzzing. In: NDSS (2018)
2. Cummins, C., Petoumenos, P., Wang, Z., Leather, H.: End-to-end deep learning of
optimization heuristics. In: PACT 2017, pp. 219–232 (2017)
3. Godefroid, P., Peleg, H., Singh, R.: Learn & fuzz: machine learning for input
fuzzing. In: ASE 2017, pp. 50–59 (2017)
4. Godefroid, P., Levin, M.Y., Molnar, D.: Automated whitebox fuzz testing. In:
NDSS 2008, pp. 151–166 (2008)
5. Google. https://www.google.com/chrome/
6. Google. https://www.pdfium.org/
7. Graves, A.: Generating sequences with recurrent neural networks. CoRR 2013,
abs/1308.0850 (2013)
484 Y. Jitsunari et al.

8. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8),
1735–1780 (1997)
9. Li, J., Zhao, B., Zhang, C.: Fuzzing: a survey. Cybersecurity 1(1), 6 (2018)
10. LLVM. https://clang.llvm.org/
11. LLVM. https://llvm.org/
12. Purdom, P.: A sentence generator for testing parsers. BIT Numer. Math. 12(3),
366–375 (1972)
13. Sutton, M., Greene, A., Amini, P.: Fuzzing: Brute Force Vulnerability Discovery
14. Burget, L., Cernocky, J., Mikolov, T., Karafiat, M., Khu-danpur, S.: Recurrent
neural network based language model. In: INTERSPEECH 2008, pp. 1045–1048
(2008)
 Predicting Elephant Flows in Internet
Exchange Point Programmable Networks

Marcus Vinicius Brito da Silva(B) , Arthur Selle Jacobs(B) ,

Ricardo José Pfitscher(B) , and Lisandro Zambenedetti Granville(B)

Institute of Informatics, Federal University of Rio Grande do Sul,

Av. Bento Gonçalves, Porto Alegre 9500, Brazil
{mvbsilva,asjacobs,rjpfitscher,granville}@inf.ufrgs.br

Abstract. Internet Exchange Points (IXPs) are high-performance net-

works that allow multiple autonomous systems to exchange traffic, with
benefits ranging from cost reductions to performance improvements. As
in any network, IXP operators face daily management challenges to pro-
mote better usage of the services provided by the network. An essen-
tial problem in IXP management concerns the identification of elephant
flows, which are characterized by having traffic size and duration sig-
nificantly higher than other flows. The current approaches to the iden-
tification of elephant flow in IXP networks depend that the analyzed
flows exceed predefined thresholds to classify them as elephants. How-
ever, although it is not perceptible initially, elephant flows are elephant
ones since their first packet. Hence, in this paper, we present a mech-
anism to predict flows behavior using historical observations and, by
recognizing temporal patterns, identify elephant flows even before they
exceed such thresholds. Our approach consists in predicting new flows
size and duration through a Locally Weighted Regression (LWR) model,
using the previous flows behavior and its temporal correlation with the
new flow. The experimental results show that our mechanism is able to
predict the volume and duration of new flows, and react to elephant flows
rapidly, approximately 50.3 ms with up to 32 historical samples in the
prediction model. These numbers are much smaller than the time each
flow would take to exceed the thresholds to classify it as an elephant. In
addition, the mechanism accurately predicts up to 80% of elephant flows
in our evaluation scenarios and approximately 5% of false positives.

Keywords: Software-defined networking · Network management ·

P4 language

1 Introduction
Internet Exchange Points (IXPs) are high-performance networks and perform
an essential role in the Internet ecosystem [1], accounting for at least 20% of all
traffic exchanged between autonomous systems (ASes) [2]. As in any network,
IXP operators face daily management challenges to promote a better usage of
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 485–497, 2020.
https://doi.org/10.1007/978-3-030-15032-7_41
486 M. V. B. da Silva et al.

the services provided by the network. An important problem in IXP manage-

ment concerns the identification of elephant flows, who are characterized by
having traffic size and duration significantly higher than other flows [3]. As a
consequence, elephant flows can significantly impact on the traffic of the smaller
flows that share the same path on the IXP infrastructure, thus compromising
the overall perceived network Quality of Service (QoS) [4].
Because of the impacts that elephant flows have on network performance,
network operators must quickly detect them to promptly perform mitigation
actions [5]. As elephant flow classification depends on the flows’ size and dura-
tion, it takes an amount of time t for these values (size and duration) to reach
a pre-established threshold; that delays the reaction in at least t units of time.
However, elephant flows are obviously elephant ones since their first packet,
although detection occurs later (t). Today, there is a lack of solutions that take
this situation into account, thus delaying decisions/actions, and consequently
creating problems for smaller flows.
Proposed solutions such as DevoFlow [6], OpenSample [7], and SDEFIX [8]
present mechanisms for identifying elephant flows in IXP networks using
sFlow [9] and managing paths (as reaction to elephant flows) using SDN/Open-
Flow [10]. As a first attempt to more quickly identify elephant flows in pro-
grammable IXP networks, we had proposed IDEAFIX [11], where flow dura-
tion and size were analyzed for each ingress packet immediately at the edge
P4 [12] switch. Although IDEAFIX reduces the detection delay when compared
to controller-based approaches, it still requires flows size and duration to reach
the thresholds to be identified as elephant flow.
In this paper, to shorten the time interval until classification thresholds
are surpassed, we present a mechanism to identify elephant flows in IXP pro-
grammable networks using historical observations. Our approach consists of pre-
dicting flow size and duration through a Locally Weighted Regression (LWR)
model [13,14]. The sample weights, in the LWR model, are attributed from a
Gaussian distribution adjusted by the network operator according to the desired
sample time window. In addition, the network operator can define a tolerance
range to validate estimations according to the calculated prediction interval for
each of them. If the prediction interval is less than the tolerance, then the inferred
values are considered valid and immediately confronted with thresholds to char-
acterize the flow as an elephant one or not, right in the beginning. When the
prediction interval is larger than the tolerance, the inference is invalidated and
the analysis proceeds to the second approach, based on the IDEAFIX identifi-
cation process.
Experimental results show that our mechanism is able to predict the volume
and duration of new flows and identify elephant flows rapidly, approximately
50.3 ms with up to 32 samples (i.e., previous flows behavior). Even when sample
numbers are significantly large, i.e., 2048 or 4096, our mechanism can predict and
react to elephant flows at an interval of up to 126.3 ms and 174.6 ms, respectively.
These numbers are much smaller than the time at which the flow would take to
exceed the thresholds to classify it as an elephant one. In addition, the packet
 Predicting Elephant Flows in Internet Exchange Point 487

processing time of software-emulated P4 switches also influences the reaction

time. On a hardware switch, packet processing time would be in nanoseconds.
Finally, the mechanism’s accuracy in well-behaved scenarios obtained at least
80% of prediction, even with conservative tolerance, and approximately 5% of
false positives.
The remainder of this paper is organized as follows. In Sect. 2, related work
is discussed. In Sect. 3, we describe the concept of elephant flows and the man-
agement challenges they require in an IXP network, as well as the statistical
method used in our proposal. In Sect. 4, we describe our novel mechanism to
predict and identify elephant flows in IXP networks. In Sect. 5, we present the
evaluation of our proposal, as well as the achieved results. Finally, in Sect. 6 we
present the main conclusions of the study and future work perspectives.

2 Related Work
In DevoFlow [6], the OpenFlow protocol is used to keep track of elephant flows
with different monitoring mechanisms, such as packet sampling and port trig-
gering. DevoFlow changes the OpenFlow switches by adding new features and
counters to the hardware in order to facilitate the identification of flows in the
network. In OpenSample [7], sFlow is used to perform flow sampling on a solu-
tion to manage SDN networks. Then, flow rates are calculated by subtracting
TCP sequence numbers from two samples of the same flow and dividing the value
by the elapsed time between them. However, the statistics of the switches are
not considered and the flows need to be sampled and sent to the control plane
to be processed. This delays elephants flow identification and add a significant
amount of sampling data in the network. Therefore, only when a threshold is
exceeded, in terms of byte count, the flow is classified as an elephant one and it
is rerouted to the least congested path between the endpoints.
In SDEFIX [8], an identification module is used to classify flows, analyzing the
collected data by sFlow according to predefined rules. When size and duration
thresholds are exceeded, the flow is classified as elephant and it is mitigated
according to policies written by the network operator. For example, elephant
flows can be routed through alternative paths in the IXP infrastructure, so as
not to affect the small flows that are sharing the same paths in the network.
As in the previous approaches, SDEFIX performs elephant flow analysis and
identification integrally in the control plane and a reaction occurs only when
thresholds are exceeded.
In our previous work [11], we proposed IDEAFIX, aiming to identify ele-
phant flows in IXPs faster by relying on programmable data planes. To do that,
IDEAFIX takes advantage of P4 features to store information about the size and
the duration of network flows. This information is extracted from each packet
that enters the network and compared to predefined thresholds for flow classifi-
cation. If flow size and duration do not exceed such thresholds, then it is marked
as not yet elephant, and the packet is routed to the default path. If thresholds
are exceeded, however, then the flow is classified as an elephant one and a noti-
fication is sent to the control plane to report the detected elephant flow. Then,
488 M. V. B. da Silva et al.

the network operator can define actions to mitigate the effects of that flow, in
accordance with, for example, QoS policies.
Although the previous approaches allow the identification of elephant flows
in IXP networks, there is a dependency on that the analyzed flows exceed the
thresholds to be classified as an elephant flow. However, although it is not per-
ceptible initially, elephant flows are elephant ones since their first packet. In this
paper, to dispense with waiting dependence for the threshold to be exceeded,
we present a mechanism to predict flows behavior using historical observations
and, from the temporal patterns recognition, identify elephant flows even before
they exceed thresholds.

3 Elephant Flows in IXP Networks and LWR

In the Internet, most of the flows have a small size and/or lifetime (i.e., mice
flows or small flows) [15,16], although there is a smaller number of flows that
accounts for the majority of the traffic, also having a longer lifetime; these are the
elephant flows [3,17]. The presence of elephant flows in networks is common, and
because they are flows with a long lifetime and size they may cause performance
issues that demand proper management actions from network operators. For
example, elephant flows can impact on smaller flows that occasionally share the
same network data path. Also, elephant flows can consume memory resources
of network devices, and this also can lead to undesirable delays, queuing, and
packet losses [3,8,18]. In the case of IXP networks, the problem turns to be even
more critical, because of the amount of traffic that IXP networks must deal with.
Thus, the faster the elephant flows are identified, the faster their effects can be
mitigated [8].
Figure 1 presents the traffic behavior in Australia IXPs [19] over the course
of a week. IX-Australia offers peering in Western Australia (WA-IX), New South
Wales (NSW-IX), and Victoria (VIC-IX). Although this is an aggregated traffic
behavior, it is possible to recognize a periodic pattern, with discrete variations.
This IXP’s behavior is also seen in the Amsterdam Internet Exchange (AMS-IX),
as well as on more than 30 IXPs networks in Brazil [20]. Other IXPs experience
the same traffic behavior as well. Even though it is not possible to explicitly spot
the elephant flows inside Fig. 1, they are there. In some cases, it is observed that
elephant flows can contribute with more than 59% of total traffic volume [16].
In addition, elephant flows traffic has a substantial, but not perfect, correlation
with traffic in total flows [21].
Considering that there is a periodicity in the traffic of an IXP network (as
shown in Fig. 1), it is possible to predict the size and duration of new flow instances
by observing the previous flows’ temporal behavior. In other words, as the traf-
fic pattern exhibits a periodicity, the events that previously resulted in elephant
flows will later probably happen again. When taking the history of flows to pre-
dict volume and duration of new ones, we also consider that, in such a history,
younger instances of flows must weight more than older instances in the predic-
tion computation. By doing so, we can monitor flows behavior in the IXP network
at run-time and predict the behavior of new flows according to their periodicity.
 Predicting Elephant Flows in Internet Exchange Point 489

Aggregate traffic (Gbps) 250

NSW-IX VIC-IX WA-IX
200

150

100

50

0
Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sunday
Day of the Week

Fig. 1. Australia IXPs aggregate traffic.

3.1 Locally Weighted Regression Prediction Method

To predict network flows behavior, we leverage a statistical method, Locally
Weighted Regression (LWR) [13,22]. LWR allows to predict the value of depen-
dent variables from a set of independent variables, through localized weighting.
LWR is based on the assumption that the neighboring values of the desired
point, in a sample range, are the best indicators for the prediction [13]. In our
context, the size and duration of previous flow instances (samples in the model)
are dependent variables, its start timeStamp are the independent variables, and
the predictions are the new flow size and duration in its start timeStamp (desired
point). The neighbors are the flow instances with timeStamp closest of the new
flow’s timeStamp.
This method fits a surface to “local” points using distance-weighted regres-
sion [14]. LWR is derived from standard linear regression [23], as shown by Eq. 1.
LRW proposal consists of defining the linear model β parameters, minimizing
the squared errors for each sample (yi − f (xi )), weighted locally by wi weight.


n
F (β0 , β1 , . . . , βm ) = wi (yi − f (xi ))2 (1)
i=1

After the derivation [13,22] of Eq. 1, the linear model β parameters, used to
predict values, are obtained by normal equations:
 −1 T T
β = XT W T W X X W Wy (2)
where X is an n × (m + 1) matrix consisting of n data point, each represented
by its m input dimensions and a “1” in the last column, y is a vector of corre-
sponding outputs for each data point, W is a diagonal matrix of order n with the
wi weights of each data point, and β is the m + 1 vector of unknown regression
parameters. Thus, prediction of the outcome of a query point xq becomes:
yq = xTq β (3)
The weights allow establishing the influence of the samples according to their
distance (i.e., age) to the desired point. Many weighting functions are proposed
in literature [13,24]. The widely used weighting function is the Gaussian kernel
weighting function [25], which can be defined as follows:
490 M. V. B. da Silva et al.

 
−d2i
wi = exp (4)
2k 2
From Eq. 4, each sample (xi , yi ) in the model will receive a weight wi in the
interval [0, 1] (as shown in Fig. 2(a)), according to its distance di = (xq − xi )2
to the desired point xq . That is, how much less the distance of the (xi , yi ) point
to the (xq , yq ) desired point (i.e., di ≈ 0), the greater the influence of (xi , yi ) on
the model because weight will be closer to 100% (i.e., wi ≈ 1). The parameter
k scales the size of the kernel to determine how local the regression will be.
Figure 2(a) shows the Gaussian kernel magnitude with k = 1.0 and k = 3.0.
That is, the greater the value of k, the larger the significant samples range in
the model. However, if k is small, then the model will be influenced only by very
close samples.
Figure 2(b) shows the model behavior influenced by the adjustment of the
magnitude k of the Gaussian kernel. The samples were obtained by: f (x) =
−(sin(x)+(0.4x)+0.26η)+9; where η is a samples random noise, and 0  x  2π.
Due to small sample variation (6.5  y  9.0) in the model, when k = 1.0, LWR
result is well behaved and smoothly. When k = 3.0, the result is similar to
traditional linear regression, which does not allow to follow the sample changes.
The fine adjustment of parameter k is essential to use LWR properly. We propose
k to be defined by the sample standard deviation of the xi independent variables.
This allows to dynamically adapt the Gaussian kernel according to changes in
the behavior of the network.

1.0
k = 3.0 9.0 k = 3.0
k = 1.0 k = 1.0
0.8 8.5

0.6 8.0
w

7.5
0.4
7.0
0.2
6.5
0.0
6.0
−6 −4 −2 0 2 4 6 0 1 2 3 4 5 6
d x

(a) Gaussian kernel magnitude. (b) LWR influenced by k parameter.

Fig. 2. Gaussian kernel and LWR prediction interval.

We use prediction intervals [22] to assess the quality of fits in predictions.

Prediction intervals Iq are expected bounds of the prediction error at a query
point xq , which can be defined as follows:

Iq = xTq β ± tα/2,n −p s 1 + xTq (X T W T W X)−1 xq (5)
 Predicting Elephant Flows in Internet Exchange Point 491

where tα/2,n −p is Student’s t-value of n − p (Eq. 6) degrees of freedom for a

100(1 − α)% prediction bound.


n
n
n = wi2 and p = m (6)
i=1
n

Figure 2(b) shows the prediction interval behavior for (xq , yq ) inferences with
k = 1.0. The prediction interval increases when there is a large sample variation
(e.g., x ≈ 2). When the model is well behaved (e.g., x ≈ 0), the forecast interval
is smaller. This allows obtaining the accuracy of the inference method even with
large sample variations.

4 Proposed Approach
In this section, we describe the proposed mechanism to predict and identify
elephant flows using historical observations and temporal correlation. Figure 3
shows the architecture of our proposal, composed of an IXP network infrastruc-
ture abstraction with a programmable data plane, a historical database, and
a prediction module, in the control plane, attached to the network controller.
When an edge switch receives the first packet of a flow, the control plane receives
a notification to compute the path by which the flow will be routed in the IXP
network. Then, running at the control plane, the mechanism uses the LWR model
to predict the flow size and duration. When predicted values are considered valid,
from the prediction interval tolerated, the mechanism characterizes the flow as
an elephant or not, according to thresholds. Such a process of flows’ parameters
prediction and related characterization is discussed following, in four phases:
sample selection, prediction, validation of predictions and flow classification.

SDN Controller
Historical Prediction
Database Module

Flow Features Control Plane Control Messages

IXP Infrastructure
Data Plane

Fig. 3. Architecture.

The prediction mechanism relies on a historical database to store information

about the behavior of previous flow instances. This information is extracted
directly from the data plane, where an agent uses P4 to account, in each edge
switch, the volume and duration of each flow that ingresses in the IXP network.
492 M. V. B. da Silva et al.

To identify each flow individually, a 5-tuple (i.e., source and destination IP

addresses, source and destination ports, and transport protocol type) is mapped
by hash functions to generate index keys. When a flow is finalized (e.g., TCP
FIN Flag is valid or timeout exceeded), a notification (on top of UDP) is sent
to the control plane to report the flow size and duration along with its 5-tuple
and start flow time (i.e., timeStamp).
The sample selection is based on the tuple that defines the new flow. Initially,
all records with the same source and destination IP address, transport protocol,
and at least one of the application ports are selected. Samples are selected within
a time window defined by the network operator to delimit the size of the historical
database. For example, it is possible to set a sampling interval in days, weeks,
months, and so on. In addition, selection can occur from hierarchical aggregation
of IP addresses, based on HHH [26]. This enables more samples in the inference
process, e.g., to sub-nets behavior analysis.
The prediction step estimates the flow parameters according to historical
information. To do that, we use the previously described LWR model (see
Sect. 3.1). We use flow sizes and duration as dependent variables yq in Eq. 3,
allowing the mechanism to predict these values individually. In both cases, the
independent variables xi are the previous flow timeStamp (samples), and xq is
current timeStamp (new flow). To establish temporal correlation and periodicity,
we defined the sample temporal distance di concerning to the new flow time of
the day (i.e., within 24 h), in according to Eq. 7. That is, the largest temporal
distance of a sample to the current flow will be 24/2 h. Thus, it is possible to cor-
relate periodic occurrences of elephant flows daily. For exemplification purposes,
the timeStamp is used in hours.

di if di ≤ 24/2
di = |xq − xi | mod 24 = (7)
24 − di if di > 24/2
The samples weighting in the LWR model is defined according to Eq. 4. We
make a modification in the Gaussian function, as shown in Eq. 8. From this,
the network operator can establish a minimum weight for a given range of sam-
ples. That is, distances in the range [0, k] may receive a minimum weight equal
to l (e.g., 30% or 40%), when di (Eq. 7) is equal to k. This allows operators to
determine how conservative the inference process will be. For example, in scenar-
ios where the network is saturated, the network operator can be more rigorous
in the samples weighing to define which temporal distance will have the most
significant influence on the predictions.
 2 
di
wi = exp · ln(l) (8)
k2
To adjust the magnitude of the Gaussian kernel, we propose k to be defined
by the sample standard deviation of the xi independent variables. This allows to
dynamically adapt the Gaussian kernel according to changes in the behavior of
the network. Finally, when sample weights are defined, it is possible to determine
 Predicting Elephant Flows in Internet Exchange Point 493

the β parameters of the linear model (Eq. 2) and then predict the new flow
volume and duration (Eq. 3) according to its start timeStamp (i.e., xq ).
After predicting flows parameters, the mechanism performs the validation of
LWR inferred values. For this validation, the network operator must define a
tolerance for the calculated prediction interval over each inference. The predic-
tion interval determines the fluctuation margin for the inferred value, as shown
in Fig. 2(b). Thus, the network operator can determine the tolerance according
to, for example, the demand of the network. That is, when the network is under-
utilized, the operator can set a softer tolerance; otherwise, in saturation cases,
the tolerance can be more conservative.
Lastly, when the inferences are not validated (i.e., pred interv > tol), it
will not be possible to predict the flow behavior, and consequently, it will not
be possible to classify it as an elephant one. In this case, analysis and possible
identification occurs based on our previous IDEAFIX identification reference. In
IDEAFIX, the packets of each flow are analyzed directly in the programmable
data plane, and elephant flow identification happens whenever the volume and
duration of a flow exceed the predefined thresholds.

5 Performance Evaluation
To assess the feasibility of our proposal, we focus on evaluating three main
aspects: (i) reaction time, i.e., the interval between the ingress time of the flow
first packet in the IXP network, the LWR predicting time, and the identified
elephant flow management time; (ii) excess data, i.e., the number of bytes that
transited in default path until the flow has been identified as an elephant one
and a reaction occurs; and (iii) mechanism accuracy, i.e., the percentage of valid
elephant flows predictions in function of the prediction interval tolerance. For
all metrics, lower values are better.
Figure 4 depicts the topology used in the evaluation experiments. We used
a topology based on the AMS-IX [27] infrastructure, as also used in the related
work [8,11], with 8 ASes connected by edge switches to a programmable data
plane IXP. Each edge switch is directly linked to a programmable switch, which,
in turn, has at least two connection paths to the core of the IXP network.
Switches were implemented in the language P416 and by using the software
switch BMv2. The infrastructure was emulated using Mininet 2.3, with a band-
width of 1 Gbps per link and no propagation delay.
We generated a workload with distinct sizes of TCP flows between the con-
nected ASes, using iPerf. The flow bandwidth was established at 10 Mbps, and
the duration was determined through a normal distribution, with a mean of
150 s, and a standard deviation of 20 s, for elephant flows. For small flows, we
used a mean of 10 s and a standard deviation of 4 s [8,11]. The IXP network traf-
fic was distributed periodically over nine weeks (i.e., ≈1,600 h). The thresholds
were defined in 10 MB and 10 s [4,8]. Each experiment lasted 10 min, repeated 32
times, and 2,048 flows were generated, of which 12% were elephant flows [11]. We
used a computer with Intel Core i7-4790 processor, 16GB of RAM, and Ubuntu
16 LTS.
494 M. V. B. da Silva et al.

ASes ASes ASes ASes

P4 Switches

ASes ASes ASes ASes

Fig. 4. IXP network topology [8].

Figure 5(a) shows the results of the reaction time to elephant flows. The
reaction time was analyzed exclusively as a function of the number of samples in
the prediction model. This allows us to observe the time it takes for a reaction
to occur with the prediction model being generated at run-time. In the y-axis,
the average reaction time (in milliseconds) has no significant difference at a 95%
confidence level for a set of up to 512 samples. The most significant difference
is evident when there are sets with at least 1,024, 2,048, and 4,096 samples.
In these cases, the average reaction time varies between 76.3 ms, 126.3 ms, and
174.6 ms, respectively.

200 400
174.6
175 350 335.3
Average Reaction Time (ms)

Average Excess Data (KB)

150 300
126.3 252.6
125 250
100 200
76.3 152.6
75 150
54.2 56.1 112.2
50.3 51.4 51.0 100.6 102.7 102.0 108.5
50 100
25 50
0 0
32 64 128 256 512 1024 2048 4096 32 64 128 256 512 1024 2048 4096
Number of samples (previous flows) Number of samples (previous flows)

(a) Reaction time. (b) Excess data.

Fig. 5. Reaction time and excess data analysis.

These numbers are much smaller than the time at which the flow would take
to exceed the thresholds to classify it as an elephant. That is, for the thresholds
we used, based on the related work [8,11], an elephant flow would take at least
10 s to be identified, at best case. Thus, it is possible to notice that predicting
 Predicting Elephant Flows in Internet Exchange Point 495

100

80

Valid Predictions (%)

60

40

20 C1
C2
C3
0
5 15 25 35 45 55 65 75 85 95
Tolerance (%)

Fig. 6. Predict mechanism accuracy.

the flows behavior early on allows one to identify the elephant flows as soon as
possible. Moreover, the packet processing time of software-emulated P4 switch
also influences the reaction time in our approach. On a hardware switch, packet
processing time would be in nanoseconds.
Figure 5(b) shows the elephant flows data volume that traveled along the
standard path until a reaction occurred. In the y-axis, the average excess data
(in Kilobytes) has no significant difference at a 95% confidence level for up to 512
samples. As in the previous analysis, the most significant difference one is evident
when there are sets with at least 1,024, 2,048, and 4,096 previous flow samples
in the prediction model. In these cases, we can consider that the elephant flows
still transmit about 152.6 KB, 252.6 KB, and 335.3 KB on the standard path in
the network, respectively.
The mechanism accuracy (in Fig. 6) has evaluated from the percentage of
valid predictions, i.e., correctly predicted and identified elephant flows. The first
scenario (C1 curve) with low traffic behavior variation and regular periodicity,
i.e., flows follow a well-defined frequency. Scenario two (C2 curve) with more
significant variation in the flows behavior and periodicity. In scenario three (C3
curve), there was a high variation in the flows behavior and low periodicity,
i.e., flows do not have a well-defined frequency pattern. Thus, we evaluate the
percentage of valid predictions according to the tolerance defined by the network
operator for the prediction interval. Prediction interval was calculated for each
predicted value at a confidence level of 90%.
Figure 6 shows that in a well-behaved scenario (C1 curve), 80% of the pre-
dictions are validated even with a conservative tolerance (e.g., 15%). In the
worst case (C3 curve), when the traffic behavior and periodicity are not regular,
approximately 20% of the predictions could be validated, with a tolerance of at
least 65%. In the mean case (C2 curve), more than 60% of the predictions were
validated at a tolerance of 65%. These results show that the method can predict
and validate values even in non-regular scenarios. These results represent, at
least, 90% of success in elephant flows identification, out of the total number
of elephant flows inserted in the network at each test (i.e., ≈245 flows), with
496 M. V. B. da Silva et al.

approximately 5% of false positives. Lastly, the elephant flows whose predictions

were not valid, was identified directly in the data plane, based on our previous
IDEAFIX [11] approach, immediately after exceeding the thresholds.

6 Conclusions
In this paper, we presented a mechanism to predict and react to elephant flows
in programmable IXP networks before thresholds are exceeded. Our approach
consists of predicting flow size and duration through a Locally Weighted Regres-
sion (LWR) model, following the temporal correlation between the behavior of
previous flows. Our experimental results show that our mechanism is able to
predict the flow features, and react to elephant flows rapidly, i.e., approximately
50.3 ms with up to 32 samples in the model. These numbers are much smaller
than the time at which the flows would take to exceed classification thresholds.
In addition, the mechanism’s accuracy was 80% of validated predictions even
with a conservative tolerance of 15%, and approximately 5% of false positives.
As future work, we will consider other methods, based on machine learning, to
predict flows behavior and mechanisms to define the thresholds dynamically.

Acknowledgement. We thank CNPq for the financial support. This research has
been supported by call Universal 01/2016 (CNPq), project NFV Mentor process
423275/2016-0.

References
1. Augustin, B., Krishnamurthy, B., Willinger, W.: IXPs: mapped? In: ACM SIG-
COMM Conference on Internet Measurement, IMC 2009, pp. 336–349. ACM (2009)
2. Cardona Restrepo, J.C., Stanojevic, R.: IXP traffic: a macroscopic view. In: Pro-
ceedings of the 7th Latin American Networking Conference, pp. 1–8. ACM (2012)
3. Guo, L., Matta, I.: The war between mice and elephants. In: 2001 Ninth Interna-
tional Conference on Network Protocols, pp. 180–188. IEEE (2001)
4. Knob, L.A.D., Esteves, R.P., Granville, L.Z., Tarouco, L.M.R.: Mitigating elephant
flows in SDN–based IXP networks. In: 2017 IEEE Symposium on Computers and
Communications (ISCC), pp. 1352–1359. IEEE (2017)
5. Gregori, E., Improta, A., Lenzini, L., Orsini, C.: The impact of IXPs on the AS-
level topology structure of the internet. In: Computer Communications, pp. 68–82.
Elsevier (2011)
6. Curtis, A.R., Mogul, J.C., Tourrilhes, J., Yalagandula, P., Sharma, P., Banerjee, S.:
DevoFlow: scaling flow management for high-performance networks. In: ACM SIG-
COMM Conference on Internet Measurement, vol. 41, pp. 254–265. ACM (2011)
7. Suh, J., Kwon, T.T., Dixon, C., Felter, W., Carter, J.: OpenSample: a low-latency,
sampling-based measurement platform for commodity SDN. In: 34th IEEE Inter-
national Conference on Distributed Computing Systems (ICDCS), pp. 228–237.
IEEE (2014)
8. Knob, L.A.D., Esteves, R.P., Granville, L.Z., Tarouco, L.M.R.: SDEFIX–
identifying elephant flows in SDN-based IXP networks. In: IEEE/IFIP Network
Operations and Management Symposium (NOMS), pp. 19–26. IEEE (2016)
 Predicting Elephant Flows in Internet Exchange Point 497

9. sFlow: sFlow.org (2018). http://www.sflow.org

10. McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford,
J., Shenker, S., Turner, J.: OpenFlow: enabling innovation in campus networks. In:
ACM SIGCOMM Conference on Internet Measurement, pp. 69–74. ACM (2008)
11. da Silva, M.V.B., Jacobs, A.S., Pfitscher, R.J., Granville, L.Z.: IDEAFIX: identify-
ing elephant flows in P4-based IXP networks. In: Proceedings of the IEEE Global
Telecommunications Conference (GLOBECOM). IEEE (2018)
12. Bosshart, P., Daly, D., Gibb, G., Izzard, M., McKeown, N., Rexford, J., Schlesinger,
C., Talayco, D., Vahdat, A., Varghese, G., et al.: P4: programming protocol-
independent packet processors. In: ACM SIGCOMM Conference on Internet Mea-
surement, pp. 87–95. ACM (2014)
13. Cleveland, W.S., Devlin, S.J.: Locally weighted regression: an approach to regres-
sion analysis by local fitting. J. Am. Stat. Assoc. 83(403), 596–610 (1988)
14. Elattar, E.E., Goulermas, J., Wu, Q.H.: Electric load forecasting based on locally
weighted support vector regression. IEEE Trans. Syst. Man Cybern. Part C (Appl.
Rev.) 40(4), 438–447 (2010)
15. Zhang, Y., Breslau, L., Paxson, V., Shenker, S.: On the characteristics and origins
of internet flow rates. In: ACM SIGCOMM Conference on Internet Measurement,
vol. 32, no. 4, pp. 309–322. ACM, New York, August 2002
16. Mori, T., Kawahara, R., Naito, S., Goto, S.: On the characteristics of internet
traffic variability: spikes and elephants. IEICE Trans. Inf. Syst. 87(12), 2644–2653
(2004)
17. Fang, W., Peterson, L.: Inter-AS traffic patterns and their implications. In: IEEE
Global Telecommunications Conference (GLOBECOM), vol. 3, pp. 1859–1868.
IEEE (1999)
18. Mori, T., Uchida, M., Kawahara, R., Pan, J., Goto, S.: Identifying elephant flows
through periodically sampled packets. In: ACM SIGCOMM Conference on Internet
Measurement, IMC 2004, pp. 115–120. ACM (2004)
19. IX Australia: Australia Internet Exchange Point (2018). https://www.ix.asn.au/
20. Internet Steering Committee in Brazil: Brazil Internet Exchange Points (2018).
http://ix.br/trafego/agregado/rs
21. Li, Y., Liu, H., Yang, W., Hu, D., Wang, X., Xu, W.: Predicting inter-data-center
network traffic using elephant flow and sublink information. IEEE Trans. Netw.
Serv. Manag. 13(4), 782–792 (2016)
22. Schaal, S., Atkeson, C.G.: Robot juggling: implementation of memory-based learn-
ing. IEEE Control Syst. 14(1), 57–71 (1994)
23. Jain, R.: The Art of Computer Systems Performance Analysis: Techniques for
Experimental Design, Measurement, Simulation, and Modeling. Wiley, Hoboken
(1990)
24. Simonoff, J.S.: Smoothing Methods in Statistics. Springer Science & Business
Media, Berlin (2012)
25. Wand, M.P., Schucany, W.R.: Gaussian-based kernels. Can. J. Stat. 18(3), 197–204
(1990)
26. Basat, R., Einziger, G., Friedman, R., Luizelli, M., Waisbard, E.: Constant time
updates in hierarchical heavy hitter. In: ACM SIGCOMM Conference on Internet
Measurement, SIGCOMM 2017, pp. 127–140. ACM (2017)
27. AMS-IX: Amsterdam Internet Exchange Infrastructure (2018). https://ams-ix.
net/technical/ams-ix-infrastructure
 ISDI: A New Window-Based Framework
for Integrating IoT Streaming Data
from Multiple Sources

Doan Quang Tu, A. S. M. Kayes(B) , Wenny Rahayu, and Kinh Nguyen

La Trobe University, Melbourne, Australia

{t.doan,a.kayes,w.rahayu,kinh.nguyen}@latrobe.edu.au

Abstract. Due to the rapid advancement in Internet of Things (IoT),

myriad systems generate data of massive volume, variety and velocity
which traditional databases are unable to manage effectively. Many orga-
nizations need to deal with these massive datasets that encounter dif-
ferent types of data (e.g., IoT streaming data, static data) in different
formats coming from multiple sources. Different data integration mech-
anisms are designed to process mostly static data. Unfortunately, these
techniques are not adequate to integrate IoT streaming data from mul-
tiple sources. In this paper, we identify the challenges of IoT streaming
data integration (ISDI). A generic window-based ISDI approach is pro-
posed to deal with IoT data in different formats and subsequently intro-
duced the algorithms to integrate IoT streaming data obtained from mul-
tiple sources. In particular, we extend the basic windowing algorithm for
real-time data integration and to deal with the timing alignment issue.
We also introduce a de-duplication algorithm to deal with data redun-
dancies and to demonstrate the useful fragments of the integrated data.
We conduct several sets of experiments and quantify the performance
of our proposed window-based ISDI approach. The experimental results,
performed on several IoT datasets, show the efficiency of our proposed
ISDI solution in terms of processing time.

Keywords: IoT streaming data integration · Timing alignment ·

De-duplication · Window-based integration

1 Introduction
Due to the rapid advancement of big data platforms, the need to improve data
access from multiple sources through data analysis and decision support systems
has grown significantly over the last few years. However, with the unprecedented
expansion of data in business, decision makers find difficult to access necessary
data for comprehensive and in-deep analysis. On the one hand, it is critical to
be able to react and respond to queries from clients accurately in a quick and
timely manner. On the other hand, it is necessary to integrate data coming from
multiple sources as the data become infinite and is generated in different streams.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 498–511, 2020.
https://doi.org/10.1007/978-3-030-15032-7_42
 IoT Streaming Data Integration from Multiple Sources 499

In the era of Internet of Things (IoTs), the term streaming data has become
more popular nowadays because of the way data generated from multiples sources
and is needed to process in order [1]. For example, web services/applications
generate big log files, social networks deliver huge information, sensors produce
streaming data and so on. These sources continuously generate streaming data
and typically transfer them into the data records. In both research and industry
arenas, we are currently interested in obtaining the useful data and information
in real-time, but streaming data may come from multiple sources in a way that
it seems to be infeasible to access them as a whole. Nowadays, we are experi-
encing the era of IoT streaming data that usually are represented in different
structures or even semi/non-structures. As such, capturing and/or transferring
these heterogeneous data in different formats (e.g., .csv, .txt, .html, .xml and so
on) into a unified form, is a challenging task [2]. In this research, our focus is to
address this challenge. This is where a framework of IoT streaming data inte-
gration plays a vital role in today’s interconnected environments. The purpose
of the framework is to facilitate the integration of IoT data.
The traditional data integration approaches, such as the schema matching
and mapping [3] and record linkage [4], can deal with the integration of data
obtained from multiple sources, however, these techniques are not adequate to
handle timing conflicts and subsequent timing alignment issues with the goal of
integrating IoT streaming data. In [5], the authors introduce a way to access
and integrate streaming data sources through an ontology-based approach. The
research work in [6] also focuses on the benefits of OBDM (Ontology-Based Data
Management) with the goal of integrating streaming data. Similar to record link-
age approaches, the issue of timing alignment is one of the common limitations
of existing ontology-based approaches to data integration. The research works
proposed in [7,8] are used different window-based approaches to process and col-
late data from different sources. However, these research works are not adequate
to deal with the timing alignment issue that has been mentioned earlier.
Other than traditional data integration solutions, recently Pareek et al. [9]
introduce their Striim engine, a streaming analytics platform, for real-time data
integration with respect to structured data using multiple datasets. This Striim
engine can be used to integrate time-series data with other structured data.
Ahad and Biswas [10] propose a distributed Hadoop-based architecture for inte-
grating data from different small files (e.g., text, document). This Hadoop-based
approach is useful to capture and process data from multiple sources. Simi-
lar to Hadoop-based research, our group has a successful history of capturing
and accessing data from a single source or from multiple distributed sources.
Towards this goal, different context-sensitive access control approaches have
been proposed [11,12] to access necessary data and information resources from
different sources. However, the existing Striim, Hadoop-based and access con-
trol approaches are not adequate to integrate IoT streaming data from multiple
sources.
This paper makes the following key contributions. Firstly, we have intro-
duced a generic model for IoT Streaming Data Integration (ISDI) from multiple
500 D. Q. Tu et al.

sources. Then, we have presented several algorithms for ISDI. We introduce

a generic window-based algorithm, extending basic windowing algorithm. It is
used to process IoT streaming data coming in sequence from multiple sources.
The timing alignment algorithm is used to align different timing conflicts of
IoT time-series data obtained from multiple sources. The de-duplication algo-
rithm is used to resolve data redundancy utilizing entity resolution for different
streaming datasets. Finally, we have conducted several sets of experiments to
demonstrate the practicality of our proposed ISDI approach. In this perspective,
we have different experiment setups, including a simple data integrator dealing
with local data schema individually and a generic ISDI integrator dealing with
all streaming datasets from multiple sources through a unified schema.
The organization of our paper is as follows. We present an application sce-
nario to motivate our research in Sect. 2. In Sect. 3, we introduce a generic Inte-
grator model for our ISDI proposal. Section 4 includes our implementation algo-
rithms for ISDI. Section 5 demonstrates the practicality of our proposed ISDI
solution through an empirical evaluation with respect to several experiment
setups. We use the real IoT datasets to evaluate our approach. Section 6 briefly
discusses the related work and finally, Sect. 7 concludes the paper.

Fig. 1. A scenario of the IoT streaming data integration

2 Motivating Scenario
In this section, we present an application scenario to motivate our ISDI research.
The scenario is illustrated in Fig. 1. Each data sample (e.g., Data1) has a
list of attributes ‘A’. The IoT streaming data can be originated from multiple
sources based on the different times (e.g., duration ‘D’ can be based on the
seconds or minutes). In such situations, the integrated data should be handled
a list of attributes, which is the intersection of all the attributes of different
 IoT Streaming Data Integration from Multiple Sources 501

streaming data instances from multiple sources, illustrated in Fig. 1. In addition,

we consider the final duration ‘D’ for the integrated data, which is the lowest
one of all ‘Ds’ (e.g., seconds, minutes or hours) in order to handle different
timing orders. In this research, we use real streaming datasets from a global
manufacturing company that is designed by many machines along with different
IoT sensors to achieve safety goals and improve operational efficiency. These IoT
sensors usually generate many different time-series data. As such, our aim is to
integrate such data with the goal of integrating them for predictive analytics
in manufacturing (e.g., to determine the critical sensors that have a significant
impact on the long-run maintenance costs). The importance of the IoT data
integration from such sensors is to react quickly to predict machine health. More
details about IoT datasets used in this research can be found in Sect. 5.
From the motivating scenario, we can see that the integration of IoT stream-
ing data along with different timing flows/orders has become critically important
in many real-life applications. Such big data are usually organized into the data
lakes or data warehouses for long-term storage and analytics, and are being
contributing to many industries and businesses nowadays. For example, in our
application scenario, a manufacturing company that runs by many IoT sensors
and globally distributed in a wide variety of regions. Data storing along with
real-time integration in this case is vital to such a manufacturing company, as
it can support to make any decisions for predictive analytics which can reduce
subsequent maintenance efforts and costs. This sort of predictive maintenance
through deeper analytics can have a greater impact on the operation of asso-
ciated machines to find the redemption before any damages, which can save
millions of dollars.

3 Our Proposed Generic ISDI Integrator Model

We present a generic model for integrating IoT streaming data (i.e., time-series
data) from multiple sources, that we call ISDI Integrator. In Fig. 2, we illustrate
the different components of our ISDI integrator model. The base model that we
consider in our research is the basic windowing model, in order to deal with IoT
streaming data in real-time from multiple sources. The ISDI integrator consists
of two different layers: (i) IoT Data sources and (ii) Integrator.

3.1 Data Sources and Managers

The data sources handle IoT streaming data in different formats (e.g., txt, csv,
xml, html and so on). In particular, this layer works as a pipeline including the
manager layer, i.e., different managers to handle different streaming data and
map to a unified format. For example, in our research, we utilize such managers
to model a unified schema as the pairs of <keys, values> format.
In addition, this layer consists of different managers to extract streaming
data from multiple sources and to deal with timing alignment as well. Real-
time incoming data from applications usually store in different formats. The
502 D. Q. Tu et al.

Data Sources

Request Data

Managers
(Data Extraction
&
Timing Alignments)

Request Windows

- Data Request (attributes, Integrator

window size) (Integrating Streaming data Summarized
- Mapping function (user-
outcome
& Windows
defined information) Timing Control)

Fig. 2. ISDI integrator model for IoT time-series data from multiple sources

managers are responsible to handle different formats of streaming data and these
are independent from the main integrator. This is very important to make the
integrator model generic so that the integrator can deal with different formats
of IoT data. We propose a timing alignment algorithm to work in this layer.
For each kind of IoT data format, it needs a manager corresponding to extract
and manipulate streaming data. It aligns streaming data based on the identical
or different time-stamps. Finally, it sends data to the integrator and marks a
pointer according to the last data/record processed. This helps the integrator to
process data in real-time.

3.2 Generic Integrator

The integrator layer collates streaming data from multiple sources and conse-
quently handles data redundancy according to identical or different time-stamps
(i.e., timing controls). We propose two algorithms, named window-based integra-
tion and de-duplication, to work in this layer. This generic integrator is responsi-
ble to control the time of incoming multiple IoT data through different managers.
It executes the relevant queries from the end-users through mapping functions
and selecting relevant data based on their requirements from the queries as
parameters.
For example, the inputs of this integrator are the attributes and the user-
identified mapping function according to the data request, including the size of
the window that is required to identify the fragments of data processed sequen-
tially from multiple IoT sources. The result of the integrator is used to store as
summarized windows.
 IoT Streaming Data Integration from Multiple Sources 503

Algorithm 1. Window-Based Integrator for ISDI

Input: keys, windowSize, summarizedFunction, sourceManagers
Output: integratedWindows
1 List iWindows;
2 while startingTime is before the current time do
3 List iRecords;
4 for each sM in sourceMangagers do
5 List recordsList = sM.getRecords(startingTime, windowSize);
6 for each r in recordsList do
7 Record newR = r.getRecord(keys);
8 iRecords.add(newR);
9 end
10 end
11 Window window = new Window(iRecords, startingTime, windowSize);
12 iWindows.add(window);
13 startingTime.plusMillis(windowSize);
14 listOfRecords;
15 end
16 return summarizedFunction.f(iWindows);

Algorithm 2. Timing alignment while Extracting Data from Multiple

Sources
Input: startingTime, windowSize
Output: listOfRecords
1 List recordsToProcess;
2 int fromIndex = 0; int toIndex = record.size();
3 if lastRecordProcessed != null then
4 fromIndex = records.indexOf(lastRecordProcessed);
5 end
6 endTime = startingTime.plusMills(windowSize);
7 for each r in records.subList(fromIndex, toIndex) do
8 if lastRecordProcessed between startingTime and endTime then
9 recordsToProcess.add(r);
10 lastRecordProcessed = r;
11 else
12 break;
13 end
14 end
15 return recordsToProcess;

4 Implementation Algorithms for ISDI

This section includes our implementation algorithms for ISDI.
The Window-Based Integrator algorithm has inputs including keys, win-
dowSize, summarizedFuntion and sourceManagers. The keys and the asso-
ciated <keys, values> pairs can be found based on the users’ queries
504 D. Q. Tu et al.

Algorithm 3. De-duplication for ISDI

input: listOfRecords, identicalKey, keyOfTime, keys
Result: remove duplicate records and return list of records
1 int recordsPointer = 0;
2 while recordsPointer < listOfRecords.size() - 1 do
3 int pointer = recordsPointer;
4 boolean continuingCondition;
5 while continuingCondition ∧ pointer < listOfRecords.size() do
6 d1 = listOfRecords.get(pointer).getKey(keyOfTime);
7 d2 = listOfRecords.get(pointer + 1).getKey(keyOfTime);
8 if d1.equals(d2) then
9 pointer += 1 ;
10 else
11 continuingCondition = false;
12 end
13 if recordsPointer < pointer then
14 for each k in keys do
15 if !k.equals(identicalKey) then
16 listOfRecords.get(pointer).getValue(
17 sov);
18 else
19 listOfRecords.get(pointer).getValue(
20 mfov);
21 end
22 end
23 else
24 end
25 for k from (recordsPointer + 1) to pointer do
26 listOfRecords.remove(k);
27 end
28 recordsPointer = pointer + 1;
29 end
30 end
31 return listOfRecords;

(i.e., requirements). The windowSize is also another requirement from the users
and the fragments of data can be formed based on what they want in each cer-
tain periods of time. The summarizedFunction is a mapping function and it is
defined based on the users’ requirements as well. This function takes different
data windows from multiple sources as inputs and returns integrated results as
an output, integratedWindows. The Line 5 of this algorithm shows the main
contribution of the sourceManagers. Each source manager is controlled by the
startingTime and the time interval (i.e., windowSize).
The data collection according to different source managers through timing
alignment is introduced later in Timing Alignment algorithm (see Algorithm 2).
After collecting data from different sources, the integrator filters data to take
 IoT Streaming Data Integration from Multiple Sources 505

only these that have relevant keys and adds them into an integrated storage
iRecords (Lines 7 and 8 in Algorithm 1). The filtering is done according to the
de-duplication process introduced later in Algorithm 3. The de-duplication algo-
rithm returns the list of records that are used in this algorithm (see the Line 15
in Algorithm 3). The windows of data are added into integrated windows before
resetting the starting time for the next integration process. During the process of
integrated windows, the output is based on the function summarizedFuntion.f(),
as shown in Line 16 in Algorithm 2.

S1 S2 S1 S2

<Key, Value> <Key, Value>

W1i W2j

W1i W2j

- Timing Alignment
- De-duplicaƟon
- IntegraƟon - Timing Alignment
- De-duplicaƟon
- IntegraƟon

Summarized
Windows

Summarized
Windows

Fig. 3. Simple integrator (without managers) vs generic ISDI integrator

The Algorithm 3 is used to process data duplication and it is also associ-

ated with the integrator model in Algorithm 1. Despite the different time-stamps
(data frequencies, such as seconds, minutes, hours or days), the time-stamps can
also be identical when collected from multiple sources. The algorithm uses two
pointers (see Lines 1 and 3 in Algorithm 3) to count the number of records
that are associated with identical time-stamps. The while loop condition in Line
5 depends on the Lines from 8 to 12, in which the identical time-stamps are
compared with an specified record and the next record. If the time-stamps are
different, the loop will stop; otherwise, the pointer will scan the rest of data
to look for another duplicate records. It then executes their sum of values (i.e.,
‘sov’ in Algorithm 3) to the non-identical attributes (i.e., keys) and aggregate
the values of the identical attributes by a user-defined mapping function (i.e.,
‘mfov’ in Algorithm 3). Finally, all the duplicate records, which are between the
recordsPointer and the pointer, are removed (Line 26 in Algorithm 3).
506 D. Q. Tu et al.

5 Experiment and Evaluation

This section demonstrates the feasibility of our ISDI approach.

5.1 Experiment Setups

In our first set of experiments, we have used a simple window-based approach
to integrate streaming data from different sources. The left side of the Fig. 3
shows the different components of this simple integrator, including two IoT data
sources (S1 and S2), source-based windows (Wi ) and summarized windows. The
simple integrator also includes three main steps, such as timing alignment, de-
duplication and integration. Instead of processing the streaming data, based
on the window-based approach, we can also consider the whole data files (csv
or xml files) at once by following traditional data integration techniques (e.g.,
schema matching). In our second set of experiments, we have considered a non-
windowing technique in terms of conducting an experimental comparison accord-
ing to our window-based approach.
The right side of the Fig. 3 shows our generic ISDI integrator, which is the
main contribution in this paper. We have conducted the third set of experiments
based on our proposed ISDI integrator. Similar to simple integrator, all the
components and steps are also same in our ISDI integrator. The only difference
here that we have different managers to extract streaming data from multiple
sources and convert them into an unified schema (i.e., pairs of <keys, values>).
These managers are also responsible for updating such pairs in real-time when
new streaming data originate from IoT sources.
The sets of experiments are carried out by running on two different machines.
The development environment of these sets of experiments is Java on NetBeans
IDE 8.0.2 and we use a time-based library, named Joda-Time, in order to process
the different time-series from multiple data sources.
M1: A Windows PC of 3.4 GHz CPU, 4 Core(s) and 8 Logical Processors with
16 GB of physical memory.
M2: A Windows Laptop of 2.4 GHz Intel Core processor, 2 Core(s) and 4 Logical
Processors with 16 GB of physical memory.
As discussed in research motivation in Sect. 2, we have used real streaming
data sets from a distributed manufacturing company, which is designed by many
machines along with IoT sensors. In this section, we have used such IoT data
that are generated from these sensors, including second and minute-based time-
series data. In particular, IoT Source 1 has 368,199 records of 94 MB size, IoT
Source 2 has 259,200 records of 62.8 MB size, and IoT Source 3 has 7,257,600
records of 3.05 GB size, respectively.

5.2 Performance Evaluation

In this section, first, we have shown the performance variation of our simple inte-
grator approach with respect to different machines. Then, we have empirically
 IoT Streaming Data Integration from Multiple Sources 507

measured performance of our generic ISDI approach comparing with our simple
integrator approach. In addition, we have demonstrated performance of our ISDI
approach, including semi-structured streaming data in different formats.

Fig. 4. Performance w.r.t. different window sizes

Performance w.r.t. Different Machines: We have considered 1-day window

size to measure the processing time, using the simple window-based approach
(what we have called the simple integrator). What we have seen, it can take 2 min
and 15 s (i.e., 135 s) to integrate IoT data from two sources. Figure 4 illustrates
the processing time of the simple integrator according to the different sizes of
the window.
We can observe that the variation of processing time can be based on differ-
ent factors: the size of the window and the processing power of the machine. In
terms of programming efficiency, these factors can be the complexity of a func-
tions, approaches, and/or number of loops. Towards this end, it is important for
revising the window size as the variation of different sizes affects the number of
loops in our algorithms. We run this set of experiments on two machines M1
and M2. Figure 4 shows that the processing time varies linearly and almost sta-
ble for window sizes from 1-day to 10-day, using our powerful machine M2. It
takes approximately 140 s for all such cases. However, the integration time rises
dramatically while the size of the window is 11-day.

Performance w.r.t. Different Window-Based Approaches: We measure

the performance of our window-based approach using machine M2. In particular,
based on the task sequences of our simple and generic ISDI integrator illustrated
in Fig. 3, we empirically demonstrate the performance of such window-based
approaches. We consider the variation of window sizes from 1-day to 10-day.
Figure 5 demonstrates the performance comparison. We can observe that our
proposed ISDI integrator has a better performance than other traditional non-
window and window-based approaches. The source managers and a unified data
508 D. Q. Tu et al.

Fig. 5. Simple integrator vs generic ISDI integrator

schema (<keys, values> pairs) are the main contributions in this variation of
performance. Overall, the source managers usually help the ISDI integrator to
collate streaming data from multiple sources with a better performance.

Performance w.r.t. Different Data Formats: We conduct our third set of

experiments, using another real dataset, IoT Source 4, that has 56.570 records
of 39.2 MB size (‘xml’ format).

Fig. 6. Performance w.r.t. semi-structured data in different formats

In this set of experiments, we only measure performance of our generic ISDI

integrator. We examine the processing time by varying the sizes of the window
and using semi-structured data in different formats (‘csv’ and ‘xml’). Figure 6
shows the performance variation in terms of 1-day to 10-day window sizes and
using both formats of streaming data. We can observe that the performance of
our proposed ISDI integrator is almost stable in both of the two cases.
 IoT Streaming Data Integration from Multiple Sources 509

6 Related Work
In this section, we provide a short overview of the state-of-the-art big data
processing and integration approaches as the related area of our research.
The earlier research work on record linkage [4] utilizes blocking techniques
to combine the identical data from different sources. This work deals with the
huge amount of data, but not the issue related to timing alignment, where align-
ing streaming data with identical or different time-stamps is one of the critical
concerns. In our work, this important issue of timing alignment is taken into con-
sideration while we need to integrate streaming data from multiple sources. We
have used a unique key for an individual time-stamp and subsequently used
<keys, values> pairs, in order to integrate data in real-time from different
sources. Similar to record linkage research, the schema matching and mapping [3]
approaches are not adequate to handle the same issue of timing alignment. Also,
these existing works are not enough to resolve the data redundancy issue while
dealing with identical or different time-stamps from multiple data sources.
In terms of IoT streaming data integration, there are several ontology-based
research works in the literature. In [5], an ontology-based data integration app-
roach with respect to collating streaming data from different sources has been
introduced. The integration task in this work [5] is based on the ontology-based
access to relational data sources. In this perspective, in [6], the authors have pro-
posed an approach to OBDM (Ontology-Based Data Management) in order to
provide a shared vision to process data from different sources. In these research
works, the OBDM model is proposed and this ontology model is considered as
a global schema for mapping different local schemas into a unified data model.
However, these ontology-based solutions are not adequate to address the issue
of timing conflicts (i.e., aligning different timing frequencies) while integrating
time-series data from multiple sources.
In [7], the authors utilize the basic windowing technique to identify the data
elements in a window and adjust its size to derive an efficient performance vari-
ations. The research work in [8] is also relevant to windowing technique for pro-
cessing time-series data. However, this work mainly focuses on the concept drift
when collecting data in sequences. These existing windowing approaches are not
enough to integrate IoT streaming data from multiple sources in real-time. In
this paper, we have acknowledged and extended the traditional windowing algo-
rithm and proposed a new window-based streaming data integration algorithm
and the associated timing alignment and de-duplication algorithms.
Recently, Pareek et al. [9] introduce a streaming analytics platform (sim-
ply, Striim engine) for real-time data integration. They also introduce several
adapters for Striim engine to extract such data, to transform SQL-based data
and for continuous data validation. This Striim engine can integrate time-series
data with other structured data. In particular, it collects datasets from different
sources and transforms them for aggregation without considering identical or dif-
ferent time-stamps. As such, Striim is not adequate to handle timing alignment
among different streaming data from multiple sources.
510 D. Q. Tu et al.

Ahad and Biswas [10] have proposed a distributed Hadoop-based architec-

ture for combining unstructured data from different small files (e.g., text, doc-
ument, pdf, and so on). They have introduced a dynamic merging mechanism
with the goal of collating data from different types of files. These Hadoop-based
research works are useful to process and manage data from multiple environ-
ments. Similar to Hadoop-based research, our group’s earlier works in the area
of context-sensitive access control can be applicable to capture data from a sin-
gle centralized source [11] or from multiple decentralized sources [12]. However,
these research works do not focus on the key issue of timing alignment of mul-
tiple time-series data from different sources. Based on our experience in this
research, the streaming or time-series data need to be processed and integrated
in real-time by dealing with timing conflicts from multiple data sources.

7 Conclusion and Future Research Directions

To date, some data integration approaches have been proposed to integrate data
from multiple sources. However, these existing approaches are not robust enough
in today’s dynamic and interconnected environments with the goal of integrat-
ing IoT time-series data due to the key issues of timing alignment and data
duplication while integrating streaming data from multiple IoT sources in real-
time. In this paper, we have proposed a new window-based approach for IoT
Streaming Data Integration (ISDI), extending the basic windowing technique.
We first introduce a generic integrator model for ISDI, including different lay-
ers to utilize streaming data integration from multiple IoT sources in real-time.
We have presented the implementation algorithms in realizing the generic ISDI
model. We have demonstrated the applicability of our ISDI approach through
conducting different sets of experiments and presenting an empirical comparison
of our proposed solution with respect to earlier simple data integration solution.
The experimental results show that our ISDI approach can be effectively used
in practice. Overall, our proposed ISDI approach can be applied to integrate
different time-series data from multiple streaming data sources.
One of the big questions still revolves around how to preserve privacy and how
much privacy is worth while streaming data integration is performed by utilizing
multiple IoT sources. Future scholars can explore different privacy control poli-
cies and solutions in this perspective, such as differential privacy, k-anonymity
and so on, and subsequently investigate the boundary between privacy and util-
ity towards IoT data integration from multiple sources.

References
1. Herland, M., Khoshgoftaar, T.M., Bauder, R.A.: Big data fraud detection using
multiple medicare data sources. J. Big Data 5(1), 29 (2018)
2. Chen, J., Chen, Y., Du, X., Li, C., Lu, J., Zhao, S., Zhou, X.: Big data challenge:
a data management perspective. Front. Comput. Sci. 7(2), 157–164 (2013)
 IoT Streaming Data Integration from Multiple Sources 511

3. Bellahsène, Z., Bonifati, A., Rahm, E.: Schema Matching and Mapping. Springer,
Berlin (2011)
4. Sagi, T., Gal, A., Barkol, O., Bergman, R., Avram, A.: Multi-source uncertain
entity resolution: transforming holocaust victim reports into people. Inf. Syst. 65,
124–136 (2017)
5. Calbimonte, J.P., Corcho, O., Gray, A.J.: Enabling ontology-based access to
streaming data sources. In: International Semantic Web Conference, pp. 96–111.
Springer (2010)
6. Daraio, C., Lenzerini, M., Leporelli, C., Naggar, P., Bonaccorsi, A., Bartolucci,
A.: The advantages of an ontology-based data management approach: openness,
interoperability and data quality. Scientometrics 108(1), 441–455 (2016)
7. Bifet, A., Gavalda, R.: Learning from time-changing data with adaptive windowing.
In: Proceedings of the 2007 SIAM International Conference on Data Mining, SIAM,
pp. 443–448 (2007)
8. Gama, J., Sebastião, R., Rodrigues, P.P.: On evaluating stream learning algo-
rithms. Mach. Learn. 90(3), 317–346 (2013)
9. Pareek, A., Khaladkar, B., Sen, R., Onat, B., Nadimpalli, V., Lakshminarayanan,
M.: Real-time ETL in Striim. In: Proceedings of the International Workshop on
Real-Time Business Intelligence and Analytics, p. 3. ACM (2018)
10. Ahad, M.A., Biswas, R.: Dynamic merging based small file storage (DM-SFS)
architecture for efficiently storing small size files in hadoop. Procedia Comput. Sci.
132, 1626–1635 (2018)
11. Kayes, A., Han, J., Rahayu, W., Dillon, T., Islam, S., Colman, A.: A policy model
and framework for context-aware access control to information resources. Comput.
J. (2018) https://doi.org/10.1093/comjnl/bxy065
12. Kayes, A., Rahayu, W., Dillon, T., Chang, E.: Accessing data from multiple sources
through context-aware access control. In: TrustCom, pp. 551–559. IEEE (2018)
 Improving Document Similarity
Calculation Using Cosine-Similarity
Graphs

Yasunao Takano1(B) , Yusuke Iijima1 , Kou Kobayashi1 , Hiroshi Sakuta1 ,

Hiroki Sakaji2 , Masaki Kohana3 , and Akio Kobayashi4
1
Aoyama Gakuin University,
5-10-1 Fuchinobe, Chuo-ku, Sagamihara-shi, Kanagawa 252-5258, Japan
{takano,sakuta}@it.aoyama.ac.jp, {a5815004,a5815039}@aoyama.jp
2
The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8654, Japan
3
Ibaraki University, 4-12-1 Nakanarusawa, Hitachi, Ibaraki 316-8511, Japan
4
RIKEN AIP Center, Nihonbashi 1-chome Mitsui Building, 15th floor,
1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan

Abstract. Data mining information using various indices and determin-

ing candidates that can be judged as having the same tendency based
on similarity between documents is common. The accuracy of similarity
largely depends on a sufficient amount of data and requires advanced
analysis using natural language processing. In this paper, we present an
approach for filtering based on the cosine similarity graph and clustering
between candidates. For filtering candidates, we focus on the inflection
point of the graphs when plotting by sorting similarities in descend-
ing order. By clustering among higher similarities, we aim at filtering
candidates that cannot be removed without analyzing advanced natu-
ral language processing. The proposed method was applied to movie
reviews and sightseeing location reviews written in Japanese. Although
this study is a work in progress, it shows that candidates can be recom-
mended without having to manually apply natural language processing,
such as preparing stopwords for each category.

Keywords: Document similarity · Cosine similarity ·

Content-based filtering

1 Introduction
Data mining information using various indices and determining candidates that
can be judged as having the same tendency based on similarity between docu-
ments is common. To improve the accuracy of judging similarity, it is necessary
to gather a sufficient amount of data, and advanced an analysis using natural
language processing is required. In particular, it is said that the accuracy of
similarity cannot be improved unless manpower is added to preprocessing, such
as selection of a stopword in natural language processing.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 512–522, 2020.
https://doi.org/10.1007/978-3-030-15032-7_43
 Improving Document Similarity Calculation Using Cosine-Similarity Graphs 513

This paper focuses on the content-based similarity between documents [1].

We present an approach for filtering based on the cosine similarity graphs and
correcting similarity by clustering between candidates. With the cosine simi-
larity graph, the proposed method focuses on the inflection points of graphs
when plotting by sorting the cosine similarities in descending order. In vectoriz-
ing documents using either TF-IDF or Doc2Vec, the graph draws characteristic
curves; hence, it was found that candidates can be filtered. By clustering among
higher similarities, we aim at filtering candidates that cannot be removed with-
out advanced natural language processing. The datasets used in this paper are
movie reviews and reviews of sightseeing locations written in Japanese, and do
not use additional meta-information such as tags or categories. Only sentences
from written reviews are used. Although this study is a work in progress, it will
lead to the development of a method in which candidates can be recommended
without manually preparing stopwords for each category.
We summarize our contributions as follows:
• We analyzed the characteristic curve of graphs for the cosine similarities
between documents.
• We proposed a method to select documents with higher similarity based on
the graph.

2 Document Similarity
To define the similarity between documents, this section describes a brief sum-
mary of the extraction method of document features using TF-IDF and Doc2Vec,
which is the basis of the proposed method, and cosine similarity.

2.1 Vectorizing Documents by TF-IDF

TF-IDF is a represents the degree of importance of each word in a document and

the term frequency (TF) and inverse document frequency (IDF) are obtained by
the following expressions.
Number of occurrences of words in sentences
TF =
The total number of words in the document
(1)
log(Number of documents)
IDF =
Number of documents containing words
(2)
TF-IDF = IF ∗ IDF

These equations indicate that TF increases the weight of frequently occurring

words in a certain document, and IDF reduces the weight of frequently occurring
words in many documents. That is, a word with a larger TF-IDF is given a higher
value as it is a word that characterizes the document.
514 Y. Takano et al.

2.2 Vectorizing Documents Using Doc2Vec

Doc2Vec [2] is a technique to vectorize documents of arbitrary length using a
neural network. A paragraph is prepared as a vector expression together with
each word, and a model is generated using learning data of all documents to
obtain the similarity using a task for predicting the next word from the vector
expression. In order to align the standard with TF-IDF, we apply the distributed
bag-of-words (DBoW) model. This model does not need to consider the order.
In DBoW, a classification task that predicts randomly sampled words from a
paragraph vector is applied.

2.3 Cosine Similarity

Using the documents vectorized by TF-IDF or Doc2Vec, the cosine similarity of
a document is obtained by the following equation.
v u · vi
cos(vu , vi ) = (3)
|vu |2 + |vi |2
vu and vi are vectors of the document. In the above equation, the proximity of
the angle formed by the vectors is expressed by taking a value from 0 to 1. A
value closer to 1 means that the similarity between them is higher.

2.4 Preprocessing of Data

When handling a document in big data, preprocessing such as word normal-
ization and deletion of unnecessary words is required after performing natural
language processing to decompose sentences into word units.
First, words are cut out as units constituting a sentence by morphological
analysis. Naturally, this analysis depends on the language in which the text is
written. Subsequently, word normalization, in which text is transformed into
a single canonical form, is applied. In general, a dictionary showing the corre-
spondence is necessary to find a canonical form. Finally, unnecessary words are
deleted to eliminate noise due to meaningless data. For example, words and parts
of speech that are not very meaningful in the text, called stopwords, are deleted.
However, meaningless words differ depending on the category of the document
and the language; hence, making the list of stopwords is difficult.

3 Data Used in This Study

3.1 Overview
In this study, we obtained data on the web about movie reviews and reviews on
sightseeing locations written in Japanese. Details are shown in Table 1.
Because each entry of data is a review about movies and sightseeing locations
written by many users, it is a collection of text composed of sentences written
in various styles.
 Improving Document Similarity Calculation Using Cosine-Similarity Graphs 515

Table 1. Data used in this research

Site name # of data Data size (KB) Remarks

Movies eiga.coma 3,857 8.4–232.5 Use only movies with
more than 10 reviews and
over 3,000 characters. Up
to 200 reviews for each
movie
Sightseeing Tripadvisorb 1,528 100 Use sightseeing locations
location in each city in Japan. If a
data entry is larger than
the data size, the entry
has been shrunk to the
data size
a
https://eiga.com/
b
https://www.tripadvisor.jp/

Fig. 1. The cosine similarity of “Star Wars The Last Jedi”

3.2 Graph of Cosine Similarity Based on TF-IDF

For each document in the target dataset, the similarity based on the TF-IDF was
obtained based on the expression shown in Sect. 2.1, and plotted on the graph
and analyzed. Parts of speech, namely “noun”, “verb”, “adjective” and “adverb”
were subject of vectorization, and unnecessary words such as stopwords were not
deleted.
In this paper, we use graphs in which the cosine similarity of each documents
to other documents is plotted on the vertical axis, and the ranking within the
documents is plotted on the horizontal axis. The similarities of a movie using
TF-IDF is shown in Fig. 1 The graph from the movie “Star Wars The Last Jedi”1
is used to illustrate. The horizontal axis indicates the similarity between “Star
1
https://www.imdb.com/title/tt2527336.
516 Y. Takano et al.

Wars: The Last Jedi” and each movie. Ranks are indicated from 1st place to
3,856 th place from left to right. The graph draws characteristic curve, where a
small number of higher rankings have higher similarity ( 1 in Fig. 1), and many
intermediate movies are less displaced ( 2 ), and it has a distribution of a small
number of subordinates ( 3 ). Actually, most observers would say that they are
related.

1. Star Wars: Episode VII - The Force Awakens2 : Cosine Similarity: 0.792
2. Rogue One: A Star Wars Story3 : Cosine Similarity: 0.608
3. ...

Similarities of reviews in 2 due to common words used in writing movie reviews

and words necessary for writing ordinary natural sentences. In many cases, cosine
similarity of movies in  2 can be disregarded, if we use stopwords and other
preprocessing method. Furthermore, the review in  3 is a movie including reviews
that are written in a strange style. These movies have a low similarity with all
movies.
(a) of Fig. 2 shows plots of similar graphs for the remaining 3,856 movies.
The graph shows that there is a similar trend for all movies.

(a) Movies (b) Sightseeing locations

Fig. 2. Cosine similarity based on TF-IDF

Furthermore, from (b) in Fig. 2, there is a similar tendency for sightseeing

locations.

3.3 Cosine Similarity Based on Doc2Vec

As in Sect. 3.2, a graph plotting the cosine similarity based on Doc2Vec is shown
in Fig. 3. These graph show that the trend for Doc2Vec is the same as for TF-IDF

2
https://www.imdb.com/title/tt2488496/.
3
https://www.imdb.com/title/tt3748528/.
 Improving Document Similarity Calculation Using Cosine-Similarity Graphs 517

(a) Movies (b) Sightseeing locations

Fig. 3. Cosine similarity based on Doc2Vec

3.4 Data Analysis

The results from Sects. 3.2 and 3.3 indicate the potential of using the graphs to
remove reviews with a low similarity. As the reviews in 2 are similar, given the
common words used to write movie reviews, it may be enough to examine the
graph without even preparing stopwords.
To better understand the graph of the cosine similarities of the reviews, we
analyzed movie reviews with higher similarities. When the cosine similarity based
on TF-IDF is obtained for “Up”4 , the top five rankings are as follows.
1. Inside Out5
2. Avatar6
3. Coco7
4. Department Q: A Conspiracy of Faith8
5. Kamera wo tomeru na!9
The review of each movie is written in Japanese, but their names are given in
English in this paper.
From above-mentioned list, “Inside Out” and “Coco” are produced by the
same company that created “Up”. This degree of similarity, that these movies
are closer to “Up”, will be appreciated by many people, although it is a very
subjective opinion. Similarly, many people will appreciate that “Department Q:
A Conspiracy of Faith” and “Kamera wo tomeru na!” are different from “Up”;
however, these movies have a high cosine similarity with “Up”. Hence, we need
to analyze the factors.
A result of examining “Department Q: A Conspiracy of Faith” shows that
because the name of the leading character was “Carl”, it turned out to be an
4
https://www.imdb.com/title/tt1049413/.
5
https://www.imdb.com/title/tt2096673/.
6
https://www.imdb.com/title/tt0499549/.
7
https://www.imdb.com/title/tt2380307.
8
https://www.imdb.com/title/tt4088268.
9
https://www.imdb.com/title/tt7914416/.
518 Y. Takano et al.

important word in the review. This shows that if the name of the character is
not used as a characteristic word, the problem will be solved. However, this has a
side effect of decreasing the similarity of the series “movies”. Similarly, it is also
possible that the names of the characters and the actors match. Therefore, con-
siderably advanced knowledge is required to eliminate this by natural language
processing alone.
Furthermore, because “Kamera wo tomeru na!” is a movie about the filming
of a movie, the terms used in the review are considered to have increased its
similarity. As a result, this movie was among the top 10 in the 2,648 movies in
the movie data collected. This indicates that if stopwords related to the movie are
prepared, the movie could possibly be removed from the top 10 in many movies.
However, it would be necessary to perform manual preprocessing depending on
the category of data.

4 Proposed Method
This paper uses the analysis in Sect. 3 to present an approach to filtering that
is based on the graph of cosine similarity. This also corrects similarities by clus-
tering between candidates by filtering them in the following two stages.
1. Removing movies with intermediate ranked similarities up to the inflection
point of the graph.
2. Clustering candidates with higher similarities.
The first stage is based on the analysis in Sects. 3.2 and 3.3. Because accuracy
of cosine similarity is tend to be close to human intutive feeling, documents
related to a document are roughly gathered upwards. The second item is based
on predicting that candidates included in the upper rank will be close each other.
Therefore, unexpected movies will be removed from among the candidates by
clustering. For example, if the similarity of a movie increases only because of
the name of its leading character, it can be clustered with other higher-ranking
candidate movies and thus eliminated.

4.1 Filtering the Candidates

The cosine similarity is obtained by vectorizing all documents on the target
dataset using TF-IDF or Doc2Vec. After that, the range for which the slope
shows a gently sloping trend, such as  2 and 3 in Fig. 1, should be excluded.
The algorithm is shown in Fig. 4. The function filtering takes two argu-
ments, the set S of the document of interest and the margin, when determining
the threshold. According to the existing general method shown in Sect. 2, the
documents are vectorized and the cosine similarity between all movies is obtained
for each movie. Therefore, |L| is a collection of sorted lists because it combines
the similarities of all movies with each movie. A threshold value was then deter-
mined using the values of the first and third quarters to find a straight line
approximating  2 . Finally, the candidates are compared in order, from the top
 Improving Document Similarity Calculation Using Cosine-Similarity Graphs 519

1: function filtering(S, margin)

2: V ← vectorize(S)
3: L ← map(sort, calculate cosine similarity(V ))
4: q1 ← |L|/4
5: q3 ← |L| ∗ 3/4
6: N ← []
7: for i = 1 to |L| do
8: threshold ← calculate threshold(Li , q1, q3)
9: for j = 1 to |Li | do
10: if Li,j ≤ threshold ∗ margin then
11: break
12: N = N ++ [j]
13: return (N , L)

Fig. 4. Algorithm for filtering candidates

Fig. 5. Image of algorithm

of the similarity, to the value on the straight line; the candidates are retained
until their values fall below that on the straight line. Figure 5 is a graph of the
algorithm. As indicated by the red line, margin taken is approximate to the
straight line, and candidates are taken up to the point at which the similarity
does not fall below that value.

4.2 Clustering
By clustering the candidates after the method shown in Sect. 4.1, we aim to filter
candidates that are not similar to others.
In terms of the candidates, the return value of filtering() in Fig. 4)
is extracted from the list of similarities and similarities between movies are
regarded as the distance between each candidate movie. An existing method like
KMeans [3] is used for clustering.
Future tasks include finding suitable parameters and concrete methods by
which to select final candidates from the clustering results obtained.
520 Y. Takano et al.

Fig. 6. Changes in the number of candidates by margin

5 Results
5.1 Results of Filtering
This section describes the result of applying the proposed method with TF-
IDF to the movie “UP”, as described in Sect. 3.4. Figure 6 shows the number
of candidate movies remaining when the margin is changed. Naturally, as the
margin increases, the number of candidates decreases. The following were the
candidates when the margin was set top 1.4.
1. Inside Out
2. Avatar
3. Coco
4. Department Q: A Conspiracy of Faith
5. Kamera wo tomeru na!
6. Your Name
7. Wakaokami ha shogakusei
8. Stand by Me
These movies are clustered in three groups using KMeans (k = 3) as follows.
• Inside Out, Coco
• Department Q: A Conspiracy of Faith
• Avatar, Kamera wo tomeru na!, Your Name, Wakaokami ha shogakusei, Stand
by Me
This result demonstrates the possibility of removing unexpected movies for which
the relation is high only with the fact that the main character name is the same,
as described in Sect. 3.4. Moreover, “Kamera wo tomeru na!” was originally in
the top 10 out of 2,648 movies when the right parameters were chosen. This
shows that filtering does not require stopwords to be prepared for each cate-
gory. However, knowledge of parameters such as K = 3 has not been sufficiently
investigated. These cannot be used as a recommender system without future
adjustment.
 Improving Document Similarity Calculation Using Cosine-Similarity Graphs 521

(a) TF-IDF (b) Doc2Vec

Fig. 7. Cosine similarity of english sightseeing location reviews

5.2 Language Dependency

All of the discussion so far has been on Japanese reviews, but the proposed
method does not require language-dependent processing except for morpholog-
ical analysis. Therefore, we confirmed whether there is a similar tendency in
reviews written in English.
To tailor the case to Japanese, English reviews of Japanese sightseeing loca-
tion on Tripadvisor.com was used. The reviews acquired for cities were the same
as for the Japanese version; however, because there were sightseeing locations
with few reviews in English, 10 KB was adopted as the data size and 1,296 data
points were obtained.
The result is shown in Fig. 7. TF-IDF shows that the trend for Japanese
reviews is the same as for English reviews. However, in the case of Doc2Vec,
there are cities for which English reviews have a high similarity, and the same
trend as in Japanese cannot be seen. The reason for this may be that sufficient
data could not be obtained for Doc2Vec learning due to the reduced data size.

6 Related Research

The Elbow method, used to determine the number of clusters in cluster analysis,
focuses on the inflection point of the figure. This research follows that method.
When plotting the relationship between the cluster number K and the residual
sum of squared prediction errors, the optimum number of clusters can be deter-
mined by paying attention to the points that turn like an elbow. This research
also proposes a method to filter candidates by focusing on inflection points in
graphs of cosine similarity.
Analysis of movie reviews is typical research material for the recommendation
systems [4]. Recent practical recommender systems are based on collaborative fil-
tering. Because this method tries to predict the items for a particular user based
on items previously rated by other users, it needs a very high item-user cor-
respondence. As content-based filtering calculates only the similarities between
522 Y. Takano et al.

documents, it is less likely to be influenced by the volume of data, although

verification of the recommendation performance is still necessary.

7 Summary

In this paper, we proposed a filtering method that minimizes preprocessing based

on natural language processing when obtaining similarity for data of multiple
categories. To filter candidates, we focused on the inflection point of the similar-
ity graph based on a characteristic graph that arranges similarities in descending
order.
The one of most important future tasks would be to investigate whether
this tendency also exists in other categories and for other languages. Figure 7
shows the possibility of similar tendency occurring in English; thus, more data
is necessary to investigate further.
Moreover, it is also necessary to enrich the evaluation of the method in the
future. Finally, because there are many subjective methods by which to evaluate
the recommended content as described in Sect. 3.4, it is necessary to evaluate
how a person feels about candidates mentioned in a questionnaire.

References
1. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems:
a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data
Eng. 17(6), 734–749 (2005)
2. Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In:
Proceedings of the 31st International Conference on International Conference on
Machine Learning, ICML 2014, vol. 32, pp. II–1188–II–1196 (2014)
3. MacQueen, J.: Some methods for classification and analysis of multivariate observa-
tions. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics
and Probability, Volume 1: Statistics, pp. 281–297 (1967)
4. Töscher, A., Jahrer, M., Legenstein, R.: Improved neighborhood-based algorithms
for large-scale recommender systems. In: Proceedings of the 2nd KDD Workshop on
Large-Scale Recommender Systems and the Netflix Prize Competition, NETFLIX
2008, pp. 4:1–4:6 (2008)
 An Efficient Event-Based Protocol for
Emergency Situations in Smart Cities

Sediane C. L. Hernandes1(B) , Marcelo E. Pellenz2 , Alcides Calsavara2 ,

and Manoel C. Penna2
1
UTFPR, Guarapuava, PR, Brazil
[email protected]
2
PUCPR-PPGIa, Curitiba, PR, Brazil
{marcelo,alcides,penna}@ppgia.pucpr.br

Abstract. Smart cities can offer a number of important and needed

services for the citizens, including smart emergency services. In an emer-
gency situation, mobile units such as police cars, fire trucks, and ambu-
lances should move to the emergency location, thus performing an ade-
quate care. A key question for achieving high performance in the service
is to define and apply criteria to select and activate mobile units for each
emergency situation. This paper presents an event-based protocol with
event filtering by subscriber’s fitness, named Event to Most Suitable
Subscribers (EMSS), by which the most appropriate mobile units are
activated. A publish-subscribe middleware that implements the EMSS
protocol was also developed. An application of the middleware to the
police and the firefighter services is presented. The results show that,
in the case where the most suitable mobile units are chosen, the ser-
vice response time can be reduced. Therefore, the emergencies are more
quickly attended, what is crucial for saving lives.

1 Introduction
Nowadays, the world population living in cities is fifty percent and by 2050
this percentage is expected to increase to seventy percent [1]. As a result, new
challenges need to be faced around the world, especially in bigger and more
populous cities. The main problems are related to providing infrastructure and
public services such as waste management, scarce resources, urban traffic man-
agement, among others. In this way, new strategies to manage cities with the
help of Information and Communication Technologies (ICTs) are needed. In this
context, the concept of Smart City consists of exploring ICTs to make better
use of public resources, to improve the quality of services offered to the citizens,
improving their quality of life [5,13]. One important service that can be improved
in Smart Cities is the emergency services.
The emergency services address the response to a natural or man-made disas-
ter and involve controlling the disaster situation [7]. The process is time-critical
because lives depend on the accuracy and speed with which emergency respon-
ders make decisions. In an emergency service, an event corresponds to the occur-
rence of an emergency like an accident, a robbery, an act of terrorism or a fire. An
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 523–534, 2020.
https://doi.org/10.1007/978-3-030-15032-7_44
524 S. C. L. Hernandes et al.

emergency situation requires transferring one or more mobile units to the event
location. A key question for achieving high performance in the service is to define
and apply criteria to select and activate mobile units for each emergency situ-
ation. A publish-subscribe middleware could manage emergency events, besides
that to facilitate the applications development. Publish-subscribe middlewares
or event notification services are appropriate for Smart Cities. An event noti-
fication service in Smart Cities [2] must manage the sending of events on a
distributed network architecture, in which the selection and event delivery activi-
ties are spread across the network. The publish-subscribe paradigm is often used
as a communication mechanism in event-based applications, like Smart Cities
services [2].
Intuitively, a simple event notification service provides a mechanism for
selecting and delivering events. However, during the delivery of an event, it
may be necessary that only a certain number of the most suitable subscribers,
the mobile units in this context, receive the event as quickly as possible. A
mobile unit may present better conditions to receive the event and handle it
than others, especially in emergency situations. In this way, this paper presents
a new event-based protocol for emergency services, called Event to Most Suit-
able Subscribers (EMSS). It considers the sending of events to the most suitable
subscribers, according to a specific criteria of the application, and its imple-
mentation in a middleware. It can be shown that sending events to the most
appropriate subscribers is better than a random choice among a percentage of
the best subscribers. This selection of the best subscribers is very important in
Smart Cities, especially in emergency services caused by natural or man-made
disasters, where human decision-making can be difficult.
This paper is organized as follows. Section 2 presents the related works. The
EMMS Event-based Protocol is describe in Sect. 3. Section 4 presents the EMSS
middleware. Section 5 presents the simulation setup and Sect. 6 presents an eval-
uation of the middleware for the police and firefighter services. Finally, Sect. 7
presents the conclusions.

2 Related Works

Several event based middlewares were studied to verify their suitability for emer-
gency services in Smart Cities. The Table 1 summaries some classical middle-
wares found in the literature, where the main characteristics are indicated. The
table indicates the type of network architecture, the routing strategy and how
the middleware does event filtering.
The middleware architecture needs to be distributed for the Smart Cities,
as pointed by [2]. That makes it possible to reduce network load, favor the
availability (fault tolerance) and increase the scalability of the system. Also, this
architecture does not present a single point of failure and no critical processing,
storage or communication overhead. Besides that, in an emergency situation as
a disaster, for example, if a centralized architecture were adopted the city could
be isolated without receiving the necessary emergency services.
An Efficient Event-Based Protocol for Emergency Situations in Smart Cities 525

Table 1. Main Publish-Subscribe Middlewares

Middleware Architecture Routing Event filtering

SIENA [4] Distributed Content Subset of participants
Hermes [15] Distributed Rendezvous Subset of participants
Steam [12] Peer-to-peer Epidemic Publisher, subscriber
SensorBus [16] Centralized Pull Subscriber
EMMA [14] Peer-to-peer Epidemic Publisher, subscriber
Mires [18] Distributed Multihop Publisher
RUNES [8] Distributed Multihop Publisher
PSWare [10] Distributed Multihop Publisher
TinnyDDS [3] Distributed DHT/ST Publisher
PRISMA [17] Centralized Pull Subscriber
PSNMVN [11] Distributed Event flooding Subscriber

The peer-to-peer architectures are suitable for the dissemination of events

between a limited number of participants and for small-scale deployment due
to the latency issues and high event throughput. Therefore, middlewares with
centralized or peer-to-peer architecture are not appropriate for the emergency
situation in Smart Cities as well as event filtering in the subscribe. Typically,
subscriptions contain intrinsically filters for events and are used in the matching
phase. The event will only follow the path to the subscriber if the subscription
matched it. All and only those who have issued a corresponding subscription
receive the event. However, regardless of whether filtering happens at the pub-
lisher or in a subset of participants, it is not possible to make a selection so that
only some receive it. Consequently, these middlewares are not able to send events
to only a subset of subscribers, as required by an emergency service. The event
model implemented by them is the publish-subscribe model by which an event
is sent to all subscribers of that type of event. They can not ensure that only a
subset of the subscribers, specifically the best ones in the context of emergency
services, is notified.
In the proposed EMSS architecture, the most appropriate available sub-
scribers are activated. Firstly, because for each subscriber, a set of fitness values
is calculated about each subscription together with the observed behavior of each
mobile unit. Each fitness value means how a mobile unit is prepared to answer a
call in a specific location. Thus, when an emergency call is made, the most suit-
able mobile units are quickly chosen. This approach differs from event filters, in
which the various signatures are compared according to a set of rules to obtain
a match before sending the event (server-side filtering approach). Such corre-
spondence does not guarantee that a subset of subscribers receives the event,
but that all subscribers in the correspondence receive it. The calculation about
the subscriptions and the observed behavior of mobile units was not addressed
in the related works. Besides that, in several event distribution systems, there is
526 S. C. L. Hernandes et al.

the possibility to define filters at the subscriber level. However, this is not appro-
priated in the emergency situations because that all subscriber will be activate.
Moreover, the mobile units doesn’t know the amount of mobile units needed
to attend the emergency and many will certainly move to the emergency loca-
tion without necessity. Second, the signatures are constantly updated because
the dynamics of the environment is taken into account. Third, because it seeks
to give human behavior to an event-based protocol to be used in emergency
situations.

3 The EMSS Event-Based Protocol

The EMSS event-based protocol is an application layer protocol that is designed

for use over a reliable transport protocol, such as the Transport Control Protocol
(TCP). The EMSS protocol describes the communication between brokers and
mobile units. The brokers receive the events sent by publishers and mobile units
issuing subscriptions consume the events according to the EMS event model.
The role of each entity in the EMSS protocol is defined as follows:

1. Publishers: The event system that publishes the events represented by an

attendant.
2. Brokers: The broker is a software process that makes the decision about which
mobile units should be activated to handle the event.
3. Subscribers: The subscriber receives the event, which is the mobile unit in the
context of emergency situations. It must move to the occurrence and provide
the necessary care.

Emergency situations happen in specific locations in the city. This location

is associated with a reference point. The reference point may be a public square,
a hospital, for example. The reference point is utilized as a guide to the mobile
unit to arrive in the emergency when it is activated to the attendance. The
set of all reference points in a city, the set of available services and the set of
brokers are denoted by R = {r1 , . . . , rp }, S = {s1 , . . . , sn } and B = {b1 , . . . , bq },
respectively. The set of mobile units for si ∈ S is denoted by Vi = {vi1 , . . . , vim }.
An emergency event is denoted as E and the corresponding reference point as
re . The number of mobile units for acting on E is denoted by Nie ∈ N.
An attendant receives a call with a precise location and the corresponding
reference point of the city, which provides the coordinates where to send the
mobile units. The brokers notify the most suitable mobile units based on their
normalized fitness value (between 0 and 1). The fitness value indicates the suit-
ability of a mobile unit to respond to emergency calls associated with a particular
reference point. It is calculated according to subscription information send by
mobile units and its observed behavior. The fitness of the mobile unit vij ∈ Vi
with respect to rk ∈ R is denoted as Fijk . The goal of the EMSS protocol is to
choose the mobile units which present the best fitness Fijk to handle E. The idea
 An Efficient Event-Based Protocol for Emergency Situations in Smart Cities 527

is to select the best mobile units from Vi in order to maximize a utility function
Ui , defined as:
|Vi |

Maximize Ui = Fijk · Aij
j
j=1

|Vi |

Subject to Aij ≤ Nie
j=1

where 
1, if vij ∈ Vi is available to handle E
Aij =
0, Otherwise
In this way, the subset of mobile units for service si ∈ S that represents the
selected units to handle an emergency situation is given by {vij ∈ Vi | Aij = 1}.
The main challenge in solving Ui for a given service si ∈ S is the determination
of Fijk for all vij ∈ Vi . The problem is in calculating the fitness of each mobile
unit of a service for an emergency situation reported at a certain reference point.
Therefore, we consider the following assumptions: (i) While the mobile unit is
operating in an emergency, it is unavailable; (ii) The closer a mobile unit is to
the reference point is, greater is its fitness value; (iii) Intense traffic means a
longer time for a mobile unit to reach a reference point; (iv) The longer the
journey of a team in a mobile unit, the greater is the level of fatigue associated
with that mobile unit.
Therefore, we define a metric for calculating the fitness value of each avail-
able mobile unit, considering the above premises, which include the following
parameters:

(a) The distance between the mobile unit vij located at (xij , yji ) and the reference
point rk located at (xk , yk ) is given by

dkij = (xk − xij )2 + (yk − yji )2

(b) The traffic intensity is categorized as low, moderate or intense, and it is

described by the parameter ckij .
(c) The team fatigue level of a mobile unit is denoted by Wij = ZE /O, where
ZE is the accumulated attendance time of the emergency team and O is
the maximum allowed attendance time of the team. The fatigue level is
normalized between 0 and 1. Thus, the fitness of each mobile unit vij for
each reference point is calculated as follows:

Fijk = (1 − (dkij · ckij )/M ) · (1 − Wij /L)

where M is a normalization factor which depends on the maximum distance
between two reference points and L is a scale factor for the fatigue level associ-
ated with the mobile unit. When the mobile unit is unavailable Fijk is zero.
528 S. C. L. Hernandes et al.

The calculation of the proposed fitness value considers the parameters above.
However, other parameters can be considered, as the different capacities of the
mobile units in terms of equipment and human resources. Although the police
and firefighters have mobile units with different capacities we assumed vehicles
of the same configuration in the evaluation scenario.

4 The EMSS Middleware

The EMSS middleware architecture presents a distributed organization. There-

fore, the EMSS middleware is implemented as a set of distributed brokers that
cooperate with each other to deliver the events to the most appropriate sub-
scribers. The brokers are distributed in the city and cover without overlapping
regions with the same coverage area. As a consequence, decision making is decen-
tralized, with each broker maintaining part of the system state. The broker ini-
tiates the decision-making process for all events that take place in it’s region,
and it is based on a consensus protocol among brokers. The decision-making
involves only the broker in the emergency region and the subset of brokers with
mobile units belonging to the requested service. Figure 1 illustrates the EMSS
middleware architecture. Brokers communicate via the Internet and mobile units
communicate with brokers using wireless communication. The broker’s commu-
nication network can be viewed as an overlay network over a physical network,
such as the Internet.

B2 broker's B1 broker's
region r6 region

vi2
Antenna
r7 r5
vi5
r4 r2
s1
Emergency
r3
r8

Internet vi1
r1
B3 broker's B4 broker's
region region

r9

r12 r16 r15

Reference point

vi3 mobile unit vi4

r10 r11 r13
r14

Fig. 1. EMSS middleware architecture.

When the attendant receives one emergency call, with the precise emergency
location, must classify the emergency situation and then decide which services
should be triggered, as well as the number of mobile units of each service. So,
the attendant must send one message, called publish-activation message, to the
broker that manages the region associated to the emergency location (reference
An Efficient Event-Based Protocol for Emergency Situations in Smart Cities 529

point), named publisher broker. The publish-activation message contains the fol-
lowing arguments: (1) The requested service, si (e.g., firefighter, police); (2)
The reference point of the emergency situation, re ; (3) The number of requested
mobile units, Nie .
For the event notification, overlay networks are created when connections
to brokers are established. Thus, for each pair of service and reference point
belonging to the broker, (si , rk ), one overlay network is created. This is justified
because it is not known in what location the emergency can happen. The function
of an overlay network is making the event to be notified to a subset of the suitable
mobile units of service si . Every broker generates one event dissemination logical
tree (EDLT) for each overlay network.
When the broker receives a subscription message, it associates the mobile
unit to the respective service si and calculates the fitness value for all reference
points of the city. When the mobile unit enters the broker’s region, it sends a
subscription message (subscribe message) and when it leaves the broker’s region
it sends an unsubscription (unsubscribe message). The subscription has the fol-
lowing arguments: (i) The service identification (si ) provided by the mobile unit;
(ii) The current position (xij , yji ) of the mobile unit; (iii) The current condition
of the mobile unit, and; (iv) The team fatigue level. After, the fitness values
belonging to the broker are inserted in each tree belonging to the pair (si , rk ).
Besides that, the fitness value for each pair (si , rk ) is propagated to all brokers,
taking into account the reference points belonging to each one, since each bro-
ker only maintains the EDLT of events belonging to the reference points of its
region. The root of each EDLT is the publisher broker.
When the publisher broker receives a publish-activation message it must
select the EDLT that corresponds to the pair (si , rk ) and then make the decision
about which mobile units must to selected for the emergency. Next, it creates
Nie messages, one message for each required mobile unit, and sends at the same
time to all of them. If the mobile unit is in the same region of the publisher
broker and it was chosen for the call, one notification message is sent directly
to that mobile unit. Otherwise, if other brokers need to be activated, a publish
message is sent to intermediary brokers that notify some of their mobile units
through notification messages. The notification and publish messages must be
confirmed. However, if the mobile unit leaves the broker region during the noti-
fication and the notification message is not confirmed, another mobile unit is
notified. Thus, the mobility of mobile units is handled. The publisher broker will
reply to the event system with one (publish-activation-confirmation) message
after the acknowledgment of the notification message. The unsubscribe message
has as an argument the identifier of the mobile unit and the service si provided
by it.
Besides that, each mobile unit must send periodically bind messages to the
broker that manages the corresponding city region to update the EDLTs. Such
a message has the same arguments of a subscribe message. The state update for
a certain service and reference point can be performed according to one of the
following approaches: (1) Proactive Update: the value of fitness is updated
530 S. C. L. Hernandes et al.

periodically so that it is always ready when necessary for solving the problem;
(2) Reactive Update: the value of fitness is updated only during the problem
resolution. The bind message does not apply for the reactive approach.

5 Simulation Setup
The simulator selected for the implementation and evaluation of the EMSS proto-
col and the middleware was the Sinalgo (Simulator for Network Algorithms) [9].
Sinalgo is free and published under the BSD license. Therefore, it was adapted
to a city scenario. A square city area of 36 km2 was simulated. The total area was
divided into nine square regions of equal size (2 km × 2 km). Each region was
divided into four square cells with an area of 1 km2 each. The center of each cell
was considered as a reference point. Thus, the broker of each region was respon-
sible for dealing with calls regarding four reference points. We assume that each
region is covered by a specific wireless communication system or cellular antenna.
The calls were realized by an Emergency Call Attendant (ECA). An ECA
for each service (police and firefighter) was defined for the whole city. The ECA
receives a series of emergency calls during a period of 24 h. The emergency calls
were synthetically generated by following a Poisson distribution to determine
the period of time between two consecutive emergency calls during the period of
one day. The parameters employed to generate the series of emergency calls were
defined according to a real-world trace from firefighter and police emergencies
calls, which were obtained from a medium-sized Brazilian city during the year of
2016 [19]. For each generated call, the following parameters are defined: (1) The
period of time for the next call (Poisson distribution); (2) The reference point
of the emergency call (Random); (3) The number of mobile units (Nie ) required
to deal with the emergency (Randomly chosen between 1 and 3 mobile units).
The time required by the selected mobile unit to attend the emergency after
their arrival at the emergency location, which is called attendance time, follows
a Beta distribution model. This model was generated based on the real-data
analysis [19]. For the firefighter service, the maximum observed attendance time
was 8 h. For the police service, the maximum observed attendance time was
40 min. Thus, the maximum allowed attendance time of the team, O, has a
default value of 8 h for firefighter e 40 min for police. The minimal attendance
time observed was 2 min and 30 s for firefighter and 20 min for police. The fatigue
level Wij is cumulative and its initialized at zero every 8 h because the unit team
is changed.
Regarding traffic conditions, the simulation assumed that the speed of mobile
units varied between 30 and 60 km/h. The traffic coefficient ckij varies between 0
and 0.30 for low traffic, between 0.30 and 0.60 for moderate traffic, and between
0.60 and 1.0 for intense traffic. The traffic conditions employed in the simulation
scenario took into account the data obtained from the real-world traffic behavior
[6]. Finally, the bind message interval, based on simulation tests, was fixed as
one minute during the whole simulation.
An Efficient Event-Based Protocol for Emergency Situations in Smart Cities 531

6 Protocol Evaluation and Results

In order to evaluate the proposed strategy and identify the benefits of using the
fitness metric, the simulations were carried out considering three approaches:

• Proactive Most Suitable (PMS:) The most suitable mobile units are
selected according to mobile unit fitness, which are proactively updated.
• Reactive Most Suitable (RMS:) The most suitable mobile units are
selected according to mobile unit fitness value which is obtained only in reac-
tion to an emergency call.
• Reactive Random (RR:) The mobile units are randomly selected amongst
the fifty percent more suitable mobile units in the city, in reaction to an
emergency call.

The metric employed to compare the performance of the simulated

approaches was the period of time taken from the moment an emergency call
happens until the moment that each requested mobile units have arrived at the
emergency location. This period of time was called of Service Response Time.
That includes the time taken to select the mobile units plus the time for each
mobile unit to move from its current location to the emergency location. The
three approaches were compared with respect to Service Response Time regard-
ing 3, 6, 9, 12 and 15 mobile units. Thus, each combination of approach and
number of mobile units represented a scenario. For each scenario, 10 rounds
of simulation were performed 10 times (i.e., 100 simulations) and obtained the
averages regarding of the Service Response Time. A total of 1500 simulations
were performed per service.

6.1 Results

In the police service, the mobility is present, since mobile units move around the
city and eventually stop at points that deserve attention. The mobile units could
be activated while they are moving in the city. The simulation results are shown
in Fig. 2. In this scenario, we have an average of 9 calls/day and, on average,
20 mobile units are requested and activated. The figure shows the behavior of
the Service Response Time in terms of the number of mobile units available. In
both cases where the most suitable mobile units were chosen (PMS and RMS),
the Service Response Time significantly decreases when the number of available
mobile units grows. The Service Response Time is higher when a random choice
(RR) is made, regardless of the number of available mobile units.
In the firefighter service, the mobile units move only when they are required
to attend the emergency. The obtained results are shown in Fig. 3.
An average of 12 calls/day was realized for the firefighter service, with
an average of 24 mobile units requested and activated. The PMS and RMS
approaches presented a Service Response Time less than the RR approach. The
time significantly decreases when the number of available mobile units grows,
while that the Service Response Time is higher when a random choice (RR) is
532 S. C. L. Hernandes et al.

3
PMS
RMS
RR
2.5

Service Response Time (minutes)

2

1.5

0.5

0
3 6 9 12 15
Number of Mobile Units

Fig. 2. Comparison of PMS, RMS and RR approaches for the police service

made, regardless the number of available mobile units. In both PMS and RMS
approaches, the Service Response Time is very near. Table 2(a) shows the aver-
age fitness and the average distance traveled by the mobile units which were
selected for the emergency attendance, for the scenarios presented in Fig. 2 and
the Table 2(b) for the scenarios presented in Fig. 3. The PMS approach is recom-
mended for the police service that presents mobility. For the firefighter service,
the fitness values were very similar in both approaches PMS and RMS. However,
the RMS approach was more appropriate because of shows a Service Response
Time slightly lower than the PMS approach.

3.5
PMS
RMS
3 RR
Service Response Time (minutes)

2.5

1.5

0.5

0
3 6 9 12 15
Number of Mobile Units

Fig. 3. Comparison of PMS, RMS and RR approaches for the firefighter service
An Efficient Event-Based Protocol for Emergency Situations in Smart Cities 533

Table 2. Average Fitness and Distances to handle emergency calls

Mobile units PMS RMS RR

Fitness - distance (km) Fitness - distance (km) Fitness - distance (km)
(a) Police
3 0.7732–2.18 0.7734–2.09 0.7681–2.13
6 0.8583–1.75 0.8345–1.92 0.8353–1.89
9 0.8978–1.48 0.8739–1.72 0.8672–1.75
12 0.9155–1.34 0.8951–1.54 0.8803–1.69
15 0.9279–1.30 0.9098–1.44 0.8835–1.67
(b) Firefighter
3 0.8101–2.47 0.8087–2.45 0.8091–2.50
6 0.8617–2.08 0.8607–2.08 0.8421–2.27
9 0.8958–1.68 0.8945–1.75 0.8593–1.98
12 0.9155–1.47 0.9146–1.48 0.8757–1.93
15 0.9211–1.60 0.9247–1.56 0.8860–1.90

7 Conclusions

In this paper it was presented an event-based protocol and it’s implementation

through the EMSS middleware. The middleware was evaluated through the sim-
ulation of emergency calls, police, and firefighter, in a hypothetical city, and the
handling of those calls by considering three distinct approaches. We have been
able to demonstrate that choosing the most appropriate mobile units is bet-
ter than a random choice between a subset of the best mobile units. With the
choice of the most suitable mobile units, the emergency is more quickly attended,
thereby saving lives. Seconds can save lives and minutes were achieved by choos-
ing the best mobile units. Finally, any event-based system for emergency services
can use the EMSS middleware to activate mobile units in emergency situations
and we envision that the fitness calculation could be increment considering oth-
ers metrics and services.

Acknowledgements. The authors would like to thank CAPES and CNPq for their
financial support.

References
1. Alkandari, A., Meshal, A., Imad, F.T.A.: Smart cities: a survey. J. Adv. Comput.
Sci. Technol. Res. (JACSTR) 2(2), 79–90 (2012)
2. Antonic, A., Marjanovic, M., Skocir, P., Zarko, I. P.: Comparison of the CUPUS
middleware and MQTT protocol for smart city services. In: 2015 13th International
Conference on Telecommunications (ConTEL), pp. 1–8. IEEE (2015)
3. Boonma, P., Suzuki, J.: TinyDDS: an interoperable and configurable pub-
lish/subscribe middleware for wireless sensor networks. In: Wireless Technologies:
Concepts, Methodologies, Tools and Applications, pp. 819-846. IGI Global (2012)
534 S. C. L. Hernandes et al.

4. Carzaniga, A., Rosenblum, D.S., Wolf, A.L.: Design and evaluation of a wide-area
event notification service. ACM Trans. Comput. Syst. (TOCS) 19(3), 332–383
(2001)
5. Cenedese, A., Zanella, A., Vangelista, L., Zorzi, M.: Padova smart city: an urban
internet of things experimentation. In: 2014 IEEE 15th International Symposium
on World of Wireless, Mobile and Multimedia Networks (WoWMoM), pp. 1–6.
IEEE (2014)
6. CET: Mobilidade no Sistema Viario Principal: Volume e Velocidade - 2016, June
2017. http://www.cetsp.com.br/media/574933/2016volumevelocidade.pdf
7. Chitumalla, P.K., Harris, D., Thuraisingham, B., Khan, L.: Emergency response
applications: dynamic plume modeling and real-time routing. IEEE Internet Com-
put. 1, 38–44 (2008)
8. Costa, P., Coulson, G., Gold, R., et al. : The RUNES middleware for networked
embedded systems and its application in a disaster management scenario. In: Fifth
Annual IEEE International Conference on Pervasive Computing and Communica-
tions, PerCom 2007, pp. 69–78. IEEE (2007)
9. Distributed Computing Group. Sinalgo Tutorial (2015). https://disco.ethz.ch/apps
10. Lai, S., Cao, J., Zheng, Y.: PSWare: a publish/subscribe middleware supporting
composite event in wireless sensor network. In: IEEE International Conference on
Pervasive Computing and Communications, PerCom 2009, pp. 1–6. IEEE (2009)
11. Leontiadis, I.: Publish/subscribe notification middleware for vehicular networks.
In: Proceedings of the 4th on Middleware doctoral symposium, pp. 1–12. ACM
(2007)
12. Meier, R., Cahill, V.: Steam: event-based middleware for wireless ad hoc networks.
In: Proceedings of the 22nd International Conference on Distributed Computing
Systems Workshops, pp. 639–644. IEEE (2002)
13. Mohanty, S.P., Choppali, U., Kougianos, E.: Everything you wanted to know about
smart cities: the internet of things is the backbone. IEEE Consum. Electron. Mag.
5(3), 60–70 (2016)
14. Musolesi, M., Mascolo, C., Hailes, S.: EMMA: epidemic messaging middleware for
ad hoc networks. Pers. Ubiquit. Comput. 10(1), 28–36 (2005)
15. Pietzuch, P.R., Bacon, J.M.: Hermes: a distributed event-based middleware archi-
tecture. In: Proceedings of the 22nd International Conference on Distributed Com-
puting Systems Workshops, pp. 611–618. IEEE (2002)
16. Ribeiro, A.R., Silva, F., Freitas, L.C., et al. : SensorBus: a middleware model for
wireless sensor networks. In: Proceedings of the 3rd International IFIP/ACM Latin
American conference on Networking, pp. 1–9. ACM (2005)
17. Silva, J.R., Delicato, F.C., Pirmez, L., et al. : PRISMA: a publish-subscribe and
resource-oriented middleware for wireless sensor networks. In: Proceedings of the
Tenth Advanced International Conference on Telecommunications, Paris, France,
p. 8797 (2014)
18. Souto, E., Guimares, G., Vasconcelos, G., et al.: Mires: a publish/subscribe mid-
dleware for sensor networks. Pers. Ubiquit. Comput. 10(1), 37–44 (2005)
19. Technical report: police and fire department. Obtained in September 2016
 Gossip Message Dissemination Protocols
in the Presence of Link Instability

Takumu Hirooka and Naohiro Hayashibara(B)

Kyoto Sangyo University, Kyoto, Japan

{i1788205,naohaya}@cc.kyoto-su.ac.jp

Abstract. In peer-to-peer networks, each node directly connects to

other nodes without access points. This type of network system is use-
ful for information sharing by using mobile devices (e.g., smartphones).
Although it is easy to construct an ad-hoc network, there are several
problems, for instance, the management of the network under node
churn. On message delivery over the network, it might be difficult to
assume the static routing if the structure of the network can be changed
frequently by node churn and link failures. In this paper, we evaluate the
message reachability by gossip protocols on random geometric graphs
under link instability. Specifically, we focus on the distribution of link
failures on the network. The result shows that the link instability caused
by nodes’ churn has a significant impact on the message reachability
compared to a uniform link instability.

1 Introduction

Peer-to-peer networks are widely used in numerous applications such as commu-

nication platforms, file sharing applications. In this type of networks, each node
directly connects to other nodes without any access points or network switches.
Thus, it is also used as a model of wireless ad-hoc networks. There are lots of
research work in this area for the past decade.
On peer-to-peer networks, the routing of messages is one of the problems if
the network allows the arrival and departure of nodes in it (i.e., the network is
dynamic). To address this problem, there are several approaches such as the use
of routing tables [13], the structured topology of nodes [1,14], the epidemic-style
message dissemination [6,11].
In this paper, we focus on gossip-style protocols, a kind of the epidemic-style
message dissemination in peer-to-peer networks. There are also many papers
related to the performance evaluation of gossip protocols [3,7].
However, there is few performance study on it with peculiar network topology
such as degree-biased network topology. We suppose that gossip-style messaging
protocols are sensitive to the network topology because those are probabilistic
protocols such that each node forwards received messages to randomly selected
neighbor nodes and increase message recipients with time. For instance, gossip
protocols are not efficient in general if the network includes many nodes but is
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 535–545, 2020.
https://doi.org/10.1007/978-3-030-15032-7_45
536 T. Hirooka and N. Hayashibara

sparse (i.e., the average degree is low). Such protocols can forward messages in
an inefficient way. It means that they can be optimized by the link structure.
So, the main contribution of this paper is to clarify the impact of the network
structure fluctuations on the message reachability by gossip protocols. The net-
work structure can be changed in the presence of link failures. Link may not be
available by node movement, node failure or instability of radio wave. We focus
on the distribution of link failures and assume two properties of link failures, local
link fluctuation and uniform link fluctuation. The former one can be occurred
by node churn or node failure and the latter one can be occurred by transient
link failure. This paper aims to clarify the impact of the distribution of link
failures on gossip protocols regarding message reachability in a P2P network. In
our simulations, we use random geometric graphs [9] as a model of the P2P net-
work. Our simulation is conducted by the discrete event simulator implemented
in Java. We show the performance result of the GMBC protocol [15] and the
fixed fanout gossip (FFG) protocol [7] and discuss on the relationship between
the efficiency of the message dissemination and the fluctuation of the network
structure by link failures.
This paper is organized as follows. Section 2 introduces several works related
to gossip protocols and Sect. 3 describes the network topology and the failures
we assume in the paper. Then, Sect. 4 explains the gossip protocols that are
used in our simulation. We conduct simulations to observe the impact of the
distribution of the link instability on the message reachability. Section 5 shows
the simulation result and discusses the impact with the link instability. Finally,
Sect. 6 summarizes the work.

2 Related Work

We now introduce several approaches for message dissemination on P2P networks

and large-scale distributed systems.
Kermarrec et al. [8] provided a theoretical perspective of two gossip protocols,
one is on a flat organization and the other is on a hierarchical structure, in the
context of group membership. The common thing among those protocols is to
forward messages to a fixed number fanout of nodes which are randomly chosen
from the set of neighbors. In particular, the hierarchical gossip protocol has
two kinds of fanout, that is intra-cluster fanout for nodes in the group and
inter-cluster fanout for nodes beyond the group. Their analysis indicates the
hierarchical protocol reduces the stress of the network with a small cost of the
latency and performance degradation.
Ferretti et al. [5] showed the experiment focused on a dissemination probabil-
ity of gossip protocols in the context of multiplayer online games on peer-to-peer
scale-free networks. The motivation of the paper is to investigate the most suit-
able protocol to spread game events with low traffic cost. They compared two
types of gossip protocols, one is the protocol that forward the fixed number of
random selected neighbors and the other adopts the certain probability on the
selection of neighbors. The result indicates the former one is not efficient on the
 Gossip Message Dissemination Protocols 537

coverage of nodes which receive messages. In the other hands, the number of
messages is proportional to the number of nodes in the latter protocol while it is
much lower in the former protocol. Unfortunately, the conclusion indicates that
both protocols are not applicable for disseminating game event to all nodes.
The authors defined the common metric, called effectual fanout, among those
protocols and realized a fair comparison of gossip protocols. Effectual fanout
is characterizes the mean dissemination power of infected sites. Moreover, on
large-scale random topologies, the effectual fanout has a linear correlation with
message complexity. Hu et al. evaluate different gossip protocols on large-scale
random topologies by effectual fanout (ef f ).
Although many papers proposed to use gossip protocols for information prop-
agation, van Renesse et al [12]. applied to failure detectors for large-scale dis-
tributed systems. It uses gossip protocols for sharing the status of monitored
nodes and addresses the scalability problem. SWIM failure detector [4] also uses
a gossip protocol for updating membership information in a scalable manner.
Philips et al. [10] presented that the expected number of neighbors to ensure
network connectivity in a model of packet radio networks. Each node is able to
connect others in range R. Nodes are distributed according to a two-dimensional
Poisson point process with density D.

(1 + )lnA
R= , > 0 (1)
Dπ
A is the size of the area in the network. D is the number of nodes in the unit of
the area, calculated by NA . N is total number of nodes in the network. Equation
(1) is to calculate the radius R on the network that is connected, where A and
N are parameters.

3 System Model
We assume unstructured peer-to-peer networks that each node is directly con-
nected other nodes. In this paper, we use a random geometric graph G = (V, E)
as a model of the network structure (topology). Each node v ∈ V (G) is located in
the Euclidean plane and an undirected link {vi , vj } ∈ E(G) between two nodes
vi and vj exists if and only if the distance of these nodes is less than r. We call
r the communication radius of each node in the rest of the paper. We assume
that any node is reachable from every other node.
Each node corresponds to a computer (e.g., server, laptop PC, mobile device)
and each link corresponds to a communication channel between computers. The
network structure might be changed by power-off and restart the computer, and
link failure.
We suppose the nodes and links may fail temporary. It means that they are
unstable and move between faulty and non-faulty states alternately. We also
assume the following two situations of link instability.
538 T. Hirooka and N. Hayashibara

Localized link instability (LLI) Links become unstable by nodes’ churn.

When a node leaves the network, links that are connected to the node are
unavailable. They become available after the node joins the network again.
Uniform link instability (ULI) Links become unstable at the given probabil-
ity in the whole network. It is a kind of dynamic networks like Time-varying
network [2].

Both of them are transient failure of links.

The goal of this paper is to analyze the impact of the locality of link failure
on the efficiency of the message dissemination by gossip-style protocols.

4 Gossip Message Dissemination Protocols

Unstructured P2P network uses the epidemic-style message dissemination

because it does not have any routing table.
Gossip-style message dissemination protocols are that each node forwards a
message to some of the neighboring nodes that are randomly selected. This type
of protocols can reduce forwarding messages compared to flooding. On the other
hand, message delivery is probabilistic. In this paper, we use two gossip-style
protocols; Fixed Fanout Gossip and a local topology aware gossip protocol.

4.1 Fixed Fanout Gossip

Fixed Fanout Gossip (FFG) [7] is a simple gossip protocol. This protocol has a
parameter f anout. This parameter is the number of messages to be forwarded.
When a node receives a message, It forward the message to the given num-
ber (f anout) of neighbor nodes. The message reachability of the gossip protocol
depends on a network structure. It deteriorates the message reachability in the
network whose density of links is biased [15].

4.2 Gossip-Style Message Dissemination Protocol Based on

Biconnected Components

Gossip-style Message Dissemination Protocol based on Biconnected Components

(GMBC) is a local topology aware protocol to improve the message reachability.
Each node has a matrix that represents the connectivity of neighboring nodes
for detecting biconnected components (MBC). Once a node detects biconnected
components by MBC, it optimizes the message forwarding. MBC is a matrix
which includes one-hop neighboring nodes and represents the distance values of
them from the node. Figure 2 is an example of a matrix M BCX of node X in the
network shown in Fig. 1. According to M BCX , the node A is connected with the
node C in two hops. Maximum value IN F means that the node A and D belong
to different components. Node X can see the two components M1 = {A, B, C}
and M2 = {D, E} from M BCX , and sends message to each components. Con-
cretely, each node preferentially forwards a message to neighboring nodes in a
 Gossip Message Dissemination Protocols 539

component different from the previous sender node of the message. The number
of forwarding messages is given as the parameter f anout. In this example with
f anout = 3. When the node X of Fig. 2 receives a message from the node A,
X preferentially forwards the message to node D and E, and one of the rest of
neighbors (i.e., B and C) that is randomly selected.

Fig. 1. Example of a P2P network

Fig. 2. M BCX represents neighboring information of X

5 Performance Evaluation
We measure the performance of the GMBC and the simple gossip protocol, called
the fixed fanout gossip (FFG) [7]. Specifically, we focus on the reachability of
a message. Gossip-style messaging protocols are probabilistic solutions. Thus,
they do not guarantee that a message sent by a node is eventually delivered
to every other node. Moreover, the reachability of messages can be changed by
the structure of the network [15]. So, we clarify the message reachability of the
proposed protocol in our simulation on random geometric graphs.
540 T. Hirooka and N. Hayashibara

5.1 Environment and Parameters

We assume the random geometric graph [9,10] such that each node has a constant
circular communication range of the radius r and connects with nodes in the
range, in our simulations. Nodes are located in the Euclidean plane at uniformly
random. Each node automatically decides its neighbors (i.e., generates links)
according to r and creates links to them. Thus, the link structure of a random
geometric graph is correlated to the geometric property of nodes on the Euclidean
plane.

(1 + )AlnA
N= (2)
r2 π
Equation (2) indicates N to be able to keep connectivity, when each node has own
communication range of the radius r to connect its neighbors in area A at high
probability [10]. Note that A is the size of the field. We configure A = 150 × 150,
r = 10 and  = 0.1 in the simulations.
We use the discrete event simulator implemented in Java. It executes sequen-
tial turns. Each node receives and forwards messages in each turn.
The network topology can be changed at the beginning of each turn according
to the parameter the link instability rate and the churn rate. The link instability
rate indicates how many links are unstable during a simulation. The churn rate
is the ratio of nodes that behave churn.
Churn rate (CR) is configured from 0.00 to 0.70. It correlated to the link
instability rate and the average number of link failures shown in Table 1. Thus,
we configure the link instability rate (LIR) from 0.00 to 0.86.

Table 1. Correlation of the churn rate, the link instability rate and the number of link
failures

Churn rate (CR) 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70
Link instability rate (LIR) 0.00 0.12 0.25 0.38 0.52 0.65 0.76 0.86
Avg. num. of link failures 0 511 1.030 1.587 2.152 2.693 3.157 3.570

5.2 Performance Metrics

Message reachability shows the proportion of nodes that received the message
sent from the source node by a gossip protocol. It is defined as the Eq. (3).
 
|Recv|
Reachability = Avg (3)
|N |
Recv(⊆ N ) is a set of nodes which received the message sent from the source
node. So, we define Reachability is the average of the rate of nodes, which
received the message, in each case of our simulation.
 Gossip Message Dissemination Protocols 541

5.3 Simulation Result

We clarify the impact of the distribution of link failures on the message reach-
ability by gossip-style protocols on random geometric graphs. We use Fixed
Fanout Gossip (FFG) and GMBC for message dissemination. The simulations
are conducted by the discrete event simulator implemented in Java.
Figure 3 shows the message reachability of FFG and GMBC with Localized
Link Instability (LLI). Nodes can leave and join repeatedly. As a result, links
connected to the nodes are unavailable during they are leaving. The x-axis of the
figure indicates the churn rate and the y-axis indicates the message reachability.
The impact of LLI on the message reachability is significantly big. FFG and
GMBC with two fanouts go down by 0.8% and the ones with four fanouts go
down by approximately 0.4% in the reachability in the case that 10% of nodes
churn. GMBC is slightly better than FFG with 4 fanouts in the case of the churn
rate 0.2 and 0.3, and with three fanouts in the case of the churn rate 0.1, 0.2
and 0.3. On the other hand, they are almost same with two fanouts.

1.0
FFG(2)
FFG(3)
FFG(4)
GMBC(2)
GMBC(3)
0.8 GMBC(4)

0.6
reachability

0.4

0.2

0.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
churn rate

Fig. 3. Message reachability of FFG and GMBC with LLI

To observe the impact of Uniform Link Instability (ULI), we use the average
number of link failures in LLI, which is correlated to the churn rate (see Table 1).
Thus, the x-axis of Fig. 4 indicates the number of link failures. The reachability
of both protocols with three and four fanouts starts to go down at more than 70%
542 T. Hirooka and N. Hayashibara

of links fluctuated that is compatible with the churn rate of 0.6. On the other
hand, the reachability of both protocols with two fanouts gradually declines by
approximately 20%.
Although GMBC with two fanouts is slightly better than FFG with two
fanouts, both protocols are almost the same regarding the tendency of their
message reachability.

1.0

0.8

0.6
reachability

0.4

0.2 FFG(2)
FFG(3)
FFG(4)
GMBC(2)
GMBC(3)
GMBC(4)
0.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
link instability rate

Fig. 4. Message reachability of FFG and GMBC with ULI

Although GMBC is better than FFG regarding the reachability when the
link structure of the network is biased [15], the reachability of them does not
have a significant difference in the uniform link structure with the link instability
according to Figs. 3 and 4.
Now, we look at the impact of the distribution of link failures. According to
the results shown in Figs. 3 and 4, link instability caused by node churn (LLI)
affects the degradation of the message reachability by gossip-style protocols sig-
nificantly compared to ULI. Since FFG and GMBC show the same tendency
regarding the reachability, we show the result of GMBC for the discussion on
the impact of LLI and ULI on the reachability. Figure 5 shows the reachability
of GMBC under both LLI and ULI with the given LIR [0.12, 0.86] that corre-
sponds to CR [0.1, 0.7] (see Table 1). In general, LLI has a significant impact
on the link structure compared to ULI. According to this, LLI has a significant
impact on the reachability of GMBC. Moreover, LLI disables nodes in addition
 Gossip Message Dissemination Protocols 543

to links through this situation is very rare in ULI. Such difference influences
the message reachability by GMBC. The result indicates that the extent of the
topology change is an important factor for the message reachability of gossip
protocols.
The number of fanouts is also crucial for the reachability with ULI. According
to the Fig. 5, the reachability of GMBC with four fanouts is improved 65.4% in
LIR = 0.12, 82.5% in LIR = 0.25 and 85.0% in LIR = 0.38 compared to the
one with two fanouts in LLI. On the other hand, it is improved 14.3%, 15.3%
and 16.5% with four fanouts, respectively compared to two fanouts in ULI. As
a result, the number of fanout is an essential factor to improve the reachability
in the presence of LLI.

1.0

0.8

0.6
reachability

0.4

0.2

0.0
0.12 0.25 0.38 0.52 0.65 0.76 0.86
link instability rate
GMBC(2) with ULI GMBC(4) with ULI
GMBC(2) with LLI GMBC(4) with LLI

Fig. 5. Comparison of the message reachability with LLI and ULI

6 Conclusion
We have evaluated the impact of the distribution of link instability on the effi-
ciency of message dissemination by using gossip protocols in random geomet-
ric graphs. We assumed two patterns of link instability; localized link instabil-
ity (LLI) and uniform link instability (ULI). The former one is caused by nodes’
churn, and the latter one is caused by transient link failure. To observe the
544 T. Hirooka and N. Hayashibara

impact, we used two gossip protocols; Fixed Fanout Gossip (FFG) and the local
topology aware gossip protocol (GMBC).
The result shows that LLI has a significant impact on the message reacha-
bility compared to ULI. This is because nodes surrounded by disabled links are
also unavailable on LLI. Thus, the link structure is affected profoundly by LLI.
On the other hand, the number of fanouts is also an important factor to
improve the message reachability. It is more efficient for the improvement under
LLI compared to the one under ULI.
Moreover, we compared FFG and GMBC regarding the message reachability
on random geometric graphs with both LLI and ULI. GMBC is slightly better
than FFG in the case of three and four fanouts under LLI. GMBC is also better
than FFG with two fanouts under ULI. However, those tendencies are quite
similar in every scenario and we do not observe a significant difference regarding
the message reachability though GMBC is better than FFG in a biased link
structure [15].
The result of the paper would be useful for the message dissemination using
gossip protocols in MANETs and unstructured P2P networks.

References
1. Aberer, K., Cudré-Mauroux, P., Datta, A., Despotovic, Z., Hauswirth, M.,
Punceva, M., Schmidt, R.: P-Grid: a self-organizing structured P2P system. SIG-
MOD Rec. 32(3), 29–33 (2003). https://doi.org/10.1145/945721.945729
2. Casteigts, A., Flocchini, P., Quattrociocchi, W., Santoro, N.: Time-varying graphs
and dynamic networks. In: Frey, H., Li, X., Ruehrup, S. (eds.) Ad-hoc, Mobile,
and Wireless Networks, pp. 346–359. Springer, Heidelberg (2011)
3. Castro, M., Costa, M., Rowstron, A.: Performance and dependability of struc-
tured peer-to-peer overlays. In: Proceedings of the 2004 International Conference
on Dependable Systems and Networks (DSN 2004), pp. 9–18 (2004)
4. Das, A., Gupta, I., Motivala, A.: SWIM: scalable weakly-consistent infection-style
process group membership protocol. In: Proceedings International Conference on
Dependable Systems and Networks, pp. 303–312 (2002). https://doi.org/10.1109/
DSN.2002.1028914
5. Ferretti, S., D’Angelo, G.: Multiplayer online games over scale-free networks: a
viable solution? In: Proceedings of the 3rd International ICST Conference on
Simulation Tools and Techniques, SIMUTools 2010, pp. 5:1–5:8. ICST (Institute
for Computer Sciences, Social-Informatics and Telecommunications Engineering),
ICST, Brussels (2010). https://doi.org/10.4108/ICST.SIMUTOOLS2010.8655
6. Frey, D., Guerraoui, R., Kermarrec, A.M., Koldehofe, B., Mogensen, M.,
Monod, M., Quéma, V.: Heterogeneous gossip. In: Proceedings of the 10th
ACM/IFIP/USENIX International Conference on Middleware, Middleware
2009, pp. 3:1–3:20. Springer, New York (2009). http://dl.acm.org/citation.cfm?
id=1656980.1656984
7. Hu, R., Sopena, J., Arantes, L., Sens, P., Demeure, I.: A fair comparison of gossip
algorithms over large-scale random topologies. In: 2012 IEEE 31st Symposium on
Reliable Distributed Systems (SRDS) (2012)
8. Kermarrec, A.M., Massoulie, L., Ganesh, A.J.: Probabilistic reliable dissemination
in large-scale systems. IEEE TPDS 3, 248–258 (2003)
 Gossip Message Dissemination Protocols 545

9. Penrose, M.: Random Geometric Graphs. Oxford University Press, Oxford (2003)
10. Philips, T.K., Panwar, S., Tantawi, A.N.: Connectivity properties of a packet radio
network model. IEEE Trans. Inf. Theory, 35(5) (1989)
11. Raman, V., Gupta, I.: Performance tradeoffs among percolation-based broadcast
protocols in wireless sensor networks. Int. J. Parallel Emerg. Distrib. Syst. 25(6),
509–530 (2010). https://doi.org/10.1080/17445760903548309
12. van Renesse, R., Minsky, Y., Hayden, M.: A gossip-style failure detection service.
In: Proceedings of the IFIP International Conference on Distributed Systems Plat-
forms and Open Distributed Processing, Middleware 1998, pp. 55–70. Springer,
London (1998). http://dl.acm.org/citation.cfm?id=1659232.1659238
13. Rowstron, A., Druschel, P.: Pastry: scalable, decentralized object location, and
routing for large-scale peer-to-peer systems. In: Middleware 2001, pp. 329–350
(2001)
14. Stoica, I., Morris, R., Karger, D., Kaashoek, M.F., Balakrishnan, H.: Chord: a
scalable peer-to-peer lookup service for internet applications. In: Proceedings of
the 2001 Conference on Applications, Technologies, Architectures, and Protocols
for Computer Communications, SIGCOMM 2001, pp. 149–160. ACM, New York
(2001). https://doi.org/10.1145/383059.383071
15. Yamamasu, D., Hayashibara, N.: On message reachability of gossip algorithms in
degree-biased peer-to-peer networks. In: 2014 20th IEEE International Conference
on Parallel and Distributed Systems (ICPADS), pp. 817–822 (2014). https://doi.
org/10.1109/PADSW.2014.7097888
 Analysis of Ethereum Smart Contracts
and Opcodes

Stefano Bistarelli1 , Gianmarco Mazzante2(B) , Matteo Micheletti2 ,

Leonardo Mostarda2 , and Francesco Tiezzi2
1
University of Perugia, Perugia, Italy
[email protected]
2
University of Camerino, Camerino, Italy
[email protected]

Abstract. Much attention has been paid in recent years to the use of
smart contracts. A smart contract is a transaction protocol that exe-
cutes the terms of an agreement. Ethereum is a widely used platform for
executing smart contracts, defined by using a Turing-complete language.
Various studies have been performed in order to analyse smart contract
data from different perspectives. In our study we gather a wide range of
verified smart contracts written by using the Solidity language and we
analyse their code. A similar study is carried out on Solidity compilers.
The aim of our investigation is the identification of the smart contract
functionalities, i.e. opcodes, that play a crucial role in practice, and single
out those functionalities that are not practically relevant.

1 Introduction
In recent years, increasing attention has been drawn towards the use of smart
contracts for various application areas, such as public registries, registry of deeds,
or virtual organisations. Smart contracts are a digitalised version of traditional
contracts which should enhance security and reduce the transaction costs that
are related to contracting. One of the most prominent platform for smart contract
definition and execution is Ethereum1 [14]. This is a blockchain-based distributed
computing platform that allows to create smart contracts by using a Turing-
complete language.
Various studies have been carried out to analyse smart contracts data from
different angles. [1] analyses smart contracts in order to detect zombie contracts,
while [3] inspects the usage of contracts with respect to their application domain.
Finally [8,10] study the contracts from technical, economic, and legal perspectives.
In this paper we present a study that gathers ten of thousands of verified
Ethereum smart contracts that has been written by using the Solidity language.2
A contract is verified when a proof that it can be obtained by compiling a (Solid-
ity usually) source code can be provided. Our study analyses the hexadecimal
1
https://www.ethereum.org/.
2
https://github.com/ethereum/solidity.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 546–558, 2020.
https://doi.org/10.1007/978-3-030-15032-7_46
 Analysis of Ethereum Smart Contracts and Opcodes 547

bytecode instructions of smart contracts, by referring to their equivalent human

readable format called opcode. We have analysed the opcodes frequency distri-
bution for the considered contracts and for various compilers, in different period
of times. We have discussed in details why some opcodes are more frequent than
others, while some others are not used at all.
Our study permits to gain a precise understanding on how the linguistic
constructs supported by Ethereum have been used in practice by contract pro-
grammers in the last two years. The results of our analysis can enable some
simple, yet effective, checks on contracts concerning anomalous usage of opcodes
(e.g., presence of opcodes never used in the practice). In addition, our study
permits to identify a set of core features laying the groundwork for defining, as
a long term goal, new formalisms and domain specific languages (DSLs) sup-
porting the development of applications based on smart contracts. On the one
hand, formalisms pave the way for the use of formal techniques for verification.
On the other hand, frequently used opcodes can be linked to a set of widely used
programming patterns related to specific domains of application. Such informa-
tion can be exploited to devise different DSLs to more conveniently define smart
contracts for specific application contexts.
The rest of the article is organised as follows. Section 2 outlines the basic
concepts of Ethereum; Sect. 3 overviews the experimental setup that has been
used to gather smart contract data and discusses the result of our analysis;
Sect. 4 reviews the related work; finally, Sect. 5 concludes the paper and outlines
future work.

2 Ethereum Background
The blockchain implements a ledger which records transactions between two
parties in a verifiable and permanent way. The blockchain is shared and syn-
chronised across various nodes (sometimes referred to as miners) that cooper-
ate in order to add new transactions via a consensus protocol [12]. This allows
transactions to have public witnesses thus making some attacks (such as mod-
ification) more difficult. In this paper we focus on Ethereum [14] which is a
blockchain-based distributed computing platform that allows the definition of
smart contract, i.e., scripting functionality.
One of the main feature of Ethereum is its Turing-complete scripting lan-
guage, which allows the definition of smart contracts. These are small applica-
tions that are executed on the top of the whole blockchain network. The code
of an Ethereum contract is written in a low-level, stack-based bytecode lan-
guage, i.e., the Ethereum Virtual Machine (EVM) code. The instructions of the
hexadecimal bytecode representation are often mapped into a human readable
form which is referred to as opcode. An exhaustive list of EVM bytecodes and
opcodes can be found in the Ethereum Yellow Paper [14]. High-level program-
ming languages are available to write smart contracts, whose code is compiled
into EVM bytecode in order to be executed in the blockchain. Currently, the
most prominent language to write Ethereum smart contracts is Solidity. Two
548 S. Bistarelli et al.

different Solidity compilers are available: solc3 and solc-js4 . The former is
written in C++, while the latter in Javascript. Our study only considers solc,
which is the official and most maintained compiler for writing smart contracts.
The solc compiler was released on the 21st of August 2015 version 0.1.2 and is
currently at version 0.5.1 released on the 3rd of December 2018.
A smart contract is added to the blockchain as a transaction. Explorers
can be used to read code and transactions of smart contracts. An explorer is a
website that tracks all the information inside the blockchain and shows it in a
human readable form. Explorers can perform various analysis on the blockchain
and allow the verification of contracts. This is a three-step process where: (i)
the author publishes the compiled code of the contract in the blockchain, then
(ii) she loads the original source code and the version of the compiler into the
explorer, and finally (iii) the explorer marks the contract as verified when the
compiled code can be indeed obtained from the source code. This process cannot
be performed by only considering the blockchain which does not store any source
code nor compiler information.

3 Analysis of Smart Contracts and Opcodes

This section overviews the experimental setup that has been used to gather
various smart contract data and the result of its analysis.

3.1 Experimental Setup

We have used Etherscan5 in order to retrieve smart contracts information.
Although several explores are available (e.g., Etherchain.org6 , Ethplorer7 and
Blockchair8 ) Etherscan is the only one that allows to obtain verified smart con-
tracts. Our study considers the following smart contract information:
• the Ethereum unique address of the contract;
• the translation of the smart contract from its bytecode form into the opcode
one;
• the Solidity compiler version that has been used to compile the smart
contract;
• all dates where at least a smart contract was verified.
We have obtained the data of all contracts that have been verified between
October 2016 and May 2018 (the date at which our data collection activity
ended). Very few contracts were verified before October 2016 thus we have not
considered these contracts.
3
https://github.com/ethereum/solidity.
4
https://github.com/ethereum/solc-js.
5
https://etherscan.io/ [13].
6
https://www.etherchain.org/.
7
https://ethplorer.io/.
8
https://blockchair.com/ethereum.
 Analysis of Ethereum Smart Contracts and Opcodes 549

We have implemented a Java program, available online9 , to scan the Ether-

scan web pages of verified smart contracts. The scanning is used to retrieve the
addresses of all verified contracts. A smart contract address can be given as an
input to an Etherscan API10 that outputs the smart contract source in an opcode
form. Our Java tool analyses the contract opcodes and store in a JSON format
the address of the smart contract, the compiler version used to compile the con-
tract and all contract opcodes with the related frequency (i.e., the number of
times the opcode appears inside the contract).

3.2 Results

In this section we present the quantitative analysis that has been performed on
the smart contract data we have described in Sect. 3.1.

3.2.1 Opcode Frequency of All Verified Contracts

Table 1 reports the number of verified contracts per month and the total number
of opcode these contracts used. Notice how the number of contracts exponentially
incfrease from 2016 to today.
The histogram of Fig. 1 instead, displays on the x-axis the hexadecimal value
of all opcodes (the entire list of opcodes can be found at [14]) while on the Y-
axis the global frequency of each opcode. This is obtained by summing up the
number of times each opcode appears inside each contract. It is worth noticing
that only 5 opcodes have a global frequency that is more then 5% of the sum of
all global frequencies (see Fig. 2 that represents the global frequencies of Fig. 1
with a logarithm scale).

Fig. 1. Histogram of opcode count on verified contracts

3.2.2 Frequently Used Opcodes

In the following we discuss why some of the opcodes are frequently used while
others do not appear very often.
Table 2 summarises the ten most frequently used opcodes. Most of these
opcodes are related to stack management operations, such as swap, push and
9
https://github.com/GianmarcoMazzante/opcodeSurv.
10
http://etherscan.io/api?module=opcode&action=getopcode.
550 S. Bistarelli et al.

Table 1. Contracta count and opcode occurrences per month

Month Verified contracts Opcode count on

verified contracts
10/2016 53 630859
11/2016 83 809555
12/2016 72 1491497
1/2017 108 1818251
2/2017 126 1830664
3/2017 120 1958167
4/2017 198 3332301
5/2017 270 3903969
6/2017 359 5436532
7/2017 702 10495739
8/2017 947 13541032
9/2017 1108 16653251
10/2017 1473 22308628
11/2017 1977 32242058
12/2017 2002 32415653
1/2018 2716 44550116
2/2018 3749 65411651
3/2018 3804 71645646
4/2018 3926 75555729
5/2018 3941 80398664

Fig. 2. Histogram of opcode count on verified contracts (log scale)

pop, since the Ethereum Virtual Machine has a stack architecture. The PUSH1
operation adds 1-byte value into the stack. This is the most frequent operation
since it is a basic stack management operation and every contract starts with
the sequence: PUSH1 0x60 PUSH1 0x40 MSTORE. This also explains the presence
of the memory storing opcode MSTORE amongst the most used opcodes. While
there are various PUSHs (Table 3 shows the First five most used PUSH opcodes)
that differ from the amount of bytes they push into the stack, there is only
one POP opcode that works equally on every element of the stack. The PUSH
 Analysis of Ethereum Smart Contracts and Opcodes 551

Table 2. The ten most used Table 3. First five most used
opcodes of verified contracts push opcodes

Most used opcodes on Most used PUSH

verified contracts opcodes on verified
1 PUSH1 contracts
2 SWAP1 1 PUSH1
3 PUSH2 3 PUSH2
4 DUP1 18 PUSH20
5 POP 23 PUSH4
6 JUMPDEST 41 PUSH32
7 DUP2
8 ADD
9 AND
10 MSTORE

and POP behaviour does not ensure that the number of POP is the same as the
number of all PUSH. In fact, the sum of all PUSH operations is 19788857 while the
number of POP ones is 4247835 (less than one quarter of the previous number).
This is consequence of the behaviour of various opcodes that automatically pop
and push parameters into the stack. For instance the MUL opcode does not just
insert the result of a multiplication on the top of the stack but it also removes
the two factors of the operation, performing a double pop and a single push
behind-the-scenes.

Fig. 3. Pie chart of JUMP,JUMPI, Fig. 4. Line chart comparison of SWAP

JUMPDEST opcodes occurences and DUP occurences

Another opcode that is widely used is the JUMPDEST one which is used to
specify the destination of the jump (JUMP) and the unconditional jump (JUMPI).
These are used to translate (i) loops, (ii) if statements and (iii) switches from
the smart contract source code which justify the high frequency of JUMPDEST
amongst the most used opcodes (see Fig. 3 for the proportion of JUMPDEST with
552 S. Bistarelli et al.

Table 4. Not used opcodes on Table 5. Occurences of environmental infor-

verified contracts mation opcodes on verified contracts

Unused opcodes on Environmental Information

verified contracts opcodes on verified contracts
131 RETURNDATASIZE 27 CALLVALUE
132 RETURNDATACOPY 28 CALLDATALOAD
133 DELEGATECALL 33 CALLER
134 INVALID 50 EXTCODESIZE
135 SELFDESTRUCT 51 CALLDATASIZE
56 ADDRESS
59 CALLDATACOPY
68 CODECOPY
70 BALANCE
100 GASPRICE
104 CODESIZE
106 ORIGIN
130 EXTCODECOPY
131 RETURNDATASIZE
132 RETURNDATACOPY

respect to the other jump operations). We can also find the ADD opcode in the
list of most used opcodes. This is not only used as an algebraic operation by the
developers, but also as an internal command to manage array positions. In other
words, ADD is used when adding an incremental value to the offset of an array
from the MSTORE opcode. The opcodes PUSH20 and PUSH32 are also frequently
used since contracts and accounts are uniquely identified by a 20-byte address
while transactions are identified by a 32-byte address. Differently, the frequency
of SWAPs and DUPs opcodes decreases as the number of bytes increase. Figure 4
shows the frequency of these opcodes as the number of bytes increase from
1 to 16.

3.2.3 Less Frequently Used Opcodes

The behaviour of less frequently used opcodes can be often simulated by using
other opcodes. For instance, the RETURNDATASIZE code has been introduced with
the Ethereum Improvement Proposal (EIP ) number 21111 and can be used to
get the size of the output data of the previous external call. The RETURNDATASIZE
opcode can be simulated by using a sequence of various opcodes (see the EIP-211
proposal for details). In the same way the RETURNDATACOPY can be simulated by
using other opcodes.

11
https://github.com/ethereum/EIPs/blob/master/EIPS/eip-211.md.
 Analysis of Ethereum Smart Contracts and Opcodes 553

There are various opcodes (see Table 4) which are rarely used since they intro-
duce a peculiar variation of an existing opcodes. For instance the DELEGATECALL
opcode is similar to the CALL one except for the context used in the call (see [14]
for details). The INVALID opcode was introduced with the EIP-141 proposal12
and it is similar to the REVERT opcode that was introduced in the EIP-140 pro-
posal13 . Both the opcodes abort the code execution but INVALID drains all the
remaining gas of the caller while REVERT does not. The INVALID behaviour is
never used since smart contract never drain all the remaining gas. In the same
way the SELFDESTRUCT opcode transfers all the ether between two accounts and
destroy the contract14 . This behaviour is never used.
There are also various environmental opcodes which are used to get financial
information. For instance the BALANCE and GASPRICE opcodes are used to get
the residual balance and the gas price, respectively. Table 5 shows that some of
these opcodes are rarely used. For instance the GASPRICE opcode sets the gas
price for transactions. This setting is rarely done since the default gas price is
often used by smart contracts.

3.2.4 Opcodes and Contracts Count over the Time

In this section we analyses the total count of opcodes of verified contracts. The
X-axis of Fig. 5 has a wide range of different months while the Y-axis shows the
following information:
• the number of contracts that have been verified
• the total count of opcodes that are contained inside verified contracts.
The 10th of October 2016 corresponds to the release date of the Solidity version
0.4.2. Figure 5 confirms that the usage of smart contracts raises in popularity.

Fig. 5. Histogram of opcode count per Fig. 6. Opcodes over contract count
month and contract count per month per month

The X-axis of Fig. 6 has a wide range of different months while the Y-axis
shows the total count of opcodes that have been used in a month divided by
12
https://github.com/ethereum/EIPs/blob/master/EIPS/eip-141.md.
13
https://github.com/ethereum/EIPs/blob/master/EIPS/eip-140.md.
14
The address will remain but any interaction with it will only waste gas or ether.
554 S. Bistarelli et al.

Fig. 7. Histogram of verified contracts per date and line chart of Ether value over time

the number of contracts verified in the same month. We kept the same range of
chart 5 to make the two charts comparable. Figure 6 clearly shows that contracts
are increasing in size.
The trend of contract deployment over the time can be better understood by
considering Fig. 7. This contains the number of verified contracts for each day
together with a line chart representing the value of the ether cryptocurrency. We
can easily see that as ether increased (it happened almost in parallel with the
bitcoin) an increasing number of users were writing Ethereum smart contracts.

3.2.5 Different Versions of Solidity Compilers

Fig. 8. Double histogram of opcode count and contract deployment on different version
of Solidity

Figure 8 shows different versions of Solidity compilers (from 0.1.1 to 0.4.25),

having on two different series the total number of contract and opcode calls
respectively, in order to have a comparative view. This shows that the compiler
version v0.4.19 is the most used. It also shows that the Solidity version usage
follows the Ethereum trend both in terms of platform popularity and value of
the currency (depicted on Fig. 7).
Figure 9 considers the Solidity compiler version v0.4.19 and shows the number
of occurrences of each opcode in the source code of the Go implementation.
 Analysis of Ethereum Smart Contracts and Opcodes 555

Fig. 9. Histogram of opcode occurrences on Solidity v0.4.19 source code

It shows that the most used are the stack management opcodes, among with
memory and storage management opcodes.

4 Related Work
In the literature there is a limited amount of works on studies of Ethereum
smart contracts and theirs analysis and statistics, with respect to other well-
known blockchains like Bitcoin [5–7].
Some of these studies focus on security issues. Atzei, Bartoletti and Cimoli
provide a survey on attacks to Ethereum smart contracts [2]. They define a
taxonomy of common programming deadfalls that may lead to different vulner-
abilities. The work provides helpful guidelines for programmers to avoid security
issues due to blockchain peculiarities that programmers could underestimate or
not be aware of. With a similar aim, Delmolino et al. provide a step by step
guide to write “safe” smart contracts [9]. The authors asked to the students
of the Cryptocurrency Lab of the University of Maryland to write some smart
contracts, and guided them to discover all the issues they had included in their
contracts. Some of the most common mistakes included: failing to use cryptog-
raphy, semantic errors when translating a state machine into code, misaligned
incentives, and Ethereum-specific mistakes such as those related to the interac-
tion between different contracts.
Anderson et al. provide a quantitative analysis on the Ethereum blockchain
transactions from August 2015 to April 2016 [1]. Their investigation focuses on
smart contracts with a particular attention to zombie contracts and contracts
referenced before creation. They performs a security analysis on that contracts
to check the usage of unprotected commands (like SUICIDE). They also inspects
the contracts code to look for similarities which could result from a contract being
written by following tutorials or from testing and variants. In the aforementioned
works, correctness of smart contracts is checked by inspecting source code for
known pattern. A more formal approach is proposed by Bhargavan et al. [4], who
provide a framework to verify Ethereum smart contracts by (i) compiling them
into F*, to check functional correctness and safety towards runtime errors, and
(ii) decompiling EVM bytecode into F* code to analyse low-level properties (e.g.
bounds on the amount of gas required to run a transaction). Even if the works
described above report analyses of smart contracts, these studies significantly
556 S. Bistarelli et al.

differ from ours, because they focus on security aspects while the aim of our study
is to identify the smart contract functionalities, i.e. opcodes, that play a crucial
role in practice, and single out those functionalities that are not practically
relevant.
Other works cover financial aspects of blockchains and their impact on the
current economy as well as introducing the blockchain technology in some exist-
ing application domains. In [10], Fenu et al. aim at finding the main factors that
influence an ICO success likeliness. First, they collect 1387 ICOs published on
December 31, 2017 on icobench.com. From that ICOs they gather information
to assess their quality and software development management. They also get
data on the ICOs development teams. Second, they study, at the same dates,
the financial data of 450 ICO tokens available on coinmarketcap.com, among
which 355 tokens are on Ethereum blockchain. Finally, they define success cri-
teria for the ICOs, based on the funds gathered and on the trend of the price of
the related tokens.
Boceck and Stiller highlights various set of functions, applications, and stake-
holders which appear into smart contracts and put them into interrelated tech-
nical, economic, and legal perspectives [8]. Examples of new applications areas
are remittance, crowdfunding, or money transfer. An existing application is Car-
goChain, a Proof-of-Concept which shows how to reduce paperwork, such as pur-
chase orders, invoices, bills of lading, customs documentation, and certificates of
authenticity.
The work in the literature closest to ours is the one by Bartoletti and Pom-
pianu in [3]. They perform an empirical analysis of Ethereum and Bitcoin smart
contracts, inspecting their usage according to their application domain and then
focusing on searching for design patterns in Ethereum contracts. Their analysis
on Ethereum contracts starts from a dataset of 811 verified smart contracts sub-
mitted to Etherscan.io between July 2015 and January 2017. The authors define
a taxonomy of smart contracts based on their application domain to quantify
their usage on each category and to study the correlation between patterns and
domains. Our work differs from theirs on some important aspects. In fact, they
study and categorise the smart contratcs transactions loaded in the blockchain
on a certain time period. Instead, we only concentrate on verified smart con-
tracts, because we are interested to find trends and patterns in their code. Our
focus indeed is not on transactions, but on opcodes.
Also, In [11] Kiffer, Levin, and Mislove examine how contracts in Ethereum
are created, and how users and contracts interact with one another. They find
that contracts today are three times more likely to be created by other contracts
than they are by users, and that over 60% of contracts have never been interacted
with. Additionally they find that less than 10% of user-created contracts are
unique and that there is substantial code re-use in Ethereum.

5 Conclusion and Future Work

In this paper we gathered and analysed the verified Ethereum smart contracts
used in the last two years. In particular, we identified most and less used opcodes.
 Analysis of Ethereum Smart Contracts and Opcodes 557

As future work, we plan to better investigate the correlation between opcodes

usage and the corresponding Solidity code to identify relevant patterns, and to
extend our study to non-verified contracts.
We also plan to study and analyse the gas consumption of the contracts
in order to try to optimize smart contract compiler on this direction. Finally,
as longer term goal, we intend to exploit these studies to (i) support formal
analyses on smart contracts and (ii) define DSLs as on top of Solidity for specific
application domains.

References
1. Anderson, L., Holz, R., Ponomarev, A., Rimba, P., Weber, I.: New kids on the
block: an analysis of modern blockchains (2016)
2. Atzei, N., Bartoletti, M., Cimoli, T.: A survey of attacks on ethereum smart con-
tracts (SoK). In: Maffei, M., Ryan, M. (eds.) Principles of Security and Trust, pp.
164–186. Springer, Heidelberg (2017)
3. Bartoletti, M., Pompianu, L.: An empirical analysis of smart contracts: platforms,
applications, and design patterns. In: Lecture Notes in Computer Science, March
2017
4. Bhargavan, K., Delignat-Lavaud, A., Fournet, C., Gollamudi, A., Gonthier, G.,
Kobeissi, N., Kulatova, N., Rastogi, A., Sibut-Pinote, T., Swamy, N., Zanella-
Béguelin, S.: Formal verification of smart contracts: short paper. In: Proceedings
of the 2016 ACM Workshop on Programming Languages and Analysis for Security,
PLAS 2016, pp. 91–96. ACM, New York (2016)
5. Bistarelli, S., Mercanti, I., Santini, F.: An analysis of non-standard bitcoin trans-
actions. In: Crypto Valley Conference on Blockchain Technology, CVCBT 2018,
Zug, Switzerland, 20–22 June 2018, pp. 93–96. IEEE (2018)
6. Bistarelli, S., Mercanti, I., Santini, F.: A suite of tools for the forensic analysis of
bitcoin transactions: preliminary report. In: Mencagli, G., Heras, D.B., Cardellini,
V., Casalicchio, E., Jeannot, E., Wolf, F., Salis, A., Schifanella, C., Manumachu,
R.R., Ricci, L., Beccuti, M., Antonelli, L., Sánchez, J.D.G., Scott, S.L. (eds.) Euro-
Par 2018: Parallel Processing Workshops - Euro-Par 2018 International Workshops,
Turin, Italy, 27–28 August 2018, Revised Selected Papers. Lecture Notes in Com-
puter Science, vol. 11339, pp. 329–341. Springer (2018)
7. Bistarelli, S., Santini, F.: Go with the-bitcoin-flow, with visual analytics. In: Pro-
ceedings of the 12th International Conference on Availability, Reliability and Secu-
rity, Reggio Calabria, Italy, 29 August–01 September 2017, pp. 38:1–38:6. ACM
(2017)
8. Bocek, T., Stiller, B.: Smart contracts – blockchains in the wings, pp. 169–184.
Springer, Heidelberg (2018)
9. Delmolino, K., Arnett, M., Kosba, A., Miller, A., Shi, E.: Step by step towards
creating a safe smart contract: lessons and insights from a cryptocurrency lab.
In: Financial Cryptography and Data Security: FC 2016 International Workshops,
BITCOIN, VOTING, and WAHC, vol. 9604, pp. 79–94, February 2016
10. Fenu, G., Marchesi, L., Marchesi, M., Tonelli, R.: The ICO phenomenon and its
relationships with ethereum smart contract environment. In: 2018 International
Workshop on Blockchain Oriented Software Engineering (IWBOSE), pp. 26–32,
March 2018
558 S. Bistarelli et al.

11. Kiffer, L., Levin, D., Mislove, A.: Analyzing ethereum’s contract topology. In: Pro-
ceedings of the Internet Measurement Conference 2018, IMC 2018, Boston, MA,
USA, 31 October–02 November 2018, pp. 494–499. ACM (2018)
12. Swan, M.: Blockchain. O’Reilly Media, Newton (2015)
13. Tan, M.: The ethreum block explorer (2018). https://etherscan.io. Accessed 09 Dec
2018
14. Wood, G.: Ethereum: a secure decentralised generalised transaction ledger (2018).
https://ethereum.github.io/yellowpaper/paper.pdf. Accessed 08 Dec 2018
 Towards 802.11g Signal Strength
Estimation in an Industrial Environment:
A Practical Study

Dalton Cézane Gomes Valadares1,2(B) ,

Joseana Macêdo Fechine Régis de Araújo2 , Ângelo Perkusich2 ,
Marco Aurélio Spohn3 , Elmar Uwe Kurt Melcher2 ,
and Natália Porfı́rio Albuquerque4
1
Federal Institute of Pernambuco (IFPE), Caruaru, Pernambuco, Brazil
[email protected]
2
Federal University of Campina Grande (UFCG), Campina Grande, Paraı́ba, Brazil
[email protected]
{joseana,elmar}@computacao.ufcg.edu.br, [email protected]
3
Federal University of Fronteira Sul, Chapecó, Santa Catarina, Brazil
[email protected]
4
Borborema Termoelétrica LTDA, Campina Grande, Paraı́ba, Brazil
[email protected]

Abstract. With Industry 4.0 and Industrial Internet of Things (IIoT),

new communication protocols are emerging or being updated. These
protocols demand technological updates at industries, mainly related
to their network infrastructures, and generally leading to large expen-
ditures. Given that the 802.11g standard is a largely used communica-
tion protocol, mostly in residential and commercial applications, many
industries also adopt such standard mainly due to its low deployment
and maintenance cost. In this scenario, there is a need to evaluate sig-
nal quality to better design the network infrastructure in order to obtain
good communication coverage. In this work, we present a practical study
about the 802.11g signal strength in a thermoelectric power plant. We
have compared our measured values with the estimated ones through the
Log-Distance Path Loss model. We concluded that it is possible to use
this model in an industrial environment to estimate signal strength with
a low error by choosing the right propagation (path loss) exponent.

1 Introduction
The technological advances have enabled an increasingly adoption of intercon-
nected devices and applications, in the most diverse areas, such as health care
monitoring, vehicle and object tracking, industrial and environmental monitor-
ing. Due to the nature of most of such scenarios, the wireless connectivity is
almost a mandatory communication requirement, since it provides a better flex-
ibility and is less invasive than wired technologies considering the devices usually
employed (sensors, actuators, embedded systems, etc.).
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 559–570, 2020.
https://doi.org/10.1007/978-3-030-15032-7_47
560 D. C. G. Valadares et al.

Many of these applications, besides of the wireless requirement, also require

a good level of reliability. For example, Industry 4.0, with the Industrial Inter-
net of Things (IIoT), usually demands some special requirements, because the
applications have to monitor machines’ operating parameters taking into account
environment constraints such as the incidence of magnetic interference. There-
fore, it is paramount having a reliable wireless infrastructure in order to allow
the responsible sector taking the right decisions in a timely manner.
This kind of application commonly runs on devices with limited process-
ing and storage resources, commonly requiring more powerful devices acting
as gateways. This scenario is known as fog/edge computing [Valadares et al.
(2018)], when gateways perform some data processing before sending them to a
server/cloud. To allow a suitable communication network for these devices, it is
important to know the behavior of the signal propagation in the specific envi-
ronment, in order to better plan and deploy the communication infrastructure
[Chebil et al. (2013)].
We have conducted a study aimed at investigating the wireless signal propa-
gation in an industrial environment. To this end, we measured the signal strength
in 20 points inside the engine room of a Brazilian thermoelectric power plant,
located at Northeast region. After gathering the measurements, we employed a
path loss model for estimating signal decaying resulting from other sources of
interference, path length, the medium, etc. Once we had both real (measured)
and estimated (path loss model) signal strength values, we compared them to
verify if this model was adequate to predict the signal propagation in an indus-
trial environment.
As there was a legacy IEEE 802.11g infrastructure in the environment
and many of the fog/edge devices (i.e., IoT gateways) support communication
through this technology, we decided to perform the measurements based on the
available infrastructure. To estimate the propagation loss and investigate its ade-
quacy to industrial environments, we decided to use the Log-Distance Path Loss
(LDPL) model, since it is a well accepted model in the literature, and it is also
the basis for many other models. The LDPL is applied to indoor and outdoor
environments with the presence of obstacles, having a propagation exponent that
indicates whether the environment has more or less obstacles, impacting on the
computed loss.
Regarding our objective, we have elaborated two research questions for guid-
ing this study:
1. Can the Log-Distance Path Loss model be applied to estimate the signal
strength at an industrial environment?
2. What is a good propagation exponent (parameter) to adopt when using the
LDPL model at an industrial environment.
This document is organized as follows: the model evaluated in this work is
briefly described in Sect. 2; in Sect. 3, we present some works that have some rela-
tion with this study; the methodology and experiments are described in Sect. 4;
the results and a brief discussion are pointed in Sect. 5; lastly, we present the
final considerations regarding this presented work and mention some suggestions
for future work in Sect. 6.
 Towards 802.11g Signal Strength Estimation in an Industrial Environment 561

2 Log-Distance Path Loss Model

In this section, we present a brief description about the Log-Distance Path Loss
model, which is the chosen propagation model to be evaluated in our industrial
environment.
A propagation model refers to the way the signal is propagated in the
medium, considering effects such as reflection, diffraction, refraction, etc. The
signal path loss, or signal power loss, usually occurs with the attenuation of
this signal, when there is a reduction in power density. Some of the reasons for
this loss/reduction are: reflection, refraction, diffraction, absorption, terrain con-
tours, propagation medium (dry or moist), distance, etc. [Ali et al. (2010); Ndzi
et al. (2012)]
A simple and well accepted propagation model, which takes into account the
existence of some obstacles, in open and closed environments, is the Log Distance
Path Loss [Ali et al. (2010); Faria (2005)], whose path loss is calculated with
Eq. 1:
L(d) = L0 + 10nlog(d) (1)
where:
• d = distance
• L0 = signal strength from 1 m of the transmitter (antenna)
• n = propagation exponent (depends of the obstacles in the environment).
Many models derive from LDPL, with adjustments in the propagation expo-
nent and addition of parameters related to the number of walls, floors, etc. In
general terms, the propagation exponent varies according the environment, as
described below [Srinivasa and Haenggi (2009)]:
• Free space - 2;
• Cellular radio in urban area - 2.7 to 3.5;
• Cellular radio in urban area with fading - 3 to 5;
• Closed environment with line of sight - 1.6 to 1.8;
• Building with obstacles - 4 to 6;
• Factory with obstacles - 2 to 3.

3 Related Works
In this section we present some works related to signal strength estimation,
mainly with the use of LDPL model.
Faria (2005) carried out a study on the modeling of signal attenuation in
802.11b networks. This study was performed considering internal and external
communications, in a building of the Stanford University. The experiment con-
sidered 41 measurement points, varying the distances between 1 and 50 m. The
estimated values were calculated with the LDPL model considering a variation in
the propagation exponent. The results validated the use of this model, after com-
parison with measured values, with 4.02 and 3.32 as propagation exponents for
562 D. C. G. Valadares et al.

internal and external communications, respectively. A similar study was accom-

plished at Kuala Lumpur University, by Ali et al. (2010). They investigated the
802.11g signal propagation in a closed environment and also used the LDPL
model to estimate values. The measured values were compared with the esti-
mated ones and the determined propagation exponents, according to floors of
the building, were 2.68, 3.22 and 4.
Lkhagvatseren and Hruska (2011) compared the various path loss models
applied to indoor environments, varying different power levels and frequencies,
and observing the impact of environmental factors. The results showed that
the LDPL model presents good approximation when compared to real values
measured. Cheffena and Mohamed (2017) investigated the path loss effects on a
wireless sensor network (WSN) in a snowy environment. They measured the path
loss at different heights from the ground and compared the results with the values
estimated by the “Two-Ray” and “Ray tracing” models. Since the comparison
presented significant difference, they proposed new empirical models based on
LDPL, which demonstrated good accuracy to apply to WSN deployments in
snowy environments.
Japertas et al. (2012) verified the 802.11g/n signal propagation considering
scenarios non-line-of-sight (NLOS) and with line of sight (LOS) inside a building
with multiple divisions (obstacles). The measured values were compared to esti-
mated values with LDPL and Free Space Path Loss models. As a result, a new
model was proposed, considering NLOS environments with signal transmission
along homogeneous walls. Fernández et al. (2012) proposed some adjustments
to LDPL model when applied to digital TV signal transmission. Three signal
strength measurement sets, obtained in Lima (Peru), were compared to values
estimated with LDPL, Okumura-Hata and ITU-R models. The two adjusted val-
ues for propagation exponent were 4.5 and 4.7, providing to LDPL model good
approximation to the real values.
Rath et al. (2017) observed that the traditional indoor path loss models are
not suitable to Indian scenarios due to some buildings characteristics such as used
materials, floor plans, etc. They proposed a non-deterministic statistical path loss
model, which was compared with LDPL and another model, becoming a good
possibility to be used in India. In an urban environment, in Valencia (Spain), a
path loss characterization of the vehicular-to-infrastructure (V2I) channel was
carried out, by Rubio et al. (2015), also based on LDPL model. The authors
investigated the correlation between the height of the antenna used and the
propagation (path loss) exponent, varying both values, and concluded that there
is no significant correlation degree.
Damsaz et al. (2017) analyzed some wireless propagation characteristics in
industrial environments to propose path loss models, focusing on Zigbee tech-
nology. The measurements were collected at various factory of factory-like envi-
ronments, such as a machine shop or an automotive assembly plant. They deter-
mined the propagation loss based on the distance, the shadowing level and the
RMS delay spread of the channel. A performance of Zigbee radio using the chan-
nels considered was presented. Karaagac et al. (2017) also used the LDPL model
 Towards 802.11g Signal Strength Estimation in an Industrial Environment 563

as a basis to estimate the path loss in an industrial environment. They consid-

ered the 802.15.4e technology, proposing three reliable and flexible architectures
with 2.4 GHz and 868 MHz.
As we can see, many works have concerned with wireless signal propagation,
always aiming at a better network infrastructure planning and deployment. The
Internet of Things, the Industry 4.0 and the Wireless Sensor Networks increase
the importance of this kind of study applied to industrial environments, what
reinforces the motivation and justification of our work.

4 Methodology and Experiment

In this section, we present the experimental design and the methodology

employed to carry out this work, describing the environment, the tools, and
the performed procedures. As mentioned before, the study considers an IEEE
802.11g infrastructure, since it was already deployed at the thermal power plant.
To help and guide the achievement of a satisfiable answer for our research
question, we have defined the business and technical problems, as follow:

• Business problem: investigate if the LDPL model can be applied reasonably

to estimate the signal strength in an industrial environment.

• Technical problems:
– Measure the signal strength throughout different points in the engine
room;
– Compute the signal strength estimated values with the LDPL model and
compare them with the measured ones.

4.1 Environment

The network architecture deployed at the power plant is compounded by four

wireless communication links, located between the administrative room and the
engine room, as seen in Fig. 1.
The UTE IN ADMIN link is located inside the administrative room, whereas
the UTE IN ENGINE is located inside the engine room. The UTE OUT link
connects both UTE IN ADMIN and UTE IN ENGINE links, and it is equipped
with two INTELBRAS WOG 212 antennas, which operate in a frequency of
2.4 GHz, with throughput up to 150 Mbps, compatible with the IEEE 802.11b/g
standards, and an integrated antenna with a gain of 12 dBi and a nominal power
of 27 dBm [Intelbras (2018)]. Both UTE IN ADMIN and UTE IN ENGINE are
characterized by the existence of a TP Link access point, with effective radiated
power of 20 dB. For our study, we just considered the TP Link access point inside
the engine room, which is responsible for the UTE IN ENGINE network.
564 D. C. G. Valadares et al.

Fig. 1. Network architecture in the power plant

4.2 Scenario

The physical scenario where tests were performed is represented in Fig. 2. As we

can see, there are two WOG 212, responsible for the UTE OUT link, and the
AP point is representing the approximate place where TP Link access point is
deployed inside the engine room. The distance between two WOG 212 is about
150 m. The engine room is approximately 150 m long.

Fig. 2. Buildings in the power plant and points inside the engine room.
 Towards 802.11g Signal Strength Estimation in an Industrial Environment 565

The great blocks inside Engine Room, in Fig. 2, represent the engines,
in a total of 20. The signal strength measurements were performed in dis-
tinct points inside the engine room, represented by P 1, P 2, P 3, ..., P 18, P 19, and
P 20 in Fig. 2. These points were chosen to enable the observation of the possible
impact of the electromagnetic interference in the data transmission. They were
positioned at the center of the engines, with 6 m between consecutive points
(except between points P 10 and P 11, which were about 12 m from each other).

4.3 Instrumentation and Tests

To carry out the tests (i.e., obtaining the signal strength in each point), a note-
book running Linux Ubuntu 14 was used. The operating system had to run from
pen drive due to electromagnetic interference in the engine room causing fail-
ures/crashes to hard drives (HDs), as reported by some engineers of the power
plant.
As the main tools, we used iwconfig and linssid. The iwconfig is a Linux
tool for the configuration of wireless network interfaces, enabling the setting up
and verification of parameters such as channel frequency and signal strength
[Community (2018)]. The linssid is a wireless network tracker for Linux, by
graphically representing sensed networks with the corresponding channels in the
frequency domain, including also the following information: signal quality, signal
strength, network ID, channel, and the noise level [WSeverin (2018)].
To gather the signal strength and quality, the notebook was positioned on the
ground in each defined point, while both tools were employed. The tools collected
the data during 40 min for each defined position. To validate our experiment, this
process was repeated five times, in different days and shifts. The data obtained
through this procedure were eventually validated given that they presented a
high degree of similarity.

5 Results and Discussion

In this section we present the results and some discussion on that. With the col-
lected data, it was possible to compute the mean value for each point, allowing the
comparison of such representative unit with the signal strength estimated with the
LDPL model, while varying the propagation exponent from 3 to 6. The estima-
tion was calculated by subtracting the value of the path loss computation at each
point (obtained with the LDPL model) from the effective power radiated by the
access point inside the engine room (20 dB). To calculate the loss at each point, a
reference loss of −20 dB was considered since 1 m from the access point, based on
the work by Faria, which presents an access point with the same characteristics of
ours and the same reference distance. In Fig. 3 we can see the real signal strength
along with the estimated ones (including results for each propagation exponent
value: 3, 4, 5, and 6). LDPLn represents the signal strength values obtained by
using the LDPL model with n as the propagation exponent.
566 D. C. G. Valadares et al.

Fig. 3. Real and estimated signal strength along the engine room.

To facilitate the visualization and interpretation of the graphs, a non-linear

logarithmic regression was performed, obtaining a coefficient of determination
0.91 (R2 = 0.91). In Fig. 4, we show the graphs comparing the logarithmic
regression with the estimated values in natural (above) and logarithmic scale
(below).
As can be seen in Figs. 3 and 4, the estimated values closest to the real ones
are those obtained with the propagation exponent 4 (i.e., LDPL 4). We calculated
the relative error between the real values and the estimated values with LDPL
4, as well as between the logarithmic regression values and the estimated ones
with LDPL 4. The relative errors calculated for each point can be seen in the
Table 1
With the exception of some discrepancies for a few points (P 1 and P 2),
relative errors are all below 15%, and they are even lower when considering
the logarithmic regression. We noticed that the engine room walls worked as a
kind of shielding, helping to keep the signal quality, and attenuating the elec-
tromagnetic interferences effect on the signal strength. A possible explanation
for the discrepancies is that, during the data collection, workers were accessing
the engine room for performing their regular activities, and as the doors were
opened the shielding effect appeared to be somehow broken, by allowing some
signal leakage while impacting the overall signal quality. This makes the results
more realistic, since the data collection was carried out in real working shifts,
and being the least possible invasive.
As the employed model was not designed to take into account electromagnetic
interferences, we are inducted to conclude that the 802.11g signal quality, while
considering the power plant environment (characteristics and “shielding”), is not
much affected by the interferences from the motors, since the estimated power
computed with propagation exponent 4 (which is used to estimate the path
loss in a regular environment with the presence of several obstacles) showed a
 Towards 802.11g Signal Strength Estimation in an Industrial Environment 567

Fig. 4. Logarithmic regression and LDPL estimated signal strength along the engine
room.

good approximation with the measured power. In other words, it seems that
the existing shielding in the engine room counterbalances the negative effect
of electromagnetic interference, characterizing the network behavior as that of
a common environment, in which the signal propagation is, many times, more
reduced by distance and obstacles.
Thus, the main contributions of this work are:
• A practical evaluation in a real industrial environment (i.e., thermoelectric
power plant), while similar studies/works usually simulate the local;
• The determination of the propagation exponent to be considered for 802.11g
signal strength estimation in industrial environments (achieving an error bel-
low 15% for most of the spots in our evaluation).
568 D. C. G. Valadares et al.

Table 1. Relative errors between measured, estimated (LDPL 4) and regression values.

6 Conclusion
The growth of IoT applications adoption in industrial environments, also known
as Industrial IoT applications, awakes some care related to wireless communica-
tions infrastructure, as seen earlier in this document. In this sense, it is important
to make a good planning, in order to establish the right places to locate the access
points so that they can provide the best coverage.
We decided to assist in this task by investigating the 802.11g signal strength
in a real industrial environment: a thermoelectric power plant. After an estab-
lished experimental design, we have collected signal strength measurements along
the engine room to compare with the calculated ones by the LDPL model.
The performed evaluation, resulted from the calculated values and comparisons,
demonstrates that the LDPL model can be used to estimate the 802.11g signal
strength in an industrial environment with an error lower than 15% to most of
the points considered. This is possible by setting the propagation exponent n
to 4.
Thus, we concluded that the LDPL model also can be used in industries,
assisting in the establishment of a good network infrastructure inside engine
rooms with similar characteristics as that we investigated, in a thermal power
plant. As future work, we envisage that some machine learning technique can be
applied to determine a model that better fits to the real path loss in each point,
resulting also in a better adjusting to the measured values of signal strength,
obtained in the engine room. Furthermore, this experiment can also be replicated
to observe the behavior of signal strength from another wireless technologies,
such as IEEE 802.11n, IEEE 802.11af and IEEE 802.11ah. Yet, together with
the applied path loss model, we suggest the new analyses can be improved by
the addition of some heat map tool, identifying points with better coverage and
points more affected by electromagnetic interferences.
 Towards 802.11g Signal Strength Estimation in an Industrial Environment 569

Acknowledgement. The authors thank Borborema Energética S.A. and Maracanaú

Geradora de Energia S.A., sponsors of the ANEEL GASIS R&D project, in which
this research was inserted, as well as CNPq, for having financed some months of the
master’s research of the main author.

References
Ali, A.H., Razak, M.R.A., Hidayab, M., Azman, S.A., Jasmin, M.Z.M., Zainol, M.A.:
Investigation of indoor WiFi radio signal propagation. In: 2010 IEEE Symposium
on Industrial Electronics and Applications (ISIEA), pp. 117–119 (2010). https://doi.
org/10.1109/ISIEA.2010.5679486
Chebil, J., Lwas, A., Islam, M.: Comparison between measured and predicted path loss
for mobile communication in Malaysia. World Appl. Sci. J. (Math. Appl. Eng.) 21,
123–128 (2013). https://doi.org/10.5829/idosi.wasj.2013.21.mae.99936
Cheffena, M., Mohamed, M.: Empirical path loss models for wireless sensor network
deployment in snowy environments. IEEE Antennas Wirel. Propag. Lett. 16, 2877–
2880 (2017). https://doi.org/10.1109/LAWP.2017.2751079
Community L: iwconfig(8) - linux man page (2018). https://linux.die.net/man/8/
iwconfig
Damsaz, M., Guo, D., Peil, J., Stark, W., Moayeri, N., Candell, R.: Channel model-
ing and performance of Zigbee radios in an industrial environment. In: 2017 IEEE
13th International Workshop on Factory Communication Systems (WFCS), pp. 1–10
(2017). https://doi.org/10.1109/WFCS.2017.7991975
Faria, D.B.: Modeling signal attenuation in IEEE 802.11 wireless LANs (2005)
Fernández, J., Quispe, M., Kemper, G., Samaniego, J., Dı́az, D.: An improvement of
the log-distance path loss model for digital television in Lima. In: XXX Simpósio
Brasileiro de Telecomunições (2012)
Intelbras: WOG 212 CPE 2.4 GHz 12 DBI (2018). http://en.intelbras.com.br/business/
outdoor-radios/cpe/wog-212
Japertas, S., Orzekauskas, E., Slanys, R.: Research of IEEE 802.11 standard signal
propagation features in multi partition indoors. In: 2012 Second International Con-
ference on Digital Information Processing and Communications (ICDIPC), pp. 1–4
(2012). https://doi.org/10.1109/ICDIPC.2012.6257267
Karaagac, A., Haxhibeqiri, J., Joseph, W., Moerman, I., Hoebeke, J.: Wireless indus-
trial communication for connected shuttle systems in warehouses. In: 2017 IEEE
13th International Workshop on Factory Communication Systems (WFCS), pp. 1–4
(2017). https://doi.org/10.1109/WFCS.2017.7991971
Lkhagvatseren, T., Hruska, F.: Path loss aspects of a wireless communication system
for sensors. Int. J. Comput. Commun. 5 (2011)
Ndzi, D., Arif, M., Shakaff, A., Ahmad, M., Harun, A., Kamarudin, L., Zakaria, A.,
Ramli, M., Razalli, M.: Signal propagation analysis for low data rate wireless sensor
network applications in sport grounds and on roads. Prog. Electromagn. Res. 125,
1–19 (2012). https://doi.org/10.2528/PIER11111406
Rath, H.K., Timmadasari, S., Panigrahi, B., Simha, A.: Realistic indoor path loss
modeling for regular WiFi operations in india. In: 2017 Twenty-third National Con-
ference on Communications (NCC), pp. 1–6 (2017). https://doi.org/10.1109/NCC.
2017.8077107
570 D. C. G. Valadares et al.

Rubio, L., Fernández, H., Rodrigo-Peñarrocha, V.M., Reig, J.: Path loss characteri-
zation for vehicular-to-infrastructure communications at 700 MHz and 5.9 GHz in
urban environments. In: 2015 IEEE International Symposium on Antennas and Prop-
agation USNC/URSI National Radio Science Meeting, pp. 93–94 (2015). https://doi.
org/10.1109/APS.2015.7304432
Srinivasa, S., Haenggi, M.: Path loss exponent estimation in large wireless networks. In:
2009 Information Theory and Applications Workshop, pp. 124–129 (2009). https://
doi.org/10.1109/ITA.2009.5044933
Valadares, D.C.G., da Silva, M.S.L., Brito, A.M.E., Salvador, E.M.: Achieving data dis-
semination with security using FIWARE and Intel software guard extensions (SGX).
In: IEEE Symposium on Computers and Communications (2018)
WSeverin: Linssid (2018). https://sourceforge.net/projects/linssid/
 Revealing Storage and Speed Transmission
Emerging Technology of Big Data

Heru Susanto1,2,4, Fang-Yie Leu3(&), Didi Rosiyadi1,

and Chin Kang Chen5
1
Research Center for Informatics, The Indonesian Institute of Sciences,
South Jakarta, Indonesia
{heru.susanto,didi.rosiyadi}@lipi.go.id
2
Information Management Department, Tunghai University,
Taichung City, Taiwan
3
Computer Science Department, Tunghai University, Taichung City, Taiwan
[email protected]
4
School of Business, University Technology of Brunei,
Bandar Seri Begawan, Brunei
5
School of Business and Economics, University of Brunei Darussalam,
Bandar Seri Begawan, Brunei
[email protected]

Abstract. The study highlighted the storage of data and the ability to transmit it
at a faster speed securely. It is acknowledged that Big Data represents a large
volume of data, and this brings a matter of ways on how to manage storage
capacity as data continues to grow exponentially, that may drive to storage
technologies to uphold its pace to the predicament. In addition, due to an
incomparably large amount of data received, processed and transferred, it also
affects the ability of its data transmission speed. In other hand, the employability
condition of Information Technology (IT) professionals, which is desperately
essential in the rapid change of the technological world. This challenge defines
the aspects, without paying necessarily much attention to the lack of Information
Technology staff and services, but is more catered towards the capability of
one’s uphold and acknowledgement of new information upon new inventions of
computing and telecommunications devices in the market. Here, the empow-
erment of Information Technology comes from the aspect of how it helps as a
platform for these fields to operate successfully and embrace positive changes in
delivering efficient as well as effective work, also benefits business companies in
decision making, marketing and business growth, and resource management,
and globalization. it is impossible to attain long term business success without
leveraging the benefits of IT in this digital age.

Keywords: Data transmission
Big Data

Computerized Axial Tomography (CAT)

Magnetic Resonance Imaging (MRI)
Data scientists

Business intelligence analysts

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 571–583, 2020.
https://doi.org/10.1007/978-3-030-15032-7_48
572 H. Susanto et al.

1 Introduction

This study entails a focus on the aspects of Information Technology and Big Data by
highlighting the two issues stated which are concurring among most current businesses
and organizations in today’s world. In this section, it depicts some foundational ideas
which are demonstrated by identifying their general key issues, brief information on its
current trends and the importance of their applications.

1.1 Key Issues – Information Technology and Big Data

Information Technology is viewed to carry predominant impacts and changes upon the
human life as a whole, and the trust to improve and enhance global technological
processes and operations especially when it comes to businesses and industries. Indeed,
the central idea agreed in the use of Information Technology comes as a role to facilitate
actions. With this, it is in a profitable sense that with Information Technology, it helps
companies and businesses to survive in the competitive world [1]. In the emerging
technological advancement and information stored, the technological landscapes tend to
experience the consideration of Big Data. It comes as the product from the demand and
high volume of information storage, and the limitation of it causing to invent further
higher storage capacity to uphold the vast information scale. Nevertheless, Big Data is
understood to have a higher capacity in profiting an organization in any kind of industry,
ubiquitously in this era of globalization. In this context, it becomes essential when it
draws to decision-making and aids to improve the financial condition of any organiza-
tion. However, in this competitive world, there are many new technologies to master and
it is evolving at a rather fast rate. Information Technology and Big Data are both in the
pursuit of innovation and so, are needed to be aligned with the changing global trends.

1.2 Current Trends

Technological advancement and innovation are always in flux causing constant
transformation of the application of Information Technology in such a way that the
computer software and hardware creations, capabilities and functions are deemed to
move parallel with this change in order to adapt to the needs of global trends. In this
perspective, some technical trends that are known come from the lowering of the price
of communication and computing devices, and the existence of digital wireless com-
munication where people in different places, regardless of near or distant, can be
reached quicker and this is done at a lower cost. In addition, Management Study Guide
[2] denotes the emergence of Cloud Computing, Mobile Application, User Interfaces
and Analytics are reckoned to leave a major influential impact upon businesses and
companies as a goal to offer services of a higher and better quality to their customers.
On another context, with the rising of Big Data potentiality and its promises in profiting
financial performance, some predictions have been made such as the rising of the Chief
Data Officer, whom holds a powerful position to the business users and Information
Technology talents [3]. Additionally, Hinchcliffe [4] perceives Big Data as one of the
consortium in the Big Five Information Technology trends of the next half decade. The
acknowledgement of this information has emphasized the value and importance of the
existence of Information Technology and Big Data.
 Revealing Storage and Speed Transmission Emerging Technology of Big Data 573

1.3 Importance of Their Applications

Information Technology has grown to support almost all aspects of society. Filho and
Ramos [5] denoted that a key criterion to a successful organization was the willingness
to apply Information Technology in the process. This has brought about a perspective by
which the importance of Information Technology is imminent. Even, Frenzel and
Frenzel [6] confer those businesses, who utilize Information Technology, to, “support or
mound strategic trusts in a competitive environment to achieve high potential in many
contemporary industries”. As for Big Data, it has become one of a ‘scheme’ towards a
potential competitive advantage as most current businesses and companies are lever-
aging data-driven strategies. Even, Schonberger and Cukier [7] represent Big Data as a,
“source of new economic and innovation.” Through this context, it is understood that
the capabilities of Big Data in gaining a lucrative image that is identified to bring an
effective approach to technological success, has driven a degree of societal importance
not just in the field of businesses and sciences but to other aspects as well, such as
education, healthcare, government and even humanities. Needless to say, the importance
of Information Technology and Big Data has been known to bring success owing to
truth that they can never astray from being embraced with certain issues.

2 Problem Statements

Although Information Technology and Big Data can bring a positive connotation to the
field of global technological development and consensus, there are, however, several
problems arising from it. In this case, this study entails the two main issues which are
the ‘norms’ within Information Technology and Big Data.
The first issue to highlight is the storage of data and the ability to transmit data at a
faster speed. It is acknowledged that Big Data represents a large volume of data, and
this brings a matter of ways on how to manage it in terms of its storage capacity as data
continues to grow exponentially. This raises a concern for storage technologies to
uphold its pace to the predicament. In addition, due to an incomparably large amount of
data received, processed and transferred, it also affects the ability of its data trans-
mission speed.
The second issue is the employability condition of Information Technology
(IT) professionals, which is desperately essential in the rapid change of the techno-
logical world. This problem defines the aspects, without paying necessarily much
attention to the lack of Information Technology staff and services, but is more catered
towards the capability of one’s uphold and acknowledgement of new information upon
new inventions of computing and telecommunications devices in the market - in this
case, it focuses on the talent pools of Information Technology professionals.

3 Literature Review

The following is literature reviews which are believed to be applicable in justifying the
study.
574 H. Susanto et al.

3.1 Empowerment of Information Technology

TechTarget [8] categories Information Technology as, “the use of any computers,
storage, networking and other physical devices, infrastructure and processes to create,
process, store, secure and exchange all forms of electronic data.” It can also be con-
sidered to be represented as hardware computers and connects the interrelated pro-
cesses in processing data. However, it is not only limited to computer devices, but
extends to other electronic communications that can run software as well. In addition,
Information Technology is the study and usage of operating systems for storage,
retrieving and sending information which is crucial in fields such as businesses, edu-
cation and even in healthcare. In this sense, the empowerment of Information Tech-
nology comes from the aspect of how it helps as a platform for these fields to operate
successfully and embrace positive changes in delivering efficient as well as effective
work [9].
Quinonez [10] notes that “since the computer was introduced, the business world
was changed forever.” With the aid of computers and software, business companies are
able to ensure that each of their departments are capable of running smoothly as well as
giving business companies the ability to view global market changes instantly and
allowing them to keep up with updates of the supply and demand of consumers.
Information Technology also benefits business companies in decision making, mar-
keting and business growth, and resource management, and globalization. Thus, Butt
[11] claims that, “…it is impossible to attain long term business success without
leveraging the benefits of Information Technology in this digital age.”
Apart from its importance in businesses, Information Technology also plays a
major role in education. For example, some lecturers may face adversity to cater an
increasing amount of students in terms of providing handouts or giving lectures. Now,
with the help of Information Technology, file sharing and online lectures are made
available and readily accessible to the students and help ease the lecturers’ workload.
Information Technology has also made a revolutionary impact in relation to the
systems and utilities used in the healthcare industry. Currently, Computerized Axial
Tomography (CAT) and Magnetic Resonance Imaging (MRI) scans from hospitals
worldwide require computers to be able to produce images of the scans. Traditional
hospitals and clinics are bound to keep a patient’s medical information for future
references which means there would be much paperwork for the doctors. In contrast,
most hospitals and clinics in the modern day have advanced by storing medical
information electronically. An example of this advancement is the BruHims1. The
BruHims system integrates and manages patients’ medical information which aims to
effectively improve time management for patients and doctors in addition to storing
patients’ confidential information [12, 13].
The importance of Information Technology does not solely rely on the three pre-
viously mentioned areas because it is also considered crucial in other fields. Hence, it is
undeniable that Information Technology is important in this new era.
 Revealing Storage and Speed Transmission Emerging Technology of Big Data 575

3.2 The Causal Experience with Big Data

Koscielniak and Puto [14] perceive from a frequent adaptation by Douglas Lane in
2001 to define Big Data as a large amount of data that are generated at a faster pace
than normal. The condition of Big Data traits comes from a category known as ‘4 V’
rule which are volume, variety, velocity and value. Essentially, Big Data is a large
amount of unorganized data that need to be sorted as these data belong to different
categories [14]. The importance of sorting data and easing out processes and operations
consolidate one’s experience. In this manner, Schmarzo [15] perceives that organiza-
tions should be leveraging with Big Data as a mean to uphold the condition and
opportunity to improve their users’ experience. In addition to this context, Schmarzo
[15] also define Big Data as something that invokes the organizations capability to, “…
identify new sources of customers, products and operational data”. This has brought
about a perceptive experience in the use of Big Data in the business field. Moreover,
Kudyba [16] denotes the era of Big Data would give customer experience by adjusting
information offerings more towards the users. This shows that the capability of Big
Data to bring an impactful decision for organizations to deliver business outcomes and
abilities. Another context relevant to this theme is the representation of data set in Big
Data. The birth of Big Data is as result of data sets collected by inexpensive and large
amount of information sensing devices, aerials, radio frequency identification meeters,
software logs, microphones, camera and wireless sensor networks.

4 Discussion

This section caters the group’s general view regarding the information obtained about
securing of information technology and Big Data which are displayed according to the
stated theme.

4.1 Views on Information Technology and Big Data

The view on Information Technology agreed is the set of methods to process the
complex large amounts of data for analysis, whereas Big Data is the term commonly
used to describe the process of applying serious computing power - the latest in
machine learning and artificial intelligence - to massive and often highly complex sets
of information. These two aspects have brought about the importance of their appli-
cations which is one of the highlights in the literature, which supports the operational
conditions of businesses and companies. This is by then the use of both entities have
brought about huge changes to the world in a sense that it minimizes cost and save
time. When it comes to the technological development in developing countries,
Information Technology and Big Data are somehow limited. One of the general issues,
the adoption of it with challenges such as infrastructure, training and qualification. In
general, Information Technology infrastructure consists of computers, software and all
components of telecommunication systems needed for data transfer and management.
This has support the relevance of lacking data storage capacity and speed of data
transfer if there are less updated technological infrastructures observed. The emergence
576 H. Susanto et al.

of technologies and devices, have predicted that there will be vast amount of data
created within the next few years. As matter of fact, a total of 90% of recent data were
established in the past few years - “a trend that will continue for the foreseeable future”
[17]. As this trend continues, Information Technology and Big Data intellects are
needed in order to progress with a systematic approach and take advantages on it for
business success. These components are crucial to exhibit technological support and
advancement in the developing countries. Not only that, developing countries also lack
skilled Internet Technologies professionals who can maintain systems and information
technology personnel to operate the systems. Specifically, this focuses on how well the
professionals do their work which, as represented in the literature, the employability of
the information technology staff should be in a good status and finding the right talent
pool is a paramount in this case. As Information Technology keeps on being updated
and introducing new applications and products to the market, the information tech-
nology staff should feel the need to educate themselves with new information regarding
it. The lack of any infrastructure as stated above will hinder the adoption of information
in developing countries, and these resources are still limited within their countries.
Relocating equipment may be easier, but it is more difficult to transfer capacity which
require human skills. Most of the times, developing countries do not have enough
graduates in science and technology academics.
The support of data analysis upon Information technology have brought about huge
changes upon the technological process. However, one of the main issues in data
processing is the lack of suitable analytical methods for use because of the rapid growth
of Big Data and the high computational complexity of algorithms for analysis. Thus,
this leads to an increase in analysis time even with regular updates of information
technology. This is a concern on the issue of the speed of transferring data to one
another which might be affected due to the fact that more data storage space is needed
to support this condition.
Big Data is an attention-driven topic among the industries, businesses and orga-
nizations around the globe. The concept of Big Data has existed since the production of
the first few computers. This was thought to be an efficient way to do work, as it is
faster and less time consuming. However, as years passed, the numbers of data by users
have increased which may have caused a few challenges for the society. Hacking for
information is a possible yet inevitable issue that could happen to many organizations
and institutions. Although, it is important for individuals to know that they are in
control of their information, privacy however may be compromised as most companies
and organizations store confidential information in their technological hardware as it is
an efficient means. On the other hand, to extract information may be of a slower
process as Big Data are usually arranged in an unstructured form that requires analysis
to differentiate from one another [18]. For instance, the data retrieved from MRI is
different from the ones retrieved from banks. Until, there is a standard procedure that is
thought of, this may still remain as an issue.
By tapping a continual stream of information from internal and external sources,
businesses today have an endless array of new opportunities for transforming decision-
making, discovering new insights, optimizing the business and innovating their
industries. Therefore, Big Data creates a new layer in the economy which is about
information, turning information, or data into revenue. This will accelerate growth in
 Revealing Storage and Speed Transmission Emerging Technology of Big Data 577

the global economy and create more jobs. Data storage, slow speed of data transmis-
sion and needing the right talent for professionals in information technology are pos-
sible yet inevitable issues that could happen to many organizations and institutions
[19–21].

4.2 Combating the Issues

With all these large datasets coming in an unprecedented scale, most organizations do
not know which data to start with and which questions to ask of their data. Such data
will remain futile, if one does not use it properly. A way to combat this is to develop
different theories relevant to the decisions being made and then use the data obtained
and analytics to demonstrate these theories. McGuire, Manyika and Chui [22] men-
tioned that sophisticated analytics can substantially improve decision-making, mini-
mize risks, and unearth valuable insights that would otherwise remain hidden.
Many organizations, however, failed to cope with the scale of data flowing into the
business, thus disabling them to figure out which technological tools to aid their
concerns. But, if the company prioritizes on a relatively small data to identify the
current and anticipated problems- this disability can be avoided. This will provide the
company a small insight on how to tackle the issues and capture the data needed.
Due to the large input of data and information by different people, storage of data
has become an unmanageable issue even though there are bigger storage mediums
created as technology advances. However, even with the increased production of newer
and more advanced technologies, the message handling speeds are slowly decreasing.
This could be because of the unstructured form that the data is arranged accordingly.
Nevertheless, information extraction is usually done manually but it is now made easier
with the existence of machines. On the same note, labeling data may still be carried out
by less-skilled laborers as the cost of maintenance is cheaper.
The shortage of information technology professions and finding talents is also one
of the main issues with Big Data. Big Data needs visualization experts, data scientists,
business intelligence analysts, data privacy experts and data warehousing professionals
to remain operational. These skills are onerous to develop in a short period but even
outsourcing them is costly and impossible in most markets. This can be overcome by
creating a center of excellence which provides a single focus and platform for sharing
of insights and information exchanges which may resolve on shortages of professionals
in information technology.
According to released studies from the Information Technology Association of
America and the Department of Commerce, it was claimed that the shortage of
Information Technology workers is increasing rapidly over the years. However, aside
from the surveys mentioned, there were no other studies conducted to support the
confirmation of the claims. Hence, there is no definite answer to the question of the
shortage of workers in information technology as there is the lack of proper evidences
and studies. Despite having said that, with the current lifestyle, digitized markets and
products, information technology workers may be one of the safest job position to be in
[23] for the meantime.
On the same note, wage inequality may instead be on the rise and it is affecting the
work enforcements. Many economists believed that the advancement of Information
578 H. Susanto et al.

Technology has contributed significantly to the rise of this problem. This means that
the demand for low-paid laborers is reducing because their work is slowly being
replaced by machines and this may cause high unemployment rate in certain job
sections if this persists. Although this does not entail the condition of the professional
employability in information technology, it is crucial to highlight this issue in which
human physical work are now starting to be replaced by machines. This foretells the
future of Information Technology so as whether there will be a rapid change in the
outlook of future work environments.
Modern communities are revolving around electronics such as the Internet and
mobile phones, which causes the methods of communication to change. As the number
of people who uses these platforms increases, the volume of data also increases through
the usage of social media and work-related means. However, the current data pro-
cessing management is unable to withstand the volume of data created [24, 25]. On the
other hand, with the easy access to the Internet through communication devices, data
may be created and distributed at a very fast pace. Due to this matter, this also means
that specialists and data analysts may have difficulties in coping with the explosion of
data created.
Once Big Data is established into the organization, the corporate executives are
needed to embrace the transformation and empower the rest of the organization to
adopt the change. This will reduce the lack of holistic approach towards Big Data.
Basically, if the speed of data cannot comprehend the transfer of large amount of data,
there will be a flood of information and data will be scattered around and this will cause
a problem in data storage (see Fig. 1).

information technology

Security
framework
big data;
speed and storage

App layers; e-health, cheminformatics, 5G network

Fig. 1. Implementing within apps layer

4.3 Securing the Information

Information security becomes very hot topic recently in the era internet of things.
Securing information content is very important for protecting the information integrity.
Any previous studies have been done in this field.
 Revealing Storage and Speed Transmission Emerging Technology of Big Data 579

Reversible image watermarking has been proposed by [26]. It was used the mod-
ified histogram based on the payload, thus the changes can be minimized. The
reversibility was obtained from the storing of changes in the virtual border LSB. In
[27], conducted the DCT watermarking scheme to comprise the level of watermark
resistance and the level of host image perceptibility against various attacks.

4.4 General Recommendations

There are a few ways to help diminish the problem of income inequality such as
education, the better management of structural policies and the implementation of
taxes. Through these methods, economies have the chance to improve by responding to
competitive pressures and opportunities that results to the development of a more
innovative society in the long run. The success of big data and analytics endeavors will
ultimately be determined based on profitability: through lower costs or higher revenues.
This is why the organization needs to come up with ways to measure the specific
impact of their data efforts and continue the business benefits. Once big data is
established, the organization will need to lessen and mitigate risk as it will be an
ongoing challenge. Risk-planning, scenario mapping and drill-type exercises will
create awareness throughout the business of key risks, emphasizing on flexibility,
ability to adapt and responsibility.
The business company will need processes and systems that will propagate the
generated insights throughout the whole organization. Without a strategic data scheme,
the company will not be successful as it does not apply to the rest of the business.

5 Case Study: Hitachi Data System (HDS), Ltd.

This section outlines a brief synopsis of how a technological company is affected by the
concurrent issues demonstrated in this study. It foretells the company on how to face
the issues and create ideas in establishing solutions to overcome them. Hitachi Data
System is an ideal sample for the case study as it promotes ample amount of infor-
mation regarding how they face challenges in terms of data storage capacity, data speed
transmission and its condition of information technology professional employability.

5.1 Background of the Company

In place of the continuity to increase the number of storage systems and give rise to
storage management complexity, the focal point should be shifted to a more eco-
nomically favorable and sustainable approach to handling Big Data. For Hitachi Data
Systems (HDS), that approach is storage virtualization [28]. Known for being a leader
in storage virtualization, Hitachi Data Systems has aided many organizations in
“transforming raw data into valuable information” [29]. Now with storage virtualiza-
tion, Hitachi Data Systems is continuing to innovate the technological field, having
shipped over 25,000 virtual storage controllers globally [28]. However, before entering
into the benefits of storage virtualization, it is important to understand why it is a
necessity, in terms of issues faced by many organizations regarding information
580 H. Susanto et al.

technology and big data, focusing primarily on storage of data, speed of transmission
and employability of professionals in information technology.

5.2 Information Technology (IT) and Big Data Issues

The first issue encountered by Hitachi Data Systems with information technology and
big data was the storage of data, more specifically, the exponential growth of data and
disruptive storage upgrades. As of 2014, 45% of organizations have announced that
their foremost problem has been the exploding growth of capacity, and by which data
growth had to be managed just beyond purchasing more storage space, with the help
from creative use of information technology [28]. If the option was to add more
storage, data migration of prolonged duration as well as extensive application dis-
ruptions should be avoided. Currently, not all data need to reside on high-performance
storage. Those used for backup, archival, test or development will perform fairly on
lower-priced, high-capacity disk storage. However, the introduction of such systems
into an existing storage infrastructure can lead to multiple storage systems that operate
in isolation from others, which, in turn, may cause increase in complications in man-
agement as well as the incapability to easily transfer data from storage of high stan-
dards to substandard systems.
Speed of data transmission is the second issue regarding information technology
and big data. Business owners grade the success of their information technology ser-
vices based on the level of performance their day-to-day applications run, including the
level of services, such as access, availability and reliability. Therefore, it is significant
to deliver quick and high-quality storage service to application and business owners.
However, information technology is not equipped to handle these diverse data needs,
and simultaneously, still be able to provide high quality service. For that reason, it is
obligatory to offer certainty of consistently high application performance and respon-
siveness, predominantly when storage resources are combined or shared with other
competing applications on the network.
The third issue Hitachi Data Systems has faced, with respect to information tech-
nology and big data, is the employability of information technology professionals.
Many organizations of information technology have been requested to look into
problems relating to budgets and staff sizes that remain constant or are decreasing [28].
Finding engineers that are talented and of quality seemed impossible as there were not
enough highly qualified candidates for the diversity of tough-to-fill positions. In
addition to that, Eidelson [30] reported that the wages for information technology
workers were about the same in 2014 as they were in 1999. This factor has majorly
contributed to repelling potential candidates for a particular position as they may feel
that, for their level of employability, they deserve to be paid more than is received.
Thus, they may rather enter into a career, which may have any connections to infor-
mation technology at all, that provides a higher salary. Growing complications in the
rudimentary infrastructure needed to support core applications has also imparted dif-
ficulty in employability of professionals in information technology, due to the fact that
more human resources are forced to handle the increased complexity in the enterprise
data center when they could be redeployed on other projects of their specialization or of
higher value to the business.
 Revealing Storage and Speed Transmission Emerging Technology of Big Data 581

6 Conclusions

Information Technology and Big Data helps advance technology and the betterment of
society lifestyle. Due to the ever changing pattern of technological climate and more
advancements are being created, the Information Technology and Big Data should
equip to align with the changes. The integral definition of Information Technology and
Big Data should be well understood and need to acknowledge how they link to one
another and how it procures to extrude opportunities and advantages. In this case, it is
relatively important to capture the benefits each one of them brings to the current world
in order to gain a successful foundation to businesses and companies. Despite its
importance to the global community, it has several issues. As discussed, data storage is
increasingly bigger to help habitat the data created and how this can as well affect the
transmission of the data in relation to its speed and processing power. The idea of this
issue challenges the needs of Information technology to adapt with the delivery and
update themselves to support high volume of data in the coming future. However,
drawbacks such as the employability of low-paid laborers are lowered and the speed to
extract information are compromised. This in turn emphasizes the need to find more
new talents upon new computer technologies being introduced in the market and how
this is able to support the mission of businesses and companies to stay upfront. The
implementation of taxes and virtualization technology are only a few of many ways to
combat the stated issues. In conclusion, it is crucial to understand the aspects of
Information Technology and Big Data in the upbringing of global technological
advancement and how they stipulate the foundation to aid in terms of operations and
processes not just in businesses but also in education, healthcare, government and other
institutions.

References
1. Carr, N.G.: Does IT Matter? Information Technology and the Corrosion of Competitive
Advantage (2004). https://books.google.com.bn/books?id=XMqlN3RKpcYC&printsec=fro
ntcover&dq=information+technology+competitive+world&hl=en&sa=X&ved=0ahUKEwix
2I-u_JTLAhXWBY4KHSOqDP0Q6AEIGjAA#v=onepage&q=information%20technology
%20competitive%20world&f=false. Accessed 24 Feb 2016
2. Management Study Guide: Emerging Trends in Information Technology (2016). http://
www.managementstudyguide.com/emerging-trends-in-information-technology.htm. Acces-
sed 20 Feb 2016
3. Davis, J.: Big Data Predictions for 2016. InformationWeek, 4 January 2016. http://www.
informationweek.com/big-data/big-data-analytics/big-data-predictions-for-2016/d/d-id/1323
671. Accessed 20 Feb 2016
4. Hinchcliffe, D.: The “Big Five” IT Trends of the Next Half Decade: Mobile, Social, Cloud,
Consumerization and Big Data. ZDNet, 2 October 2011. http://www.zdnet.com/article/the-
big-five-it-trends-of-the-next-half-decade-mobile-social-cloud-consumerization-and-big-
data/. Accessed 24 Feb 2016
582 H. Susanto et al.

5. Filho, L.A.M.M., Ramos, A.S.M.: The perception of managers on the impacts of the internet in
Brazilian hotels: an exploratory study. In: Kamel, S. (ed.) Managing Globally with Information
Technology, pp. 244–259 (2003). https://books.google.com.bn/books?id=3sb5PUqBAiMC&
pg=PA245&dq=importance+of+information+technology&hl=en&sa=X&ved=0ahUKEwjL
xJXFm5XLAhWXWo4KHeObBA0Q6AEIGjAA#v=onepage&q=importance%20of%20inf
ormation%20technology&f=false. Accessed 23 Feb 2016
6. Frenzel, C.W., Frenzel, J.C.: Management of Information Technology (2004). https://books.
google.com.bn/books?id=nQgLAAAAQBAJ&pg=PA49&dq=information+technology
+importance&hl=en&sa=X&ved=0ahUKEwiXmt_
FoZXLAhVQjo4KHeK0C6oQ6AEIJTAC#v=onepage&q=information%20technology%
20importance&f=false. Accessed 23 Feb 2016
7. Schonberger, V.M., Cukier, K.: Big Data: A Revolution That Will Transform How We Live,
Work, and Think (2013). https://books.google.com.bn/books?id=HpHcGAkFEjkC&printse
c=frontcover&dq=big+data+prediction+2016&hl=en&sa=X&ved=0ahUKEwitmu7GhpXL
AhWJBI4KHXDoBQsQ6AEIPTAD#v=snippet&q=importance&f=false. Accessed 22 Feb
2016
8. TechTarget: Information Technology (IT) (2016). http://searchdatacenter.techtarget.com/
definition/IT. Accessed 20 Feb 2016
9. Susanto, H., Almunawar, M.N.: Information Security Management Systems: A Novel
Framework and Software as a Tool for Compliance with Information Security Standard.
CRC Press, Boca Raton (2018)
10. Quinonez, N.: The Importance of Information Technology and Its Disadvantages (2014).
https://blog.udemy.com/importance-of-information-technology/. Accessed 17 Feb 2016
11. Butt, A.A.: The Role of Information Technology in Business Success (2015). https://www.
linkedin.com/pulse/role-information-technology-business-success-abid-afzal-butt. Accessed
17 Feb 2016
12. Susanto, H.: Electronic health system: sensors emerging and intelligent technology
approach. In: Smart Sensors Networks, pp. 189–203 (2017)
13. Susanto, H., Chen, C.K.: Macromolecules visualization through bioinformatics: an emerging
tool of informatics. In: Applied Physical Chemistry with Multidisciplinary Approaches,
p. 383 (2018)
14. Koscielniak, H., Puto, A.: BIG DATA in decision making processes of enterprises. Proc.
Comput. Sci. 65, 1052–1058 (2015). http://www.sciencedirect.com/science/article/pii/
S1877050915028835. Accessed 18 Feb 2016
15. Schmarzo, B.: Driving Business Strategies with Data Science Big Data MBA (2016). https://
books.google.com.bn/books?id=tI8gCwAAQBAJ&pg=PA66&dq=experience+with+big
+data&hl=en&sa=X&ved=0ahUKEwjpoOS44pXLAhUCGI4KHU1NB38Q6AEIKzAD#v=
onepage&q=experience%20with%20big%20data&f=false. Accessed 24 Feb 2016
16. Kudyba, S.: Big Data, Mining, and Analytics: Components of Strategic Decision Making
(2014). https://books.google.com.bn/books?id=nuoxAwAAQBAJ&pg=PA160&dq=experi
ence+with+big+data&hl=en&sa=X&ved=0ahUKEwjpoOS44pXLAhUCGI4KHU1NB38Q
6AEISjAI#v=onepage&q=experience%20with%20big%20data&f=false. Accessed 24 Feb
2016
17. Al-Jarrah, O.Y., Karagiannidis, G.K., Muhaidat, S., Taha, K., Yoo, P.D.: Efficient machine
learning for big data: a review. Big Data Res. 2, 87–93 (2015). Big Data, Analytics, and
High-Performance Computing
18. Somro, T.R., Thavet, N.: Big data challenges. J. Comput. Eng. Inf. Technol. 5 (2015).
https://www.researchgate.net/profile/Tariq_Soomro/publication/282281171_Big_Data_
Challenges/links/560a53c908ae4d86bb137402.pdf. Accessed 20 Feb 2015
 Revealing Storage and Speed Transmission Emerging Technology of Big Data 583

19. Liu, J.C., Leu, F.Y., Lin, G.L., Susanto, H.: An MFCC-based text-independent speaker
identification system for access control. Concurr. Comput.: Pract. Exp. 30(2), e4255 (2018)
20. Leu, F.Y., Liu, C.Y., Liu, J.C., Jiang, F.C., Susanto, H.: S-PMIPv6: an intra-LMA model for
IPv6 mobility. J. Netw. Comput. Appl. 58, 180–191 (2015)
21. Susanto, H.: Biochemistry apps as enabler of compound and dna computational: next-
generation computing technology. In: Applied Chemistry and Chemical Engineering,
Volume 4: Experimental Techniques and Methodical Developments, p. 181 (2017)
22. McGuire T., Manyika J., Chui M.: Why Big Data is the new competitive advantage. Ivey
Bus. J. (2012). https://iveybusinessjournal.com/publication/why-big-data-is-the-new-
competitive-advantage/
23. Thibodeau, P.: New job realities ahead for IT workers (2016). http://www.computerworld.
com/article/3023343/it-careers/new-job-realiities-ahead-for-it-workers.html. Accessed 25
Feb 2016
24. Riahi, A., Riahi, S.: The big data revolution, issues and applications. Int. J. Adv. Res.
Comput. Sci. Softw. Eng. 5 (2015). http://www.ijarcsse.com/docs/papers/Volume_5/8_
August2015/V5I8-0231.pdf. Accessed 25 Feb 2016
25. Almunawar, M.N., Anshari, M., Susanto, H., Chen, C.K.:. How people choose and use their
smartphones. In: Management Strategies and Technology Fluidity in the Asian Business
Sector, pp. 235–252. IGI Global (2018)
26. Rosiyadi, D., Horng, S.J., Lestriandoko, N.H.: A resistant digital image watermarking
scheme based on masking model. In: International Carnahan Conference on Security
Technology, ICCST (2015)
27. Lestriandoko, N.H., Juhana T., Munir, R.: Security System for Surveillance Radar Network
Communication Using Chaos Algorithm (2015). https://www.researchgate.net/publication/
283518322_Security_system_for_surveillance_radar_network_communication_using_
chaos_algorithm
28. Hitachi Data Systems: Solve the Top 6 enterprise Storage Issues (2014). https://www.hds.
com/assets/pdf/hitachi-white-paper-solve-the-top-six-enterprise-storage-issues.pdf. Accessed
23 Feb 2016
29. Indeed: Hitachi Data Systems (2016). http://www.indeed.com/hire/case-study/hitachi-data-
systems. Accessed 19 Feb 2016
30. Eidelson, J.: Bloomberg. The Tech Worker Shortage Doesn’t Really Exist, 24 November
2014. http://www.bloomberg.com/bw/articles/2014-11-24/the-tech-worker-shortage-doesnt-
really-exist. Accessed 25 Feb 2016
 A Web-Based Artwork Editing System
Empowered by Neural Style Transfer

Kenta Goto1 and Hiroaki Nishino2(&)

1
Graduate School of Engineering, Oita University, Oita, Japan
[email protected]
2
Faculty of Science and Technology, Oita University, Oita, Japan
[email protected]

Abstract. A technique called neural style transfer is an effective method for

generating artistic images based on a deep learning technique. It can extract a
mood of a specific painting and blends it with a different image. The original
method, however, needs a high-performance computer to get an output image
within a practical response time since the neural style transfer involves heavily-
loaded processing. To solve the problem, we develop a web-based image editing
system enabling users to readily access the function only by using a mobile
device with a standard web browser and a network connection. The proposed
system allows the users to easily generating a wide variety of artistic images like
logos and image clips using the neural style transfer anywhere they have a
connection to the Internet. We implement the system as a web application and
conduct some experiments to verify the effectiveness of the system. We elab-
orate the implementation method, experimental results, and observations in this
paper.

1 Introduction

There is a technique called neural style transfer for automatically generating artistic
images based on a deep learning technique [1]. It extracts nonquantifiable style features
of a specific painting and blends them with entirely-different image contents. It allows
users to produce aesthetically pleasing images irrespective of their artistic sense and
skill. Figure 1 shows example images created by the neural style transfer. It uses a deep
neural network for extracting the features of a content image (a building image) and
three style images (a water splash image, a sunset image, and an ukiyo-e painting “The
Great Wave off Kanagawa” by Katsushika Hokusai, 1847). The deep neural network
extracts feature representations from both of the content and style images, and syn-
thesizes a new image by blending the acquired style features into the content. It
generates a variety of building images with different moods extracted from three style
images as outputs as shown in the figure. The neural style transfer iteratively explores
the best output image that have image features closest to both of the content and the
style images’ features. The output images shown in the figure exemplify that higher
quality images can be generated as the number of iterations increases.
The original neural style transfer technique, however, is difficult to use because it
requires the user to prepare a set of images as inputs and tune up many parameters of

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 584–594, 2020.
https://doi.org/10.1007/978-3-030-15032-7_49
 A Web-Based Artwork Editing System Empowered by Neural Style Transfer 585

the deep neural network to get satisfactory results. Users also need to prepare a high-
performance computer equipped with a GPU node since the neural style transfer
involves heavily-loaded processing. To solve these problem, we develop a web-based
image editing system enabling users to readily access the neural style transfer function
only by using a notebook PC, a tablet, or a smartphone with a standard web browser
and a network connection.
The proposed system allows the users for exploring useful base images to use as
inputs in freely available image archives on the Internet, and finding good results by
trying the system through simple web application interface. The system also provides a
function for opening successful results to others, so the users can easily share and reuse
variety of produced images as fresh materials for editing their own images or new
inputs to the system. Since the neural style transfer is a heavily-loaded and time-
consuming process, we install dual servers for efficiently supporting the system
functions: a web server for dealing with user management, image database manage-
ment, and interactive editing operations with users, and a separate back-end server
specialized for asynchronously executing the neural style transfer processing.
The rest of the paper is organized as follows. After we describe related research
projects in Sect. 2, we describe the functional overview of the proposed system in
Sect. 3. Then, we elaborate the implementation method of the system and the detailed
design of its GUI in Sect. 4. After that, we show some experimental results acquired
and observations found in the preliminary evaluations in Sect. 5. Finally, we conclude
the paper with some issues for further improvements in Sect. 6.

* the number of iterations for image optimization.

Fig. 1. Example images generated by neural style transfer function.

586 K. Goto and H. Nishino

2 Related Work

The original style transfer method was proposed by Gatys et al. They showed a method
for extracting features from a pair of content and style images and generating a new
image by blending the features of both images [1]. It uses a type of convolutional
neural network (CNN) called VGG-network which is trained for recognizing objects
appeared in a picture with high-performance feature extraction capability [2]. Although
the method produces attractive results for some kinds of images such as scenery photos,
it does not work well for some other targets containing characters such as logo images.
A method for clearly extracting specific parts in an image such as characters and
applying the neural style transfer to the parts is an effective approach to solve the
above-mentioned problem. Champandard et al. developed a technique called semantic
map for separating some elements in an image from the other portions with a colored
map [3]. It allows to convert and reconstruct the image into a new one with different
objects’ layout. While their method focuses on reconstructing an image in the neural
style transfer, Luan et al. proposed another method to apply the semantic map function
with the traditional neural style transfer [4]. Their method provides an easy way to
represent the image layout by specifying the objects’ locations in the image such as a
building, a mountain, and sky.
We enhanced the original neural style transfer method by incorporating the Luan’s
approach for explicitly locating specific objects in an image and applying the effects of
the neural style transfer to these objects [5]. We designed and implemented the system
as web application software enabling users to easily access the enhanced neural style
transfer function from their mobile terminal.

3 Functional Overview

Figure 2 shows the processing overview of the enhanced neural style transfer method.
A user needs to prepare at most four input data: a content image, a content semantic
map, a style image, and a style semantic map. The content and style semantic maps are
optional data. The figure exemplifies the author’s organization name logo consisting of
four characters CSIS (Computer Science and Intelligent Systems) generated by the
proposed system.
The logo image of four characters (CSIS) with white background is the content
image to present the core structure of the logo. The style image is a warm-colored
sunset image texture for decorating the logo characters or the background of the content
image. The style semantic map defines the portions in the style image used for dec-
orating the content image. In this example, the system clips a central portion of the style
image corresponding to the sun and its surrounding area and applies its features for
decorating the content image. If a user does not specify the content semantic map, the
system blends the extracted style features into the background part of the content image
that is the same white color as the style semantic map. This process creates the output
image A, the upper right image in the figure, as a result. If the user specifies the content
semantic map with white characters (a reversed content image), the system blends the
 A Web-Based Artwork Editing System Empowered by Neural Style Transfer 587

Fig. 2. Overview of proposed style transfer method.

style features into the character parts and produces the output image B, the lower right
image in the figure.
The user can select his/her preferred artistic picture as a style image for generating
an aesthetically-refined content image. We use the VGG-Network, a type of convo-
lutional neural network trained on object recognition, for extracting the features of the
style and content images. The VGG-Network has been achieved remarkable perfor-
mance in image classification problems. It extracts fine and local features in the lower
layers of the network such as the original pixel values of the input image, and extracts
abstract and global features in the higher layers such as the whole organization and
coloring of the image.

4 System Architecture and Implementation

In this section, we elaborate the implementation method of the proposed image editing
system. We made it as a web application system allowing users to access the neural
style transfer function via standard web environment.

4.1 Web Application Architecture

The main purpose of the system is to enable users to easily access the neural style
transfer functions from their mobile terminals such as PCs and smart phones with a
standard web browser and a connection to the Internet. Therefore, we firstly define a
functional architecture as shown in Fig. 3 and implement the system based on this
architecture. It consists of two servers, the web interaction server (WI server) and the
main task server (MT server). The WI server directly communicates with a web
588 K. Goto and H. Nishino

browser running on the user’s mobile terminal for providing a set of web services such
as user registration and authentication, data upload to and download from the system,
tuning an execution environment, and activating a specific function. The MT server
executes the system functions requested by the user through the WI server. The MT
server deals with the image generation based on the neural style transfer in the pro-
posed system.

Fig. 3. Functional architecture of web application system.

Whereas the WI server manages real-time dialogues with the user, the MT server
focuses on the execution of image generation functions. When the system especially
performs heavily-loaded and time-consuming processing such as deep learning, the MT
server executes the operations asynchronously with the user’s activities. The WI server
acts as a broker to adjust the asynchronous executions between the user and the MT
server. The system manages all data communications between the user and two servers
by using the HTTP protocol.

4.2 System Implementation

Figure 4 shows the software configuration designed and implemented based on the
architecture as explained in the previous section. The two servers are Linux machines
and the WI server is configured as a web server by installing Nginx, an open source
web server program. The whole processing flow is as follows. Firstly, the WI server
receives input images used for the neural style transfer from a user, and registers them
in the database as a new job. The WI server simultaneously allocates a specific time to
start the registered job. Next, the MT server periodically inquires the WI server on the
existence of jobs registered to start execution at that time. Then, the WI server submits
all the jobs to the MT server if exist. Then, the MT server executes all the submitted
jobs in a batch processing manner using the equipped GPU node (NVIDIA
GeForce GTX 1080 with 8 GB memory). After that, the MT server returns a set of
generated image to the WI server and the WI server records them in the database.
Finally, the user checks the job completion after the execution start time passed and
browses the results if available.
The process called Web Service Broker (WSB) running on the WI server is a main
control program for managing all system operations. It is written in Python language
 A Web-Based Artwork Editing System Empowered by Neural Style Transfer 589

Fig. 4. Software configuration of the proposed system.

and consists of the following three modules: User Management (UM), Image Editing
Management (IEM), and Data Transfer Management (DTM). The UM module is
responsible for registering and authenticating user information. The IEM module is
responsible for registering and presenting image data, setting parameters and start time
for the neural style transfer jobs, and activating the jobs. The DTM manages all
communications with the MT module for submitting the registered jobs and receiving
generated images.
Figure 5 shows the entity relationship diagram of the system database managed by
the WSB process. We adopted Maria database [6], an open source relational database,
for maintaining three entities such as user, input data, and output data. The WSB
process manages the database via ORM (Object-Relational Mapping) that is a generic
API for providing an access method to the relational database system in object-oriented
formats. The DTM module in the WSB process uses WSGI (Web Service Gateway
Interface), a unified API for Python, enabling the DTM module to communicate with
the client (MT server) in a standardized manner.

Fig. 5. Entity relationship diagram of system database.

590 K. Goto and H. Nishino

4.3 GUI Design and Implementation

A user firstly needs to register his/her own information (name, mail address, and
password) in the registration screen shown in Fig. 6(a) before using the system. After
that, he/she can access the system from a mobile device anywhere he/she has a network
connection. Figure 6(b) shows the login screen of the system. The UM module in the
WSD process deals with these user management tasks.

(a) User registration window (b) Login window

Fig. 6. GUI for user registration and authentication.

Next, prior to the neural style transfer process, the user needs to prepare at most
four images (content, content semantic map, style, and style semantic map) and register
a neural style transfer job with these images. Figure 7(a) shows the job registration
window for this task. The IEM module in the WSD process sets the job start time of the
registered job indicating when the job will be submitted to and started on the MT
server. The user can check the registered job’s detail information on the job state
monitoring window as shown in Fig. 7(b). On this window, all the registered jobs are
shown in the registered order and each job is presented on each line.
The user can move to the parameter adjustment screen for controlling the output of
the neural style transfer job by pushing the rightmost button indicated in the upper right
part in Fig. 7. To avoid complicated tuning especially for novices, the parameters for
adjusting the neural style transfer job are the power of influence of the style and content
images applied for blending these two images. When a job is submitted to the MT
server and the images generated in the job are returned, a button linking to the output
image browsing window appears at the right end in the job state monitoring window.
When the user presses the button, the browsing window is displayed and the list of all
the generated images are listed as shown in Fig. 7(c). The default number of maximum
iterations for exploring the optimum blended image in each neural style transfer job is
1000. The system outputs one image every 100 times of iterative processing, so it
displays ten generated images in total for each job in the output image browsing
window. Since this upper limit depends on the content (complexity) and resolution of
the images to process, the user can change the number as necessary.
 A Web-Based Artwork Editing System Empowered by Neural Style Transfer 591

Fig. 7. GUI for job registration, job state monitoring, and output image browsing.

5 Preliminary Experiments

We conducted some experiments for verifying the usefulness of the proposed method
by generating some images. We prepared four different style images (water splash,
sunset, flower, and ukiyo-e) as shown in Fig. 8. All the images are downloaded from
free image archive sites on the Internet. The four single style pair sets shown in Fig. 8
(a) are used to generate images in the same way as the mechanism shown in Fig. 2. We
also prepared the two dual style pair sets shown in Fig. 8(b) for generating finer output
images than the case with the single pair sets.
Figure 9 illustrates an image generation example using a dual style image, the
splash and sunset. In this case, we need to prepare the style semantic map with two
colors representing the physical layout of two style images, and the content image
colored by the same two colors of the style semantic map. In this case, the sunset image
features corresponding to the red part of the semantic map are applied to the content
image’s background which has the same red color. The splash image features are
applied to the blue characters in the content image. Finally, the system produces a finer
image with different styles of its background and characters. The user can easily
exchange the features applied to the background and characters by simply inverting the
colors of the style semantic map. It is also possible to generate more sophisticated
images with more features applied simultaneously by using the same methodology.
592 K. Goto and H. Nishino

splash sunset flower u-kiyoe

(a) Four pairs of single style image and its semantic map

sunset+splash sunset+u-kiyoe
(b) Two pairs of dual style image and its semantic map

Fig. 8. Style images and their semantic maps used in experiments.

Fig. 9. Image generation using a pair of dual style image and content image.

Figure 10 shows example images generated by a single style image and Fig. 11
depicts other images generated by a dual style image, respectively. They are the images
generated after 1000 times iterations of the optimum image exploration in the neural
style transfer jobs. We put the style image name underneath each image used to
generate the output image. The subjects could prepare the semantic maps quite easily
by using some standard image editing tools that can automatically recognize and select
a specific object appeared in an image such as Window’s 3D paint tool.

splash flower splash flower

(a) generated images without content map (b) generated images with content map

Fig. 10. Example images generated by using single style image.

 A Web-Based Artwork Editing System Empowered by Neural Style Transfer 593

splash+sunset sunset+ukiyo-e splash+sunset sunset+ukiyo-e

(a) generated images without content map (b) generated images with content map

Fig. 11. Example images generated by using dual style image.

We, further, evaluate the system by generating the two-color logo images as shown
in Fig. 12. They are the logo images of authors’ laboratory and made by using a color
scheme design tool [7] for giving opposite impressions with cold and warm colors to
each other. We perform the neural style transfer for these two logo images by using the
single style image pairs shown in Fig. 8(a) and the content images shown in Fig. 13.
We define the cold and warm content image pairs shown in Fig. 13 for the areas to
which the style image features are applied are reversed. Figure 14 shows example
images generated by the system. Even in this case, we could create aesthetically
pleasing artistic logo images emphasizing the moods of coldness and warmth only by
creating simple map images, registering our preferred style images, and requesting the
execution.
We examined the effectiveness of the system by creating some new logo images
whose colors and texture patterns have some impacts on the impression of artwork.
Although the results are still preliminary ones, the observed results are quite promising.
We would like to continue the verification of the system’s practicality by generating
larger-scale outputs such as event posters and product catalogs in the near future.

(a) cold-colored logo (b) warm-colored logo

Fig. 12. Two original logo images giving opposite feelings to each other.

cold-A (cA) cold-B (cB) warm-A (wA) warm-B (wB)

(a) content images for cold-colored logo (b) content images for warm-colored logo

Fig. 13. Content images for producing new logos used in experiments.
594 K. Goto and H. Nishino

cA+splash cB+splash wA+ukiyo-e wA+flower wB+flower wA+sunset

Fig. 14. Example logo images generated by using single image pairs.

6 Conclusions and Future Work

We proposed a web-based image editing system enabling users to readily access the
heavily-loaded neural style transfer function performed based on a deep learning
technique. The users can perform art work anywhere by accessing the system from
their mobile device via a connection to the Internet. The system enables them to easily
creating artistically appealing outputs by selecting their preferred images and submit
the image production jobs to the system. We designed and implemented the system as a
web application and verified the effectiveness through some experiments.
In the near future, we would like to conduct more comprehensive experiments to
verify the practicality for producing more realistic targets such as event posters and
product catalogs. The deep neural network can also effectively be used for evaluating
and predicting the aesthetics of images [8, 9]. A function for evaluating the aesthetics
of generated images and recommending highly-rated ones to the user is another critical
issue as future enhancements.

References
1. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural
networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pp. 2414–2423 (2016)
2. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image
recognition. arXiv:1409.1556, pp. 1–14 (2015)
3. Champandard, A.J.: Semantic style transfer and turning two-bit doodles into fine artworks.
arXiv:1603.01768, pp. 1–7 (2016)
4. Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, pp. 6997–7005 (2016)
5. Goto, K., Nishino, H.: A method of neural style transfer for images with artistic characters. In:
Proceedings of the 12th International Conference on Complex, Intelligent and Software
Intensive Systems, pp. 911–920 (2018)
6. The MariaDB Foundation. https://mariadb.org/. Accessed Jan 2019
7. Miura, S., Nishino, H.: A practical colour scheme explorer for designers. Int. J. Space-Based
Situated Comput. 7(3), 155–165 (2017)
8. Bianco, S., Celona, L., Napoletano, P., Schettini, R.: Predicting image aesthetics with deep
learning. In: Proceedings of the 17th International Conference on Advanced Concepts for
Intelligent Vision Systems, pp. 117–125 (2016)
9. Lu, X., Lin, Z., Jin, H., Yang, J., Wang, J.Z.: Rating image aesthetics using deep learning.
IEEE Trans. Multimedia 17(11), 2021–2034 (2015)
 Mobile App and Malware Classifications
by Mobile Usage with Time Dynamics

Yong Zheng(B) and Sridhar Srinivasan

Illinois Institute of Technology, Chicago, USA

[email protected], [email protected]

Abstract. Smartphones have become a popular target for cyberattacks.

Malware can be embedded into the mobile applications. Several tech-
niques have been proposed to alleviate these problems. However, these
solutions may perform experiments by using simulated data, or may
require root system privileges, or did not take advantage of the dis-
covered patterns to build more effective malware detection methods. In
this paper, we use the SherLock data which is a labeled smartphone
dataset that captures ongoing attacks within the low-privileged moni-
torable features. We analyze the usage behaviors, discover temporal and
usage patterns, and further examine multiple classification techniques
to predict the type and the running state (i.e., benign and malicious)
of the mobile apps by using different combinations of feature sets. Our
experiments identified the best feature sets and methods to detect mal-
wares, and we demonstrate the usefulness of temporal information in the
predictive analysis.

1 Introduction
The popularity of smartphones has made them a target for malware, especially
for the Android phones. Android users may not always carefully review or under-
stand the app permissions during installation [5], which enables trojans to be
installed and gain access to sensitive resources without exploiting vulnerabilities.
And even the users may acquire root privileges, which makes the rooted devices
be exposed to more vulnerabilities. Malware [8] can be embedded into the mobile
applications or devices where they can further consume mobile resources, drain
the battery, steal our information, and so forth. For example, an app may present
itself as a messaging application that sends SMS messages when the users click
the “send” button. But actually it may steal the contact information and send
out over the mobile networks. A spyware embedded into a Web browser app
may spy on location, Web history and traffic information, and even place Ads,
popups or banners in the system.
Researchers have developed several solutions to detect malware in the mobile
devices. One of these solutions may perform dynamic analysis on the device
itself [2,4], however these solutions may require root privileges or make changes
to the kernel codes. It may cause security architectures to lose their effectiveness.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 595–606, 2020.
https://doi.org/10.1007/978-3-030-15032-7_50
596 Y. Zheng and S. Srinivasan

Another type of solutions may take advantage of network usage (e.g., data sent
out or received over network), but they may also rely on specific database (such
as “mobile ping”) which may store a set of signatures [15]. And some others
may utilize classification techniques to detect malware [1,11], however the labels
(such as benign and malicious) may not be always available and they have to
use simulated data sets.
To alleviate these issues, we perform malware analysis, app classifications and
malware detections by using the SherLock data [10] which is a labeled smart-
phone dataset that captures ongoing attacks within the low-privileged moni-
torable features. They acquire a wide variety of low-privileged monitorable fea-
tures (such as CPU, memory, battery, WIFI, etc.) at a high sample rate, and
assign one agent to create labeled data which can tell that an app is running in
benign or malicious states. We first perform exploratory analysis on the usage
data, where we identify interesting usage and temporal patterns. We further
perform multi-class and multi-label classifications to predict the type and the
running state (i.e., benign or malicious) of the mobile apps by using different
combinations of usage and temporal features. We figure out the best feature sets
and methods for malware detections finally, and they are consistent with the
patterns we found in the exploratory analysis.

2 Data and Problem Statement

The SherLock data [10]1 is a labeled smartphone dataset that captures ongoing
attacks within the low-privileged monitorable features. They provided Samsung
Galaxy S5 smartphone to 50 volunteers for the purpose of data collections. Dif-
ferent mobile apps were installed on the phone, and 12 of these apps can simu-
late malicious behaviors. Namely, these apps can randomly run in normal (i.e.,
benign) and abnormal (i.e., malicious) states. The system deploy two agents –
SherLock and Moriarity. The SherLock is used to collect low-privileged moni-
torable features at a high sample rate, including CPU, memory, battery, network
usages, and so forth. The Moriarty agent perpetrates attacks on the user’s device
while creating labels for the SherLock dataset. As a result, each record of mobile
usage is associated with a timestamp and the running states (either benign
or malicious). The whole dataset currently contains around 10 billion records,
including different types of files, such as usage, logs, events, descriptions, etc.
In this work, we perform malware analysis and detections [7], as well as mobile
app classifications. The malware analysis is used to exploit usage patterns and
the relations between the mobile usage and the type and/or the running states of
the apps. The patterns discovered from the malware analysis will be reused for
malware detection which is actually a process of classifications. Particularly, we
have two predictive tasks – predicting the type of the apps and the running states
by given different combinations of the features. We would like to examine which
feature sets are the most useful ones for predictions, and whether the mobile usage
and/or the temporal features are helpful in these classifications or detections.
1
http://bigdata.ise.bgu.ac.il/sherlock/.
 Mobile App and Malware Classifications by Mobile Usage 597

3 Usage and Temporal Analysis

There are two major analytical work we are interested in. The first one is usage
analysis [19] in which we perform statistical tests to learn whether the mobile
usages (i.e., the low-privileges features, such as CPU, memory, battery usages)
are significantly different among different apps, and whether they are different
between the benign and malicious running state of each app. Furthermore, we try
to correlate these patterns with the malware behavior (such as stealing informa-
tion, photo theft, click-jacking, etc.), and observe whether they are consistent. In
addition, we exploit the temporal analysis, where we are particularly interested
in how users use these apps over time, and whether there are different usage pat-
terns among the apps. The malicious behaviors will be randomly simulated on
the apps. It is higher possible to observe malicious activities if users use an app
more frequently in a specific time period. These time dynamics may be helpful
in the predictive tasks (i.e., app classification and malware detections) we are
going to perform in our work.

3.1 Data Sampling

There are 12 apps which simulate malicious activities, where we select 8 of
them (i.e., Puzzle Game, Web Browser, Utilization Widget, Sports App, Game1,
Game2, Music Player, Media Player ) in our work, since the data related to other
4 apps is very limited. These mobile apps were released in different time, which
results in different amount of the usage data associated with each app. It is also
worth mentioning that the Moriarity agent records the sessions and also creates
labels for them. A session is defined as a period that an app is running in a
specific state, and each session is labeled with either “benign” or “malicious”.
The duration of a session could be very short (e.g, less than 1 min), or long (e.g.,
multiple hours). We sample the session data to obtain the average mobile usage
per app per running state. We remove the sessions which are too short (i.e., less
than 1 min). Finally, our sample data contains 150,753 records (each record is
associated with the app’s usage in CPU, memory, battery and networks) which
are involved with 932 sessions (around 928 h in total) associated with the 8 apps
that we selected. The sampling process tried to balance the records per app and
per running states (i.e., benign and malicious) in order to alleviate the imbalance
issue in the classification tasks.

3.2 Usage Analysis

In terms of the mobile usages, we are interested in CPU (usage in percentage),
memory (in Kibibytes), battery (the battery temperature in tenths of a degree
Centigrade) and network usage (data transferred in bytes). These information
were collected for each app at a high sample rate. For example, the CPU, memory
and battery usage will be collected every 5 s, while network usage was collected
every one minute. Our sampled data make sure that we have CPU, memory,
battery and network usage by given a timestamp.
598 Y. Zheng and S. Srinivasan

CPU Memory Battery Network

Puzzle Game

6e+07
Web Browser

400
80
Utilization Widget

3000
Sports App
60

4e+07
Game1

300
Game2
40

Music Player

0 1000
Media Player
20

2e+07

200
0

Fig. 1. Comparison of mobile usage among apps (in Benign state)

First of all, we compare the mobile usage among different apps when they are
running in the benign state, as shown by Fig. 1. It is used to examine whether
the mobile apps have significant and different usages when they are running
normally. We can observe that the two game apps, “Game1” and “Game2”,
may consume more CPU, memory and battery, while the puzzle game may not
cost that much. It may be because that these two games are large-scale and
more complicated games, while the puzzle game is a local and easier one. The
sports app uses less memory than others significantly, and the variances of usage
in battery and network are quite small by the media player. These observations
infer that the differences in the mobile usage may be helpful for us to distinguish
and classifier different mobile apps.

Table 1. Comparison of mobile usage between benign and malicious states

Mobile apps # of sessions [mins] More usage in Malicious behaviour

malicious state
Puzzle Game 196 [5,212] Battery, Network Steals and send out contact information
Web Browser 87 [4,904] CPU, Memory Spiecs location and audio
Utilization Widget 106 [16,082] CPU Steals photos
Sports App 18 [1,289] Memory, Network Captures & reports on SMSs with verification
codes
Game1 231 [9,205] Network Gathers private information & places
shortcuts notifications, and attempts to install
new apps
Game2 266 [2,251] Network Gathers information, places Ads, popups &
banners
Music Player 87 [13,760] CPU, Battery Performs DDOS attacks and SMS botnet
activities
Media Player 20 [2,978] CPU, Memory, Maps the connected local network and
Battery searches for files and vulnerabilities

Moreover, we explore whether there is a significant difference in the mobile

usage for each app, when the app is running in benign and malicious states.
We examine the significance by using a two-sample hypothesis testing at 95%
confidence level. Table 1 presents the results. In the table, we present the number
of sessions associated with the total number of minutes in our sampled data for
each app. The hypothesis testing identifies the significant more usage when an
 Mobile App and Malware Classifications by Mobile Usage 599

app is running in the malicious state. We also list the malicious behaviors of
each app as the last column in the table in order to find the correlations.
We can observe that the Web browser, utilization widget, and two players
may consume more CPU in malicious, since their malicious behaviors are asso-
ciated with either large-file operations (e.g., stealing photos or spying audio) or
complicated task (e.g., DDOS attacks). The music and media players may also
need to simulate multiple and longer-time, complicated tasks, which may drain
the battery significantly. Most of the malicious behaviors may utilize more net-
work in comparison of the network usage when they are running normally, unless
the mobile apps will also consume a lot of network (e.g., network connections,
data transfer) in benign states. For example, the Web browser and utilization
widget may have frequent connections with network, which results in their usage
in network is not significantly higher in malicious state than in the benign state.
These observations imply that it may be useful to utilize the mobile usage for
malware detections which can tell an mobile app is running in either the benign
or the malicious state.

3.3 Temporal Analysis

Each app may be installed on different devices, and therefore a same app may be
used by multiple users. We are particularly interested in the usage frequency of
the apps over different temporal levels. Tang et al. [17] performed similar analy-
sis and they found that some apps tend to be opened at specific times on a daily
basis. In our work, we use the usage probability to discover the temporal pat-
terns. Take the day of the week for example, this usage probability for Monday
refers to the percentage of hours the users may use a specific app on Monday. The

By Day of the Week By Weekend/Weekday

0.25 0.9
0.8
0.2 0.7
0.6
0.15
0.5
0.4
0.1
0.3

0.05 0.2
0.1
0 0
Mon Tue Wed Thu Fri Sat Sun Weekday Weekend

Puzzle Game Web Browser UƟlizaƟon Widget Game1 Puzzle Game Web Browser UƟlizaƟon Widget Game1

By Time Segment By Hours In a Day

0.4 0.16
0.35 0.14
0.3 0.12
0.25 0.1
0.2 0.08
0.15 0.06
0.1 0.04
0.05 0.02
0 0
Morning AŌernoon Evening Night 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Puzzle Game Web Browser UƟlizaƟon Widget Game1 Puzzle Game Web Browser UƟlizaƟon Widget Game1

Fig. 2. Usage analysis by temporal levels

600 Y. Zheng and S. Srinivasan

usage probabilities can be calculated for other time levels accordingly. Note that
these probabilities can also be computed based on the percentage in terms of the
sessions. However, the duration of the sessions may vary, which results in mis-
leading results. We only present the usage probabilities based on the percentage
of hours used by each app.
The patterns by using four apps as case studies can be depicted from Fig. 2.
We investigate four temporal levels – day of thee week, weekday or weekend,
hours in a day and a special time segment. More specifically, there are four
segments we defined – morning (6 AM to 12 PM), afternoon (12 PM to 5 PM),
evening (5 PM to 9 PM) and night (9 PM to 6 AM). We can observe that
users may heavily use the apps in a week except Friday and Saturday, while the
“Game1” app may still be frequently used on Friday which is different from other
apps. There are no clear different patterns in the level of weekend and weekday
among these apps. We do find that the puzzle game may be used in different
time segments in comparison with others. For example, it is less used on the
morning but used more often at evening, which presents different patterns than
other apps on the morning and evening. The pattern can be further confirmed
in the hour-based plot. The different usage patterns can be useful for use to
predict the type of the apps by given the usage information. In addition, the
malicious activities may happen more frequently if the users are using an app in
specific time period. Potentially the temporal information may also be valuable
to predict the running states, such as benign or malicious.

4 App Classifications and Malware Detections

We have explicit labels associated with each running record, which enables us
to transform the task of malware detection [7] to a classification process. More
specifically, we have three predictive tasks: one is a multi-class app classification,
in which we predict the type of the apps by given a set of features, such as mobile
usage. Another one is a binary classification (i.e., malware detection) in which
we predict the running state of the apps (i.e., benign or malicious). The last one
is the multi-label classification [18] in which we use the feature sets to predict the
values in two labels (i.e., app and the running state). The purpose of applying
multi-label classification is to examine whether the correlation between the two
labels can be helpful to improve the predictions, since there are some multi-
label classification methods that can take the correlation between labels into
consideration. In this section, we introduce the feature sets we build, evaluation
settings and the experimental results.

4.1 Feature Sets

Our major goals is detecting malware by classifications based on the low-privilege

monitorable features which refer to the usage information in our data. In addi-
tion, we have the time stamp associated with the usage data, and we find out that
there are temporal effect in the usage patterns based on our temporal analysis
 Mobile App and Malware Classifications by Mobile Usage 601

Table 2. Individual usage as features Table 3. Usage + Time as features

CPU Battery Memory Network App State Day Weekend Segment CPU ...
53.37 372 25964786 0 Game1 Malware TUE Weekday Morning 53.37 ...
3.14 304 31631870 172 Widget Benign THU Weekday Evening 3.14 ...
27.83 308 54643848 0 Game1 Malware WED Weekday Night 27.83 ...
32.68 366 52759824 1185 Browser Malware SAT Weekend Night 32.68 ...
42.76 385 27837368 56 Game2 Benign MON Weekday Night 42.76 ...

in the Sect. 3.3. Therefore, our feature sets are basically different combinations
of these usage and temporal features which can be introduced as follows.
(1) Individual Usage. Each record associated with the usage will be consid-
ered as the features. An Example can be shown as Table 2, where the “App”
and “State” are the two labels in the classifications. There are a total of 150,753
records.
(2) Individual Usage + Time. We can add the temporal features as the new
feature set, as shown by Table 3. We decompose the time stamp into different
temporal features, including day of the week, whether it is weekend, and different
time segments, as mentioned in the previous section. There are a total of 150,753
records too. We did not use higher level of temporal features, such as year and
month. It is because our analyses reveal that some apps were used in a specific
year and month and these two features may overwhelm the contributions by
other ones and result in overestimated classification accuracies.
(3) Usage Statistics by Blocks. By using individual usage records as the
features, we are going to make predictions by given the usage in a specific date
and time. However, the prediction accuracy may be decreased if there is a large
variance in the usage during a day. In this case, we propose to use usage statistics
instead of the individual usage itself. More specifically, we sort the usage records
per app per running state based on the time stamp, and then we equally decom-
pose the data into N blocks. Also, we make sure that the data in each block
must be the usage records in a same day. For each block, we will calculate the
descriptive statistics of each mobile usage, and use them as the features. These
descriptive statics refer to the minimum, maximum, mean value, 1st quartile, 2nd
quartile, 3rd quartile, and variance of each mobile usage. We vary the block size
from 25 to 125 with an increment of 25 in each step, and finally we find out that
100 is the best choice in our data. There are 1,914 records/lines if we use 100 as
the block size in the experiment. Note that, the size of some blocks is close to
100 instead of exactly 100, since we need to make sure each block is associated
with a specific app running in a specific state (i.e., benign or malicious) so that
we can still assign the two labels “App” and “State”. An example of the data
can be shown by Table 4.
602 Y. Zheng and S. Srinivasan

Table 4. Block statistics as features

CPU Min CPU Max CPU Mean CPU q1 CPU q2 CPU q3 CPU var Battery Min ... App State
12.69 94.15 44.19 27.74 43.94 58.57 19.87 394.93 ... Browser Malware
0.00 95.49 18.44 1.56 11.09 27.09 21.05 443.13 ... Widget Malware
0.00 85.27 25.58 2.90 19.41 45.26 24.71 610.47 ... Sports Benign
0.00 81.32 23.07 4.89 17.45 35.71 21.32 454.64 ... Game1 Malware
0.00 95.05 19.26 3.77 13.01 29.09 19.51 380.71 ... Game1 Benign
2.08 98.04 42.42 19.95 39.62 62.58 25.48 648.98 ... Game2 Malware

(4) Usage Statistics by Blocks + Time. Accordingly, we can use temporal

features in addition to the block statistics. Note that, there may be different
timestamps in a same block, but we make sure they are the records in a same
day when we create the blocks. Therefore, we are able to use the temporal
features including the day of the week, and whether it is weekend.
(5) Usage Statistics by Sessions. Another way to compute the usage statis-
tics is performing the calculations based on the pre-defined sessions, instead of
creating blocks by ourselves. In most cases, each session is a set of time periods in
a same day. Sometimes, we may have an overnight session, while we just ignore
the usage information after 11:59 PM to make sure the records in a session are
associated with a same day. We compute the same descriptive statistics based
on the session data, and use them as features. There are 932 sessions in total in
the sampled data.
(6) Usage Statistics by Sessions + Time. Similarly, we can add the tempo-
ral information as the new features to the usage statistics based on the sessions.

4.2 Evaluation Protocols

We apply both classical and modern classification methods in the multi-class

(i.e., mobile app classification), binary (i.e., malware detections) and multi-
label classifications. More specifically, we first apply Naive Bayes, decision trees
and random forest. In addition, we try the traditional artificial neural networks
(ANN) and the recurrent neural networks with long short-term memory (LSTM)
which specifically deals with data in a sequence. Both ANN and LSTM have been
applied to malware classification recently [9,12] and they may perform better
than the traditional classification approaches. n terms of the multi-label classifi-
cation method, we apply classifier chains (CC) [13] and conditional dependency
networks (CDN) [6] which are the multi-label methods that consider the corre-
lation among labels. We would like to examine whether the correlation between
the two labels (i.e., App and State) can help achieve better predictive perfor-
mance. For each classification algorithm, we try to tune up different parameters
in order to obtain the best performing results. We utilize the Keras library2 for
the experiments based on ANN and LSTM. We use a 5-fold cross validation for
2
https://keras.io/.
 Mobile App and Malware Classifications by Mobile Usage 603

the evaluation purpose when we use different feature sets for the classification
purpose. We use precision, recall and accuracy as the evaluation metrics. Due
to that fact that they present similar patterns, we just discuss our results based
on accuracy in the following sections.

4.3 Experimental Results

The results of accuracy for the app prediction and malware detection are shown
in Fig. 3. And the findings can be summarized as follows.

App ClassificaƟon Malware DetecƟon

1 1.000
0.9 0.900
0.8 0.800
0.7 0.700
0.6 0.600
0.5 0.500
0.4 0.400
0.3 0.300
0.2 0.200
0.1 0.100
0 0.000
Naïve Bayes Decision Trees Random Forest ANN LSTM Naïve Bayes Decision Trees Random Forest ANN LSTM

Individual Usage Usage+Time Block StaƟsƟcs Individual Usage Usage+Time Block StaƟsƟcs
Block StaƟsƟcs+Time Session StaƟsƟcs Session StaƟsƟcs+Time Block StaƟsƟcs+Time Session StaƟsƟcs Session StaƟsƟcs+Time

Fig. 3. Comparison of accuracy for app prediction and malware detection

First of all, the Random forest technique becomes the best performing
method in both app classification and malware detection, even if the results
between random forest and ANN/LSTM are close, especially when we use the
usage statistics by the blocks as the features. Among the approaches using neu-
ral networks, ANN performs better than LSTM. LSTM only achieves acceptable
results when we use the block statistics as the features. It makes sense that
LSTM may perform worse in the app classifications, since the label “App” may
not have sequential effects. However, LSTM is still underperformed in the mal-
ware detection. It may be because the mobile apps would randomly simulate the
malicious behaviors during a day.
Recall that we have six feature sets. Based on the classification results, we
can observe that the usage statistics by the blocks become the best feature sets in
order to obtain higher classification accuracy. Individual usage data is the worse
one, which make sense since it may be difficult to make predictions on such a
finer-grained level. The accuracy can be generally improved if we collect the usage
statistics. However, the results based on the usage statistics by the sessions are
underperformed in comparison with the results using block statistics. However,
the correct block size should be identified. Figure 4 presents the classification
accuracy by using decision tree and random forest, while we try different block
size. Our experiments identify that 100 is the optimal size to divide the sampled
data into blocks.
Recall that we performed basic temporal analysis of the data and observe that
users may prefer to use different apps in specific time period. Our assumption
604 Y. Zheng and S. Srinivasan

ClassificaƟon By Decision Trees ClassificaƟon By Random Forest

0.92 0.96
0.9 0.94
0.88
0.92
0.86
0.84 0.9
0.82 0.88
0.8
0.86
0.78
0.76 0.84

0.74 0.82
App ClassificaƟon Malware DetecƟons App ClassificaƟon Malware DetecƟons

25 50 75 100 25 50 75 100

Fig. 4. Classification accuracy by using block statistics with different block size

is that the information of time dynamics could be useful to classify the mobile
apps and even the running states. Based on the results in Fig. 3, we can observe
that by adding the temporal information as additional features, the classification
performance can be generally improved. Take the feature set of individual usage
for example, the classification accuracy is significant increased once we addition-
ally utilize the temporal features. The same pattern can be identified when we
use usage statistics by the blocks. This pattern is not significant when we use
session statistics and apply the ANN and LSTM as the predictive models. From
an overall perspective, the temporal features are helpful in the app classification
and malware detections.
Furthermore, we also perform multi-label classification in which we consider
app and the running state as two labels in the task. We apply classifier chains
(CC) [13] and conditional dependency networks (CDN) [6] which are the multi-
label methods that consider the correlation among labels. We would like to
examine whether multi-label classifications which consider label correlations can
help improve the app classification and malware detections.

Table 5. Multi-label classification accuracy by using random forest

Usage Usage Block Block statistics Session Session statistics

+ Time statistics + Time statistics + Time
App Multi-class 0.366 0.74 0.938 0.95 0.655 0.786
classification
Classifier 0.349 0.704 0.933 0.949 0.668 0.786
chains
Malware Binary-class 0.609 0.816 0.945 0.958 0.761 0.758
detection
Classifier 0.609 0.816 0.936 0.957 0.763 0.767
chains

Table 5 presents the experimental results by using random forest as classifier

over the six feature sets. In terms of the multi-label classification, we only show
the results by classifier chains since it outperforms the CDN method. We compare
the classification accuracy for each label by using classifier chains and the original
independent classifications (i.e., a process of multi-class classification for app
classification, and a process of binary classification for malware detections). It
 Mobile App and Malware Classifications by Mobile Usage 605

seems that classifier chains can only provide slight improvement when we use
session statistics as feature sets. It implies that the correlation between the two
labels (i.e., App and Running states) is not that helpful in the predictive models.

5 Related Work
Usage analysis and malware detection are popular research methods in mobile
computing. Tang et al. [17] performed similar analysis and they found that some
apps tend to be opened at specific times on a daily basis, and each app may have
its own life or usage cycle. Chang et al. [3] utilized the radar chart to visualize the
resource consumptions in order to observe the usage patterns. By contrast, Shin
et al. [16] made their attempts to predict mobile usages. Usage information or
patterns are also helpful in malware detection or classifications. Shamili et al. [14]
detected malware on mobile devices based on high-level usage, such as calls and
SMSs, as well as data communication logs. Alternatively, Burguera et al. [2] took
advantage of the interactions between malware and system calls (such as read,
write, open, chmod, chown operations) for malware detections. There are limited
work which utilize the low-privilege and monitorable features for the detection
purpose.

6 Conclusions and Future Work

In this work, we take advantage of the Sherlock data, perform usage and tem-
poral analysis, and further figure out the best feature sets and the classification
methods to predict the type of the apps and the running state. Our experimental
results reveal that the temporal features are useful to improve the classification
performance significantly. The most effective feature set is the usage statistics
based on blocks but the size of the block should be appropriately selected. Ran-
dom forest is identified as the best classifier in our experiments. LSTM failed
to outperform the random forest, which may infer that there are no significant
sequential patterns in the running states. And the multi-label classification meth-
ods which consider the correlation among labels failed to offer more improve-
ments on the classification results. In our future work, we will try to utilize the
full large-scale data set instead of performing experiments by using a sampled
data. In addition, we will exploit other ways to take advantage of the temporal
features to further improve the predictive performance.

References
1. Amos, B., Turner, H., White, J.: Applying machine learning classifiers to dynamic
Android malware detection at scale. In: 2013 9th International Wireless Communi-
cations and Mobile Computing Conference, IWCMC, pp. 1666–1671. IEEE (2013)
2. Burguera, I., Zurutuza, U., Nadjm-Tehrani, S.: Crowdroid: behavior-based mal-
ware detection system for Android. In: Proceedings of the 1st ACM Workshop on
Security and Privacy in Smartphones and Mobile Devices, pp. 15–26. ACM (2011)
606 Y. Zheng and S. Srinivasan

3. Chang, S.-W., Cheng, S.-W., Hsiu, P.-C., Kuo, T.-W., Lin, C.-W.: Application
behavior analysis in resource consumption for mobile devices. In: Proceedings of
the 29th Annual ACM Symposium on Applied Computing, pp. 1469–1474. ACM
(2014)
4. Enck, W., Gilbert, P., Han, S., Tendulkar, V., Chun, B.-G., Cox, L.P., Jung, J.,
McDaniel, P., Sheth, A.N.: TaintDroid: an information-flow tracking system for
realtime privacy monitoring on smartphones. ACM Trans. Comput. Syst. (TOCS)
32(2), 5 (2014)
5. Felt, A.P., Ha, E., Egelman, S., Haney, A., Chin, E., Wagner, D.: Android permis-
sions: user attention, comprehension, and behavior. In: Proceedings of the Eighth
Symposium on Usable Privacy and Security, p. 3. ACM (2012)
6. Guo, Y., Gu, S.: Multi-label classification using conditional dependency networks.
In: IJCAI Proceedings, vol. 22, p. 1300 (2011)
7. Idika, N., Mathur, A.P.: A survey of malware detection techniques, p. 48. Purdue
University (2007)
8. Jiang, X., Zhou, Y.: Dissecting Android malware: characterization and evolution.
In: 2012 IEEE Symposium on Security and Privacy, pp. 95–109. IEEE (2012)
9. Kolosnjaji, B., Zarras, A., Webster, G., Eckert, C.: Deep learning for classification
of malware system call sequences. In: Australasian Joint Conference on Artificial
Intelligence, pp. 137–149. Springer (2016)
10. Mirsky, Y., Shabtai, A., Rokach, L., Shapira, B., Elovici, Y.: Sherlock vs moriarty:
a smartphone dataset for cybersecurity research. In: Proceedings of the 2016 ACM
Workshop on Artificial Intelligence and Security, pp. 1–12. ACM (2016)
11. Narudin, F.A., Feizollah, A., Anuar, N.B., Gani, A.: Evaluation of machine learning
classifiers for mobile malware detection. Soft Comput. 20(1), 343–357 (2016)
12. Nix, R., Zhang, J.: Classification of Android apps and malware using deep neural
networks. In: International Joint Conference on Neural Networks, pp. 1871–1878.
IEEE (2017)
13. Read, J., Pfahringer, B., Holmes, G., Frank, E.: Classifier chains for multi-label
classification. In: Joint European Conference on Machine Learning and Knowledge
Discovery in Databases, pp. 254–269. Springer (2009)
14. Shamili, A.S., Bauckhage, C., Alpcan, T.: Malware detection on mobile devices
using distributed machine learning. In: 2010 20th International Conference on Pat-
tern Recognition, ICPR, pp. 4348–4351. IEEE (2010)
15. Shen, Y., Evans, N., Benameur, A.: Insights into rooted and non-rooted Android
mobile devices with behavior analytics. In: Proceedings of the 31st Annual ACM
Symposium on Applied Computing, pp. 580–587. ACM (2016)
16. Shin, C., Hong, J.-H., Dey, A.K.: Understanding and prediction of mobile appli-
cation usage for smart phones. In: Proceedings of the 2012 ACM Conference on
Ubiquitous Computing, pp. 173–182. ACM (2012)
17. Tang, L.-Y., Hsiu, P.-C., Huang, J.-L., Chen, M.-S.: iLauncher: an intelligent
launcher for mobile apps based on individual usage patterns. In: Proceedings of the
28th Annual ACM Symposium on Applied Computing, pp. 505–512. ACM (2013)
18. Tsoumakas, G., Katakis, I.: Multi-label classification: an overview. Int. J. Data
Warehous. Min. (IJDWM) 3(3), 1–13 (2007)
19. Zheng, Y., Srinivasan, S., Taehun, K.: Exploratory malware analysis of mobile
usages. In: Proceedings of the 19th Annual SIG Conference on Information Tech-
nology Education, pp. 158–158. ACM (2018)
 Improvement of Self Position Estimation
of Electric Wheelchair Combining Multiple
Positioning Methods

Fumiai Sato(&)

Department of Information Science, Toho University,

Miyama 2-2-2, Funabashi, Chiba 274-8510, Japan
[email protected]

Abstract. A self-driving electric wheelchair must estimate its own position,

identify the travelable area, and determine the travel route. For indoor self-
position estimation, the location can be estimated by wireless LAN or beacon.
However, for autonomous operation, more accurate position estimation is
required. Conventionally, we have used augmented reality (AR) markers with
high positioning accuracy as a method for self-position estimation of a wheel-
chair in a relatively narrow space. However, the positioning error of the angle
when the AR marker was recognized from a frontal direction was large, and its
improvement was a problem. In this research, we propose a method to correct
the errors of positions obtained with AR markers using wheelchair odometry
information and object detection. Since estimates using AR markers and
odometry information both have errors, we propose a method for correcting the
AR marker positioning by odometry information and distance information
obtained by object detection. Preliminary experiments showed that using
odometry for correction improves the positioning error and allows for stable
control of the wheelchair.

1 Introduction

Electric wheelchairs are widely used for supporting the daily life of the elderly and
disabled. Experiments for self-driving electric wheelchairs are conducted as a means to
allow elderly and physically impaired people to travel to destinations within public
facilities, such as airports. In these experiments, the wheelchair estimated its own
position using a beacon [1] and detected obstacles using light detection and ranging
(LIDAR) [2]. However, the position estimation accuracy using a beacon is low (one to
several meters), and the system configuration using LIDAR is relatively expensive. In
the field of wheelchair self-driving, there is research using an omni-directional camera
and four laser range scanners [3]. However, such a system has high energy con-
sumption and becomes a burden to a battery-powered wheelchair.
To overcome these limitations, we have developed self-positioning and travelable
area detection based on monocular camera images. In the system used in our study, as
an electric wheelchair autonomously travels through an area, it estimates its own
position and direction using deep learning [4]. This system operates without using a

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 607–618, 2020.
https://doi.org/10.1007/978-3-030-15032-7_51
608 F. Sato

special camera to obtain an input image so as to make it as simple as possible. Instead,

the system uses the screen image obtained by a smartphone camera. Near walls, where
the travelable area does not appear on the screen image, augmented reality
(AR) markers are used to estimate the position. In addition, this system has indoor map
information and can calculate the route from the initial location to a destination.
A rough estimation of the current location is obtained using Bluetooth low-energy
(BLE) beacons. We developed a prototype of this system and confirmed that it can
autonomously travel from a given location to a destination [5].
Our system is applicable in a relatively narrow space, such as a toilet. We used AR
markers with high positioning accuracy as a method of self-position estimation by a
wheelchair in a relatively narrow space. However, the positioning error of the angle
when the AR marker was recognized from the front was large, and its improvement
was a problem [6–8]. In this research, we propose a method to correct the error of
positions obtained with markers by self-position estimation using wheelchair odometry
information. Since positions obtained by AR markers and odometry information both
have errors, the method to correct the AR marker-derived position by odometry
information is based on the distance to the target object obtained by an object detection
system. Preliminary experiments showed that by using odometry for correction, the
positioning error was improved and the wheelchair could be controlled stably.
The composition of this paper is as follows. In Sect. 2, we provide an overview of
the AR marker and how it might be enhanced. In Sect. 3, we describe the configuration
of the proposed system and its features. In Sect. 4, we describe the evaluation results
for self-positioning and correction with odometry information. In Sect. 5, we describe
conclusions and future work.

2 Positioning and Precision Problem with AR Marker

An AR marker-based positioning system recognizes AR markers using the camera in a

user’s device and constructs a three-dimensional coordinate system relative to the
marker (Fig. 1). The system uses the coordinates to represent virtual objects in physical
space. The ARToolKit [9] is a typical library used to implement these systems, which
both detect the markers and calculate the user’s position and orientation. Thus, the
system can display a directional indicator on the user’s device, which is difficult to
accomplish in conventional indoor positioning methods.
When estimating its own position using a marker, it is important to correctly detect
the change in the two-dimensional appearance of the marker in the captured image.
This is because detection errors for two-dimensional features, such as marker edges and
corners, may greatly affect the estimation accuracy, depending on the position and
orientation of the camera with respect to the marker plane. In particular, when tracking
the direction of the axis (Z axis) perpendicular to the marker plane (XY plane), it is
known that the angle of the projection axis of the camera has great influence with
respect to the marker plane. As shown in Fig. 2(a), when the camera is capturing an
image obliquely with respect to the marker, it is possible to correctly estimate the
parameters representing the position and attitude of the camera, and the three-
dimensional coordinate system on the marker can be identified correctly [8]. However,
 Improvement of Self Position Estimation 609

Fig. 1. AR marker recognition and three-dimensional coordination.

(a) An image sequence captured from a direction oblique to the marker plane.

(b) An image sequence in which the projection axis of the camera is substantially
perpendicular to the marker plane.

Fig. 2. Example images on tracking 3D coordinate system [8].

610 F. Sato

as shown in Fig. 2(b), when the projection axis of the camera is close to vertical with
respect to the marker plane, in spite of not moving the camera at all, tracking of the axis
may change greatly. The reason is that, when the camera is shooting at such an angle, a
slight error in the detection of two-dimensional features, such as the marker vertex and
edge, will have large influences on the calculation of the parameters representing the
three-dimensional position.
To solve this problem, a method for predicting the destination of the camera from
its previous position change and reducing the error has been proposed [8]. Methods for
extending the marker itself were also proposed. One of these methods is to add four
dots around the marker and reducing the error by detecting the dots [6]. In addition,
LentiMark has proposed a moiré pattern to the marker to accurately detect the camera
angle [7].
This research aims at estimating the position of an electric wheelchair, and we can
use its odometry information. Therefore, we propose an error mitigation method that
combines odometry information with AR imaging information.

3 Configuration of Proposed System

3.1 Proposed System Configuration

The configuration of the electric wheelchair system proposed in this study is shown in
Figs. 3 and 4. A smartphone application detects the AR markers and estimates the self-
position. A Jetson TX2 board is used for object detection based on deep learning and
for calculation of the difference between the route and the self-position. To reduce the
difference and drive a wheelchair along the route, the control unit calculates the left and
right motor rotation speeds and sends the command to the left and right motor drivers.
Simultaneously, the smartphone can estimate its position by receiving radio waves
from a BLE beacon [10], which is part of the external infrastructure. BLE signals are
used for rough location estimation over a wide area.

Fig. 3. Proposed electric wheelchair system.

 Improvement of Self Position Estimation 611

External image
(1)BLE beacon information (floor, markers)
Electric wheelchair

(2) Smartphone
(Self-position detection)
(6) Left motor

(3) Raspberry Pi
(Route information) (4) Control unit (5) Motor driver (7) Right motor

Fig. 4. Configuration of electric wheelchair self-driving system.

3.2 Control Flow from AR Marker Detection to Command Generation

Figure 5 shows the flow of AR marker detection and command generation. First, the
smartphone application detects the marker. The AR Toolkit library calculates the 3D
space and transformation matrix. The self-position is estimated from the inverse matrix
of the transformation matrix. After the estimation of the self-position, the difference
between it and the route information is calculated. After that, the control board gen-
erates a command to reduce the difference.

Route direc on

Generates the
A Difference command of the
control board to
reduce the
difference.

(a)Marker detec on (b)Es ma on of the self-posi on and

difference from route informa on.

Fig. 5. Flow of AR marker detection and command generation.

3.3 Correction of Marker-Based Positioning by Odometry and Object

Detection
Figure 6 shows the outline of the self-position estimation correction procedure using
the odometry information of the wheelchair. When the self-position is estimated only
with the AR marker, an error occurs, as indicated by the dotted line in Fig. 6. If the next
target position is determined based on the position information with the error and the
612 F. Sato

wheelchair moves accordingly, a larger error occurs. Therefore, the position after
movement from the previous self-position is estimated by odometry and integrated with
the self-position estimated from the AR marker. As a result, the self-position is cor-
rected, as shown by the solid line in Fig. 6, and the error from the actual self-position
becomes small.
However, since marker-based positioning is often highly accurate, correction by
odometry needs to be applied only when the AR marker positioning error is large. To
evaluate the reliability of AR marker-derived position, we apply object detection to the
images.
We adopted you only look once (YOLO) [11] as a model of deep learning for
object detection. Because YOLO has a fast recognition speed and is relatively accurate,
it fits the purpose of controlling a wheelchair in real time. After detecting the size of the
object with the bounding box in the object detection, the distance from the camera to
the object is calculated based on its size and position on the screen. Figure 7 is an
example of object detection by YOLO where the target object is a toilet seat.

Marker

The posi on from the self-

posi on es ma on where the
odometry is corrected to the
marker recogni on result

Self posi on
Travel posi on from self
posi on es ma on based t2 es ma on based on
marker recogni on
on marker recogni on
result.
result
t1
Actual posi on

t0 Self-loca on es ma on
by odometry correc on
Wheelchair for marker recogni on
result.

Fig. 6. Outline of the self-position correction by the wheelchair odometry information.

Figure 9 shows a flowchart of the correction of self-position by using odometry

information. The AR marker and odometry information each have an error, and the
method to correct the AR marker position with odometry information is based on the
distance to the target object determined by the object detection system. First, self-
positioning based on AR is performed. Next, self-positioning is performed based on the
 Improvement of Self Position Estimation 613

Fig. 7. Object detection by YOLO (toilet seat).

odometry information and the previous self-position. The distance to the destination is
calculated by object detection. On the basis of the distance by object detection, the
positional error due to the marker and the positional error due to the odometry are
calculated. Based on the ratio of error variance, the position based on the marker is
corrected by odometry at an appropriate rate. After the position correction, a command
to move the wheelchair is generated.
The specific calculation procedure is as follows. Let XAR and YAR be the position
information detected by the AR marker, and let XOD and YOD be the position infor-
mation estimated by odometry. DAR, DOD, and DOB are the distances to the target based
on the position information detected by the marker, based on the position information
estimated by odometry, and obtained by the object detection, respectively (Fig. 8).
Let EAR and EOD be the errors of the distances to the target.

EAR ¼ DAR  DOB

EOD ¼ DOD  DOB

Let VAR and VOD be the variances of these errors and WAR be WOD the weights for
synthesizing position information, which are calculated as follows.

WAR ¼ VOD =ðVAR þ VOD Þ

WOD ¼ VAR =ðVAR þ VOD Þ

Then, the position information based on the marker is corrected as follows.

XCORR ¼ WAR XAR þ WOD XOD

YCORR ¼ WAR YAR þ WOD YOD
614 F. Sato

AR Marker
Target object

DOD
AR marker
DAR detection

DOB (XAR, YAR)

(XOD, YOD)

Object Odometry
detection Information

Fig. 8. Object detection by YOLO (toilet seat).

START

Self-positioning based on AR markers.

Self-positioning based on odometry.

Calculation of the distance to the

destination by object detection.

Calculation of positioning errors.

Position correction by odometry.

Destination
arrival?

Traveling Wheelchair.
END

Fig. 9. Flow of the self-position correction by the wheelchair odometry information.

 Improvement of Self Position Estimation 615

4 Evaluation

To evaluate the effectiveness of the proposed correction method for self-position

estimation, the error of the self-position before and after correction was measured. The
results of accuracy of the Z axis (direction vertical to the marker) are shown in Fig. 10.
The measured error range is less than 5% of the distance. The results for the X axis
(direction horizontal to the marker) are shown in Fig. 11. The measured error ranges of
Fig. 11(a) and (b) are about 80% and 8% of the distance, respectively. From Fig. 11, it
was found that the proposed self-position correction is effective.
Next, we measured the accuracy of the final position of the electric wheelchair.
Figure 12 shows the experiment setting. We moved the wheelchair from a position 2 m
away to a position 50 cm away from the marker. Since the wheelchair is close to
vertical with respect to the marker plane, marker-only self-positioning errors are large.
Figure 13 shows the results for final position accuracy. Table 1 shows the basic sta-
tistical values of the measurement results. Figure 13 and Table 1 show that the posi-
tioning error is improved and the wheelchair can be controlled stably (Fig. 14).

Marker size 8 cm Marker size 5.6 cm

300 180
160
250
140
Es ma on (cm)
Es ma on (cm)

200 120
100
150
80
100 60
40
50
20
0 0
30 40 60 80 100 120 140 160 180 200 220 240 260 280 30 40 60 80 100 120 140 160
Distance (cm) Distance (cm)

(a) Marker size is 8 cm. (b) Marker size is 5.6 cm.

Fig. 10. Positioning accuracy by marker for Z axis.

Marker size 5.6 cm

100
80
60
Es ma on (X axis) (cm)

40
20
0
-20 0 50 100 150 200 250
-40
-60
-80
-100
-120
Distance (cm)

X axis ave. X axis min. X axis max.

(a) Without self-position correction (b) With self-position correction

by odometry. by odometry.

Fig. 11. Positioning accuracy by marker for X axis.

616 F. Sato

Fig. 12. Experiment setting for final position accuracy.

Table 1. Basic statistical values of stop position accuracy.

X average (cm) X variance Z average Z variance
Without correction −2.12 28.9 −0.65 13.7
With correction 1.19 6.3 0.85 6.1

Stop posi on of wheelchair

10

2
X (cm)

0 Without correc on
-10 -8 -6 -4 -2 0 2 4 6 8 10 With correc on
-2

-4

-6

-8

-10
Z (cm)

Fig. 13. Measurement results for final position accuracy.

 Improvement of Self Position Estimation 617

Y １7０cm(1200,6000)
G
① ②
F (1700,5400)
１６０cm ９２cm

７４cm Go Back ①
(2300,4400)

１００cm C

１１０cm

(100,3300) (1700,3200) B
③ ④ １7０cm

(1700,2200) A
EV EV ④
１３０cm
X

Fig. 14. Wheelchair toilet space for running experiment.

5 Conclusions

In this study, we developed a self-driving system for an electric wheelchair based on

position estimation by AR marker and odometry information. Self-position estimation
based on AR marker only has a large angle error when the AR marker is recognized
from the front. In this research, we proposed a method to correct the error of posi-
tioning from AR markers by using wheelchair odometry information. From our eval-
uation results, the proposed correction method is effective, and the success rate for
entering and exiting a wheelchair toilet increased.
Our future work is to integrate this system with a wide-area self-driving system for
an electric wheelchair, which was our previous work [5]. Furthermore, this system will
be integrated with an outdoor self-driving system.

Acknowledgments. Part of this work was carried out under the Cooperative Research Project
Program of the Research Institute of Electrical Communication, Tohoku University.

References
1. Shah, S.F.A., Srirangarajan, S., Tewfik, A.H.: Implementation of a directional beacon-based
position location algorithm in a signal processing framework. IEEE Trans. Wirel. Commun.
9(3), 1044–1053 (2010)
2. Yao, Y., Lou, M., Yu, P., Zhang, L.: Integration of indoor and outdoor positioning in a three-
dimension scene based on LIDAR and GPS signal. In: 2nd IEEE International Conference
on Computer and Communications (ICCC) (2016). https://doi.org/10.1109/compcomm.
2016.7925006
618 F. Sato

3. Mehl, R.: The automotive industry as a digital business. NTT Innovation Institute Inc.
(2016)
4. Pongratz, M., Mironov, K.: Accuracy of positioning spherical objects with a stereo camera
system. In: 2015 IEEE International Conference on Industrial Technology (ICIT) pp. 1608–
1612 (2015)
5. Sato, F., Koshizen, T., Matsumoto, T., Kawase, H., Miyamoto, S., Torimoto, Y.: Self-
driving system for electric wheelchair using smartphone to estimate travelable areas. In:
2018 IEEE International Conference on Systems, Man, and Cybernetics, pp. 298–304 (2018)
6. Tanaka, H., Ogata, K., Matsumoto, Y.: Improving the accuracy of visual markers by four
dots and image interpolation. In: 2016 International Symposium on Robotics and Intelligent
Sensors (IRIS2016), pp. 178–183 (2016)
7. Tanaka, H., Sumi, Y., Matsumoto, Y.: A visual marker for precise pose estimation based on
lenticular lenses. In: 2012 IEEE International Conference on Robotics and Automation,
pp. 5222–5227 (2012)
8. Uematsu, Y., Saito, H.: Camera tracking with 2D rectangular markers for augmented reality.
In: 10th Meeting on Image Recognition and Understanding (MIRU 2007), pp. 1272–1276
(2007). (in Japanese)
9. Kato, H., Billinghurst, M.: Marker tracking and HMD calibration for a video-based
augmented reality conferencing system. In: Proceedings of the 2nd IEEE and ACM
International Workshop on Augmented Reality (IWAR 1999), October 1999
10. Bluetooth SIG: Bluetooth specification version 4.0, June 2010. http://www.bluetooth.org
11. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time
object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 779–788 (2016)
 A Roadside Unit Placement Scheme
for Vehicular Ad-hoc Networks

Seif Ben Chaabene1(B) , Taoufik Yeferny1,2 , and Sadok Ben Yahia3

1
Faculty of Sciences of Tunis, LIPAH-LR11ES14, University of Tunis El Manar,
Tunis, Tunisia
[email protected], [email protected]
2
College of Science, Northern Border University, Arar, Kingdom of Saudi Arabia
3
Department of Software Science, Tallinn University of Technology,
Akadeemia tee 15a, 12618 Tallinn, Estonia
[email protected]

Abstract. Road Side Units (RSUs) have a crucial role in maintaining

Vehicular Ad-hoc Networks (VANETs) connectivity and coverage, espe-
cially, for applications gathering or disseminating non-safety information.
In big cities with complex road network topology, huge number of costly
RSUs must be deployed to collect data gathered by all moving vehicles.
In this respect, several research works focusing on RSUs deployment, the
so called coverage problem, have been proposed. The thriving challenge
would be to (i) minimize as possible the number of used RSUs, which
leads to reduce the deployment cost; and (ii) to maximize the coverage
ratio. In this paper, we introduce a new spatio-temporal coverage scheme
consisting of two main components, namely “mobility patterns mining”
and “coverage computing”. The first one computes a sequential mobility
patterns set of moving vehicles by mining their trajectories. whereas the
second computes a minimal set of junctions, where RSUs have to be ide-
ally placed. The performed thorough experiments show that our scheme
outperforms its competitors in terms of cost as well as coverage ratio.

1 Introduction
Vehicular Ad-hoc Networks (VANETs) have emerged as an interesting way
to enable Intelligent Transportation System (ITS) services. Indeed, various
ITS applications related to road safety and transport efficiency are nowadays
deployed through VANET. The main thrust of these applications stand in pro-
viding useful information to drivers, passengers, transportation departments,
to name but a few. Indeed, this network has grasped the attention of major
international automakers (e.g., BMW, MERCEDES, etc.) as well as transporta-
tion agencies. As a result, several VANET projects have been proposed for this
hot topic, such as FleetNet [6], NOW (Network on Wheels) [15], CarTALK 2000,
PReVENT, etc. In this respect, the United States Department of Transportation
(DOT) has allocated a spectrum of 75 MHz between the frequencies 5.850 GHz
and 5.925 GHz. This spectrum is split into 7 channels of 10 MHz, including
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 619–630, 2020.
https://doi.org/10.1007/978-3-030-15032-7_52
620 S. Ben Chaabene et al.

a channel called Control Channel (CCH) specifically dedicated for Intelligent

Transportation Systems (ITS). The remaining 6 channels are used for other
types of applications, such as multimedia applications [20]. The DSRC protocol
(Dedicated Short Range Communication) is currently the appropriate standard
for VANETs wireless communications. It supports vehicle-to-vehicle communi-
cations (V2V), where vehicles directly communicate without any infrastructure,
as well as vehicle to infrastructure (V2I), where vehicles communicate with the
infrastructure or Road Side Units (RSUs) installed along the roads network. In
addition, RSUs can communicate with each other in a wired backbone. Indeed,
RSUs have a crucial role in maintaining the network connectivity and coverage,
especially, for applications gathering or disseminating non-safety information [1–
3] and information sharing services [4,22,23]. During their trips in a given city,
vehicles, equipped by different sensors, can gather different kinds of non-safety
data (e.g., traffic information, weather conditions, pollution, etc.). When they
make contact with a RSU, vehicles transfer to the RSU all the gathered data. In
turn, the RSU sends the received data from vehicles to other tiers (e.g., trans-
portation agency clouds). Gathered data will then be transformed to valuable
knowledge that helps government authorities make better decisions. It is worth
noting that vehicles can also receive information (e.g., commercial announce-
ments) from RSUs. In large cities with complex road network topology, a very
important number of costly RSUs need to be deployed to collect data gathered
by all moving vehicles. In this respect, several research works focusing on RSUs
deployment, aka coverage problem, have been proposed. Two thriving challenges
need to addressed, to (i ) minimize as much as possible the number of used RSUs,
which leads to a reduction of the deployment cost; and (ii) to maximize the cover-
age ratio (i.e., number of vehicles that get in touch at least with one RSU during
their trips). A glance to the literature unveils that the existing approaches can be
categorized as spatial coverage [7,10,11]; temporal coverage [9,16,18] and spatio-
temporal coverage [5,19,21,24]. Spatio-temporal coverage approaches are most
promising since they rely on both spatial attributes and temporal characteris-
tics. In this respect, we introduce here a new spatio-temporal coverage scheme
consisting of two main components, namely “sequential patterns mining” and
“coverage computing”. The first one mines a sequential mobility pattern set of
moving vehicles from the historical of their trajectories. A mobility pattern in
our approach stand for a frequent set of crossed junctions by a certain number
of vehicles. The second component, computes a minimal set of junctions, where
RSUs will be placed, that cover the extracted sequential mobility patterns set.
The remainder of this paper is organized as follows. In Sect. 2, we review
related work in the literature. In Sect. 3, we detail the main idea of our coverage
scheme and its two components. The performance evaluation of the introduced
scheme and a comparison with other main coverage approaches is presented in
Sect. 4. Section 5 concludes this paper and pins down future directions.
 A Roadside Unit Placement Scheme for Vehicular Ad-hoc Networks 621

2 Related Work
The existing RSU placement approaches can be categorized as spatial cover-
age, temporal coverage and spatio-temporal coverage approaches. We focus here
on spatio-temporal coverage approaches since our scheme belong to this family
of approaches. Spatio-temporal coverage approaches first extract the mobility
patterns of moving vehicles and then compute the appropriate RSU locations
based on the extracted patterns. Indeed, finding the relevant mobility patterns
is a step of paramount importance. In the literature, three mobility descriptions
based on the traffic flow theory, namely microscopic description, the kinetic
description and the macroscopic description [5], have been proposed. Within
the microscopic description, all vehicles are individually identified. Position and
velocity of each vehicle define the state of the system as dependent variables
of the time variable. In the kinetic description the state of the system is still
identified by position and velocity of the vehicles. However, their identification
does not refer to each vehicle but to a suitable probability distribution. Within
the macroscopic description, the state is described by locally averaged quanti-
ties, i.e. density, mass velocity and energy, considered as dependent variables of
time and space. Some of the existing coverage approaches tried to adapt these
traffic flow model in order to depict the mobility patterns of vehicles. However,
the application of these models is very difficult. Therefore, other surveyed cov-
erage schemes mined the mobility patterns of vehicles from their trajectories.
In this respect, Xiong et al. [19] introduced a new graph model to characterize
the traffic stream. The mobility model is then detected by splitting the road
system to uniform areas (shape and surface). In addition, the authors relied
on the Markov chain theory to compute RSUs locations. In the same trend,
Zhu et al. [24] introduced a new method for detecting the forthcoming vehicu-
lar movements based on the Markov chain. In this respect, they extracted the
mobility patterns of vehicles from their log files and then predicted their future
movements. Fiore et al. [17] proposed an approach for collaborative downloading
of data between vehicles. They suggested placing RSUs at the crossing volume
zones so that vehicles play the role of relays. To this end, they introduced a
new traffic flow topology by modeling the road map as a graph, where roads
and intersections represent vertices and edges, respectively. Yeferny et al. [21]
introduced a new interesting method for depicting the mobility patterns of vehi-
cles from their trace files based on the Formal Concept Analysis (FCA) theory.
Thereafter, they computed the adequate RSU locations to cover the extracted
mobility patterns by a minimal possible number of RSUs. They formulated the
latter issue as a problem of extracting a special type of minimal transversals of
a hypergraph [12]. The proposed strategy decreased the deployment cost and
achieved good results in terms of coverage ratio, latency and overhead. How-
ever, the adapted FCA technique used for computing the mobility patterns was
based on a transactional database representing relationships between vehicles’
trajectories and crossed junctions. This representation considers the trajectory
of a vehicle as an unordered set of crossed junctions. Therefore, it ignores a key
factor which is the order of junction vehicle crossing. Hence, an extra number
622 S. Ben Chaabene et al.

of mobility patterns could be extracted leading to an increase in the number of

RSUs used to cover the mobility patterns set. To overcome this shortage, we rely
in this paper on a sequence database consisting of sequences of ordered junctions
crossed by each vehicle.

3 Proposed RSUs Placement Scheme

Our spatio-temporal coverage scheme consists of two main components:
• Sequential Mobility Patterns Mining (SMPM): it mines a sequen-
tial mobility patterns set of moving vehicles from the historical of
their trajectories. A mobility pattern in our approach stand for a
maximal frequent sequence of crossed junctions by a certain number of
vehicles;
• Coverage Computation (CovC): it computes a minimal set of junctions,
where RSUs will be placed, that cover the extracted sequential mobility pat-
tern set extracted by the SMPM component.
In the following, we thoroughly describe the aforementioned components.

3.1 SMPM Component

Mining mobility patterns from vehicle trajectories has recently received increas-
ing attention to solve some issues chiefly in non-safety VANET applications.
In this respect, we introduce here a new method for depicting vehicle mobility
patterns. Our method relies on the exploitation of the historical of vehicle tra-
jectories to predict the traffic flow or the mobility patterns in the future. To this
end, we rely on the sequential patterns theory. In what follows, we recall and
adapt some notions of sequential patterns mining.

Key Notions
Definition 1 (Sequence database). A sequence database is a set of ordered
sequences. In our case, a sequence is an ordered list of junctions crossed by a
vehicle, denoted <j1 j2 . . . jk >.
Example 1. Table 1 illustrates a sequence database D, where V =
{v0 ; v1 ; v2 ; v3 ; v4 ; v5 ; v6 } is a set of vehicle trajectories and S represent the
sequences of junctions crossed by vehicles.
Definition 2 (Sequence support). Let s be a sequence of junctions. The sup-
port of s, denoted supp(s), is an indication of how frequently the sequence s
appears in D. It is computed as follows:
|{si ∈ D|s  si }|
supp(s) =
|D|
The symbol  stands for an orderly inclusion, for example (j3 j4 ) 
(j2 j3 j5 j6 ; j4 ) but (j3 j4 ) . (j2 ; j4 j5 j6 ; j3 )
 A Roadside Unit Placement Scheme for Vehicular Ad-hoc Networks 623

Table 1. Sequence database D

V S
v0 j6 j5 j3
v1 j6 j5 j3 j7
v2 j5 j3 j7
v3 j3 j6
v4 j3 j6 j4
v5 j2 j6 j7
v6 j2 j6 j7 j1

Definition 3 (Frequent sequence). The sequence s is said to be fre-

quent if the support of s in D is greater than or equal to minsup
(user specified threshold). Formally, the set FS of frequent sequences in D is:

FS = {s ∈ D|supp(s)  minsup}

Example 2. If we consider a threshold minsup = 2/7 and the sequence database

D of the Example 1 then the sequence s = <J2 J6 J7 > is frequent since its support
supp(s) = 2/7  minsup.

Definition 4 (Maximal frequent sequence). Let FS be the set of frequent

sequences in D. We define the set MFS of maximal frequent sequences in D as:

MFS = {s ∈ FS|s ∈ FS, s  s }

Application of the Sequential Pattern Theory for Extracting Sequen-

tial Mobility Patterns of Vehicles
We introduce here our algorithm for extracting the mobility patterns set from
vehicle trajectories, called Maximal Mobility Patterns (MMP). It is worth noting
that, in the literature, a mobility pattern stands for a sequence of junctions.
The idea behind MMP is to extract the frequent mobility patterns, which
appear at least in certain number minsup of vehicle trajectories. Furthermore,
they must be maximal mobility patterns (i.e., each frequent mobility pattern
must not be included within another frequent pattern). Indeed, we look for the
longest frequent patterns. To this end, we firstly rely on the prefixSpan algo-
rithm [14] for extracting the frequent mobility patterns (i.e., frequent sequences
of junctions) from vehicle trajectories. In addition, we define the getMaximal-
Frequent() algorithm (c.f. Algorithm 1), which selects only the maximal patterns
from the frequent ones.
In Example 3, we show how we extract the maximal frequent mobility pat-
terns from a sequence database of vehicles’ trajectories.
624 S. Ben Chaabene et al.

Example 3. Considering a threshold minsup = 2/7 and the sequence database D

of the Example 1, to extract the maximal frequent mobility patterns our MMP
algorithm proceeds as follows:
1. Firstly, it invokes the prefixSpan algorithm, which will extract the following
frequent sequences or patterns set:
FS = {J2 ; J3 ; J5 ; J6 ; J7 ; J2 J6 ; J2 J7 ; J5 J3 ; J3 J6 ; J3 J7 ;
J6 J7 ; J6 J3 ; J6 J5 ; J2 J6 J7 ; J5 J3 J7 }
2. Secondly, it call our getMaximalFrequent() algorithm, which will extract from
the FS set the following maximal frequent sequences set:
MFS = {J3 J6 ; J6 J3 ; J6 J5 ; J2 J6 J7 ; J5 J3 J7 }

Algorithm 1. getMaximalFrequent()
Input: FS: set of frequent sequences.
Output: MFS: set of maximal frequent sequences.
1 foreach E of FS do
2 deleteSubSequences(E, MFS)
3 insert(E, MFS)
4 end
5 return (MFS)

3.2 CovC Component

Before describing this component, we briefly recall in the following some basic
definitions of a hypergraph and a minimal transversal.

Preliminaries
Definition 5 (Hypergraph). A hypergraph H is an ordered pair H = (J ; E),
where J = {j1 , . . . , jn } is a finite set of elements and E = {E1 , . . . , Em } is a
family of subsets of J such that:
1. Ei = ∅ (i = 1, ldots, m) and
i=1
2. i=m Ei = J .
The elements of J are called vertices, while the elements of E are called hyper-
edges of the hypergraph H [12].
Definition 6 (Transversal). Let H = (J ; E) be a hypergraph. A set T ⊆ J
is called a transversal (or, hitting set) of H if it intersects all its hyperedges,
i.e., T ∩ Ei = ∅ ∀Ei ∈ E. A transversal T is called minimal if no proper subset
T  of T is a transversal of H. Afterwards, we denote MH the set of minimal
transversals of H” [13].
 A Roadside Unit Placement Scheme for Vehicular Ad-hoc Networks 625

Coverage Computation
As mentioned previously, the goal of the CovC component is the computation of
a minimal set of junctions, where RSUs will be placed, that cover the sequential,
maximal and frequent mobility patterns set MFS extracted by the SMPM com-
ponent. Indeed, the MFS set can be represented by a hypergraph H = (J ; E),
where:

J = Si
Si ∈MF S

E = MFS
Roughly speaking the vertices and the hyperedges of the hypergraph H cor-
respond, respectively, to a set of junctions and a set of mobility patterns (i.e.,
MFS). Figure depicts a hypergraph representing the maximal frequent patterns
set of Example 3.

Fig. 1. Hypergraph representing the maximal frequent patterns set of Example 3

It is now clear that the computation of the minimal set of junctions that cover
the mobility patterns set MFS comes back to the computation of a minimal
transversal of the hypergraph H corresponding the MFS set. In our case, we
use the algorithm of Murakami et al. [12] for extracting the minimal transversals
set T rM in from the MFS set (c.f. Example 4). It is worth noting that when-
ever there is more than one minimal transversal we choose the smallest one in
cardinality terms.

Example 4. From the hypergraph of Fig. 1 representing the MFS set of Exam-
ple 3, we extract the following minimal transversal:
(1) T rM in1 = {J6 , J5 } (2) T rM in2 = {J6 , J3 }
(3) T rM in3 = {J5 , J3 } (4) T rM in4 = {J6 , J7 }
Then, we choose the one of the smallest minimal transversal in cardinality
terms, for example: T rM in1 = {J6 , J5 }
626 S. Ben Chaabene et al.

4 Performance Evaluation
In this section, we evaluate the performance of our coverage scheme and we
compare its results to that of MPC. Indeed, we have chosen MPC since it is recent
spatiotemporal coverage strategy. In addition, MPC adapted the FCA technique
for computing the mobility patterns, therefore, it ignores the junctions order.
However, in our scheme we consider that the order of junctions is of paramount
importance.

4.1 Simulation Settings

The urban mobility is simulated with microscopic and continuous road traffic
simulation using the SUMO [8]. In addition, we use OMNeT++ environment
to carry out the network simulation. The simulation settings are summarized in
Table 2.

Table 2. Simulation settings

Somme physical layer Frequency band 5.9 GHz

Transmission power 40 mW
Bandwidth 10 MHz
Link layer Bit rate 6 Mbit/s
CW 15.1023
Slot time 13 us
SIFS 32 us
Scenarios Message size 2312 Bytes
Message frequency 0.5 Hz
Slot time 13 us
#Runs 30 times

As dataset, we choose the road map of from the north of Tunisia, downloaded
from OpenStreetMaps.1 This city has a very high complex road topology.

4.2 Evaluation Metrics

To assess our scheme, we use the following metrics:

1. Coverage ratio: It assesses the percentage of coverage. The coverage ratio

for a given event message e is defined as follows:

|V  |
M essageCoverageRatio =
|V |
1
www.openstreetmap.org.
 A Roadside Unit Placement Scheme for Vehicular Ad-hoc Networks 627

• V  : set of informed vehicles.

• V : all moving vehicles.
2. Cost: We mean by cost the total number of used RSUs to cover a considered
area.
3. Latency: It refers to the amount of time which is needed to deliver a message
from a RSU to a vehicle. The average latency, AL, is defined as follows:

(ti − T )
AL =
N umberOf V ehicles
4. Overhead: It refers to the number of sent messages.

4.3 Results
Figure 2(a) shows the evolution of the deployment cost of our scheme and that
of MPC under different dimensions of the covered areas. Indeed, the deployment
cost proportionally increases to the covered area. Indeed, more RSUs are needed
to cover larger area. Furthermore, we observe that our scheme has a low deploy-
ment cost than that of MPC for all variations in covered area. Indeed, it decreases
the overall cost of MPC by around 38%. This is owe the fact that our scheme con-
siders the junctions order when extracting the mobility patterns, which undoubt-
edly leads to reduce the total number of extracted patterns. Therefore, less RSUs

100 1
0.98
80 0.96
0.94
Coverage Ratio
RSUs number

60 0.92
0.9
40 0.88
0.86
20 0.84
0.82
0 0.8
600 800 1000 1200 1400 1600 200 300 400 500 600 700 800 900 1000
Covered area / m² Number of vehicles
Our scheme MPC Our scheme MPC

(a) Coverage cost (b) Coverage ratio

6 900
5.5 800
5 700
Latency (S)

4.5 600
Overhead

4 500
3.5 400
3 300
2.5 200
2 100
200 300 400 500 600 700 800 900 1000 200 300 400 500 600 700 800 900 1000
Number of vehicles Number of vehicles
Our scheme MPC Our scheme MPC

(c) Latency (d) Overhead

Fig. 2. Performance of our scheme vs MPC

628 S. Ben Chaabene et al.

are needed to cover the extracted patterns leading to a decrease in the deploy-
ment cost. Figure 2(b) depicts the evolution of the coverage ratio with respect to
the number of moving vehicles. It is normal that the coverage ratio is inversely
proportional to the traffic density, since increasing the number of vehicles in the
network will lead to raise up the ratio of lost packets. Therefore many vehicles
wont receive messages sent by the RSUs. Moreover, we observe that the cov-
erage ratio of both coverage approaches are more or less the same. Figure 2(c)
depicts the evolution of the latency of our scheme compared to MPC under dif-
ferent vehicle densities. We note that the latency of the two approaches increases
whenever the number of vehicles grow up. Indeed, raising the number of vehicles
drastically overloads the RSUs, which leads to a rise in the latency. In addi-
tion, Fig. 2(c) shows that both approaches have nearly latency for all variations
in vehicle densities. Figure 2(d) shows the overhead of our scheme and that of
MPC under different vehicle densities. We remark that the overhead increases as
far as the number of vehicles grows up. Indeed, the higher the number of vehicles
is, the higher network communications and overhead are. In addition, Fig. 2(d)
depicts that our scheme flags out a minimum overhead than do MPC. This is
owe the fact that our scheme relies on less number of RSUs than do MPC, which
leads to reduce communications between RSUs and vehicles.
To sum up, the results of simulation highlight that our scheme provides a
better trade off between cost, coverage ratio, overhead and latency than MPC.
Indeed, it decreases the overall cost and overhead of MPC by around 38% and
25%, respectively while providing nearly results in terms of coverage ratio and
latency.

5 Conclusion and Future Work

In this paper, we have introduced a spatio-temporal coverage scheme for RSUs
deployment over VANETs. The proposed scheme consists on two components,
namely SPMP and CovC. Indeed, SMPM mines a maximal frequent patterns
set from vehicles’ trajectories. To this end, we have relied on a data mining tech-
nique. CovC finds the adequate RSU locations by computing a set of junctions
that cover the extracted mobility patterns within the SMPM component. Simu-
lation results have highlighted that the proposed scheme has allowed to reach a
high coverage ratio as well as lower deployment cost. Obvious pointers for future
work include highlighting certain shortcomings in the evaluation of our scheme.

References
1. Allani, S., Yeferny, T., Chbeir, R.: A scalable data dissemination protocol based
on vehicles trajectories analysis. Ad Hoc Netw. 71, 31–44 (2018)
2. Allani, S., Yeferny, T., Chbeir, R., Yahia, S.B.: DPMS: a swift data dissemina-
tion protocol based on map splitting. In: Computer Software and Applications
Conference, COMPSAC, vol. 1, pp. 817–822 (2016)
3. Allani, S., Yeferny, T., Chbeir, R., Yahia, S.B.: A novel VANET data dissemination
approach based on geospatial data. Proc. Comput. Sci. 98, 572–577 (2016)
 A Roadside Unit Placement Scheme for Vehicular Ad-hoc Networks 629

4. Arour, K., Yeferny, T.: Learning model for efficient query routing in P2P informa-
tion retrieval systems. Peer-to-Peer Netw. Appl. 8(5), 741–757 (2015)
5. Bellomo, N., Delitala, M., Coscia, V.: On the mathematical theory of vehicular
traffic flow I: fluid dynamic and kinetic modelling. Math. Models Methods Appl.
Sci. 12(12), 1801–1843 (2002)
6. Boukenadil, B.: Importance of realistic mobility models for VANET network sim-
ulation. CoRR abs/1410.2450 (2014)
7. Kafsi, M., Papadimitratos, P., Dousse, O., Alpcan, T., Hubaux, J.: VANET con-
nectivity analysis. CoRR abs/0912.5527 (2009)
8. Kaisser, F., Gransart, C., Berbineau, M.: Simulations of VANET scenarios with
OPNET and SUMO. In: Proceedings of the Communication Technologies for Vehi-
cles - 4th International Workshop, Nets4Cars/Nets4Trains 2012, Vilnius, Lithua-
nia, 25–27 April 2012, pp. 103–112 (2012)
9. Kaur, R., Kaur, R.: Scalable TDB based RSUs deployment in VANETs. Int. J.
Innov. Appl. Stud. 3(4), 1025–1032 (2013)
10. Kchiche, A., Kamoun, F.: Traffic-aware access-points deployment strategies for
VANETs. In: 6th International ICST Conference, pp. 15–26 (2014)
11. Lee, J., Kim, C.: A roadside unit placement scheme for vehicular telematics net-
works. In: Advances in Computer Science and Information Technology, pp. 196–202
(2010)
12. Murakami, K., Uno, T.: Efficient algorithms for dualizing large-scale hypergraphs.
CoRR abs/1102.3813 (2011)
13. Murakami, K., Uno, T.: Efficient algorithms for dualizing large-scale hypergraphs.
Discrete Appl. Math. 170, 83–94 (2014)
14. Pei, J., Han, J., Mortazavi-Asl, B., Wang, J., Pinto, H., Chen, Q., Dayal, U., Hsu,
M.C.: Mining sequential patterns by pattern-growth: the PrefixSpan approach.
IEEE Trans. Knowl. Data Eng. 16(11), 1424–1440 (2004)
15. Singh, S., Agrawal, S.: VANET routing protocols: issues and challenges. In: Pro-
ceedings of 2014 RACES UIET Panjab University, Chandigarh. 17th IEEE Inter-
national Symposium on A World of Wireless, Mobile and Multimedia Networks
(2014)
16. Sun, Y., Lin, X., Lu, R., Shen, X., Su, J.: Roadside units deployment for efficient
short-time certificate updating in VANETs. In: Proceedings of IEEE International
Conference on Communications, pp. 1–5 (2010)
17. Trullols-Cruces, Ó., Fiore, M., Barceló-Ordinas, J.M.: Cooperative download in
vehicular environments. IEEE Trans. Mob. Comput. 11(4), 663–678 (2012)
18. Wu, H., Fujimoto, R.M., Riley, G.F., Hunter, M.: Spatial propagation of informa-
tion in vehicular networks. IEEE Trans. Veh. Technol. 58(1), 420–431 (2009)
19. Xiong, Y., Ma, J., Wang, W., Tu, D.: RoadGate: mobility-centric roadside units
deployment for vehicular networks. IJDSN 9 (2013)
20. Ye, W., Jang, S., Lee, J., Lee, K.: MDTMAC: a medium access control protocol
for multimedia data transmission service over IEEE 802.11 wireless LAN. In: Pro-
ceedings of the International Conference on Parallel and Distributed Processing
Techniques and Applications, pp. 144–150 (1999)
21. Yeferny, T., Allani, S.: MPC: a RSUs deployment strategy for VANET. Int. J.
Commun. Syst. 31(12) (2018)
22. Yeferny, T., Arour, K.: Efficient routing method in P2P systems based upon train-
ing knowledge. In: Advanced Information Networking and Applications Workshops,
WAINA, pp. 300–305 (2012)
630 S. Ben Chaabene et al.

23. Yeferny, T., Arour, K., Bouzeghoub, A.: An efficient peer-to-peer semantic overlay
network for learning query routing. In: Advanced Information Networking and
Applications, AINA, pp. 1025–1032 (2013)
24. Zhu, Y., Bao, Y., Li, B.: On maximizing delay-constrained coverage of urban vehic-
ular networks. IEEE J. Sel. Areas Commun. 30(4), 804–817 (2012)
 Personality-Aware Collaborative
Learning: Models and Explanations

Yong Zheng(B) and Archana Subramaniyan

Illinois Institute of Technology, Chicago, USA

[email protected], [email protected]

Abstract. Personality traits have been demonstrated as one of the effec-

tive human factors in the process of decision making. Personality-aware
recommendation models have been built for different applications. How-
ever, the models and research for educations are still under investigation.
In this paper, we utilize the educational learning as a case study, exploit
and summarize different approaches which take advantage of personality
traits in collaborative personalized recommendations. Furthermore, we
extend the existing personality-aware recommendation models and pro-
pose two other alternative recommendation algorithms which can utilize
the personality traits. We perform empirical comparisons and studies
over an educational data set, and explain the effects of personality from
the perspective of the characteristics of each algorithm, in order to dis-
cover more insights in the data and models.

1 Introduction
Recommender systems have been applied to multiple areas, in order to alleviate
the problem of overfitting and assist users in the process of decision making. The
idea behind is that the recommender systems are able to learn user preferences
from the past and build the predictive models to suggest a list of items tailored
by the user tastes. In recent years, human factors, such as emotions, trust and
personality, have been recognized as influential factors in the recommender sys-
tems. For example, emotional states and reactions [25,26,33,34] can be treated
as strong implicit feedbacks that can additionally represent user tastes. Trust
network [18] can provide additional property to infer the user preferences. User
personalities [20,23,28,29] may directly affect a user’s decision, since people
with different personalities may present distinct behavior patterns or tastes in
the real-world practice.
With the development of the technology-enhanced learning [17], recom-
mender systems have also been introduced to educations to provide the service
of personalized learning. It has been applied to different applications, such as
suggesting books for K-12 users [21], recommending after-school programs [5],
or suggesting appropriate citations [11] in paper writings. Moreover, personal-
ity has been proved to be influential in educational learning, e.g., Komarraju
et al. [14] identify the impact of personality on the academic achievements in
GPA.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 631–642, 2020.
https://doi.org/10.1007/978-3-030-15032-7_53
632 Y. Zheng and A. Subramaniyan

Personality-based recommendation models have been developed and applied

in several applications, such as music [9,12], tourism [8], images [23], social net-
works [2], etc. The effect of personality has been investigated in educations,
such as its impact on learning performance (e.g., marks or GPA [14]). However,
there are limited work that explore whether and how personality-aware recom-
mendation models are useful in the learning area. In this paper, we exploit and
summarize different recommendation approaches which take advantage of per-
sonality traits, extend existing models to propose new techniques, and finally
perform empirical comparisons over an educational data set [30], in order to
interpret the effects by personality traits and discover more insights in the pro-
cess of educational learning.

2 Related Work

Recommender systems have been developed for several decades. Multiple rec-
ommendation algorithms have been proposed and developed to better learn user
tastes and build the recommendation models. Most of these recommendation
algorithms can be categorized into three classes – content-based recommenda-
tions [16], collaborative filtering (CF) [3] and the hybrid recommendations [4].
CF is one of the most popular and effective recommendation models, while most
of the personality-aware recommendation algorithms were built on top of it.
Personality can be measured by the personality traits. Several frameworks
were developed to collect the personality traits. One of the most popular frame-
works is the Big Five Factor (Big5) [19] framework, in which the personality
traits can be described by five dimensions [14]: Openness is reflected in a strong
intellectual curiosity and a preference for novelty and variety. Conscientious-
ness is exemplified by being disciplined, organized, and achievement-oriented.
Extraversion is displayed through a higher degree of sociability, assertiveness,
and talkativeness. Agreeableness refers to being helpful, cooperative and sym-
pathetic towards others. Neuroticism indicates the degree of emotional stability,
impulse control, and anxiety. To collect the Big5 traits, survey is usually adopted,
such as the Ten-Item Personality Inventory (TIPI) [10] questionnaire.
Personality is useful in recommender systems for four major reasons:

• First of all, it is helpful to alleviate the problem of new users which is the
well-known cold-start problem in recommender systems. With personality
information, we can assume that users with similar personality traits may
have similar preferences on the items. And corresponding techniques [12] have
been developed to take advantage of the personality information to produce
the item recommendations, even if we do not have the preference data for a
new user.
• Additionally, personality is useful to increase diversity in recommender sys-
tems. Wu and Chen et al. [6,24] found that personality information was help-
ful to adjust diversity and balance between the recommendation accuracy and
diversity.
 Personality-Aware Collaborative Learning: Models and Explanations 633

• In addition, personality information could be useful in the group work. For

example, we found that personality traits could be used to identify group
leaders and followers [29], so that we can assign different weights to the con-
tributions of each group member, and help produce more accurate group
recommendations.
• Finally, it has been demonstrated that personality can generally help improve
the recommendation performance, no matter whether there are new users or
for the purpose of increasing diversity in the item recommendations.

In this paper, we ignore the issue of new users, diversity or the group recom-
mendations. We focus on the general recommendation performance, summarize
and apply multiple personality-aware recommendation models in the educational
data to discover and understand the effects of personality in the area of learning.

3 Preliminary: Collaborative Filtering

In this section, we introduce the collaborative filtering technique, in order to

better discuss the personality-aware recommendation models in the next section.

3.1 User-Based Collaborative Filtering

The user-based collaborative filtering refers to the well-known Resnick’s algo-

rithm for user-based kNN method (UserKNN), as shown in Eq. 1, where u is a
user, i is an item, and Nu is a neighborhood of k users similar to u. The algo-
rithm calculates Pu,i , which is the predicted rating that user u is expected to
assign to item i. r̄u denotes user u’s average rating over the items. And sim(a, u)
is similarity function which gives the similarity between the two users. Usually,
we can use the similarities based on the ratings associated with the co-rated
items by these two users, i.e., simr (a, u). We can use different popular similarity
measures to calculate user-user similarities, such as the cosine similarity, Pearson
correlations, and so forth. Cosine similarity is identified as the best choice in our
experiments, and it is shown in Eq. 1, where − ra and −
→ →
ru denote the rating vectors
by user a and u respectively.

(ra,i − r̄a ) × sim(a, u) →
−
a∈Nu ra · −→
ru
Pu,i = r̄u +  , sim(a, u) = simr (a, u) = −
sim(a, u) || r || · ||−
→
a
→
r ||
u
a∈Nu
(1)
The UserKNN method assumes u’s taste on the item i is similar to the taste
on the same item by the user neighbors. The function above first seeks the top-k
nearest neighbors, aggregate the neighborhoods’ ratings and predict u’s rating
on the item i.
634 Y. Zheng and A. Subramaniyan

3.2 Matrix Factorization

Matrix factorization (MF) [15] is a latent-factor based learning technique. It

is able to alleviate the sparsity issue in the rating data set and improve the
recommendation performance. The biased matrix factorization is shown by Eq. 2.

r̂ui = μ + bu + bi + pTu qi (2)

μ refers to the global average rating, while bu and bi are the bias associ-
ated with user u and item i respectively. pu and qi are the latent-factor vector
which can represent u and i respectively. The MF will learn these parameters by
minimizing sum of squared errors by using stochastic gradient descent (SGD)
as the optimizer. The L2 norms are usually added into the loss function as the
regularization terms in order to alleviate overfitting. The loss function can be
described by Eq. 3, where u, i is an entry in the training set T , and λ is the
regularization rate. rui and r̂ui are the real rating and predicted rating for the
entry u, i respectively.

M inimize (rui − r̂ui )2 + λ(||pu ||2 + ||qi ||2 + b2u + b2i ) (3)
p∗,q∗,b∗
(u,i)∈T

4 Personality-Aware Recommendation Models

In this section, we exploit and summarize the personality-aware recommendation

models, and discuss two other possible ways to extend the existing methods.

4.1 Existing Approaches

4.1.1 UserKNN with Similarities Based on Personality (KNNβ )

The assumption in this method [12] is that personality traits may help find
better user neighborhood. The predictive function in this model is the same as
shown by Eq. 1. However, the similarity between two users will come from two
contributions, as shown by Eq. 4.

sim(a, u) = β × simr (a, u) + (1 − β) ∗ simp (a, u) (4)

More specifically, simr (a, u) represents the original way to calculate user-
user similarities. In this case, simr (a, u) will be computed base on the ratings
of the co-rated items by user a and u. By contrast, the personality traits of each
user can be represented by a vector based on the Big5 personality framework
mentioned before. Therefore, simp (a, u) denotes the similarity between two users
based on the vector of personality traits. β is a float value between 0 and 1, and
it is used to adjust the weights of the two contributions, in order to find the
optimal setting of the model.
 Personality-Aware Collaborative Learning: Models and Explanations 635

4.1.2 Personality-Aware Matrix Factorization (MFp )

Elahi et al. [8] have successfully incorporated personality traits into the matrix
factorization technique. The predictive function can be shown by Eq. 5.


5
r̂ui = μ + bu + bi + qiT .(pu + ya ), where a = Bui (5)
i=1

Assume we use Big5 personality framework, Bu represents the vector of per-

sonality traits for user u. We use Bui to represent the value in the ith dimension of
the personality vector. For each value in each personality dimension, we initial-
ize a latent-factor vector ya which has the same length with pu . Therefore, the
optimization process will additionally learn ya in addition to other parameters.
The loss function can be generated accordingly, as shown by Eq. 6.

 
5
M inimize (rui − r̂ui )2 + λ(||pu ||2 + ||qi ||2 + ||ya ||2 + b2u + b2i ) (6)
p∗,q∗,b∗
(u,i)∈T i=1

In this method, each situation or value in a personality dimension is repre-

sented by a latent-factor vector, and all of the users who have the same situation
or value in that personality dimension will share a same personality vector.

4.2 Other Possible Methods

In this section, we extend the existing approaches and introduce two other possi-
ble methods to take advantage of the personality in the recommendation models.

4.2.1 Matrix Factorization with Personality Regularization (MFr eg )

First of all, the personality information can be utilized as regularization terms
in the matrix factorization technique. In this case, the predictive function is the
same as shown by Eq. 2. However, the loss function is updated by adding the
personality-based regularizations, as denoted by Eq. 7.
 
M inimize (rui − r̂ui )2 +λ(||pu ||2 +||qi ||2 +b2u +b2i + simp (a, u)×||pu −pa ||2 ) (7)
p∗,q∗,b∗
(u,i)∈T a∈Nu

Nu represent the top-k nearest neighborhood of user u, and user a is one of

the neighbors. Note that this neighborhood is identified by the personality traits.
In other words, the similarity between two users, simp (a, u), is computed based
on the personality vector associated with the two users. The assumption behind
is that users with similar personalities may have similar tastes. As a result,
the latent-factor vectors associated with the two users, pu and pa , should be
as close as possible. That’s the reason why the personality regularization term,
||pu −pa ||2 , is used to minimize the difference between the two user vectors, while
simp (a, u) can be considered as the weight term. pu and pa should be much more
similar, if the similarity between user a and u is larger.
636 Y. Zheng and A. Subramaniyan

It also makes sense to combine the M Fp model with the personality regu-
larization. Accordingly, we use Eq. 5 as the rating prediction function, and add
the personality regularization term into the loss function shown by Eq. 6. This
method is denoted as M Fp+reg in our paper.

4.2.2 Item Splitting by Personality Traits (iSplitting)

Item splitting [1] is one of the context-aware splitting approaches [32] which can
be applied in context-aware recommendations. There are three major splitting
methods – item splitting [1] which splits the items, user splitting [22] which splits
users and UISplitting [32] which combines user and item splittings.

Table 1. Example of item splitting by using contexts

Take the movie ratings data in Table 1(a) for example, there are one user
U1 , one item T1 and two ratings (the first two rows) in the training and one
unknown rating that we are trying to predict (the third row). There are three
contextual dimensions – time (weekend or weekday), location (at home or cin-
ema) and companion (friend, girlfriend or family). In the following discussion,
we use contextual dimension to denote the contextual variable, e.g. “Location”
in this example. The term contextual condition refers to a specific value in a
contextual dimension, e.g. “home” and “cinema” are two contextual conditions
for “Location”.
The underlying idea of item splitting is that the nature of an item, from the
user’s point of view, may change in different contextual conditions, hence it may
be useful to consider it as two different items. Therefore, item splitting tries
to find a contextual condition on which to split each item. The split should be
performed once the algorithm identifies a contextual condition in which items
are rated significantly differently. In the movie example above, there are three
contextual conditions in the dimension companion: friend, girlfriend and family.
Accordingly, there are three possible binary conditions: “friend vs not friend ”,
“girlfriend vs not girlfriend ”, “family vs not family”. Impurity criteria [1] are
used to determine whether and how much items were rated differently in these
alternative conditions, e.g. a t-test or other statistical metrics can be used to
evaluate if the means differ significantly across conditions. The reason why we
use binary splits is because it can alleviate the data sparsity problem so that the
statistical tests are more reliable.
Item splitting iterates over all contextual conditions in each context dimen-
sion and evaluates the splits based on the impurity criteria. It finds the best
split for each item. Assume that the best contextual condition to split item T1
 Personality-Aware Collaborative Learning: Models and Explanations 637

in Table 1(a) is “Location = home vs not home”, T1 can be split into T11 (movie
T1 being seen at home) and T12 (movie T1 being watched at other locations).
The original rating matrix therefore can be transformed to a 2D matrix as shown
by Table 1(b). The contexts have been fused into the new items, and any tradi-
tional recommendation models (e.g., UserKNN, MF) can be applied to produce
recommendations.
Both Baltrunas et al. [1] and us [32] have claimed that not only the con-
text information, but also the user demographic information (e.g., age, gender,
nationality) can be used to split the items. Alternatively, the values in the per-
sonality traits can be used in item splittings too. However, there are no existing
research exploring this method yet. More specifically, there are 7 values (i.e.,
1–7) in each dimension of the personality traits which can be viewed as the
degree of the characteristics in each personality trait, e.g., strongly agree, agree,
and so forth. These values can be viewed as the “context conditions” to split the
items. Note that, user information, including personality traits, cannot be used
to split the users, since these information are fixed by given a specific user. In
our experiments, we tried the item splitting method, and used p-value as well
as the t value in the t-test as the impurity criteria to identify the best split for
each item.

5 Experimental Results and Findings

5.1 Data Set and Evaluation Protocols

Note that it is difficult to collect educational data sets, not to mention the data
with user personality traits. As mentioned by Drachsler et al. [7], “collecting data
(in the learning area) was limited in terms of cost, time requirements, scope and
authenticity of the data, as this was typically done using single groups or classes
for an experiment”. Therefore, the data set in this area is relatively smaller in
comparison with the data in other domains, such as movies or music.
We use an educational data [30] which was collected from our Web-based
learning portal that improves the process of teaching and learning for faculties
and students. One of the components in our portal is project recommendations.
Students are required to complete a project for some courses in which they need
to find a data set and topics, define research problems, and utilize their skills
to solve the proposed research problems. We pre-define the potential topics by
giving a list of data sets on Kaggle.com in our learning portal.
Each student was asked to fill the questionnaire by himself or herself. Each
subject should select at least three liked and disliked topics of the projects, and
provide an overall rating (in scale 1 to 5) to them. In addition, we obtained the
student personality traits by using the TIPI and Big5 framework as introduced in
Sect. 2. We have collected the data for two years. There are a total of 3,306 ratings
given by 269 students on 70 items (i.e., the topics of the projects). The density
is 17.6% so it can be considered as a dense data in the area of recommender
systems.
638 Y. Zheng and A. Subramaniyan

We use 5-fold cross validation since the data set is small. We evaluate dif-
ferent approaches based on both rating prediction and top-N recommendation
tasks. In terms of the rating predictions, we use mean absolute error (MAE) as
the evaluation metrics. And we use precision, recall and normalized discounted
cumulative gain (NDCG) [13] metrics for the top-10 recommendation task. Pre-
cision refers to the fraction of positive predictions over the top-N recommenda-
tions, while recall is the fraction of positive predictions with respect to the list of
relevant items by a specific user. NDCG is a ranking metric which additionally
examines the quality of the ranks in the top-N recommendations. To determine
“relevance”, we set 3 as the rating threshold. In other words, items in the test
sets with ratings no less than 3 will be considered as the relevant items as the
ground truth in our experiments. The reason why is we requested students to
give a rating no less than 3 if they like the items in the process of data collections.
In the UserKNN based approach, we applied cosine similarity as the similarity
measure, and vary the number of neighbors from 10 to 100 with each step of 10,
in order to find the best settings. In the MF based algorithms, SGD is selected
as the optimizer and we vary the number of factors from 10 to 50 with each step
of 10, and change the learning and regularization rates accordingly to find the
optimal parameters. In terms of the iSplitting, we apply the matrix factorization
as the recommendation approach after the transformations.

5.2 Results, Findings and Explanations

The experimental results can be shown by Table 2 and Fig. 1, where we categorize
these methods into three classes – KNN based approaches, MF based methods,
and the iSplitting algorithm.

Table 2. Experimental results 0.450 0.100

0.095
0.400
0.090
MAE Precision Recall NDCG
0.350 0.085
Recall & NDCG

KNNβ=1 1.133 0.082 0.394 0.242

0.080
Precision

KNNβ=0 1.123 0.087 0.411 0.283 0.300

0.075
KNNβ=0.6 1.108 0.093 0.438 0.291 0.250 0.070
MF 1.176 0.077 0.367 0.235
0.200 0.065
MFp 1.067 0.081 0.371 0.246
0.060
MFreg 1.106 0.080 0.370 0.241 0.150
0.055
MFp+reg 1.067 0.086 0.400 0.248
0.100 0.050
iSplitting 1.067 0.087 0.410 0.255 KNNβ=1 KNNβ=0 KNNβ=0.6 MF MFp MFreg MFp+reg iSpli ng
Recall NDCG Precision

Fig. 1. Visualized results

First of all, we discuss the results based on the methods of KNNβ which utilize
the prediction function as shown by Eq. 1 and the similarity function as noted by
Eq. 4. KNNβ=1 and KNNβ=0 are two baseline methods, since they represent the
UserKNN approach which calculates the user-user similarity based on the rating
vectors only and the personality vectors only respectively. KNNβ=0.6 represent
 Personality-Aware Collaborative Learning: Models and Explanations 639

the method that was introduced in Sect. 4.1.1, while the optimal β value is 0.6 in
our experiments. We can observe that KNNβ=0.6 performs the best among these
approaches in both the rating prediction and the top-10 recommendation tasks.
More specifically, it is able to reduce the MAE by 2.3%, and increase precision,
recall and NDCG by 11.8%, 10% and 16.8% respectively, in comparison with the
base method KNNβ=1 . It tells that using the rating vectors only or the person-
ality vectors only is not the best way to correctly estimate user-user similarities.
The linear combination of these two similarities imply that 60% contributions
from the similarities based on the rating vectors and 40% contribution from the
similarities based on personality traits could be the best option in this data set.
Therefore, personality traits have been demonstrated as useful information to
improve the recommendation performance according to these results based on
the KNN based algorithms.
Moreover, we discuss the results based on the MF recommendation tech-
niques. MFp incorporates personality traits in the rating prediction function,
while MFreg utilizes personality traits in the regularization terms. MFp+reg inte-
grates MFp and MFreg together. In comparison with the standard MF method,
we can observe that all of these approaches can improve the predictive perfor-
mance, though the improvement by MFreg is small. MFp+reg performs the best,
which infers that it is useful to combine these two methods.
In addition, the iSplitting method which utilize MF as the rating predictions
can further outperform MFp+reg in the top-10 recommendations. Among all of
these techniques, MF-based methods, including the iSplitting approach, perform
the best in the rating prediction task. However, the KNNβ=0.6 outperforms other
approaches in the top-10 recommendations in terms of the observations in pre-
cision, recall, and NDCG. As mentioned before, this educational data has only
70 items, while the density is 17.6%. MF-based methods deal well with sparse
data, and it may not outperform the KNN based algorithms when it comes to
dense data sets. And an algorithm that works well in the rating predictions is
not necessary to also perform well in the top-N recommendations. These are
potentially the reasons why we have such observations.
Finally, we tried to explain the effects of the personality traits according to
the performance and the characteristics of these personality-aware collaborative
recommendation algorithms.

• KNNβ=0.6 performs the best, while the optimal contribution of user similar-
ities based on the personality traits is 40%. It confirms the effects of person-
ality in better estimating user-user similarities and looking for appropriate
neighbors for the UserKNN recommendation algorithms.
• MFreg assumes that the latent-factor vector representations for two users
should be similar if these two users have similar personality traits. MFreg
works better than the standard MF, which confirms the patterns identified
by the KNNβ=0.6 too. MFp extends the user vectors by adding personality
traits. It works even better since it updates the rating prediction function in
the recommendation process. On one hand, it assumes two users should be
similar if these two users have similar personality traits. On other hand, it
640 Y. Zheng and A. Subramaniyan

allows the matrix factorization process to update its latent factors accordingly.
Both of these MF based methods can confirm the effects of personalities.
However, it is difficult to interpret more due to that the latent factors in MF
cannot be well explained.
• iSplitting performs the best among the MF based methods. Recall that there
are 70 items in total in our data set. A further investigation on the splitting
operations tells us that 40 out of these 70 items were splitted. We are partic-
ularly interested in which personality traits are the most useful ones to split
the items. We found that 30%, 17.5%, 22.5%, 15% and 15% of these 40 items
were splitted by openness, conscientiousness, extraversion, agreeableness, neu-
roticism respectively. The top-3 personality traits used in the splittings are
“extraversion = 7 vs not 7”, “openness = 6 vs not 6” and “neuroticism =
1 vs not 1”. Recall that the values in each personality dimension is from 1
to 7. Interestingly, we can observe that these best splits are the personality
traits in extreme conditions, i.e., the values are very close to either 1 or 7. It
implies that the items will be treated differently for students with extreme or
distinguished personality traits.

6 Conclusions and Future Work

In this paper, we utilize the educational learning as a case study, exploit and
summarize different approaches which take advantage of personality traits in
collaborative personalized recommendations. Additionally, we propose two alter-
native methods which can also be used to build personality-aware recommen-
dation models. Accordingly to the empirical comparisons and studies over an
educational data set, we confirm the effects of the personality traits in personal-
ized learning. Particularly, using personality traits in the similarity functions in
UserKNN becomes the best performing approach in collaborative recommenda-
tions. Furthermore, we interpret the role of personality traits bases on the charac-
teristics of different algorithms. In our future work, on one hand, we would like to
investigate the chains effect [26,27] of personality traits. On other hand, we will
further explore the impact of personality on multi-criteria preferences [27,31],
since we also have multi-criteria ratings in our educational data set.

References
1. Baltrunas, L., Ricci, F.: Context-based splitting of item ratings in collaborative
filtering. In: Proceedings of ACM Conference on Recommender Systems, pp. 245–
248 (2009)
2. Bian, L., Holtzman, H.: Online friend recommendation through personality match-
ing and collaborative filtering. In: Proceedings of UBICOMM, pp. 230–235 (2011)
3. Breese, J.S., Heckerman, D., Kadie, C.: Empirical analysis of predictive algo-
rithms for collaborative filtering. In: Proceedings of the Fourteenth Conference
on Uncertainty in Artificial Intelligence, pp. 43–52. Morgan Kaufmann Publishers
Inc. (1998)
 Personality-Aware Collaborative Learning: Models and Explanations 641

4. Burke, R.: Hybrid recommender systems: survey and experiments. User Model.
User-Adap. Inter. 12(4), 331–370 (2002)
5. Burke, R., Zheng, Y., Riley, S.: Experience discovery: hybrid recommendation of
student activities using social network data. In: Proceedings of the 2nd Interna-
tional Workshop on Information Heterogeneity and Fusion in Recommender Sys-
tems, pp. 49–52. ACM (2011)
6. Chen, L., Wu, W., He, L.: How personality influences users’ needs for recommenda-
tion diversity? In: Extended Abstracts on Human Factors in Computing Systems,
CHI 2013, pp. 829–834. ACM (2013)
7. Drachsler, H., Verbert, K., Santos, O.C., Manouselis, N.: Panorama of recom-
mender systems to support learning. In: Recommender Systems Handbook, pp.
421–451. Springer (2015)
8. Elahi, M., Braunhofer, M., Ricci, F., Tkalcic, M.: Personality-based active learn-
ing for collaborative filtering recommender systems. In: Congress of the Italian
Association for Artificial Intelligence, pp. 360–371. Springer (2013)
9. Ferwerda, B., Yang, E., Schedl, M., Tkalcic, M.: Personality traits predict music
taxonomy preferences. In: Proceedings of the 33rd Annual ACM Conference
Extended Abstracts on Human Factors in Computing Systems, pp. 2241–2246.
ACM (2015)
10. Gosling, S.D., Rentfrow, P.J., Swann Jr., W.B.: A very brief measure of the Big-
Five personality domains. J. Res. Pers. 37, 504–528 (2003)
11. He, Q., Pei, J., Kifer, D., Mitra, P., Giles, L.: Context-aware citation recommen-
dation. In: Proceedings of the 19th International Conference on World Wide Web,
pp. 421–430. ACM (2010)
12. Hu, R., Pu, P.: Using personality information in collaborative filtering for new
users. In: Recommender Systems and the Social Web, p. 17 (2010)
13. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of ir techniques.
ACM Trans. Inf. Syst. (TOIS) 20(4), 422–446 (2002)
14. Komarraju, M., Karau, S.J., Schmeck, R.R., Avdic, A.: The Big Five personality
traits, learning styles, and academic achievement. Pers. Individ. Differ. 51(4), 472–
477 (2011)
15. Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender
systems. IEEE Comput. 42(8), 30–37 (2009)
16. Lops, P., De Gemmis, M., Semeraro, G.: Content-based recommender systems:
state of the art and trends. In: Recommender Systems Handbook, pp. 73–105.
Springer (2011)
17. Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., Koper, R.: Recommender
systems in technology enhanced learning. In: Recommender Systems Handbook,
pp. 387–415. Springer (2011)
18. Massa, P., Avesani, P.: Trust-aware recommender systems. In: Proceedings of the
2007 ACM Conference on Recommender Systems, pp. 17–24. ACM (2007)
19. McCrae, R.R., John, O.P.: An introduction to the five-factor model and its appli-
cations. J. Pers. 60(2), 175–215 (1992)
20. Nunes, M.S., Hu, R.: Personality-based recommender systems: an overview. In:
Proceedings of ACM Conference on Recommender Systems, pp. 5–6 (2012)
21. Pera, M.S., Ng, Y.-K.: What to read next?: making personalized book recommenda-
tions for K-12 users. In: Proceedings of the 7th ACM Conference on Recommender
Systems, pp. 113–120. ACM (2013)
22. Said, A., De Luca, E.W., Albayrak, S.: Inferring contextual user profiles – improv-
ing recommender performance. In: ACM RecSys, the 4th Workshop on Context-
Aware Recommender Systems (2011)
642 Y. Zheng and A. Subramaniyan

23. Tkalcic, M., Kunaver, M., Tasic, J., Košir, A.: Personality based user similar-
ity measure for a collaborative recommender system. In: Proceedings of the 5th
Workshop on Emotion in Human-Computer Interaction-Real World Challenges,
pp. 30–37 (2009)
24. Wu, W., Chen, L., He, L.: Using personality to adjust diversity in recommender
systems. In: Proceedings of the 24th ACM Conference on Hypertext and Social
Media, pp. 225–229. ACM (2013)
25. Zheng, Y.: Adapt to emotional reactions in context-aware personalization. In:
EMPIRE Workshop @ ACM Conference on Recommender Systems (2016)
26. Zheng, Y.: Affective prediction by collaborative chains in movie recommendation.
In: Proceedings of the International Conference on Web Intelligence, pp. 815–822.
ACM (2017)
27. Zheng, Y.: Criteria chains: a novel multi-criteria recommendation approach. In:
Proceedings of the 22nd International Conference on Intelligent User Interfaces,
pp. 29–33. ACM (2017)
28. Zheng, Y.: Exploring user roles in group recommendations: a learning approach.
In: Adjunct Publication of the 26th Conference on User Modeling, Adaptation and
Personalization, pp. 49–52. ACM (2018)
29. Zheng, Y.: Identifying dominators and followers in group decision making based
on the personality traits. In: Companion Proceedings of the 23rd International on
Intelligent User Interfaces: 2nd Workshop on Theory-Informed User Modeling for
Tailoring and Personalizing Interfaces (2018)
30. Zheng, Y.: Personality-aware decision making in educational learning. In: Proceed-
ings of the 23rd International Conference on Intelligent User Interfaces Companion.
ACM (2018)
31. Zheng, Y.: Utility-based multi-criteria recommender systems. In: Proceedings of
the 34th Annual ACM Symposium on Applied Computing. ACM (2019)
32. Zheng, Y., Burke, R., Mobasher, B.: Splitting approaches for context-aware rec-
ommendation: an empirical study. In: Proceedings of the 29th Annual ACM Sym-
posium on Applied Computing, pp. 274–279. ACM (2014)
33. Zheng, Y., Mobasher, B., Burke, R.: Emotions in context-aware recommender sys-
tems. In: Emotions and Personality in Personalized Services, pp. 311–326. Springer
(2016)
34. Zheng, Y., Mobasher, B., Burke, R.D.: The role of emotions in context-aware rec-
ommendation. In: Proceedings of the 3rd Workshop on Human Decision Making
in Recommender Systems @ ACM RecSys, pp. 21–28 (2013)
 Interfacer: A Model-Driven Development
Method for SDN Applications

João Eurı́pedes Pereira Júnior(B) , Flávio de Oliveira Silva,

João Henrique de Souza Pereira, and Pedro Frosi Rosa

Federal University of Uberlândia, Uberlândia, Brazil

{joao,flavio,joaohs,pfrosi}@ufu.br

Abstract. In order to deliver network services using Software-Defined

Networking (SDN) elements of the three layers, namely infrastructure,
control, and application are necessary. Infrastructure and Control layer
elements are in a more mature state. They had received more extensive
research and already count with some product options available in the
market. SDN Applications, however, are not commodity software that
can be deployed into different control and infrastructure layers. Appli-
cations should be built in a more customized fashion, and seamless inte-
grated with existing SDN infrastructure and control. This paper explores
the Interfacer, a Model-Driven Development (MDD) approach to SDN
application development and integration in order to deliver high-quality
network services. This method uses ontology-driven conceptual modeling
to capture essential aspects of existing and to be developed components
of a SDN network architecture as well how they should better interface
and integrate into models. These models are transformed into source code
that respects the requirements of existing components and enforce the
requirements of SDN applications been developed leading to higher con-
tinuity and lower time to market and maintenance cost of SDN services.
These models also promote communication and learning improvements
in developer community accelerating the development process and mini-
mizing risks. A case study is reported illustrating the application of the
method. In this case study we perform the analysis and refactoring of the
Entity Title Architecture (ETArch), an SDN based network architecture
that is deployed over an OpenFlow capable infrastructure.

1 Introduction

Software Defined Networking (SDN) facilitates the implementation of network

services by combining components in a bottom-up approach. However, high qual-
ity network services rely on complex SDN applications which also demand a
top-down approach to materialize a unified service view throughout the network
while compatibility with underlying components is enforced by the bottom-up
approach. Moreover, such SDN application implementation strategy requires
additional processes and artifacts to manage and automate the development.
Generally, this features are delivered by Model-Driven Development (MDD)
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 643–654, 2020.
https://doi.org/10.1007/978-3-030-15032-7_54
644 J. E. Pereira Júnior et al.

methods. To our knowledge there is no model-driven development approach spe-

cific for developing complex SDN applications. In this paper we explore the
Interfacer, a model-driven development approach for SDN applications.
SDN starts from three simple ideas: generalize network hardware establishing
common interfaces in order to provide a standard collection of packet-processing
functions, a logically centralized and decoupled control layer makes decisions
upon an up-to-date global view of network state and summarizes the network
state for applications and translates application requirements to low-level rules.
Additionally, applications are programmable and network aware so that they
can enforce their communication requirements. These principles make it possi-
ble to evolve the network infrastructure without having to change the underlying
hardware, to evolve network applications with fewer changes in the underlying
control software and enable expressing network algorithms in terms of appro-
priate abstractions for particular applications [1,2]. However, obtaining these
appropriate abstractions is not an easy task. Despite several initiatives in the
related work the problem still is considered challenging.
This paper explores the fact that proposing good abstractions frequently
require deep knowledge and long experience in a system or domain. It pro-
poses a structure to support knowledge and experience, helping to materialize
a top-down approach to manage SDN application development complexity. it
uses Ontology-Driven Conceptual Modeling (ODCM) to build shared and for-
mal conceptualizations of the problems, the method helps to isolate develop-
ment responsibilities by introducing artifacts and promotes more core reuse by
transformations. Additionally it provides a formal and shared understanding
of the domain that developers can use learn, communicate, and write and test
code. ODCM is the activity of formally capturing community knowledge about
a domain or system. The aim of this method is to introduce representations of
knowledge in the form of artifacts simplifying interface and abstraction definition
and accelerating the development process.
The remainder of this work is organized as follows: Sect. 2 presents the state
of the art by comparing our approach with the related work. Section 3 describes
the Interfacer, a method that applies model-driven development to SDN appli-
cations. Section 4 shows a case study that applied the Interfacer into an SDN
based network architecture, called Entity Title Architecture (ETArch). Finally,
Sect. 5 presents some concluding remarks and future work.

2 Model-Driven Development for Networks

This Section presents the related literature available. We have first realized that
research addressing the problem of SDN application development are frequently
related to the concept of interface. Much research has been made in order to
improve SDN application development (i.e. NetKAT [3], Frenetic [4], Ravel [5],
Gavel [6]) and ONF initiatives such as Intents [7] Northbound Interface (NBI)
and the Open Information Modeling (CIM) of Open Network Foundation (ONF).
Besides the advancements in programming languages and interfaces, complex
SDN application development continues to be a challenge.
 Interfacer: A Model-Driven Development Method for SDN Applications 645

Complexity of SDN application development makes modeling an ideal candi-

date for improvements as already verified by related works. Although the SDN
architecture enables network programmability, SDN does not make it easy for
developers and network operators [8]. The Model-Driven Networking also applies
Model-Driven Development (MDD) to SDN development recognizing the ben-
efits (e.g., reduced complexity, less error-prone, meaningful validation) to the
development and management of applications but it aims at delivering simpler
generated source code that can be executed. In other related work, MDD is
applied to cloud computing. It defines MDD as a methodical software develop-
ment approach based on three primary activities i.e. modeling, model transfor-
mation, and verification. It acknowledges that the application of such methods
incorporates the features of fast development, reusability, and portability. Such
sophisticated features are highly supportive and aligned with the implementation
requirements of cloud computing. Consequently, MDD is considered an effective
and attractive development approach for cloud computing [9].
This related work joins Infrastructure as a Code technique with MDD in
order to deliver data-intensive architectures. It advocates that this trend of using
software engineering techniques that reduce the space, time, and efforts between
software development and operations, as well as the technical and organizational
distance between these two types of software teams, is known as DevOps. As
part of the DevOps menu, many practices entail re-using standard tools from
software development (e.g., code-versioning, code-revision management, etc.) to
manage what is known as the Infrastructure-as-Code (IaC) approach. In this
context, it has experimented with MDD and seen that power of abstraction
and its automation potential simplified IaC development [10]. Although model-
driven development has been already proposed in other works, none of then focus
specifically on SDN application that is currently a very important issue.

3 The Interfacer

The Interfacer is a Model-Driven Development approach that helps to manage

SDN application development complexity by introducing artifacts in the devel-
opment process. One type of artifact that the Interfacer introduces is the model.
Models capture high level (architectural) abstractions, giving a complete view
of the solution and the relationship between its parts, establishing a formal and
shared vocabulary that promotes developers team communication and learning.
Another type of artifact the Interfacer introduces is the transformation code.
Transformation code uses models as input and produces SDN application source
code as output. The goal of transformation codes is to reduce abstraction from
model level to programming language level (i.e. object-oriented, functional, pro-
cedural). These artifacts are introduced and improved in a cyclic way divided
into 4 phases: Modeling, Transformation, Implementation, and Testing as shown
in Fig. 1.
Modeling captures essential knowledge of community about the system using
ontology-driven conceptual modeling [11] and stores it into models expressed
646 J. E. Pereira Júnior et al.

in OntoUML language. Transformation incorporates architectural decisions and

transform model elements into generated code that instantiate good architectural
patterns using Eclipse Acceleo plugin and OntoUML meta-model [12]. Imple-
mentation is the phase where SDN developers actually write some code. With
knowledge and design issues isolated in precedent phases, this phase encompasses
only programming language specific decisions that must obey earlier phases con-
strains simplifying and accelerating the implementation. In testing phase net-
work services are tested and its compliance with the model and the architectural
elements can be verified. The results are used as feedback to fix implementation
or improve the model in subsequent cycles. Testing concerns not only code tests
but also network service tests (such as performance, security) that can also be
generated from models and transformations.

Fig. 1. Interfacer: model-driven development for SDN applications

The models created in the modeling phase must work in a complimentary way
with SDN reference architecture. The concepts defined in these models should
represent some SDN plane element and concepts from different planes must com-
municate through standard interfaces. Interfacer helps to seamless integrate SDN
applications into existing infrastructures as it can model the interfaces of reused
software (i.e. SDN Controllers) and abstraction of applications been developed.
Additionally, the Interfacer helps to define interfaces that do not compromise the
general purpose of SDN networks by leveraging a special application to manage
application interfaces, so that network control can use standardized interfaces
while (user) applications can define more specific interfaces when needed. Once
development project is mature Interfacer provides much less knowledge to ser-
vice time for huge alterations in the project meeting SDN demands for agility
in network service deployments.
 Interfacer: A Model-Driven Development Method for SDN Applications 647

3.1 Knowledge to Model

In order to represent knowledge into models, we use ontology-driven conceptual

modeling [13]. Ontology-driven conceptual modeling is the act of capturing and
formalizing how a community perceives a domain or system of interest, using
modeling primitives inherited from a foundational ontology. Moreover, we use
the Unified Foundational Ontology (UFO) as foundational ontology to identify
the fundamental distinctions between types and individuals, taking into con-
sideration ontological properties, such as rigidity, identity, and dependence. In
software-defined networking, the individuals can be the programmable switches,
routers, or any hardware composing the network, and processes running software
that controls the network, or provides some application to his users.
Unified Foundational Ontology [14] provides a set of universals that behaves
as a set of meta-properties and are used as class or association stereotypes.
A model is constructed by grouping some stereotyped classes and associations
so that the classes and associations of the models will follow a formal meta-
property behavior that represents an actual network behavior. Visually a model
will appear like a UML class diagram, but it carries a much more expressive
semantic due to the use of unified foundational ontology as stereotypes.
Models contain classes, which can be stereotyped as kind, subkind, phase, role
or relator. The classes are connected by associations, which can be stereotyped
as component of, member of, material, or mediation. The use of ODCM allows
the simultaneous modeling of diverse aspects of the network like architecture,
performance, security, energy-efficiency and user quality of service producing an
agile, cheap, compliant and consistent architecture design that can be refined in
successive cycles while automatically generates code that reflects model changes.
It isolates the high-level aspects of design concerns formally promoting its com-
pleteness and soundness. A formal representation of knowledge about network
leverages developer community communication and learning leading to more
precise development process and results.
Figure 2 is a model diagram that depicts our vision of a reference SDN appli-
cation consisting of several application components such as main, controller and
client components. The main component resides in the core of SDN applica-
tion layer. The controller component resides inside the SDN controller and is
responsible for the translation between application and controller abstractions.
The client component runs inside the user device, it requests and uses network
services provided by the main component. These components have different func-
tions and may have different behaviors under different circumstances. We model
these using roles (e.g. a controller component of the application may behave as
a cloud controller component or as a fog controller component depending if it
is located in the core or in the edge of the network). This models help to sepa-
rate knowledge about the SDN application from source code allowing knowledge
reusability and improving developer community learning and communication.
648 J. E. Pereira Júnior et al.

Fig. 2. Example of SDN application model

3.2 Model to Code

We use the Eclipse Acceleo [15,16] plugin and the OntoUML meta-model in
order to transform network models into architecture and test code. Writing a
template in Acceleo model transformation language we can instruct Acceleo
how to transform network models into programming language interfaces, classes,
methods, and variables. Transformation code [17] isolates intermediary design
concerns, separating architectural from implementation decisions.
The Menthor [18] tool is used to guide the model construction process and
to verify OntoUML syntax. Menthor is an OntoUML language editor to build
and validate the network models. OntoUML is semantically supported by Uni-
fied Foundation Ontologies (UFO). The OntoUML metamodel contains all the
UFO universals that are used to interpret the network models. The metamodel
is imported into an Eclipse Acceleo Project where templates must be written
with the instructions to transform model to code. OntoUML is a UML pro-
file extension that integrates well founded ontological semantic with a graphical
modeling language. Listing 1 shows an excerpt of a model transformation lan-
guage of Eclipse Acceleo plugin.
 Interfacer: A Model-Driven Development Method for SDN Applications 649

1 [comment encoding = UTF-8 /]

[module ExampleModule(’OntoUML’)]
3

[template public generateControllerModule(aKind : Kind)]

5

[file (aKind.name + ’.java’, false, ’UTF-8’)]

7 public [aKind.name/] implements IOFMessageListener,
IFloodlightModule

9 @Override
public String getName() {
11 // TODO Auto-generated method stub
return null;
13 }

15 @Override
public boolean isCallbackOrderingPrereq(OFType type, String
name) {
17 // TODO Auto-generated method stub
return false;
19 }
}
21 [/file]
[/template]
Listing 1. Acceleo model transformation language example

public ControllerModule implements IOFMessageListener,

IFloodlightModule {
2

@Override
4 public String getName() {
// TODO Auto-generated method stub
6 return null;
}
8

@Override
10 public boolean isCallbackOrderingPrereq(OFType type, String
name) {
// TODO Auto-generated method stub
12 return false;
}
14 }
Listing 2. Excerpt of SDN application source code generated in file
ControllerModule.java

A profile in the Unified Modeling Language (UML) provides a generic exten-

sion mechanism for customizing UML models for particular domains and plat-
forms. Extension mechanisms allow refining standard semantics in a strictly
650 J. E. Pereira Júnior et al.

additive manner, preventing them from contradicting standard semantics. Pro-

files are defined using stereotypes, tag definitions, and constraints which are
applied to specific model elements, like Classes, Attributes, Operations, and
Activities. A Profile is a collection of such extensions that collectively customize
UML for a particular domain.
Using a modeling transformation language SDN architects can specify which
type of programming structures should replace each occurrence of a determined
stereotype as shown in Listing 1 and steer developers to implement known best
pattern as shown in Listing 2. For example, each element of the model stereo-
typed with the universal kind should be replaced by a Java interface or class
with the same name of the model element, and different role universals associ-
ated with this kind should be replaced by specific methods within his respective
interfaces or classes.

3.3 Code to Service

This phase embraces a more traditional development style, however, it bene-

fits from a formal and shared vocabulary delivered by the models of the first
phase. This vocabulary must be broadcast inside the developers’ team in order
to facilitate communication and speed the learning. Some code convention may
also be delivered from this vocabulary so that the whole code reflects the model
concepts.
In order to transform the code into service developers must complete manda-
tory Java class methods and have a little understanding of how SDN infrastruc-
ture works. With this methodology interfaces and software architecture decisions
can be separated from implementation decisions, and model improvements can
be quickly transmitted to code in a top-down approach.
An Infrastructure-as-Code is used to enforce control of model and code ver-
sion to allow reuse of implementation code compatible with different model ver-
sions. Additionally, modeling and transformation can also be used to provide the
testing code to verify the service.

4 Applying Interfacer to the Entity Title Architecture

(ETArch)
This section presents a case of study where we apply the Interfacer: Model-Driven
Development for SDN Application method to the Entity Title Architecture,
discussing his limitations and some results.

4.1 Entity Title Architecture (ETArch)

ETArch [19] is a clean-slate architecture with a natural match with SDN [20]
concept. ETArch’s goal is to support the new applications requirements such
as multicast, mobility, QoS, and security. ETArch’s design goals consider a new
 Interfacer: A Model-Driven Development Method for SDN Applications 651

horizontal addressing scheme that solves the ambiguity of identification and loca-
tion, then making easier mobility and multicast communication and mitigates
the need of various addresses (MAC/IP/TCP), wherein ETArch uses only a
unified address for communication between entities. A flexible Communication
layer, semantic oriented, capable of supporting different applications require-
ments and an advanced network control plane to establish a communication
context that provides guarantees to the communication requirements over time
for satisfactory user experience. In more depth, ETArch has a communication
context, namely workspace, which is a logical bus, naturally multicast. In prac-
tical terms, the workspace supports the dialog of a set of entities, which take
part in a communication context. This communication context provides security,
QoS, QoE, multicast communication, and mobility for these entities over time.
Another important concept of ETArch is the Domain Title Service (DTS),
which is a distributed environment that manages all of the control mechanisms
at a global scale and solves issues, such as management of entity lifecycle, cal-
culation of the best route for the extension of workspaces, and guarantee of
communication quality. These functions are materialized by a DTS Agent called
Domain Title Service Agent (DTSA), which is an ETArch controller based on
the SDN concept. To sum up, the key element design applied to ETArch is
flexibility, which allows this architecture to handle a broad array of require-
ments, while it can still be generic enough to support the technological evolu-
tion. Through these considerations, ETArch can uphold different services from
distinct paradigms such as Service-Centric, User-Centric, Information-Centric,
and the current Host-Centric.

4.2 Limitations of Entity Title Architecture

Despite ETArch’s design goals consider a horizontal addressing scheme that
makes mobility and multicast easier, and its capacity to uphold distinct net-
work service paradigms. It has only delivered a content-centric transport service
known as Workspace. Although, initial implementation was straightforward, fur-
ther development proved the architecture has serious service limitations. The
first, regards to service extensibility, the use of automata to define a communi-
cation between what instantiates client and main components of SDN applica-
tion leads to a simple language, however every feature addition implies in code
changes in both components. The second, regards to service scalability, architec-
ture application and control are tightly coupled meaning that every DTSA is an
application and also a controller. Additionally, this condition makes the archi-
tecture interface abusive (i.e. prohibiting new versions of the SDN controller),
as it do not use the standard northbound interface, without mention the effort
maintain the distributed applications of DTS synchronized.

4.3 Refactoring ETArch

By refactoring the architecture we eliminated these limitations, whose causes
were only identified as we created models to understand how the code base
652 J. E. Pereira Júnior et al.

reflected upon SDN architectural principles. We enforced the use of northbound

interface between the main component of SDN application and the controller
modules by introducing transformation code. This measure have made applica-
tion and control loosely coupled and respected standard northbound interface.
Further, we have centralized the application that can now communicate with
several controllers through this interface, thus making the specific protocol for
DTSA inter communication no more necessary.
Figure 3 shows the main components of ETArch before Interfacer. Entity-
SBB, Workspace-SBB, Mobility-SBB, NEConnector-SBB, and Openflow-RA are
JAIN SLEE components and represent the main component of SDN application.
TopologyManager, FloodlightProvider, StorageSource, and GenericTranslator
are Floodlight SDN controller modules. The dashed lines delimit SDN planes
showing how ETArch components fit into SDN architecture. Figure 4 shows the

Fig. 3. ETArch components before interfacer

Fig. 4. ETArch components after few interfacer iterations

 Interfacer: A Model-Driven Development Method for SDN Applications 653

main components of ETArch after Interfacer was briefly applied to architecture,

resulting in an analysis and refactoring. Note that there are only controller mod-
ules in control plane and only application modules in application plane. They
must always communicate through northbound interface.

5 Concluding Remarks and Future Work

In this paper we have explored the Interfacer, a method for developing complex
SDN applications in order to deliver high-quality network services. The method
was applied to ETArch, an analysis, that identified ETArch’s limitations, and
a refactoring, that qualitatively improved ETArch’s scalability, were performed
as a case study. Although, service extensibility issues were not improved by
the refactoring, it opened the possibility for future investigation on how the
replacement of automata by grammars can help to produce more extensible and
expressive communication amid SDN application components while minimizing
client application changes.
Another future concern is apply this method to other well known open source
SDN architectures, evaluate how its applications perform in terms of SDN archi-
tectural principles, and propose some guide lines to fixing eventual issues using
models and transformation code.

Acknowledgements. This work was has been partially funded by Coordination for
the Improvement of Higher Education Personnel (CAPES), FAPEMIG and PROP-
P/UFU.

References
1. Casado, M., Foster, N., Guha, A.: Abstractions for software-defined networks. Com-
mun. ACM 57(10), 86–95 (2014). https://doi.org/10.1145/2661061.2661063
2. Open Networking Foundation: Northbound interfaces, June 2018. https://www.
opennetworking.org/sdn-definition/
3. Anderson, C.J., Foster, N., Guha, A., Jeannin, J.-B., Kozen, D., Schlesinger, C.,
Walker, D.: NetKAT: semantic foundations for networks. SIGPLAN Not. 49(1),
113–126 (2014). https://doi.org/10.1145/2578855.2535862
4. Foster, N., Harrison, R., Freedman, M.J., Monsanto, C., Rexford, J., Story, A.,
Walker, D.: Frenetic: a network programming language. SIGPLAN Not. 46(9),
279–291 (2011). https://doi.org/10.1145/2034574.2034812
5. Wang, A., Mei, X., Croft, J., Caesar, M., Godfrey, B.: Ravel: a database-defined
network. In: Proceedings of the Symposium on SDN Research, SOSR 2016, pp.
5:1–5:7. ACM, New York (2016). https://doi.org/10.1145/2890955.2890970
6. Barakat, O.L., Koll, D., Fu, X.: Gavel: software-defined network control with graph
databases. In: 2017 20th Conference on Innovations in Clouds, Internet and Net-
works, ICIN, March 2017, pp. 279–286 (2017)
7. Open Networking Foundation: Intent NBI - definition and principles, October
2016. https://www.opennetworking.org/images/stories/downloads/sdn-resources/
technical-reports/TR-523 Intent Definition Principles.pdf
654 J. E. Pereira Júnior et al.

8. Lopes, F.A., Santos, M., Fidalgo, R., Fernandes, S.: Model-driven networking: a
novel approach for SDN applications development. In: 2015 IFIP/IEEE Interna-
tional Symposium on Integrated Network Management, IM, May 2015, pp. 770–773
(2015)
9. Muzaffar, A.W., Mir, S.R., Anwar, M.W., Ashraf, A.: Application of model driven
engineering in cloud computing: a systematic literature review. In: Proceedings
of the Second International Conference on Internet of Things, Data and Cloud
Computing, ICC 2017, pp. 137:1–137:6. ACM, New York (2017). https://doi.org/
10.1145/3018896.3036380
10. Artac, M., Borovšak, T., Nitto, E.D., Guerriero, M., Perez-Palacin, D., Tamburri,
D.A.: Infrastructure-as-code for data-intensive architectures: a model-driven devel-
opment approach. In: 2018 IEEE International Conference on Software Architec-
ture, ICSA, April 2018, pp. 156–15609 (2018)
11. Verdonck, M., Gailly, F., Pergl, R., Guizzardi, G., Martins, B., Pas-
tor, O.: Comparing traditional conceptual modeling with ontology-driven
conceptual modeling: an empirical study. Inf. Syst. 81, 92–103 (2019).
http://www.sciencedirect.com/science/article/pii/S0306437918303727
12. Pergl, R., Sales, T.P., Rybola, Z.: Towards OntoUML for software engineering:
from domain ontology to implementation model. In: Cuzzocrea, A., Maabout, S.
(eds.) Model and Data Engineering, pp. 249–263. Springer, Heidelberg (2013)
13. Guarino, N., Welty, C.A.: An overview of OntoClean, pp. 201–220. Springer, Hei-
delberg (2009). https://doi.org/10.1007/978-3-540-92673-3 9
14. Guizzardi, G.: Ontological foundations for structural conceptual models. CTIT,
Centre for Telematics and Information Technology (2005)
15. Mtsweni, J.: Exploiting UML and acceleo for developing semantic web services. In:
2012 International Conference for Internet Technology and Secured Transactions,
December 2012, pp. 753–758 (2012)
16. Benouda, H., Azizi, M., Esbai, R., Moussaoui, M.: MDA approach to automate
code generation for mobile applications. In: Kim, K.J., Wattanapongsakorn, N.,
Joukov, N. (eds.) Mobile and Wireless Technologies 2016, pp. 241–250. Springer,
Singapore (2016)
17. Rose, L.M., Matragkas, N., Kolovos, D.S., Paige, R.F.: A feature model for model-
to-text transformation languages. In: Proceedings of the 4th International Work-
shop on Modeling in Software Engineering, MiSE 2012, pp. 57–63. IEEE Press,
Piscataway (2012). http://dl.acm.org/citation.cfm?id=2664431.2664440
18. Guerson, J., Sales, T.P., Guizzardi, G., Almeida, J.P.A.: OntoUML lightweight edi-
tor: a model-based environment to build, evaluate and implement reference ontolo-
gies. In: 2015 IEEE 19th International Enterprise Distributed Object Computing
Workshop, September 2015, pp. 144–147 (2015)
19. de Oliveira Silva, F., Goncalves, M., de Souza Pereira, J., Pasquini, R., Rosa, P.,
Kofuji, S.: On the analysis of multicast traffic over the Entity Title Architecture.
In: 2012 18th IEEE International Conference on Networks, ICON, pp. 30–35 (2012)
20. Cox, J.H., Chung, J., Donovan, S., Ivey, J., Clark, R.J., Riley, G., Owen, H.L.:
Advancing software-defined networks: a survey. IEEE Access 5, 25487–25526
(2017)
 Reducing the IEEE 802.11 Beacon
Overhead in Low Mobility Networks

Gabriel de Carvalho Ferreira(B) , Priscila Solis Barreto, and Eduardo Alchieri

Department of Computer Science, University of Brasilia (UnB),

Brasilia, DF 70910-900, Brazil
[email protected], {pris,alchieri}@unb.br

Abstract. Wireless networks based on the IEEE 802.11 standard use

management traffic for advertising the network, promoting service qual-
ity, device association/dissociation and maintaining connections. Among
the management traffic, the beacon frame is responsible for network
advertising and synchronization. The beacon broadcast is designed for
scenarios where the network devices are assumed to be mobile, but in sce-
narios with low mobility, the aggressive transmission of beacon frames
by the access points (APs) may be unnecessary, as the probability of
changing between networks is very low. This type of management traffic
can be identified as unwanted traffic, defined as a set of unproductive
transmissions on the shared medium. As the number of APs sharing the
same channel increases, the unwanted traffic increases, producing more
collisions and reducing the network efficiency and its potential through-
put. In this work, we present a proposal to modify the 802.11 protocol
to support the dynamic adjustment of the transmission interval between
beacon frames. The proposal is based on a method to identify the mobil-
ity of devices that share the same channel which is used to character-
ize the mobility of the network. The comparison of simulation results,
considering different scenarios, with other related methods shows that
our proposal can be effective to reduce unwanted traffic and increase
throughput in networks with low mobility and medium to high traffic
intensity.

1 Introduction
Nowadays wireless networks are ubiquitous, serving the network for mobile
devices without requiring wires and costly installations. Although the wireless
networks were made presuming the devices would be moving, most of the devices
and their users remain relatively static most of the time, and eventually roam
across different access points (APs).
The major limitations in wireless networks are caused by (1) a small number
of channels, that increases chance of sharing a channel among applications, and
(2) decentralized medium access control, prone to collisions or requiring collision
avoidance that increases overhead and latency. Several works try to increase the
efficiency of spectrum usage for better throughput and network performance,
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 655–666, 2020.
https://doi.org/10.1007/978-3-030-15032-7_55
656 G. de Carvalho Ferreira et al.

either by sharing the channel more efficiently (e.g., [1,7]) or by reducing the
number of collisions and congestion (e.g., [4,19,23]).
The term unwanted traffic is used to define traffic originated by unproduc-
tive activities [17], like keep-alive transmissions that maintain connections. The
reduction of unwanted traffic may improve the performance of congested net-
works, particularly in office and residential buildings, where each office or res-
idence has one or more APs. A traffic analysis at the network layer, based
on the real traffic data collected at the 67th Internet Engineering Task Force
Meeting, shows that more than half of the network traffic in 802.11 networks
comes from control and management transmissions [3], that are used for associ-
ation/dissociation and connection maintenance. The beacon frame, responsible
for network advertisement, is transmitted frequently and always at the lowest
rate supported by the AP for backwards compatibility.
Aiming to reduce unwanted traffic and increase network efficiency, this paper
proposes a modification of the 802.11 protocol considering a low mobility net-
work. In summary, this paper makes the following contributions:

• It proposes a simple method to characterize network mobility in indoor 802.11

networks, without creating network overhead;
• It proposes an algorithm that dynamically adjusts the frequency of beacon
transmissions and is compatible with the standard version. This algorithm
reduces unwanted traffic in 802.11 networks, increasing network efficiency.

This paper is organized as follows. Section 2 presents basic concepts and

related work. Section 3 presents our proposal of mobility characterization and
adjustment of the beacon transmission interval. Section 4 describes and ana-
lyzes simulation results, comparing them with other protocols. Finally, Sect. 5
concludes the paper.

2 Theoretical Concepts and Related Work

2.1 Network Management in IEEE 802.11 Networks

The IEEE 802.11 [10] standard defines three types of frames: management,
control and data. Control frames are used to guarantee frame delivery or
request/grant time to transmit. The management frames are used by devices
to find and establish connections. In this group, the beacon frame is responsi-
ble for network advertisement. If the AP does not advertise its SSID (Service
Set Identifier), it will generate traffic by making devices in the coverage area
transmitting Probe Requests to discover hidden SSIDs and their connection
parameters. The beacon frame is also used for synchronization of the network
using timestamps, waking up devices in energy saving mode so that they can
receive buffered frames, and to control part of the transmission window without
conflicts using the traffic indication maps [10].
Although the beacon frame has multiple important functions, it causes a
significant overhead in specific situations since the beacon transmission interval
 Reducing the IEEE 802.11 Beacon Overhead in Low Mobility Networks 657

is static independently of the network usage and devices mobility. Beacons can be
as high as 16% of the total traffic, depending on the AP model implementation
and network settings [3]. The traffic analysis collected in real environments where
users are predominantly static, i.e., remain at the same place most of the time,
shows that network efficiency may increase by reducing aggressive roaming and
searching for new APs associations [17].

2.2 Network and Device Mobility

Mobile users need to communicate without interruption while moving across
different networks. Efficient support of mobile users requires seamless handover
between network attachment points and network layer mechanisms for uninter-
rupted IP connectivity. In the last years, many works focused on provide support
to complex requirements for mobile environments, considering constraints such
as handover latency time, packet loss during the handover process, signaling
overhead and throughput.
Considering our case of study as an indoor environment with several APs and
network users, to acquire position changes of any device is a realistic alternative
to characterize the network mobility. Although 802.11 networks have not been
designed for positioning, several techniques are available for location estimation.
For example, to measure the distance between two nodes in a wireless network,
the Time-of-Flight (ToF) [21] uses the delay between the start of a transmission
at the first node and the start of receiving the transmission at the other node to
estimate the distance between a pair of antennas in those two nodes. This kind
of measurement is complex as each nanosecond of error translates into 30 cm of
error, and off-the-shelf devices usually only provide high-level access to network
interfaces and can also have higher latency than the transmission itself. Another
technique is the fingerprinting [8], that uses multiple analysis on signal power
and network traffic to estimate the distance and position of the nodes.
Although the previously mentioned indoor techniques may have a good pre-
cision and are very useful for some applications, they require multiple antennas
for ToF (or other approaches based on the Angle-of-Arrival (AoA)), or addi-
tional message exchanges for ToF, or high processing power for fingerprinting.
In applications where the exact position is not crucial, a less precise and simpler
approach can be used to measure distance between the AP and stations. One of
those techniques, which does not impose more traffic overhead in the network,
is based on the received signal strength indication (RSSI), that can be used by
promiscuous devices listening to the channel and estimating distance to nearby
devices that are transmitting [15].

2.3 Related Work

The aggressive transmission of beacons in static networks results in an unneces-
sary overhead, that reduces the overall throughput by compromising the spec-
trum efficiency [12,18] and increases noise to nearby networks that share the
same channel [6]. The default beacon interval used by most manufacturers is of
658 G. de Carvalho Ferreira et al.

a 100 ms. Previous works evaluate the impacts of reducing the beacons trans-
mission frequency, such as increasing the latency of handoff and data transmis-
sion [20], or increasing power consumption since devices power saving strategies
may rely on the beacon frequency [9,20]. However, a contradictory work sug-
gests that the reduction on unnecessary broadcasts (e.g. beacons) can reduce
the power consumption from 18% up to 83%, depending on the application and
device [16].
One technique widely used in sensor networks to reduce the amount of traffic
and preserve battery is the Trickle algorithm (RFC6206), where transmission
windows are adjusted quickly to transmit new and relevant information while
unaltered information allows the network to reduce the number of transmitted
messages. The Trickle algorithm proposes adjusting the interval between bea-
con transmissions accordingly to a timer limited by an arbitrary range and the
‘consistency’ of a received transmission, that serves as a heuristic to trigger the
algorithm.
Other type of technique proposes the creation of a centralized power saving
protocol [22] that dynamically adjusts network parameters, including the beacon
interval, but the analysis was limited to power savings.
Other works [5,14] used RSS (Received Signal Strength) measurements to
calculate the distance between a device and nearby APs, and then derive the
average speed of the device to choose the best moment to start the handoff.
Another way to reduce the overhead caused by beacons other than reducing
their frequency of transmission is by reducing their size [11].

3 Dynamic Adjustment of Unwanted Traffic

Our proposal considers an indoor network 802.11 and defines a dynamic adjust-
ment of beacon transmission frequency using an algorithm that calculates station
(STA) distance. Using the measured distance, the AP identifies two mobility pat-
terns to adapt the beacon transmission frequency: an associated STA is moving
away or a dissociated STA is moving towards the AP. Our proposal must imple-
ment the dynamic adaptation while keeping the association/dissociation delays
within a threshold that allows mitigating the interruption of services during
handoffs.
The network mobility is characterized by Eq. 1. The distance, speed and
direction are used to identify STAs at the border of the service area that may
require special attention. STAs associated to an AP and moving away from it
have a higher probability to disassociate from it. Equation 1 also considers STAs
that are not associated but moving into the AP coverage area, that might request
association with it in the near future.
Algorithm 1 executes the device mobility characterization on the channel.
The procedure starts by calculating the distance between STAs and the AP using
samples collected while listening to devices transmissions (in case of RSSI, AoA
or ToF based approaches) or using an indoor location protocol (lines 2–4). In our
simulations, the distance measurements are derived from the RSSI samples of
 Reducing the IEEE 802.11 Beacon Overhead in Low Mobility Networks 659

Algorithm 1. Adjustment of Beacon Frequency

1:  Sample distance measurements, using RSSI, AoA, ToF, etc
2: for j ← 0 → numSamples do
3: for sta in ST As do
4: sta.dstSamp[j] ← M easureDst(sta)
5:  Consolidate distance measurements
6: for sta in ST As do
7: sta.dst[i] ← Avg(sta.dstSamp)
8: sta.dstErr[i] ← StdDev(sta.dstSamp)
9: sta.dstDif f ← sta.dst[i] − sta.dst[i − 1]
10: sta.dstDif f Err ← AbsSum(sta.dstErr, i, i − 1)
11: sta.moving ← Abs(dstDif f ) > dstDif f Err
12:  Search for moving stations inside the region of interest
13: for sta in ST As do
14: insideROI ← sta.dst > APrange × ROI
15: leave ← sta.dstDif f − sta.dstDif f Err > 0
16: approach ← sta.dstDif f + sta.dstDif f Err < 0
17: if sta.moving ∧ insideROI then
18: if (leave ∧ sta.associated) ∨ (approach ∧ ¬sta.associated) then
19: return True
20: return False

received messages. After sampling distance measurements, we consolidate them

in an average distance and standard deviation of the samples (lines 6–8). With
the average distance and the deviation, we calculate the movement of the station
comparing to the measurement made in the previous interval and compute the
accumulated error (lines 9–10). After, we mark the STA as moving or not (line
11). Finally, we search for a STA moving inside the region of interest – ROI
(line 13–17), and check if it has one of the behaviors we are looking for. If it is
either associated and moving away or dissociated and moving towards the AP
(lines 18–19), then Algorithm 1 returns True, increasing the beacon frequency.
Otherwise, it returns False, keeping the beacon frequency as it is. Notice the
algorithm is executed at the AP, that usually does not have power restrictions.
Equation 1 is equivalent to lines 14–20 of Algorithm 1. As the 802.11 protocol
relies heavily in the beacon interval, we can transmit a new beacon whenever we
need to reduce the interval between beacons, but we have to wait to increase the
interval without introducing more overhead. Equation 3 describes the wait for the
beacon duration before increasing the interval as fbeacon [n−1]×(n−nlastbeacon ),
being n the current interval, nlastBeacon the interval where the last beacon was
sent, and fbeacon [n − 1] the beacon frequency computed in the last interval. The
Eq. 3 also keeps the fbeacon limited to a range, to ensure proper response time
of the algorithm to perceived movement.
Equation 4 defines the dynamic adjustment of the beacon transmission fre-
quency, based on the presence of stations with a specific mobility behavior
(Eq. 1). The frequency of the beacon increases if Eq. 1 returns True and Eq. 3
return False. The frequency of the beacon decreases if Eq. 1 returns False for the
660 G. de Carvalho Ferreira et al.

period of the interval of the last beacon sent (e.g., a beacon interval of 100 ms
will be increased to 200 ms if there is no STA moving with the desired behavior
for 100 ms).
The speed of frequency increase is defined in Eq. 1 by the duration of the sam-
pling interval, by the AP range and the ROI. ROI is a percentage that determines
a ring which STAs contained in it affect the beacon adjustment, reducing false
positives when stations are nearby the AP. The equations guarantee that the
average transmission frequency has an upper limit at the highest frequency (the
initial frequency Eq. 2) and a lower limit at the lowest frequency. The average
frequency of beacon transmission is given by Eq. 5.

mobility[n] = ST A.dst[n] ≥ APrange × ROI

∧[(ST A.dstDif f [n] − ST A.dstDif f Err[n] > 0 ∧ ST A.associated)
∨(ST A.dstDif f [n] + ST A.dstDif f Err[n] < 0 ∧ ¬ST A.associated)] (1)
1
fbeacon [0] = fmaxBeacon = = 10 Hz (2)
0.1 s

transmittedbeacon [n] =
⎧  
⎪
⎪
⎨1 fbeacon [n − 1] × (n − nlastBeacon ) = 1 ∧ fbeacon [n − 1] > fminBeacon
 
⎪
⎪
⎩0 fbeacon [n − 1] × (n − nlastBeacon ) = 1 ∨ fbeacon [n − 1] = fminBeacon
(3)

fbeacon [n − 1] × (1 + mobility[n])
fbeacon [n] = (4)
1 + transmittedbeacon [n − 1]

N −1
1 
fbeaconAverage = fbeacon [n] (5)
N n=0

4 Simulations and Results

This section presents the simulation scenarios and then proceed to the results.
Simulation Scenarios. To evaluate our proposal, we defined three typical sce-
narios of IEEE 802.11 networks: the first scenario (Fig. 1(a)) emulates a domestic
environment, composed of 10 static STAs distributed randomly around the main
AP, a secondary AP connected to the first one and broadcasting the same SSID,
plus a moving STA crossing the service area of the main AP. The second scenario
(Fig. 1(b)) doubles the number of APs and emulates an environment of a small
office. The third scenario (Fig. 1(c)) emulates an office or residential building,
 Reducing the IEEE 802.11 Beacon Overhead in Low Mobility Networks 661

with rooms of 5 × 5 m. Each office has an AP with range limited to 30 m. Each

AP initially has 7 associated STAs, being 5 of them static and 2 mobile, moving
at 30 cm per second. In total, the third scenario has 10 APs and 70 STAs.

(a) Scenario 1 (b) Scenario 2

(c) Scenario 3

Fig. 1. Simulation scenarios

The parameters that control the beacon transmission frequency were config-
ured as follows: the ROI was defined as 85% of the maximum known distance
from the APs; and the sampling frequency, used in Algorithm 1, was set to run
every 10 ms, allowing the beacon transmitting frequency to be increased 64 times
in 70 ms, resulting in an interval between beacons from 6.4 s to 100 ms.
All scenarios were implemented using the NS-3 (Network Simulator 3) and
the traffic is generated with an echo application (RFC862) that sends packets
with a specific size to the main AP, that hosts an echo server application and that
replies with the client payload. For scenarios 1 and 2, the packet sizes are fixed
having a uniform distribution. For scenario 3, the packet sizes and transmission
interval have a normal distribution. The traffic types selected were picked to
represent network traffic with consistent high load, using large packets with a
relatively small inter-arrival time [2,13]. A summary of the traffic workloads used
in simulations is listed in Table 1.
To validate and compare the results, we simulated in all scenarios the static
beacon, named Static Beacon, and the proposed dynamic adjustment, named
Dynamic Beacon. In the third scenario, we also simulated other two dynamic
adjustments based on the Trickle algorithm (RFC6206). A Trickle based algo-
rithm, named Pure Trickle, uses as the trigger heuristic the detection of trans-
662 G. de Carvalho Ferreira et al.

Table 1. Traffic workloads

Type Distribution Packet transmission interval Packet size

A Uniform 100 ms 1 kB
B Uniform 100 ms 10 kB
C Normal 25 ms 1 kB
D Normal 100 ms 1 kB
E Normal 200 ms 1 kB
F Normal 100 ms 5 kB

missions from dissociated STAs that entered the service area. The other Trickle
based algorithm, named Dynamic Trickle, uses our proposed heuristic based on
mobility. The Trickle based approach decreases the interval between beacons to
the minimum when triggered, while our proposed algorithm halves the interval
every sampled interval until reaching the minimum interval.
Analysis of Results. The simulation results for the first two scenarios are
shown in Figs. 2 and 3 and Tables 2 and 3, while the results for the third scenario
are shown in Fig. 4. For Scenario 1 and Traffic A, there was a 77.5% reduction
in the number of transmitted beacons, reducing the transmission time portion
from 5.77% to 1.1%. In this scenario, for Traffic B, there was a 94% reduction
in the number of transmitted beacons, reducing the transmission time 1.81%
to 0.12%. The average throughput increased 3.1% with traffic A and decreased
0.95% with traffic B.

(a) Beacon transmissions (b) Average transmitted rate

Fig. 2. Simulation results for Scenario 1

For Scenario 2 and Traffic A, there was a 93.62% reduction in the number
of transmitted beacons, reducing the related transmission time from 52.54% to
5.42%. For Traffic B, there was a 92.2% reduction in the number of transmit-
ted beacons, reducing the related transmission time from 5.91% to 1.32%. The
average throughput increased 61.9% and 31.57% with traffics A and B.
For Scenario 3, the airtime occupied by beacons (Fig. 4(a)) for the traffic D
dropped from 95%, with the Static Beacon interval of 100 ms, to 94%, 4% and
 Reducing the IEEE 802.11 Beacon Overhead in Low Mobility Networks 663

(a) Beacon transmissions (b) Average transmitted rate

Fig. 3. Simulation results for Scenario 2

(a) Beacon transmissions (b) Average transmission rate

(c) Average association delay (ms) (d) Maximum association delay (ms)

Fig. 4. Simulation results for Scenario 3

3% for the Pure Trickle, Dynamic Trickle and Dynamic Beacon, respectively.
The average transmitted rate (Fig. 4(b)) followed that pattern, jumping from
2.6 Mbps to 3.02, 22.55 and 23.56 Mbps, respectively. For Traffic E, the airtime
occupied by beacon transmissions dropped from 97%, to 96%, 38% and 37%
for the Pure Trickle, Dynamic Trickle and Dynamic Beacon, respectively. The
average transmitted rate followed that pattern, jumping from 1.86 Mbps to 2.09,
21.8 and 17.95 Mbps, respectively.
Another important metric is how changing the beacon affects the handoff.
For this, we calculate the average and maximum association delays. Association
delay is the difference between an association response and the previous beacon
of an AP. The Scenario 3 average association delays show that the Dynamic
Beacon is better than the Trickle when using the same heuristic. Figures 4(c)
and (d) also show some spikes of the association delay for traffic D, indicating
that the mobility heuristic for this traffic workload has a higher handoff latency,
which will increase the need to execute active scanning.
664 G. de Carvalho Ferreira et al.

Table 2. Scenario 1 results

Static Dynamic
Beacon interval 100 ms 200 ms 100 ms-6,4 s 200 ms-6,4 s
A Transmitted Beacons 400 200 90 57
Beacon transmission time (ms) 259.2 129.60 58.32 36.94
Total transmission time (s) 4.49 4.38 4.46 4.35
Time portion (%) 5.77 2.96 1.31 0.85
Beacon traffic (kB) 33.20 16.60 7.47 4.73
Total traffic (MB) 3.93 3.93 4.02 3.92
Traffic portion (%) 0.85 0.42 0.19 0.12
Throughput (Mbps) 6.99 7.17 7.21 7.21
B Transmitted Beacons 400 200 24 19
Beacon transmission time (ms) 259.20 129.60 15.55 12.31
Total transmission time (s) 14.28 14.96 12.68 13.48
Time portion (%) 1.81 0.87 0.12 0.09
Beacon traffic (kB) 33.20 16.60 1.99 1.58
Total traffic (MB) 13.17 13.36 11.58 12.63
Traffic portion (%) 0.25 0.12 0.02 0.01
Throughput (Mbps) 7.37 7.14 7.30 7.50

Table 3. Scenario 2 results

Static Dynamic
Beacon interval 100 ms 200 ms 100 ms-6,4 s 200 ms-6,4 s
A Transmitted Beacons 2400 1200 153 140
Beacon transmission time (ms) 1555.20 777.60 99.14 90.72
Total transmission time (s) 2.96 2.55 1.83 1.71
Time portion (%) 52.54 30.53 5.42 5.29
Beacon traffic (kB) 202.40 101.20 12.90 11.80
Total traffic (MB) 1.55 1.64 1.56 1.49
Traffic portion (%) 13.03 6.15 0.83 0.79
Throughput (Mbps) 4.20 5.16 6.80 6.97
B Transmitted Beacons 2158 1139 190 173
Beacon transmission time (ms) 1398.38 738.07 123.12 112.10
Total transmission time (s) 23.65 9.21 9.32 8.38
Time portion (%) 5.91 8.01 1.32 1.34
Beacon traffic (kB) 181.83 96.01 16.01 14.59
Total traffic (MB) 19.11 9.73 9.90 9.72
Traffic portion (%) 0.95 0.98 0.16 0.15
Throughput (Mbps) 6.46 8.45 8.50 9.28
 Reducing the IEEE 802.11 Beacon Overhead in Low Mobility Networks 665

The results for the scenarios show that the Dynamic Beacon is capable of
reducing unwanted traffic while maintaining full network functionality. More-
over, the results show that a higher network efficiency can be obtained without
significantly impacting the handoff latency for transmitting devices.

5 Conclusions

In this work, we proposed an algorithm for 802.11 networks to support the

dynamic adjustment of the beacon transmission interval. The proposed algorithm
uses the distance between STAs and APs to adjust the interval between beacon
transmissions. The algorithm was evaluated in three simulation scenarios and
compared with other proposals. The simulation results shown that the dynamic
adjustment of the beacon transmission interval is capable of reducing unwanted
traffic and improve the network efficiency, mainly in densely populated locations
with low mobility, as office and residential buildings.

Acknowledgements. This work was partially supported by RNP/CTIC (Brazil)

through project 5G-RANGE.

References
1. Bellalta, B.: IEEE 802.11ax: high-efficiency WLANS. IEEE Wirel. Commun. 23(1),
38–46 (2016). https://doi.org/10.1109/mwc.2016.7422404
2. Benson, T., Akella, A., Maltz, D.A.: Network traffic characteristics of data centers
in the wild. In: 10th Annual Conference on Internet Measurement. ACM Press
(2010)
3. Bento, T.F.: Um estudo das propriedades do tráfego em redes sem fio e seu impacto
na avaliação de desempenho. Master’s thesis. Universidade de Brası́lia (2011)
4. Chatzimisios, P., Boucouvalas, A.C.: Optimisation of RTS/CTS handshake in
IEEE 802.11 wireless LANs for maximum performance. In: IEEE Globecom Work-
shops (2004)
5. Chien, S.F., Liu, H., Low, A.L.Y., Maciocco, C., Ho, Y.L.: Smart predictive trigger
for effective handover in wireless networks. In: 2008 IEEE International Conference
on Communications. IEEE (2008). https://doi.org/10.1109/icc.2008.416
6. Duda, A.: Understanding the performance of 802.11 networks. In: 19th IEEE
International Symposium on Personal, Indoor and Mobile Radio Communications
(2008)
7. Gilani, M.H.S., Sarrafi, I., Abbaspour, M.: An adaptive CSMA/TDMA hybrid
MAC for energy and throughput improvement of wireless sensor networks. Ad
Hoc Netw. 11(4), 1297–1304 (2013). https://doi.org/10.1016/j.adhoc.2011.01.005
8. He, S., Chan, S.H.G.: Wi-Fi fingerprint-based indoor positioning: recent advances
and comparisons. IEEE Commun. Surv. Tutor. 18(1), 466–490 (2016)
9. Hess, A., Hyytia, E., Ott, J.: Efficient neighbor discovery in mobile opportunistic
networking using mobility awareness. In: 2014 Sixth International Conference on
Communication Systems and Networks, COMSNETS. IEEE (2014)
666 G. de Carvalho Ferreira et al.

10. IEEE: IEEE standard for information technology–telecommunications and infor-

mation exchange between systems local and metropolitan area networks–specific
requirements - part 11: Wireless LAN medium access control (MAC) and physical
layer (PHY) specifications. https://doi.org/10.1109/ieeestd.2016.7786995
11. Khorov, E., Lyakhov, A., Krotov, A., Guschin, A.: A survey on IEEE 802.11ah:
an enabling networking technology for smart cities. Comput. Commun. 58, 53–69
(2015). https://doi.org/10.1016/j.comcom.2014.08.008
12. Kwak, J., Lee, H., Lee, K.: A study on the airtime occupied by beacon frame in
802.11 hotspot environment. Proc. KICS 12, 11–12 (2012)
13. Megyesi, P., Molnár, S.: Analysis of elephant users in broadband network traffic.
In: Lecture Notes in Computer Science, pp. 37–45. Springer, Heidelberg (2013)
14. Naeem, B., Nyamapfene, A.: Seamless vertical handover in WiFi and WiMAX
networks using RSS and motion detection: an investigation. Pac. J. Sci. Technol.
12(1), 298–304 (2011)
15. Parameswaran, A.T., Husain, M.I., Upadhyaya, S., et al.: Is RSSI a reliable param-
eter in sensor localization algorithms: an experimental study. In: Field Failure Data
Analysis Workshop, F2DA 2009, vol. 5. IEEE (2009)
16. Peng, G., Zhou, G., Nguyen, D.T., Qi, X., Lin, S.: HIDE: AP-assisted broadcast
traffic management to save smartphone energy. In: 2016 IEEE 36th International
Conference on Distributed Computing Systems, ICDCS. IEEE (2016)
17. Raghavendra, R., Belding, E.M., Papagiannaki, K., Almeroth, K.C.: Unwanted link
layer traffic in large IEEE 802.11 wireless networks. IEEE Trans. Mob. Comput.
9(9), 1212–1225 (2010). https://doi.org/10.1109/tmc.2010.88
18. Rodrig, M., Reis, C., Mahajan, R., Wetherall, D., Zahorjan, J.: Measurement-based
characterization of 802.11 in a hotspot setting. In: Proceeding of the 2005 ACM
SIGCOMM Workshop on Experimental Approaches to Wireless Network Design
and Analysis, E-WIND 2005. ACM Press (2005). https://doi.org/10.1145/1080148.
1080150
19. Sanabria-Russo, L., Barcelo, J., Bellalta, B., Gringoli, F.: A high efficiency MAC
protocol for WLANs: providing fairness in dense scenarios. IEEE/ACM Trans.
Netw. 25(1), 492–505 (2017). https://doi.org/10.1109/tnet.2016.2587907
20. Sati, S., Graffi, K.: Adapting the beacon interval for opportunistic network commu-
nications. In: 2015 International Conference on Advances in Computing, Commu-
nications and Informatics, ICACCI. IEEE (2015). https://doi.org/10.1109/icacci.
2015.7275576
21. Schauer, L., Dorfmeister, F., Maier, M.: Potentials and limitations of WIFI-
positioning using time-of-flight. In: International Conference on Indoor Positioning
and Indoor Navigation. IEEE (2013). https://doi.org/10.1109/ipin.2013.6817861
22. Xie, Y., Luo, X., Chang, R.K.C.: Centralized PSM: an AP-centric power sav-
ing mode for 802.11 infrastructure networks. In: IEEE Sarnoff Symposium. IEEE
(2009)
23. Zhou, H., Li, B., Yan, Z., Yang, M.: A channel bonding based QoS-aware OFDMA
MAC protocol for the next generation WLAN. Mob. Netw. Appl. 22(1), 19–29
(2016). https://doi.org/10.1007/s11036-015-0670-8
 AXARPS: Scalable ARP Snooping Using
Policy-Based Mirroring of Core Switches

Motoyuki Ohmori1(B) , Naoki Miyata1 , and Ichiroh Suzuta2

1
Tottori University, Koyama-minami, Tottori 680-8550, Japan
{ohmori,miyata}@tottori-u.ac.jp
2
ALAXALA Networks Corporation, Kawasaki, Kanagawa 212-0058, Japan
[email protected]

Abstract. In order to handle a computer security incident or network

failure, it is important to grasp a list of pairs of IP and MAC addresses
of hosts. Ones may then traditionally poll an Address Resolution Pro-
tocol (ARP) table of a core switch at an interval using SNMP or other
methods. This traditional method based upon polling, however, has two
major drawbacks that (1) some pairs of IP and MAC addresses may not
be obtained and (2) incurs a heavy load on a core switch. This paper then
proposes AXARPS that is the novel scalable ARP snooping to build a
list of pairs of IP and MAC addresses. AXARPS can avoid missing pairs
of IP and MAC addresses by monitoring all ARP traffic instead of the
traditional method. AXARPS also can reduce a CPU load on a recent
high-end core switch by approximately 20% in comparison with the tra-
ditional method. AXARPS is scalable because AXARPS incurs no addi-
tional CPU load even the number of hosts increases. AXARPS employs
a policy-based mirroring of a switch that mirror traffic that matches a
specified filter. The policy-based mirroring can then mirror ARP traffic
only, and reduce a load on an ARP parsing server.

1 Introduction
In order to handle a security incident or network failure, it is important to grasp
a list of pairs of IP addresses and MAC addresses of hosts. To this end, ones
may poll a Address Resolution Protocol (ARP) [9] table of a core switch at an
interval using SNMP or other methods. We have, however, experienced that a
MAC address of an IP address was not correctly obtained. The cause was that
the IP address was duplicately assigned to different hosts. One of them was
authorized but another was not. The unauthorized host then induced a security
incident, and it was difficult to identify the host. It can be said that polling an
ARP table is not accurate.
This paper then proposes AXARPS that is the novel scalable ARP snooping
to build a list of pairs of IP addresses and MAC addresses. A challenge of this
paper is to build the list without any extra load on a core switch. AXARPS
employs a policy-based mirroring [1] of a core switch that mirror traffic that
matches a specified filter. AXARPS is, therefore, passive snooping, and incur
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 667–676, 2020.
https://doi.org/10.1007/978-3-030-15032-7_56
668 M. Ohmori et al.

almost no load on a core switch while an active snooping or active MAC address
table polling incur much loads on a core switch. In addition, AXARPS can reduce
a load on an ARP parsing server because the policy-based mirroring can mirror
ARP traffic only. AXARPS is scalable because AXARPS incurs no additional
CPU load even the number of hosts increases.
The rest of this paper is organized as follows. Section 2 presents existing ARP
table polling and ARP table snooping and their problems. Section 3 proposes
AXARPS. Section 4 explains a prototype implementation of AXARPS. Section 5
then evaluates AXARPS using the prototype implementation in an actual cam-
pus network. Section 6 discusses validity of pre-conditions in this paper and fur-
ther applied usage of AXARPS. Section 7 refers to related work. Section 8 finally
concludes this paper.

2 Problems of Existing ARP Table Polling and Watching

2.1 Problems of ARP Table Polling
A traditional way to obtain a list of pairs of IP addresses and MAC addresses
in a network is to poll an ARP table of a core switch. A polling server remotely
polls an ARP table of a core switch using SNMP or Command Line Interface
(CLI). Problems of this method are:
1. an ARP table polling cannot track a quick IP address change less than an
polling interval, and
2. imposes a heavy load on a core switch.
Regarding the former, a polling interval is traditionally 5 min or longer. 5 min
would be minimal interval, and even the 5-minute interval cannot track a quick
IP address change. We have actually experienced that an unauthorized host was
assigned an IP address to its NIC for very short duration within 5 min while an
authorized host was always assigned the IP address. A pair of a MAC address
and an IP address of the unauthorized host could not be then listed in our ARP
table tracking. The unauthorized host, unfortunately, caused a security incident,
and we could not immediately identify the unauthorized host this year, 2018.
Regarding the latter, we have experienced more than 20% CPU load increase
of a core switch when we polled an ARP table from the core switch that was
AX8616 which is made by AlaxalA Networks Corporation. Figure 1 shows a CPU
load of a core switch without ARP table polling a day where about unique 6000
IP addresses were assigned to hosts. In Fig. 1, CPU load 1/1 and 1/2 represent
main and backup CPUs, respectively. The backup CPU load is always about 1%
because backup CPU does not process any traffic in authors’ configuration. We,
hereafter, focus on the main CPU load only. As shown in Fig. 1, the CPU load
was basically 1% or less. On the other hand, Fig. 2 shows a CPU load of the
same core switch with ARP table polling using SNMP a day. An polling interval
was 5 min. As shown in Fig. 2, the CPU load dramatically increased, and the
maximum CPU load of that day was 31%. It can be then said that ARP table
polling imposes a heavy load on a core switch.
 AXARPS: Scalable ARP Snooping 669

As described above, an ARP table polling table has critical problems, and
these problems should be solved or mitigated.

Latest Min. Avg. Max.

avg.
avg.

Fig. 1. CPU load of a core switch without ARP table polling.

Latest Min. Avg. Max.

avg.
avg.

Fig. 2. CPU load of a core switch with ARP table polling using SNMP.

2.2 Problems of ARP Watching

Another way to obtain a list of pairs of IP addresses and MAC addresses in a

network is to watch an ARP. The most famous implementation of this method
is arpwatch [7]. Employing arpwatch, ones can collect pairs of IP addresses and
MAC addresses of hosts that broadcasts ARP requests into a network. An ARP
request is usually broadcasted and it includes a MAC address and IP address
of a host which is sending an ARP request. This way, however, cannot avoid a
malicious host to escape from being collected its MAC address. This is because
that a malicious host can statically configures an ARP entry for a default gateway
in order to avoid to emit an ARP request.
670 M. Ohmori et al.

In order to avoid this kind of a malicious host escape, ones may be able to
capture all traffic and snoop all ARP requests and responses. It is, however,
difficult to implement because all traffic in a network should be snooped and
many snooping servers are necessary.
As described above, an ARP watching has flaws of security and scalability.

3 AXARPS

This section presents AXARPS that is the novel scalable ARP snooping and
solves all issues described in Sect. 2. A basic idea of AXARPS is very simple:

1. mirroring only ARP traffic by employing policy-based mirroring [1], and

2. parsing ARP requests and replies.

Policy based mirroring is a traffic mirroring function which improves a tra-

ditional mirroring function of a core switch. A traditional mirroring can mirror
all traffics on a specific VLAN or specific physical port. A traditional mirroring,
however, cannot mirror only a specific protocol traffic, i.e., cannot restrict mir-
rored traffic based upon a protocol. On the other hand, policy-based mirroring
can selectively mirror traffic by protocol.
Figure 3 depicts an overview of AXARPS system. In Fig. 3, a core switch
has multiple VLANs configured, and those VLANs are necessary to be snooped.
ARP requests and replies on those VLANs are mirrored to an ARP snooping
server. An ARP snooping server parses ARP packets, generates syslog messages,
and sends these syslog messages to a logging server. A logging server then stores
syslog messages in order to provide a user with a pair of an IP address and a
MAC address when an incident or network failure occurs.

Fig. 3. Overview of AXARPS system.

 AXARPS: Scalable ARP Snooping 671

4 Implementation
This section presents our first prototype implementation of AXARPS.
We have configured policy-based mirroring on AX8616 which is made by
AlaxalA Networks Corporation. We have configured 220 VLANs on which all
ARP traffic were to be mirrored to an ARP snooping server via a physical GbE
link using advanced access-list as shown in Fig. 4.

Fig. 4. An example of policy-based mirroring configuration.

We have then prepared a physical server, not virtual machine, as an ARP

snooping server. We have then implemented an ARP snooping server using
python3 [10]. In order to parse ARP packets, we have utilized Scapy [8] that
provides simple but strong APIs to parse ARP packets. An ARP snooping server
then extracts a source MAC address and IP address included in an ARP packet.
An ARP snooping server then generate a syslog message as shown in Fig. 5. An
ARP snooping server then sends a generated syslog message to a logging server.
We have implemented a logging server using fluentd [4] and mongoDB [6] in
order to shorten a search time on an incident or network failure.

Fig. 5. An example of syslog output of AXARPS.

5 Evaluations
This section evaluates AXARPS using our prototype implementation. This
section firstly introduces evaluation environment. This section then shows the
number of ARP requests and replies in an actual environment in order to make
sure that our prototype implementation works properly. This section then proves
that core switch load increase of AXARPS is smaller. This section then presents
how big storage is necessary to hold all pairs of IP addresses and MAC addresses
of all hosts.
672 M. Ohmori et al.

5.1 Evaluation Environment

We have implemented the scalable ARP snooping at a core switch, AX8616, in
our Koyama campus network in Tottori city where about unique 6000 hosts can
be seen at maximum. The core switch accommodates all wireless traffic using
wireless LAN controller and all (i.e., wired and wireless) traffic goes through
this core switch to the Internet. The external link to the Internet is 1 Gbps and
almost all internal links are 10 Gbps in the core switch. An ARP expiry timer
on the core switch is configured to be 5 min. The reason of this configuration is
not clear, and previous network vendor might configure in order to minimize an
ARP table size. We have then installed one physical ARP snooping server, and
its specification is shown in Table 1.

Table 1. ARP snooping server specification.

CPU Intel(R) Xeon(R) CPU E5506 2.13 GHz (8 core)

Memory 8 GB
OS CentOS Linux release 7.5.1804 (64bit)

5.2 ARP Requests and Replies

In order to make sure that our prototype implementation works properly, we
here dig into the number of ARP requests and replies that are snooped and
stored. Figure 6 shows the number of ARP requests and replies per hour from
2018/10/11 9:00 to 2018/11/3 4:00. The maximum number is 2,047,676 on
2018/10/25 15:00. As shown in Figure 6, there are more ARP requests and replies
on week days than ones on holidays. We have made sure that all MAC addresses
were properly stored, and there was no dropped MAC address except for a mis-
operation.
On the other hand, we have made sure an effectiveness of policy-based mirror-
ing. When policy-based mirroring was not employed, i.e., all traffic was mirrored,
many packets were dropped and many ARP packets were then dropped. It can
be, therefore, said that AXARPS works better than a traditional ARP watching
method that needs all traffic is mirrored.

5.3 Increase of Core Switch Load

As described in Sect. 2.1, a traditional ARP table polling imposes a heavy load on
a core switch. In order to make sure that AXARPS incurs no heavy load on a core
switch, we here check to see if a CPU load is increased or not when AXARPS
is enabled. Figure 7 shows a CPU load of a core switch on a day, 2018/9/3,
when AXARPS was enabled. AXARPS was enabled on approximately 13:00. In
Fig. 7, ones cannot see any spikes around 13:00. It can be, therefore, said that
AXARPS incurs no heavy load on a core switch while a traditional ARP table
polling increases a CPU load by more than 30% under some circumstances.
 5.4
4:00).
500000

0
1000000
2000000
2500000

1500000
2018/10/11 9:00
2018/10/11 13:00
2018/10/11 17:00
2018/10/11 21:00
2018/10/12 1:00
2018/10/12 5:00
2018/10/12 9:00
2018/10/12 13:00
2018/10/12 17:00
2018/10/12 21:00
2018/10/13 1:00
2018/10/13 5:00
2018/10/13 9:00
2018/10/13 13:00
2018/10/13 17:00
2018/10/13 21:00
2018/10/14 1:00
2018/10/14 5:00
2018/10/14 9:00
2018/10/14 13:00

Data Size
2018/10/14 17:00
2018/10/14 21:00
2018/10/15 1:00
2018/10/15 5:00
2018/10/15 9:00
2018/10/15 13:00
2018/10/15 17:00
2018/10/15 21:00
2018/10/16 1:00
2018/10/16 5:00
2018/10/16 9:00
2018/10/16 13:00

avg.
avg.
2018/10/16 17:00
2018/10/16 21:00
2018/10/17 1:00
2018/10/17 5:00
2018/10/17 9:00
2018/10/17 13:00
2018/10/17 17:00
2018/10/17 21:00
2018/10/18 1:00
2018/10/18 5:00

Latest
2018/10/18 9:00
2018/10/18 13:00
2018/10/18 17:00
2018/10/18 21:00
2018/10/19 1:00
2018/10/19 5:00
2018/10/19 9:00

Min.
2018/10/19 13:00
2018/10/19 17:00
2018/10/19 21:00
2018/10/20 1:00
2018/10/20 5:00
2018/10/20 9:00
2018/10/20 13:00
2018/10/20 17:00
2018/10/20 21:00

Avg.
2018/10/21 1:00
2018/10/21 5:00
2018/10/21 9:00
2018/10/21 13:00
2018/10/21 17:00
2018/10/21 21:00
2018/10/22 1:00
2018/10/22 5:00

Max.
2018/10/22 9:00
2018/10/22 13:00
2018/10/22 17:00
2018/10/22 21:00
2018/10/23 1:00
2018/10/23 5:00
2018/10/23 9:00
2018/10/23 13:00
2018/10/23 17:00
2018/10/23 21:00
2018/10/24 1:00
2018/10/24 5:00
2018/10/24 9:00
2018/10/24 13:00
2018/10/24 17:00
2018/10/24 21:00
2018/10/25 1:00
2018/10/25 5:00
2018/10/25 9:00
2018/10/25 13:00
2018/10/25 17:00
2018/10/25 21:00
2018/10/26 1:00
2018/10/26 5:00
2018/10/26 9:00
2018/10/26 13:00
2018/10/26 17:00
2018/10/26 21:00

30 GB of storage in order to hold about one month data.

2018/10/27 1:00
2018/10/27 5:00
2018/10/27 9:00
2018/10/27 13:00
2018/10/27 17:00
2018/10/27 21:00
2018/10/28 1:00
2018/10/28 5:00
2018/10/28 9:00
2018/10/28 13:00
2018/10/28 17:00
2018/10/28 21:00
2018/10/29 1:00
2018/10/29 5:00

one of MongoDB even though syslog text file stores longer data.
2018/10/29 9:00
2018/10/29 13:00
2018/10/29 17:00
2018/10/29 21:00
2018/10/30 1:00
2018/10/30 5:00
2018/10/30 9:00
2018/10/30 13:00
Fig. 7. CPU load of a core switch with scalable ARP snooping. 2018/10/30 17:00
2018/10/30 21:00
2018/10/31 1:00
2018/10/31 5:00
2018/10/31 9:00
2018/10/31 13:00
AXARPS: Scalable ARP Snooping

2018/11/1 16:00
2018/11/1 20:00
2018/11/2 0:00
2018/11/2 4:00
2018/11/2 8:00
2018/11/2 12:00
2018/11/2 16:00
2018/11/2 20:00
2018/11/3 0:00
2018/11/3 4:00

The syslog text file size is about 19 GB that stores all ARP requests and replies
indices but that is future work. As shown in Table 2, we need approximately
673

from 2018/9/3 13:11 to 2018/11/7 22:00. The size of syslog text file is less than
sizes of mongoDB. Note that we here have not configured mongoDB to have
Fig. 6. The number of ARP requests and replies (from 2018/10/11 9:00 to 2018/11/3

We here examine how big store is necessary to hold all pairs of IP addresses and

On the other hand, we also stored the same data in normal syslog text file.
indices such as an IP address and MAC address. In order to reduce a searching
data from 2018/10/11 9:00 to 2018/11/3 4:00. Table 2 presents data and index
MAC addresses. Our logging server that employs fluentd and mongoDB held

time on an incident and network failure, we would have better to have these
674 M. Ohmori et al.

Table 2. Data size of the scalable ARP snooping on mongoDB.

The number of logs 256,942,822

Data size 24,167,277,036 B (approx. 24 GB)
Index size 3,713,732,608 B (approx. 3.7 GB)

6 Discussions

6.1 Operation Cost

As shown in Fig. 4, we currently have to specify each VLAN ID for each VLAN.
There is no way to specify all VLAN to be mirrored, and this configuration might
be forgotten by mistake when a new VLAN is created. This kind of mistakes
might cause the case where a MAC address cannot be resolved from a given IP
address on an incident. In order to avoid this kind of mistakes, we may need
to automate an operation to create new VLAN. Or a configuration of an access
list for policy-based mirroring should be improved so that a VLAN ID can be
omitted when a configure policy-based mirroring on a physical port.

6.2 Data Size and Contraction

In this paper, AXARPS stores all ARP requests and replies logs. These logs may
be, however, contracted. For example, regarding an incident, the most important
is which IP address is assigned to which host at the specified time. All logs of
all ARP requests and replies are unnecessary for this purpose. Before an ARP
entry expires, the ARP entry continues to exist on an ARP table unless the
ARP entry is overridden by another MAC address. Only latest ARP request or
reply is, therefore, necessary before an ARP entry expires. If such contraction
is implemented, data size can be smaller than ones presented in this paper. The
data size contraction is future work.
On the other hand, ones may be able to detect anomaly when all ARP
requests and replies are stored. It is still unclear if the contraction must be
implemented in all cases. We will dig into the ARP requests and replies as
future work in order to make sure that all ARP requests and replies are worthy
of being held.

6.3 MongoDB and File

As described in Sect. 5.4, MongoDB requires more storage than simple file log
while MongoDB search is relatively faster then ones of simple file log. Both of
MongoDB and file log have pros and cons, and it is difficult to determine which
one is better. We here discuss when we should use mongoDB or simple file log.
Regarding an incident or network failure, it should be handled as soon as
possible. In such case where a quick handling is necessary, MongoDB might be
 AXARPS: Scalable ARP Snooping 675

better. On the other hand, simple file log is better to check very old log for some
reasons such that ones need to check to see what purpose an IP address is used
for.

7 Related Work
PFU limited has released iNetSecSF [5] that snoops ARP packets for security on
2017. Their sensor that snoops ARP packets can accommodate only 32 VLANs
at same time. If ones would like to snoop ARP packets on more VLANs, ones
need to install more their sensors. AXARPS is, therefore, more scalable than
iNetSecSF.
AlaxalA Networks Corporation has released AX-Security-Controller (AX-
SC) [2] on 2017 that can store a list of an IP address and a MAC address. AX-SC,
however, employs a traditional method using SNMP to periodically polls ARP
table from a core switch. AX-SC cannot then track a quick IP address change
as described in Sect. 2.1.
Tokyo University of Agriculture and Technology has also implemented auto-
mated suspicious host isolation [3,11]. They, however, adopted, AX-SC, and have
the similar problems as described above.

8 Concluding Remarks
This paper has proposed AXARPS that it the novel scalable ARP snooping
using policy-based mirroring. AXARPS solves critical issues that traditional
ARP table polling and watching methods have. Employing policy-based mirror-
ing, only ARP traffic can be mirrored, and then it is getting easier to parse all
ARP packets without any packet loss.

References
1. AlaxalA Networks Corporation: Policy based mirroring (2016). (in Japanese).
http://www.alaxala.com/jp/solution/network/pbm/pbm/. Accessed 7 Nov 2018
2. ALAXALA Networks Corporation: Ax-security-controller (2017). http://www.
alaxala.com/jp/news/press/2017/20170601.html. Accessed: 03 June 2017
3. ALAXALA Networks Corporation: Case study: Tokyo university of agriculture
and technology (2017). https://www.alaxala.com/jp/introduce/case36/. Accessed
01 Oct 2018
4. Fluentd Project: fluentd (2010). https://www.fluentd.org/. Accessed 07 Oct 2018
5. iNetSecSF. Sinetsec sf: Detecting security risk and blocking (2017). (in japanese).
https://www.pfu.fujitsu.com/inetsec/products/sf/. Accessed 7 Nov 2018
6. MongoDB, Inc.: mongodb (2018). https://www.mongodb.com/. Accessed 07 Oct
2018
7. Network Research Group: Lawrence Berkeley National Laboratory. arpwatch
(2009). https://ee.lbl.gov/. Accessed 7 Nov 2018
8. Philippe Biondi and the Scapy community. Scapy: Packet crafting for Python2 and
Python3 (2018). https://scapy.net/. Accessed 7 Nov 2018
676 M. Ohmori et al.

9. Plummer, D.: Ethernet address resolution Protocol: or converting network Protocol

addresses to 48.bit Ethernet address for transmission on Ethernet Hardware. RFC
826 (Standard), November 1982. Updated by RFCs 5227, 5494
10. Python Software Foundation: Python (2001). https://www.python.org/. Accessed
7 Nov 2018
11. Tsujisawa, T., Sakurada, T., Segawa, H., Kawamura, Y., Mishima, K., Hagiwara,
Y.: A challenge for full deployment both 802.1x authentication and an automati-
cally isolation function in a campus network Task and correspondence in the oper-
ation. J. Acad. Comput. Netw. 22(1), 36–43 (2018)
 DCS-MAC: A Distributed Cross-Layer
Communication Protocol for Directional
Sensor Networks

Shamanth Nagaraju1(B) , V. Sreejith1 , Nipun Sood1 , Lucy J. Gudino2 ,

Mehul Kasliwal1 , and Rajas Kejriwal1
1
Department of Computer Science and Information Systems,
BITS Pilani K. K. Birla Goa Campus, Zuarinagar, Sancoale, Goa, India
[email protected], [email protected], [email protected],
[email protected], [email protected]
2
Department of Computer Science and Information Systems, BITS Pilani,
Pilani, India
[email protected]

Abstract. Employing directional antenna in Wireless Sensor Networks

(WSNs) constitute Directional Sensor Networks (DSNs) which provides
potential benefits such as longer coverage range, reduced interference,
diminished power requirement and spatial reusability. In this paper, we
propose a Distributed Cross-layer Sectored-antenna Medium Access Con-
trol (DCS-MAC) protocol. The proposed DCS-MAC employ distributed
Time Division Multiple Access (TDMA) for MAC and Distributed Slow-
Scan Routing (DSSR) protocol for routing purpose. The DCS-MAC elim-
inates the need for an exclusive neighbor discovery protocol as DSSR
protocol itself serves this purpose. The performance of DCS-MAC is
simulated using the Cooja simulator with directional antenna support.
The simulation results reveal that DCS-MAC display reduced slot allo-
cation delay, lower communication overhead and higher packet delivery
ratio.

1 Introduction

A Wireless Sensor Network (WSN) is a type of ad-hoc network which consists of

nodes that are capable of sensing the environment, processing the sensed data
and communicating processed data to other nodes in the network. This inno-
vative technology is exciting with unlimited potential for abundant application
areas including military, environmental, medical, home automation, transporta-
tion, entertainment, crisis management etc. The use of the directional antenna
in the field of WSN constitutes the Directional Sensor Network (DSN). In gen-
eral, the directional antenna radiates in a particular direction with limited
beamwidth. Hence, the perks of using DSN include lower interference, dimin-
ished power requirements, longer transmission range and better spatial reuse
[1]. The DSN employs two types of directional antennas: steerable beam and
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 677–689, 2020.
https://doi.org/10.1007/978-3-030-15032-7_57
678 S. Nagaraju et al.

switched beam. In Steerable beam, the beam can be steered to any direction of
interest while simultaneously nulling interfering signals and maximizing recep-
tion quality of the intended signal which comes at the cost of employing complex
signal processing algorithms. On the other hand, switched beam antenna is easy
to implement, as it is made up of multiple fixed beams. A simple Radio Fre-
quency (RF) switch can be used to access a particular beam. The switched
beam antenna also known as sectored antenna avoids the need for carrying out
complex signal processing operations.
Establishing communication among neighboring nodes is more challenging
in DSN due to the limited coverage of the directional antenna. This essentially
leads to two major challenges: neighbor discovery and deafness. In traditional
WSN which employs omnidirectional antenna, neighbor discovery is trivial to
implement as simple broadcast mechanism is sufficient to identify the neighbor-
ing nodes, whereas, DSN needs an explicit neighbor discovery algorithm. Three
types of neighbor discovery are in practice for DSN, namely, (i) one-way [2,3] (ii)
two-way [4,5] and (iii) three-way handshaking [6] mechanisms. In the one-way
broadcast mechanism, the nodes with sector antenna broadcast HELLO pack-
ets containing its location and antenna orientation information through all the
sectors. The receiver node upon reception of these packets identifies the sender
node and the orientation in which it received the packet. However, due to the
lack of acknowledgment from the neighboring nodes, the sender node fails to
discover the neighboring nodes. This issue necessitated the need for a feedback
mechanism, which is accomplished through two-way or three-way handshaking
mechanisms [7]. In two-way handshaking mechanism, for every HELLO packet
transmitted by the sender, the receiver node responds with the REP LY packets,
facilitating the neighbor discovery at both the nodes. However, this mechanism
may lead to the collision of packets, when two or more REP LY packets are
responded by the receivers to the same sector from which they received HELLO
packets. To avoid this, the three-way handshaking mechanism is employed. Here,
tokens are exchanged along with HELLO and REP LY packets. In this scheme,
only the token holder initiates the neighbor discovery. Due to the serialization
process involved, it results in higher discovery latency with the increase in net-
work size. In addition to resolving the neighbor discovery issue, the DSN should
also address the deafness. Deafness is a common problem found in DSN, wherein
during data communication, the packets are lost if the receiver node’s sector is
not oriented towards the sender node’s sector [8]. Deafness issue can lead to the
collision of packets resulting in increased energy consumption, poor Quality of
Service (QoS) and lower packet delivery ratio [9].
The rest of the paper is organized as follows. An overview of related work is
presented in Sect. 2. The proposed DCS-MAC is presented in Sect. 3. Section 4
presents the performance of DCS-MAC with similar protocols and the conclusion
drawn in Sect. 5.
 DCS-MAC 679

2 Related Work
In [10], a MAC protocol for DSN with sectored antenna known as SAMAC is
proposed. SAMAC is a cross-layer protocol which follows a centralized approach
where it performs neighbor discovery using Sectored Antenna Neighbor Discov-
ery (SAND) [6] protocol. Once neighbor information regarding all the nodes is
collected by the base station. A centralized TDMA scheme is executed at the
base station. The base station allocates time-slot for each node lying on the
shortest hop routing tree path obtained by Bellman-Ford algorithm. The time-
slot information is passed on to the nodes in the network by following SAND
protocol, wherein the token is passed to each node in a serialized manner. The
token holder will communicate the time-slot allocation information to its neigh-
bor and this process continues until the token is passed to all the nodes. The
nodes will be awake during assigned time-slot during which it communicates
data to the next-hop node and goes to sleep in unassigned time-slot. SAMAC
displayed higher throughput, packet delivery ratio and lower delay. However, if
a node fails in the network then to recover, the base station has to recompute
the routing path and time-slot, then redistribute the time-slots to all the nodes
through the SAND protocol.
Nur et al. proposed a low Duty Cycle Directional MAC (DCD-MAC) protocol
for DSN [11]. DCD-MAC is also a cross-layered protocol which uses a Collabora-
tive Neighbor Discovery (COND) protocol [12] to discover the neighboring nodes.
A data forwarding tree from source to sink node is constructed using any static or
dynamic routing protocol based on neighboring node information collected. The
parent-child pair nodes along the data forwarding tree negotiate the time-slot
prior to data communication. The DCD-MAC similar to SAMAC performs duty-
cycling as they wake up and sleep periodically in accordance with their schedule.
Whenever the source node has a data to send it transmits pilot tones at par-
ticular time-slots towards the parent and other nodes which are idle listen to
these tones in omnidirectional mode. The node up on listening to tone intended
to it switches to the directional mode and negotiates the time-slot allocation,
followed by which data transmission also happens in directional mode. In con-
trast to SAMAC which employed centralized TDMA, the DCD-MAC introduced
distributed TDMA which in turn reduces communication overhead between the
base station and deployed nodes. However, using a combination of directional
and omnidirectional antennas for a single node increases energy requirement and
are complex to implement practically.
The proposed DCS-MAC is developed exclusively for DSN employing a sec-
tored antenna. Unlike DCD-MAC there is no need of using the combination of
directional and omnidirectional antennas. The implementation of a distributed
TDMA scheme in the proposed MAC protocol makes it dynamic. Unlike SAND,
the proposed approach avoids the need for performing centralized operations
such as neighbor discovery and schedule allocation which further increases the
communication overhead between the base station and sensor nodes. In DCS-
MAC, the Distributed Slow-Scan Routing (DSSR) performs the operation of
680 S. Nagaraju et al.

neighbor discovery and schedule allocation in distributed manner thereby reduc-

ing the packets exchanged between the base station and sensor nodes.

3 DCS-MAC: An Overview

The proposed approach, avoids the need for explicit neighbor discovery and
functions in a distributed manner to establish the route from source to sink,
thus making it lesser complex compared to existing cross-layer MAC protocols
of DSN.

3.1 Network Model

In the proposed DCS-MAC protocol, N number of nodes are assumed to be

deployed in DSN. Each node is mounted with M sector antennas. Only one
sector is active at any particular time interval and it periodically switches from
one sector to another sequentially for communicating with its neighboring nodes.
While setting up the Distributed Slow-Scan Routing (DSSR) scheme, some of
the nodes will be in Slow − Scan (SS) mode and remaining nodes will be in
F ast − Scan (FS) mode. The nodes in SS mode scans slowly in such a way
that the node switches from one sector to another only when the nodes in F S
mode completes switching among all M sectors i.e. the node completes sweep-
ing all the sectors. The Slow − scan mode is performed twice, once to discover
the neighbors and for the second time, to distribute the schedule to its neigh-
bors discovered in the first round of SS. At the initial round of SS mode, it
ensures that by sending packets in a particular sector for a predetermined time,
all the intended receivers which operate in F S mode will be notified and dis-
covered. In SS mode, whenever the node switches to a new sector, it transmits
HELLO packet and the neighboring node in F S mode that receives it responds
with a REP LY packet. As depicted in Fig. 1, the HELLO and REP LY packet
consists of different fields such as node id, (X, Y) coordinates and sector num-
ber from which HELLO or REP LY packet was transmitted or received. Also,
a T ime slot field is used for assigning time-slot information. The T ime slot
information is set to -1 at the initial round of SS indicating no schedule is
allocated and during the second round of SS, actual time-slot information is
assigned. Whenever either of HELLO or REP LY packets is received the node
updates its neighbor table which records information in the form of 5-tuple
<node id, X coordinate, Y coordinate, sector number, T ime slot>.

Fig. 1. Packet format of HELLO/REP LY packets.

 DCS-MAC 681

The DCS-MAC developed for DSN which employs a distributed approach

is segregated into the following three phases: Distributed Slow-Scan Routing,
Schedule Distribution and Data Delivery.

3.2 Distributed Slow-Scan Routing

Algorithm 1 explains Distributed Slow-Scan Routing (DSSR) protocol. Initially,

the sink node will be in SS mode and will be listed in sn array and the nodes not
listed in sn array will resume operating in F S mode. Prior to starting the DSSR,
the parent variable is initialized to −1 for all the deployed nodes, indicating that
no parent is assigned to each node. Once SS mode completes, the nodes which
are in the F S mode, which receives HELLO packets, responds with REP LY
packet and hence marked as discovered. These discovered nodes update the node
from which it received HELLO packet as its parent and establishes the link with
its parent.

Algorithm 1. Distributed Slow-scan Routing

1: procedure Distributed Slow-scan-Routing( )
2: nsn[] ← next Slow − Scan nodes
3: sn[] ← Slow − Scan nodes
4: for all i ← 1 to N do
5: parent ← −1
6: end for
7: for all i ← 1 to N do
8: if Sweeping of all sectors in f irst round of Slow − Scan completed then
9: if parent == −1 then
10: parent ← node id of Slow − Scan node
11: update routing path with parent as its next − hop node
12: Insert neighboring nodes in nsn
13: T ime Slot Assignment(nsn)
14: end if
15: end if
16: if Sweeping of all sectors in second round of Slow − Scan completed then
17: N eighboring nodes discovered operate in Slow − Scan mode
18: sn ← nsn
19: parent resumes F ast − Scan mode
20: end if
21: end for
22: end procedure

The neighboring nodes discovered are now listed in nsn array and the parent
node resumes operating in F S mode. Once, the neighboring nodes are discov-
ered at the completion of SS operation, the T ime Slot Assignment() procedure
(Algorithm 2) is called in the second round of SS mode to allocate time slot-
between the parent and its child i.e. neighboring nodes discovered. Further, the
nsn listed nodes are now listed in sn array and they start operating in SS mode
and then the process continues until all the nodes are subjugated to SS mode.
682 S. Nagaraju et al.

Fig. 2. DSSR process: (a) sink node in Slow-Scan mode and remaining nodes in Fast-
Scan mode (b) sink and non-discovered nodes in Fast-Scan mode and the neighboring
nodes discovered in Slow-Scan mode. (c) Routing tree constructed.

Figure 2 depicts the DSSR protocol. As illustrated in Fig. 2a, at the first round
of SS mode, the sink node initiates the DSSR protocol by sending a discov-
ery packet (HELLO) among all the sectors. When a node ni send a HELLO
packet, the nodes which receive these packets identifies node ni as its parent and
establishes the link with its parent as shown in Fig. 2b. In the second round of
SS mode, time-slots for data communication are distributed by executing Algo-
rithm 2. Once the SS mode of the parent node is completed, the neighboring
nodes which established the link with parent node start operating in the SS
mode and the process repeats as shown in Fig. 2b and c.

3.3 Schedule Distribution

In the proposed protocol, establishing time-synchronization among the deployed

nodes is outside the scope of the paper and prior to schedule distribution among
all the nodes, these nodes are assumed to be time synchronized by using the
protocols similar to the TDMA synchronization employed in [10]. The schedule
distribution occurs during the second round of SS mode.
Once REP LY packets are received by the parent node or the node operating
in SS mode, the parent node computes the schedule for data communication
among neighboring nodes as per Algorithm 2. The time-slot assignment problem
is similar to edge coloring problem found in graph theory. Algorithm 2 employs
a heuristic coloring algorithm which uses minimal colors i.e. time slots. Since the
distributed approach is followed in schedule distribution it reduces computation
and communication overhead.
In the proposed time-slot schedule algorithm, the time-slots/color are dis-
tributed from the parent node to its child nodes i.e. neighboring discovered
nodes. Whenever assigning a color to the link between parent and child node,
suppose the existing maximum color index is exhausted, then the maximum
color index i.e. maximum number of time-slots allocated are incremented, thus
making this algorithm flexible for even large-scale network.
In Algorithm 2 based on the neighboring nodes discovered in the first round
of SS mode, the maximum color index or the maximum number of time-slots s
 DCS-MAC 683

Algorithm 2. Time Slot Assignment

1: N ← number of nodes
2: nodes[N ][N ] ← Array that indicates color assigned to each node
3: for all i ← 1 to N do
4: for all j ← 1 to N do
5: nodes[i][j] ← −1
6: end for
7: end for
8: procedure Time Slot Assignment(nsn)
9: s ← size of nsn
10: for all i ← 1 to s do
11: L[s] ← neighboring node in sn[i]
12: Color Edges of N ode(sn[i], L)
13: end for
14: end procedure

is determined. Initially, a 2-D array named nodes is used to allocate time-slots to

each and every node in the network and is initialized to −1. The array L lists all
the neighboring nodes discovered of the respective SS node. Then Algorithm 3
is called to color or allocate the time-slot for the link between parent and child
nodes.

Algorithm 3. Edge Color

1: procedure Color Edges of Node(current node,L)
2: for all j ← 1 to L.length do
3: neighbor ← L[j]
4: next color ← F ind N ext Color(nodes, current node, neighbor)
5: nodes[current node][neighbor] ← next color
6: nodes[neighbor][current node] ← next color
7: end for
8: end procedure

In Algorithm 3, the neighbor variables which represent individual neighboring

node is assigned a color for its link established with parent by calling Algorithm 4.
Once a particular color is returned then that particular link between parent and
child node will be assigned the returned color.
In Algorithm 4, we find the color which is unique to both the token
holder/parent node as well as its neighboring node. For this, a boolean array
is used whose entries will be initialized to F ALSE. All the indices correspond-
ing to the colors assigned to parent and neighboring nodes will be assigned
T RU E in the array X. This algorithm returns the color which is unique by
finding the index in array X which has been assigned F ALSE. Suppose there
is no unique color between the two nodes, then it returns 0 indicating that
maxim color index needs to be incremented. The selected color or time-slot of
684 S. Nagaraju et al.

Algorithm 4. Assign Color

1: procedure Find Next Color(nodes,current node,neighbor)
2: X[N ] ← boolean array initialized with all entries as f alse
3: for all i ← 1 to nodes[current node].length do
4: if nodes[current node][i]! = −1 then
5: color ← nodes[current node][i]
6: X[color] ← T RU E
7: end if
8: end for
9: for all i ← 1 to nodes[neighbor].length do
10: if nodes[neighbor][i]! = −1 then
11: color ← nodes[neighbor][i]
12: X[color] ← T RU E
13: end if
14: end for
15: for all i ← 1 to X.length do
16: if X[i] == F ALSE then
17: return i
18: end if
19: end for
20: return 0
21: end procedure

Table 1. Time-slot allocation table

Node id TS1 TS2 TS3 TS4

1 2 3 4 5
2 1 X X X
3 X 1 X X
4 X X 1 X
5 X X X 1

the individual neighboring node will be transmitted by the parent node to its
child in the second round of SS mode through HELLO packet.
Figure 3 represents the network-scenario for time-slot allocation. Here, node
1 is the parent node and the neighbor nodes discovered such as node 2, 3, 4
and 5 are the children nodes. Based on Algorithms 2, 3 and 4, the time-slots
are computed and stored in 2D-array named nodes as depicted in Table 1. In
Table 1, T S denotes the time-slot and X indicates that particular time-slot is
unallocated to the node. The node 1 transmits the allocated time-slot to its child
node through the HELLO packet in the second round of SS mode. Figure 3b
depicts the different color assigned to links between parent and child node as
different time-slots allocated between them.
 DCS-MAC 685

Fig. 3. Network scenario (a) prior to time-slot allocation (b) after time-slot allocation.

Table 2. Configuration parameters for Cooja simulator

Parameter Description
Deployment area 750 × 750 m2
Number of sensor nodes deployed 20 to 80
Number of source nodes 10
Time slot duration 200 ms
Packet transmission rate 1 packet per sec (pps)
Radio frequency 2.4 GHz
Number of sectors 8
Beamwidth of sector antenna 45◦
Range of sector antenna 135 m
Receiver sensitivity −95 dBm
Transmission power 0 dBm

3.4 Data Delivery

At the end of DSSR, each and every node will be allocated time-schedule. The
data delivery phase starts once DSSR process ends, where the sensor nodes
follow their respective schedule and switches to the sector pertaining to that
schedule to communicate with its next-hop node. The nodes which are not active
in communication at that particular time-slot will sleep thus enabling energy
saving.

4 Results

The performance of the proposed DCS-MAC protocol is evaluated using the

Cooja simulator with directional antenna support [13]. For performance analy-
sis, the proposed DCS-MAC protocol is compared with the SAMAC [10] pro-
tocol. The simulation results are obtained by averaging the results of multiple
simulation runs. Table 2 depicts the Cooja simulator’s configuration parameters.
686 S. Nagaraju et al.

The simulations are carried out with the following assumptions:

1. The static sensor nodes are randomly deployed in an area of 750 × 750 m2 .
2. Each static sensor node is mounted with an antenna having eight sectors.
The individual sector antenna is assumed to be a rectangular patch antenna
array (RPA) [14]. The RPA has the peak gain of 2.65 dBi and communication
range of 135 m [13].
3. In F S mode, the sector switching interval is set to Δt and in SS mode, the
sector switching interval is set to 8Δt.
4. For performance analysis of the protocols, 10 random edge source nodes are
selected to transmit packets at the rate of 1 packet per second (pps). The
simulation results for the packet delivery ratio and delay is calculated by
considering a run-time of 2000 s after the completion of the initial phase of
time-slot allocation.

Fig. 4. (a) Schedule allocation latency for varying network size. (b) Per node control
packet overhead for schedule allocation of varying network size.

The schedule allocation latency comparison of cross-layer protocols for the

varying network size is depicted in Fig. 4a. Here, the schedule allocation latency is
inclusive of the routing tree formation and distribution of the time-slot among all
the nodes lying along the routing tree. In SAMAC, the routing tree construction,
as well as the time-slot allocation, are computed in the sink node. In SAMAC,
we have considered the time taken by SAND neighbor discovery protocol [6]
to distribute time-slot among all the nodes. The DCS-MAC protocol outper-
forms SAMAC protocol over schedule allocation latency as it follows distributed
approach. Since the SAMAC uses SAND protocol which uses a token passing
serialized mechanism to distribute the time-slot allocation information regarding
all the nodes in the network it further increases the delay. Also, an entire time-
slot allocation information of all the nodes is distributed via a large number of
packets through the SAND protocol. As a result, the control packet overhead of
SAMAC protocol is higher as shown in Fig. 4b in contrast to DCS-MAC proto-
col. In the DCS-MAC protocol, only the time-slot allocation information from
 DCS-MAC 687

the parent to the child node needs to be distributed, which is achieved through
the smaller number of packets exchanged which also results in significant energy
saving.
The SAMAC uses the optimized routing tree which is determined using the
standard Bellman-Ford algorithm that follows a centralized approach, whereas
the DCS-MAC uses heuristic DSSR protocol which follows a distributed app-
roach and is a near optimal solution. As a result, SAMAC has the lower number
of hops per link than the DCS-MAC protocol as depicted in Fig. 5a. As a result
of the lower number of hops from the source node to sink node, SAMAC displays
lower end-to-end delay than DCS-MAC protocol as illustrated in Fig. 5b. Even
though SAMAC has the slightly lower delay, DCS-MAC is suitable for large-scale
DSNs as it follows distributed approach and has lower communication overhead
in long run.

Fig. 5. (a) Average number of hops for the established links in the network. (b) End-
to-end delay from source to sink node for varying network size.

Fig. 6. Packet delivery ratio for varying network size.

688 S. Nagaraju et al.

Figure 6 shows that the packet delivery ratio of DCS-MAC is almost on par
with the SAMAC protocol as both the protocols display packet delivery ratio
of more than 98%. Since both these protocol follows the TDMA approach, each
child node has dedicated time-slot to communicate with its parent which further
reduces contention.

5 Conclusion

The DSNs provide huge performance improvement over traditional WSNs in

terms of throughput, delay and interference. In this paper, the DCS-MAC pro-
tocol is proposed for DSNs which eliminates the need for an exclusive protocol to
perform neighbor discovery and routing operations. For the proposed DCS-MAC,
the DSSR protocol simultaneously performs the task of neighbor discovery and
establishing the link with neighboring nodes. A distributed TDMA approach
is used in the proposed protocol to compute the time-slot schedule between
parent and child node. The simulation results reveal that the proposed DCS-
MAC protocol requires minimum time for schedule allocation and has very low
communication overhead when compared to the SAMAC protocol. The packet
delivery ratio of the DCS-MAC protocol is almost on par with the SAMAC pro-
tocol. Even though the end to end delay of the DCS-MAC is slightly higher than
the SAMAC protocol, the distributed approach followed by DCS-MAC protocol
makes it suitable for large-scale WSNs.

References
1. Santivanez, C., Redi, J.: On the use of directional antennas for sensor networks.
In: IEEE Military Communications Conference, pp. 670–675 (2003)
2. Wei, L., Zhou, B., Ma, X., Chen, D., Zhang, J., Peng, J., Luo, Q., Sun, L., Li, D.,
Chen, L.: Lightning: a high-efficient neighbor discovery protocol for low duty cycle
WSNs. IEEE Commun. Lett. 20(5), 966–969 (2016)
3. Burghal, D., Tehrani, A.S., Molisch, A.F.: On expected neighbor discovery time
with prior information: modeling, bounds and optimization. IEEE Trans. Wirel.
Commun. 17(1), 339–351 (2018)
4. Cai, H., Wolf, T.: On 2-way neighbor discovery in wireless networks with directional
antennas. In: IEEE Conference on Computer Communications (INFOCOM), pp.
702–710 (2015)
5. Nur, F.N., Sharmin, S., Habib, M.A., Abdur Razzaque, Md., Shariful Islam, Md.,
Hassan, A.A.M.M., Alamri, A.: Collaborative neighbor discovery in directional
wireless sensor networks: algorithm and analysis. EURASIP J. Wirel. Commun.
Netw. 2017(1), 1–15 (2017)
6. Murawski, R., Felemban, E., Ekici, E., Park, S., Yoo, S., Lee, K., Park, J., Mir,
Z.H.: Neighbor discovery in wireless networks with sectored antennas. Ad Hoc
Netw. 10(1), 1–18 (2012)
7. Liu, B., Rong, B., Rose, Q.H., Qian, Y.: Neighbor discovery algorithms in direc-
tional antenna based synchronous and asynchronous wireless ad hoc networks.
IEEE Wirel. Commun. 20(6), 106–112 (2013)
 DCS-MAC 689

8. Choudhury, R.R., Vaidya, N.H.: Deafness: a MAC problem in ad hoc networks

when using directional antennas. In: 12th IEEE International Conference on Net-
work Protocols (ICNP), pp. 283–292 (2004)
9. Sharma, R., Kadambi, G., Vershinin, Y.A., Mukundan, K.N.: A survey of MAC
layer protocols to avoid deafness in wireless networks using directional antenna.
Mob. Comput. Wirel. Netw. Concepts Methodol. Tools Appl. 1758–1797 (2016)
10. Felemban, E., Vural, S., Murawski, R., Ekici, E., Lee, K., Moon, Y., Park, S.:
SAMAC: a cross-layer communication protocol for sensor networks with sectored
antennas. IEEE Trans. Mob. Comput. 9(8), 1072–1088 (2010)
11. Nur, F.N., Sharmin, S.R., Abdur Razzaque, Md., Shariful Islam, Md., Hassan,
M.M.: A low duty cycle MAC protocol for directional wireless sensor networks.
Wirel. Pers. Commun. 96(4), 5035–5059 (2017)
12. Nur, F.N., Sharmin, S., Habib, M.A., Abdur Razzaque, Md., Shariful Islam, Md.:
Collaborative neighbor discovery in directional wireless sensor networks. In: Region
10 Conference (TENCON), pp. 1097–1100 (2016)
13. Nagaraju, S., Rege, V., Gudino, L.J., Ramesha, C.K.: Realistic directional antenna
suite for cooja simulator. In: 23rd National Conference on Communications (NCC),
pp. 1–6 (2017)
14. Nagaraju, S., Kadam, B.V., Gudino, L.J., Nagaraja, S.M., Dave, N.: Performance
analysis of rectangular, triangular and E-shaped microstrip patch antenna arrays
for wireless sensor networks. In: 5th International Conference on Computer and
Communication Technology (ICCCT), pp. 211–215 (2014)
 An MQTT-SN-Based QoS Dynamic
Adaptation Method for Wireless Sensor
Networks

Helbert da Rocha1(B) , Tania L. Monteiro2 , Marcelo Eduardo Pellenz3 ,

Manuel C. Penna3 , and Joilson Alves Junior2
1
Federal University of Technology - Paraná,
Avenue Monteiro Lobato, s/n - Km 04, Ponta Grossa 84016-210, Brazil
[email protected]
2
Federal University of Technology - Paraná,
Avenue Sete de Setembro, 3165 - Rebouças, Curitiba 80230-901, Brazil
{tmonteiro,joilson}@utfpr.edu.br
3
Pontifical Catholic University of Paraná,
Imaculada Conceição, 1155 - Prado Velho, Curitiba 80215-901, Brazil
{marcelo,penna}@ppgia.pucpr.br

Abstract. The Internet of Things (IoT) is a concept that has dissemi-

nated in the last few years. The idea is to connect smart devices through a
network solution. IoT will be present in everyday objects and in people’s
lives. The economic impact of IoT solutions is expected to be, annually,
billions of dollars. To provide data exchange from smart devices, some
protocols are being used. The Message Queuing Telemetry Transport
Protocol (MQTT) is one of the most common application protocols for
IoT and Machine-to-Machine (M2M) communications. The MQTT uses
the publish/subscribe paradigm, which provides three Quality of Service
(QoS) to ensure communication between the smart devices. There is a
version of MQTT for Sensor Network (SN), named of MQTT-SN, devel-
oped especially for messages exchanging in Wireless Sensors Networks
(WSNs). As many smart devices can be connected in the same network,
it can result in network overload and message loss. To ensure a better
message delivery, a QoS Dynamic Adaptation Method (DAM) for WSNs
was developed. The DAM is focused on selecting the proper QoS level,
based on network latency conditions. The proposed method showed good
performance when compared with the normal QoS strategy implemented
in the MQTT-SN protocol.

1 Introduction
Internet of Things (IoT) is a concept where smart devices communicate through
the Internet. It is a new paradigm which has technical, social and economic
impact [12]. The smart devices can see, hear, think, talk to each other, and share
information to coordinate decisions [1]. Nowadays, there are many areas that
are using connected sensors, like environmental monitoring, medical solutions,
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 690–701, 2020.
https://doi.org/10.1007/978-3-030-15032-7_58
 An MQTT-SN-Based QoS Dynamic Adaptation Method for WSNs 691

military applications, transportation systems, crisis management, and industrial

automation [18]. The worldwide impact of IoT and Machine-to-Machine (M2M)
is related to the connection of billions of devices, resulting in a global economic
impact between $3.5 and $11 trillions by 2025 [10,12].
The Message Queuing Telemetry Transport (MQTT) protocol is one of the
most appropriated to exchange messages in the IoT and M2M world. It is able
to provide message routing for small, cheap, low power and low memory devices
and in low bandwidth networks [1].
The MQTT protocol was developed in 1999 by Andy Clark of IBM and
Arlen Nipper (Eurotech). It was standardized in 2013 at the OASIS [1,2]. The
MQTT protocol implements the publish/subscribe paradigm that provides a
communication mechanism (one-to-one, one-to-many, and many-to-many) built
on top of the TCP protocol. In this paradigm, the publisher publishes to a
topic in a middleware named broker. A subscriber subscribes in a topic of his
interest in the broker. When the publisher publishes a message to the topic in
the broker, the broker transfers the message from the topic to the subscriber.
The messages are delivered using one of the three Quality of Service (QoS):
QoS 0 sends a message without confirmation (fire or forget), the QoS 1 sends
a message with a confirmation message, but duplicate messages can arrive, and
QoS 2 sends exactly one message with three confirmation messages. There is
a version of MQTT developed specially for Wireless Sensor Networks (WSN)
named of MQTT-SN [15].
A typical WSN consists of a large number of devices equipped with a lim-
ited amount of storage and processing capabilities [15]. The MQTT requires
underlying protocols like the TCP/IP, which are too complex for simple, small
footprint, and low-cost sensor devices. The MQTT-SN was adapted to the par-
ticularities of wireless communication environments. It was developed to work
with low bandwidth and under unreliable network links, by transmitting mes-
sages with a shorter length [15]. Therefore, the MQTT-SN protocol is a good
option to be used in the WSN’s application layer. As many smart devices may
be connected in the same network, the network can be overloaded and message
losses increase. In order to improve the communication process, we propose a
new QoS Dynamic Adaptation Method (QoS DAM). The DAM optimizes the
process of message exchange between a publisher and a broker or between a
publish and a middleware using the MQTT-SN protocol.
The algorithm runs at the publisher’s side. Initially, the user chooses one
of the three available QoS levels provided by the MQTT-SN and the algorithm
will adapt the QoS, if necessary, according to the network latency. The results
show that the proposed method can reduce the message losses compared with
the original MQTT-SN QoS strategy.
The rest of the paper is organized as follows. Section 2 discusses the back-
ground and related works. The problem formulation is presented in Sect. 3. The
results are shown in Sect. 4. Finally, conclusions are presented in Sect. 5.
692 H. da Rocha et al.

2 Background and Related Work

2.1 Background

The basic MQTT-SN architecture is composed of a client (publisher), a gateway

(GW), and a forwarder. The MQTT-SN protocol exchange messages between
a client and a broker using an MQTT-SN GW as a middleware. The function
of an MQTT-SN GW is to translate messages from MQTT to MQTT-SN or
vice-versa if the MQTT-SN GW is in a stand-alone topology. The MQTT-SN
Forwarder is configured when the MQTT-SN GW is not present in the same
network that the broker is present [15].
To start the communication process, a CONNECT message is sent from the
publisher to the broker. To finish the communication, a DISCONNECT message
is sent from the publisher to the broker. During the communication, the messages
are sent using one of the three QoS levels (QoS 0, QoS 1, and QoS 2) provided
by the MQTT-SN [2,3,7,8].
QoS 0 (at most once): a PUBLISH message is sent according to network
capabilities. There is no retransmission if the sent message is lost. The message
can arrive once at the receiver or not arrive [2].
QoS 1 (at least once): a PUBLISH message is sent from the publisher to
the broker. The broker answer with a confirmation message (PUBACK). The
message is kept in the publisher until it receives the PUBACK. In case the
message does not arrive, the publisher will send a PUBLISH message again to
the broker until a PUBACK message is received or the time to retransmit the
message is over. This QoS ensures that the message arrives at the receiver at least
once. However, duplicate messages can arrive to the receiver until the publisher
receives the PUBACK [2].
QoS 2 (exactly once): a PUBLISH message is sent from the publisher to
the broker. The broker responses with a confirmation message (PUBREC). The
PUBLISH message is kept in the publisher until it receives a PUBREC. When
the PUBREC message arrives or the time to retransmit is over, the publisher
discards the PUBLISH message and keeps the PUBREC message. Then, the
publisher sends a confirmation message (PUBREL) indicating that the PUBREC
message was received. The PUBREL is kept at the publisher until it receives
a confirmation message (PUBCOMP) from the broker, what means that the
broker received the PUBREL message from the publisher, which finally discards
the PUBREL message. The QoS 2 is the highest QoS and it is appropriated
to be used when neither loss nor duplication messages are acceptable for the
application. This QoS level increases the overhead [2,3]. The QoS levels are
illustrated in Fig. 1.
The MQTT-SN specification defines best practices that can be adopted when
the MQTT-SN is implemented e.g., a client should wait for a time Twait ≥
5 min before restarting the registration procedure, the time to retry a message is
between 10 to 15 s and starts when a client sends a message and stops when the
expected GW’s reply, and the number of retries to send a message is between 3
to 5.
 An MQTT-SN-Based QoS Dynamic Adaptation Method for WSNs 693

Publisher/Subscriber Broker Publisher/Subscriber Broker Publisher/Subscriber Broker

Client Client Client

PUBLISH PUBLISH PUBLISH

PUBACK PUBREC

PUBREL

PUBCOMP

QoS 0 QoS 1 QoS 2

Fig. 1. Packet transmission QoS levels.

The Really Small Message Broker (RSMB) was implemented with MQTT-
SN. The RSMB is a small server with 80 KB of storage that can run in 200 KB
of memory. The reduced size of the RSMB enables messaging from very small
devices like sensors and actuators in networks with low bandwidth. Also, the
RSMB can run in embedded systems to provide infrastructure in remote instal-
lations using the protocols TCP/IP or UDP [4].

2.2 Related Works

The authors in [5] presented a mechanism that establishes different QoS lev-
els using the publish/subscribe model for wireless networks, to provide reliable
delivery of packet and timeliness. They proposed three QoS levels. The QoS 1
calculates the Retransmission Timeout (RTO) and adjusts the RTO depending
on the Packet Delivery Ratio (PDR). The QoS 2 provides data aggregation to
reduce network congestion. The QoS 3 promotes the timeliness on the deliv-
ery packet. The authors proposed new QoS based on Dynamic Retransmission
Scheme using a simulated topology to test, whereas in this paper, was proposed
to use the QoS provided by MQTT-SN adapting the QoS based on the network
latency.
An implementation of the MQTT-SN protocol in an end-to-end e-health
system was presented in [6]. The study promoted end-to-end delay assurance and
sufficient probability of content delivery in WSN and IoT systems. The study
wanted to provide content delivery for sensors mounted on the human body to
transfer data by the MQTT-SN and a gateway. In the paper the authors analyzed
the delay using QoS 1 and provided an extension to QoS 0 and QoS 2 model.
However, they do not change the QoS during the communication process and
validate the theoretical modeling with simulations.
The author in [8] analyzed the three QoS implemented on the MQTT pro-
tocol in wired and wireless networks using various sizes of payload. The author
analyzed the delay and message loss. The conclusion was that the delay and
694 H. da Rocha et al.

message loss had a strong relationship with the QoS level which was chosen to
transfer the messages. The QoS 2 had the highest delay, but had the least loss of
messages. While the QoS 0 had the lowest delay and highest message loss. How-
ever, the study was developed using the MQTT protocol, the proposed study
analyzed the QoS implemented by MQTT-SN protocol, also the MQTT-SN using
the QoS DAM method.

3 Methodology
3.1 Problem Statement

Both protocols MQTT and MQTT-SN provide three QoS levels to exchange the
messages. The impact of the QoS level in the communication is presented by the
authors [3,7].
Using QoS 0 and assuming that pt is the time to send a PUBLISH message
and n is the number of messages, the total time to exchange messages is given
by n · pt . For QoS 1, which has an additional PUBACK message with time
mt , the time taken to transfer n messages is n · (pt + mt ). The QoS 2 has three
confirmation messages PUBREC, PUBREL, and PUBCOMP. So, the time taken
to transfer n messages is n · (pt + 3mt ).
Based on these analyses, the QoS 1 takes 40% more time to send the same
number of messages than QoS 0, while QoS 2 takes about 120% more time.

3.2 Proposed Solution

A method was developed to improve and guarantee the number of delivered

messages in a WSN scenario with unreliable wireless links and high traffic load.
Three scenarios with different latency test intervals were proposed to test the
QoS adaptation method, based on the current WSN performance. The latency
intervals for the test scenario were based in the studies of the latency on wireless
networks in [11,16,17]. The method gets information about the network latency
and decides the proper QoS to be currently used.
Initially, the user informs the desired QoS level to communicate. During a
specific period of time, the method estimates a set of network latency values.
Based on these values, it computes the latency median (LM) to select a more
suitable QoS value, if necessary.
The LM is calculated as in [13]. The median was chosen to soften the network
latency variation every time that a new latency was measured and added to the
latency buffer.
When the method changes the QoS of one or more messages, it will transfer
all these messages using the new selected choice, until the latency is measured
again. Based on the new value of the LM, the method can assign or not a new
QoS.
According to the LM value and the intervals presented in Table 1, If it is
considered as “Good”, the method changes the QoS selected by the user. If the
 An MQTT-SN-Based QoS Dynamic Adaptation Method for WSNs 695

Table 1. Latency interval for each test scenario.

Test scenario Good Normal Unstable

1 LM ≤ 10 ms 10 ms < LM ≤ 20 ms 20 ms < LM
2 LM ≤ 15 ms 15 ms < LM ≤ 30 ms 30 ms < LM
3 LM ≤ 30 ms 30 ms < LM ≤ 60 ms 60 ms < LM
Selected QoS QoS 0 User defined QoS QoS 0

initial type was QoS 1 or QoS 2 it changes to QoS 0, meaning that this change
continues to guarantee messages delivery without overload the network. It is
important to note that even with this change, the message will be delivered
once.
If the network is considered as Unstable, the initial QoS selected by the user
is changed to QoS 0. The idea is to reduce the network load. To this unstable
moment, by choosing QoS 0 the method gives more chances the message be
delivered if compared to QoS 1 and QoS 2. In case the confirmation message
is lost and the time to retransmit the message is over, the client aborts the
procedure and assumes that the gateway is disconnected [15].
When LM is in the interval considered “Normal”, the method returns the
current QoS to the original one selected by the user at the beginning of the
communication. For example, if the user initially has chosen QoS 2 to exchange
messages. If during the communication process the LM is classified as “Good”
or “Unstable”, the method changes the QoS 2 to QoS 0. When the LM returns
to “Normal” the method returns to use QoS 2. In case the user initially selects
QoS 0, the method does not do any change. The algorithm of the Quality of
Service Dynamic Adaptation Method (QoS DAM) is presented in Fig. 2.
The method computes which QoS will be used every time based on the esti-
mated latency median. So the goal is not to overload the network by providing
message delivery with a proper QoS.

3.3 Evaluation Scenario

The tests were performed in a real network topology and the scenario was con-
structed to simulate a network with lossy links and high traffic. The experi-
mental evaluation scenario is composed by a computer acting as an MQTT-SN
publisher connected to a single broker. The MQTT-SN broker was installed in a
Raspberry Pi3. The method was developed to select the best QoS to send mes-
sages from the publisher to a broker. The broker works as an MQTT-SN gateway
and broker. Also, the same Raspberry was configured as MQTT subscriber using
a Mosquitto client tool. A second Raspberry Pi3 was used to overload the net-
work using the software iPerf and injecting UDP packets at a rate of 10 Mbits/s.
These devices are connected through a wireless network using a wireless router.
The test scenario is illustrated in Fig. 3.
696 H. da Rocha et al.

Start
User informs
the QoS
(0, 1, or 2)

Store
user's QoS
Measure
network
latency

Store latency in a
variable size buffer

QoS DAM

Classify
Get median
latency
value
median

Sort buffer in ascending

order
Good Normal Unstable

User's
QoS 0 QoS 0
QoS

Are there more Send messages

messages with the new QoS
to send?

Finish No Yes Estimate latency every 100 messages

Fig. 2. QoS dynamic adaptation method.

It was considered a distance of 30 m between the MQTT-SN publisher and

the wireless router [14]. The second Raspberry Pi3 was positioned close to the
wireless router. In this equipment was also executed the iPerf software, which
generates the network traffic load. The test script was written using Python
language provided by the client library to RSMB [4]. A Mosquitto client tool [9]
is used to receive messages using the MQTT protocol when the RSMB is used
as an MQTT-SN gateway and MQTT broker at the same time.
The test was conducted by sending 10 groups of 5,000 messages of 64 bytes,
using each of the three QoS levels. Every 100 messages the method estimate the
latency using a ping command between the publisher and the broker. The system
computes the latency median using a sliding window with variable size. The
 An MQTT-SN-Based QoS Dynamic Adaptation Method for WSNs 697

Broker Gateway Subscriber iPerf

MQTT-SN MQTT-SN MQTT Server

Raspberry Pi 3 Raspberry Pi 3

Publisher
MQTT-SN
5m
Computer 30 cm
iPerf
Client

30 m
Wireshark

Wireless Router

Fig. 3. Test scenario.

latency buffer is ordered in ascending order. The LM is selected and compared

with the values presented in Table 1. At this point, the method chooses the best
QoS to transfer the messages. When the LM value is in the interval considered
as “Normal”, the QoS selected by the user is used. In the case of the LM value
is classified as “Good” or “Unstable”, the initial QoS changes to QoS 0. An
overview of the test scenario is presented in Fig. 3. The iPerf tool generates
background traffic in order to increase the latency and generate packet losses.
The goal of the test scenario was to verify how the proposed QoS DAM works
under high traffic and latency variation.

4 Results

4.1 Performance of the Original MQTT-SN

The average performance of the MQTT-SN without the QoS DAM is presented
in Fig. 4. The tests were repeated 10 times for each type of QoS (QoS 0, QoS 1,
and QoS 2). The results show that MQTT-SN using QoS 0 and QoS 1 have the
same percentage of delivered messages (99%). However, the MQTT-SN using
QoS 2 has a message delivered rate of only 41%. It means that 58% of the
messages were lost in the communication process compared with QoS 0 and
QoS 1.
Even with more latency during the communication process (70 ms) the QoS
0 delivered the same percentage of messages than the QoS 1 (62 ms). Although
the QoS 2 presented the lowest latency, the number of delivered messages were
worse. The time to publish 5,000 messages were closer for all QoS types. It is
possible to point out that QoS 0 and QoS 1 presents a similar performance in
698 H. da Rocha et al.

QoS 0
120
QoS 1
QoS 2

100

80

60

40

20

0
Delivery Ratio (%) Average Latency (ms) Time to Publish (s)

Fig. 4. Performance of the MQTT-SN without QoS DAM.

the experimental scenario. The QoS 2, which requires a more complex commu-
nication process, presents a reduced efficiency in terms of messages delivered.

4.2 Results with QoS DAM – QoS 1

The Fig. 5 shows the results considering messages exchange from MQTT-SN,
without any changes, and the results from QoS DAM using the three different
latency intervals shown in Table 1, where QoS 1 was initially considered. In this
scenario, the method changes QoS 1 to QoS 0 if the latency median (LM) is
considered as “Good or Unstable” and return to the original one, QoS 1, if
the latency interval returns to the LM value considered as “Normal”. When
considered latency intervals 1 and 2 it was delivered 100% of the messages. The
average latency using MQTT-SN implementing latency intervals 1 and 2 was
23 ms and the time to publish all the messages was 106 s and 105 s, respectively.
However, using MQTT-SN with latency interval 3 had the best performance
because the QoS DAM kept the percentage of messages delivered about 99%,
with a higher latency of 100 ms.

4.3 Results with QoS DAM – QoS 2

In this scenario, the method changes from QoS 2 to QoS 0 in case of the latency
interval is classified as “Good” or “Unstable” and returns from QoS 0 to QoS 2
when the latency interval returns to be classified as “Normal”. Figure 6 shows the
results for the MQTT-SN protocol using QoS 2, without QoS adjustment during
all the communication process, and the use of the QoS DAM. The QoS DAM
achieves better performance when processing messages of QoS 2 from MQTT-SN.
By using latency intervals 1 and 2 from Table 1 results in the best performance in
 An MQTT-SN-Based QoS Dynamic Adaptation Method for WSNs 699

QoS 1 - Dynamic Adaptation

Without Latency Interval
120 Latency Interval 1
Latency Interval 2
Latency Interval 3
100

80

60

40

20

0
Delivery Ratio (%) Average Latency (ms) Time to Publish (s)

Fig. 5. Comparison between MQTT-SN QoS 1 and QoS DAM.

QoS 2 - Dynamic Adaptation

Without Latency Interval

120 Latency Interval 1
Latency Interval 2
Latency Interval 3
100

80

60

40

20

0
Delivery Ratio (%) Average Latency (ms) Time to Publish (s)

Fig. 6. Comparison between MQTT-SN QoS 2 and QoS DAM.

delivery ratio, 98% and 100%, respectively. Also, it obtains lower latencies (32 ms
and 16 ms) and the lowest time to publish all the messages. The latency interval
3 presented the worst latency but delivered about 74% of the messages with a
latency of 68 ms and a time to publish around 113 s. The standard MQTT-SN
using QoS 2 presented the worst performance, with only 41% of the messages
delivered. So, we can observe that the number of messages delivered with the
standard MQTT-SN using QoS 2 was 33% lower than QoS DAM using latency
interval 3.
700 H. da Rocha et al.

5 Conclusions
We developed a new strategy for dynamic QoS adaptation in MQTT-SN proto-
col. The tests showed that by using the estimated network latency to dynami-
cally changes the QoS levels, the communication performance can be improved
in WSN applications. We have performed an experimental and theoretical study
of the MQTT-SN protocol and defined an appropriated strategy to monitor the
network latency and to control the QoS. The number of delivered messages can
be improved in overloaded networks, mainly when QoS 2 level is initially selected.

References
1. Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., Ayyash, M.: Internet
of things: a survey on enabling technologies, protocols, and applications. IEEE
Commun. Surv. Tutor. 17(4), 2347–2376 (2015). https://doi.org/10.1109/COMST.
2015.2444095
2. Banks, A., Gupta, R.: MQTT version 3.1. 1 29 (2014)
3. Boyd, B., Gauci, J., Robertson, M., Van Duy, N., Gupta, R., Gucer,
V., Kislicins, V., Redbooks, I.: Building Real-Time Mobile Solutions
with MQTT and IBM MessageSight. IBM Redbooks, Poughkeepsie (2014).
https://books.google.com.br/books?id=R3tNBQAAQBAJ
4. Craggs, I.: Really Small Message Broker (2014). https://www.ibm.com/
developerworks/community/alphaworks/tech/rsmb
5. Garcı́a Davis, E., Calveras Augé, A.M.: Publish/subscribe protocol in wireless
sensor networks: improved reliability and timeliness. KSII Trans. Internet Inf. Syst.
12(4), 1527–1552 (2018). https://doi.org/10.3837/tiis.2018.04.008
6. Govindan, K., Azad, A.P.: End-to-end service assurance in IoT MQTT-SN. In:
2015 12th Annual IEEE Consumer Communications and Networking Conference
(CCNC), pp. 290–296 (2015). https://doi.org/10.1109/CCNC.2015.7157991
7. Lampkin, V., Leong, W.T., Olivera, L., Rawat, S., Subrahmanyam, N., Xiang, R.,
Kallas, G., Krishna, N., Fassmann, S., Keen, M., et al.: Building smarter planet
solutions with MQTT and IBM websphere MQ telemetry. IBM Redbooks, Pough-
keepsie (2012)
8. Lee, S., Kim, H., Hong, D., Ju, H.: Correlation analysis of MQTT loss and delay
according to QoS level. In: 2013 International Conference on Information Network-
ing (ICOIN), pp. 714–717. IEEE (2013)
9. Light, R.A.: Mosquitto: server and client implementation of the MQTT protocol.
J. Open Source Softw. 2(13) (2017). https://doi.org/10.21105/joss.00265
10. Manyika, J.: The Internet of Things: Mapping the Value Beyond the
Hype (2015). https://www.mckinsey.com/business-functions/digital-mckinsey/
our-insights/the-internet-of-things-the-value-of-digitizing-the-physical-world
11. Patro, A., Govindan, S., Banerjee, S.: Observing home wireless experience through
WiFi APs. In: Proceedings of the 19th Annual International Conference on Mobile
Computing & Networking, pp. 339–350. ACM (2013)
12. Rose, K., Eldridge, S., Chapin, L.: The internet of things: an overview, pp. 1–50
(2015)
13. Shrivastava, N., Buragohain, C., Agrawal, D., Suri, S.: Medians and beyond: new
aggregation techniques for sensor networks. In: Proceedings of the 2nd Interna-
tional Conference on Embedded Networked Sensor Systems, SenSys 2004, pp. 239–
249. ACM (2004). https://doi.org/10.1145/1031495.1031524
 An MQTT-SN-Based QoS Dynamic Adaptation Method for WSNs 701

14. Siboni, S., Shabtai, A., Elovici, Y.: Leaking data from enterprise networks using a
compromised smartwatch device. In: Proceedings of the 33rd Annual ACM Sym-
posium on Applied Computing, SAC 2018, pp. 741–750. ACM (2018). https://doi.
org/10.1145/3167132.3167214
15. Stanford-Clark, A., Truong, H.L.: MQTT for sensor networks (MQTT-SN) proto-
col specification 1 (2013)
16. Sui, K., Zhou, M., Liu, D., Ma, M., Pei, D., Zhao, Y., Li, Z., Moscibroda, T.:
Characterizing and improving WiFi latency in large-scale operational networks.
In: Proceedings of the 14th Annual International Conference on Mobile Systems,
Applications, and Services, pp. 347–360. ACM (2016)
17. Syed, A.A., Heidemann, J.S., et al.: Time synchronization for high latency acoustic
networks. In: Infocom, vol. 6, pp. 1–12 (2006)
18. Ullah, M.H., Park, S.S., No, J., Kim, G.H.: A collaboration mechanism between
wireless sensor network and a cloud through a Pub/Sub-based middleware service.
In: The Fifth International Conference on Evolving Internet, pp. 38–42. Citeseer
(2013)
 Centrality Based Geocasting for
Opportunistic Networks

Jagdeep Singh1(B) , Sanjay K. Dhurandher2 , Isaac Woungang3 ,

and Makoto Takizawa4
1
CAITFS, Division of Information Technology,
Netaji Subhas Institute of Technology, University of Delhi,
New Delhi, India
[email protected]
2
CAITFS, Department of Information Technology,
Netaji Subhas University of Technology, New Delhi, India
[email protected]
3
Department of Computer Science, Ryerson University, Toronto, ON, Canada
[email protected]
4
Department of Advanced Sciences, Faculty of Science and Engineering,
Hosei University, Tokyo, Japan
[email protected]

Abstract. Opportunistic networks (OppNets) are a subclass of delay

tolerant networks characterized by unstable topology, intermittent con-
nectivity, and no guarantee of the existence of an end-to-end path to
carry the messages from source to destination because of node’s mobil-
ity. For these reasons, the routing of data in such networks occurs in
a store-carry-and-forward approach. Following this approach combined
with geocasting as message transmission strategy, this paper proposes
a geo-casting protocol for OppNets named Centrality based Geocasting
for OPPortunistic networks (CGOPP for short). The proposed scheme
combines uni-casting and flooding by first transferring a message to an
individual geographic location, then intelligently flood the message to all
nodes inside the area by using a degree centrality function. Simulation
results are provided, demonstrating the efficiency of the proposed pro-
tocol against traditional geo-casting protocols for OppNets, in terms of
average latency, delivery ratio, and number of messages forwarded, under
varying number of nodes, buffer sizes, and TTL.

Keywords: OppNets · Geo-casting · Degree of centrality · Flooding

1 Introduction
OppNets have been designed for operation in environments such as battlefields,
underwater communications, and deep space exploration, which are character-
ized by the lack of end-to-end connection, high delays, intermittent connectivity,
and high error rates. They aim to maintain reliable connectivity where there is
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 702–712, 2020.
https://doi.org/10.1007/978-3-030-15032-7_59
 Centrality Based Geocasting for Opportunistic Networks 703

no fixed end-to-end association between the source and destination nodes. Usu-
ally, the nodes in OppNets have highly dynamic low density, limited power, short
transmission range; and they are prone to various types of attacks by malicious
nodes. Due to these attributes, OppNets present notable research challenges,
the foremost ones being the routing and transmission of messages from source
to destination. In the recent literature [1], researchers have presented solutions
for routing messages towards the destination cast. However, designing routing
protocols to support the geo-casting mechanism in the context of OppNets is
still in the primary stage. A geo-casting protocol is expected to transmit the
data to all nodes within a geographic cast as compared to transmitting the same
to a single node. Geo-casting can empower new services and applications, for
instance, geographic warning for crisis circumstances.
In this paper, a novel geocasting routing algorithm for OppNets (called Cen-
trality based Geocasting for OPPortunistic Networks (CGOPP)) is proposed,
which is able to transmit the messages towards the destination cast using a
combined uni-casting and flooding mechanisms. When a message is to be sent
from source to destination, it is first sent to the assigned geographical location,
then afterwards, it is flood to all nodes within the cast. Transferring a message
toward the path to the goal area is the main target to deal with. The proposed
scheme consists of two parts: (1) The messages are forwarded to the destination
cast and (2) All the messages are flooded on the basis of the degree centrality
of nodes in the cast. This is a multi-copy based routing algorithm designed to
control the non-deterministic versatility and the network. It utilizes the node’s
own particular sequence id and contact points of interest and it does not require
any additional routing elements to be transmitted over the network. Along these
lines, data confidentiality and secrecy are maintained and can be changed as the
network size increases.
In geo-casting, it is not only required that the message be delivered in the
destination area, but also that the message be disseminated within the cast effi-
ciently. Geo-casting in OppNets is a difficult task, which requires that temporal
and spatial domains be considered, along with the restrictions with respect to
nodes’ resources. The proposed routing protocol follows the current member
delivery membership model, where all the messages are scheduled to the partic-
ular area; therefore some form of target area information must be incorporated
into the messages’ block; for instance, the endpoint identifier. Whenever a node
accepts a message, it first matches its own location with the endpoint identifier
of the message, then it forwards the message to other nodes.
The rest of the paper is organized as follows. In Sect. 2, some related work on
geo-casting routing protocols for OppNets are discussed. In Sect. 3, the proposed
CGOPP routing algorithm is presented. In Sect. 4, the performance evaluation
of the CGOPP scheme is presented. Finally, Sect. 5 concludes the paper.

2 Related Work
Some of the key features [2] that are considered when designing routing proto-
cols for OppNets include: (a) contact opportunity - this feature can be exploited
704 J. Singh et al.

opportunistically to achieve data forwarding, but its use requires the design of
an efficient management strategy to deal with eventually limited node storage
capability. More recently, the introduction of the Complex Network Analysis
(CNA) concept [3] has helped predicting the contact opportunities in OppNets.
Following this concept, CNA-based protocols have been proposed in the litera-
ture, which exploit a community-based structure to decide whether a node can
participate or not in the data forwarding process, (b) cooperation level - as the
nodes in an OppNet are typically controlled by human users, who are mostly
unwilling to share their resources for public usage, nodes usually tend to save
their buffer space and energy in idle situations rather than sharing their resources
with others. Often, this problem is addressed by implementing incentives mech-
anisms [4] to encourage nodes to participate in the data forwarding process, (c)
mobility modeling - this is a context-specific feature that is difficult to charac-
terize; nonetheless, it has some influence on the way that mobility patterns are
analyzed [5] and, (d) energy efficiency - in OppNets, nodes connectivity usually
happens temporary due to the fact that energy consumption of a node does not
occur only during the data transmission or reception, but also during processing.
Based on the aforementioned features, routing protocols for OppNets have
been classified into three categories: flooding-based [6], context-aware [7] and
social relationships-based [8]. Following this classification, representative recent
work on geo-casting routing protocols for OppNets are as follows.
In [9], Liu et al. proposed a routing scheme for transferring the messages
from a mobile sender to a stationary destination in delay tolerant networks.
This protocol is based on the encounter between the sender and the neighboring
nodes that are within its geographical location. In order to accept the messages
being routed, nodes have to interchange their so-called visiting and meeting
records with each other.
In [10], Lu proposed a location-based geo-casting protocol called LOOP,
which transmits the messages towards the destination area in a greedily fash-
ion. In this scheme, a Bayes’ prediction model is built to determine to estimate
the node’s versatility history, which is then taken into account in the message
forwarding phase of the protocol.
In [11], Ott et al. proposed a geographically limited content sharing scheme
for OppNets, which depends on the availability of nodes within the area where
the message is created. In this scheme, a large number of users in that area results
in a higher sustainability of the shared content, leading to a fully distributed
content sharing service for OppNets.
In [12], Ma and Jamalipour proposed a geo-casting protocol named Expected
Visiting Rate (EVR), where the focus is on routing the messages to the desti-
nation location instead of considering the routing of the message to all devices
within that location (cast). EVR is calculated based on the historical information
of the visited areas in the network, which involves selecting the next carrying
node. The node with the highest EVR rate is selected as the next carrying hop.
For this scheme, it should be noted that with respect to the EVR evaluation,
no investigation has been carried out regarding the temporal membership of the
nodes within the considered cast.
 Centrality Based Geocasting for Opportunistic Networks 705

3 CGOPP: Centrality Based Geo-Casting

for OPPortunistic Networks
The proposed protocol gets its motivation from epidemic and spray-and-wait
routing protocols, which are unicast in nature. The endpoint identifier is used
to identify the particular devices. But in geo-casting, it is entailed to adapt
the endpoint identifiers so that they can distinguish between casts or regions.
Geographical casts are defined as the geographical areas which can be covered
by the network. Any user of the network that is located within a defined cast is
a potential recipient. In geo-casting, messages are addressed to a cast and every
user located in that cast receives the message. It is necessary to define the cast
within the network, and messages should be addressed to that cast. Further,
every node which located inside that area should receive that message. First,
we investigate how to define a geographical cast in OppNets. Geographical casts
are a group of hosts which are within the same region and can be addressed as
the recipient of a message. The region is a set of coordinates in the 2-D space of
the network. Next, using this mechanism, hosts can forward the messages to the
arbitrary defined cast. This provides high flexibility, potentially minimizing the
number of devices that are receiving undesirable messages compared to other
geo-casting algorithms that use regions as circles.
When a node receives a message, it is required to examine whether it is
situated inside or outside the specified area. This identification is executed for
every encountered message. The cast is designed by using any of the following
tests. (a) Crossing test: A straight line is to be drawn from the examined point
to any fixed direction. The number of intersections between the ray and the
boundaries of the polygon is the key to the identification. If the number of
intersections is even, the point is outside of the polygon; otherwise, it is inside
the region. (b) Angle summation test: An angle needs to be drawn by a ray being
a line from the examined point towards a vertex of the polygon and another ray
being a line from the point towards the next neighbor vertex. If the total sum
of all neighboring angles in the polygon equals 360◦ , the point is located inside;
otherwise it is located outside. (c) Triangle test: The polygon needs to be divided
into a number of triangles. The point is inside the polygon if it is inside one of
the triangles; if not it is outside. In [13], Haines reported that the crossing test
algorithm is the most fastest one to achieve the aforementioned features since
it requires less processing power in comparison to all other algorithms. Thus, in
our proposed CGOPP scheme, we will make use of that crossing test algorithm
to identify if a node is inside the area or lies outside it.

3.1 Proposed CGOPP Protocol

CGOPP entails a two-part approach to forward the messages in the direction to
their destination cast; and in latter part, proliferating them to receiver devices
within the destination cast. In the proposed approach, the messages carry the
cast definition, actual payload, sequence id and a pair of epoch times that define
its lifetime.
706 J. Singh et al.

Part 1 - Forwarding the Message to its Destination Cast: In the first part, a
multi-copy spraying algorithm is utilized, which is effective in terms of message
delivery ratio and overhead ratio. Initially, a sequence id S is set to the message.
Each time a message is duplicated and transferred to another node, S is decre-
mented by 1. At the point when the S value reaches zero, the message can’t be
handed-off any longer and it will be expelled when the local buffer gets occupied
completely or when the message runs out.
Part 2 - Delivering Messages to all Devices Inside the Cast: In the next part, the
message is distributed to all nodes within the cast by an intelligent centrality-
based flooding strategy. This centrality-based algorithm has been designed ear-
lier in the GSAF and GeoEpidemic protocols [15] to decrease the overhead by
embedding the social features in the routing of the message in OppNets. We have
replicated this same technique in our proposed scheme by introducing two mech-
anisms. First, the message time-to-live (TTL) is adapted based on the degree
centrality of the nodes. Second, a message blocking method is implemented to
prevent receiving the replications of run out TTL messages in active nodes. The
detailed steps are described in Algorithm 1.
Degree of Centrality of a Node: This represents the popularity of the node in the
network, i.e. its social relative importance. A higher degree centrality indicates
that the node connects with a larger numbers of nodes in the network. The
degree centrality of a node is defined as:
N

DCi = a(i, n) (1)
n=1

where N represents the groups of nodes in the network and there exists a direct
link exists between node i and node n, a(i, n) = 1. We used the CW indow cal-
culation algorithm to calculate the degree centrality of anode. The main reason
behind this calculation is that it takes into account the changes in node’s cen-
trality over time and averages the nodes centralities for few window intervals.
This is suitable for people’s behaviour in the network. Each node Ni records the
encountered nodes in the network. When node Ni meets Nj , the degree central-
ity values are evaluated using CW indow . Then, centrality values are exchanged
between both nodes. Node Ni compares its centrality value DCi against that of
node Nj is centrality value DCj . If DCj is greater than DCi , it means that Nj is
socially more active than Ni ; then for each message Mi in the Ni buffer, the TTL
value of message i is decreased by dividing it by the node Nj ’s centrality value
DCj . The ID of each socially infected message is registered in the Nj blocking
register block N.

4 Performance Evaluation

The proposed CGOPP is simulated using Opportunistic Network Environment

(ONE) simulator [14]. We have compared the performance of CGOPP against
 Centrality Based Geocasting for Opportunistic Networks 707

Algorithm 1. Centrality-based Geocasting For Opportunistic Networks

1: Begin
2: for Each encounter between two nodes in the network do
3: one takes the sender role N1 and the other takes the receiver role N2
4: if the node is the sender N1 then
5: drop expired messages from buffer
6: apply buffer scheduling policy
7: for each message in the buffer do
8: if the message is already existed in the N2 ’s buffer then
9: pass to the next message in the for loop
10: end if
11: if S > 0 then
12: S = S-1
13: forward a copy to N2
14: else if S = 0 then
15: if N1 is inside the destination cast then
16: forward a copy to N2
17: end if
18: end if
19: end for
20: else if the node is destination node N2 then
21: for each received message do
22: if N2 is inside the destination cast then
23: Calculate DC1 , DC2
24: compare degree centrality of both nodes
25: if DC2 > DC1 then
26: decrease message TTL
27: register message in N2 block register
28: deliver to Application Layer
29: S = S-1
30: end if
31: end if
32: end for
33: end if
34: change roles (sender N1 & receiver N2 ) and go back to line 3 just for one time
35: end for

that of two other benchmark algorithms: GeoEpidemic and GSAF [15]. In the
GeoEpidemic scheme, the messages are flooded in the desired region of the net-
work, leading to an upper bound on the network connectivity. On the other
hand, in the GSAF scheme, messages are forwarded to their destination region
(cast) in two parts. In the first one, a message is duplicated using a sequence id
mechanism associated with the encountered devices. When the message reaches
its destination area, the second part is permitted and the message is flooded
within all nodes of the destination area.
In the evaluation of our proposed protocol, three performance metrics are
considered: the average latency, delivery ratio, and the number of messages for-
708 J. Singh et al.

warded, under varying number of hops, buffer sizes, and TTL. In geo-casting,
the per message delivery ratio is calculated, which is the segment of the available
devices in the region throughout the duration of life of the message; also, the
epoch it takes for a message to outreach the desired region device is considered,
and the per message delivery latency is computed.

4.1 Simulation Setup

The considered OppNet is modelled as a set of mobile nodes in which nodes may
join and leave the network at any time. It consists of six groups of moving ele-
ments: pedestrian, electric motor car, vehicles, bicycles, office worker and tram.
Every group of elements in motion follows the map-based movement model with
different speeds. Random destinations are selected by the bicycles and vehicles
in their reach on the map. The elements in motion can be varied, which has no
affect on the characteristics of communication. The simulation framework is on a
Helsinki city map with region 4500 m * 3400 m. This is segmented into 16 casts. In
this framework, the nodes utilize Bluetooth interface with a transmission speed
of 2 Mbps and a range of 10 m. The simulation time for one run is 57600 s. The
warm-up and cool-down periods for every simulation are 2 h. In addition, eight
levels of host density (65, 130, 195, 260, 325, 390, 455, 520, default number of
hosts is 195) are considered.

4.2 Simulation Results

In our simulations, we have considered varying buffer sizes (5, 10, 15, 20, 25 and
30 MB; default buffer size is 10 MB) and distinctive message lifetimes (30, 60,
90, 120, 150, 180, 210 and 240 min, the default is 120 min). Each simulation is
rehashed five times, using various movability models.

Delivery Ra o vs. Number of hosts

0.6
0.5
Delivery Ra o

0.4
0.3
0.2
0.1
0
65 130 195 260 325 390 455 520
Number of hosts

CGOPP GSAF GeoEpidemic

Fig. 1. Message delivery ratio vs. number of nodes

 Centrality Based Geocasting for Opportunistic Networks 709

Average Latency vs. Number of hosts

5000

4000
Average latency
3000

2000

1000

0
65 130 195 260 325 390 455 520
Number of hosts

CGOPP GSAF GeoEpidemic

Fig. 2. Average latency vs. number of nodes.

Number of forwarded messages vs. TTL

Number of forwarded Messages

12000
10000
8000
6000
4000
2000
0
30 60 90 120 150 180 210 240
TTL (Minutes)

CGOPP GSAF GeoEpidemic

Fig. 3. Number of messages forwarded vs. TTL.

First, the default values for the buffer size (10 MB) and message lifetime
(120 min) are considered. The number of nodes is increased and the impact of
this increase on the message delivery probability (DP ) and average latency are
investigated. The results are captured in Figs. 1 and 2. In Fig. 1, it is observed
that the DP for GeoEpidemic, GSAF and the proposed CGOPP protocols also
increases when the number of nodes increases.
The average message delivery probability is 49.07% for CGOPP, 38% for
GSAF and 32% for GeoEpidemic when the number of hosts is varied. In Fig. 2,
we can be inferred that the average latency is varied with the number of hosts
for CGOPP, GSAF and GeoEpidemic. In fact, Fig. 2 clearly indicates that our
proposed routing algorithm outperforms GSAF and GeoEpidemic in terms of
average latency. Indeed, when the buffer capacity enlarges, new routes from
source to destination might emerge, hence, more nodes are involved in delivering
a message.
710 J. Singh et al.

Delivery Ra o vs. Buffer Size

0.4
0.35
0.3
Delivery Ra o

0.25
0.2
0.15
0.1
0.05
0
5 10 15 20 25 30
Buffer Size (MBps)

CGOPP GSAF GeoEpidemic

Fig. 4. Message delivery ratio vs. buffer size

Average Latency vs. Buffer Size

6000
5000
Average latency

4000
3000
2000
1000
0
5 10 15 20 25 30
Buffer Size (MBps)

CGOPP GSAF GeoEpidemic

Fig. 5. Average latency vs. buffer size.

Second, the number of nodes and message lifetime remains unchanged to

their default values. The buffer size is increased and the impact of this increase
on the message delivery probability (DP ) and average latency are analyzed.
The results are captured in Figs. 4 and 5. In Fig. 4, it is observed that in terms
of message delivery ratio performance, CGOPP is 30% better than GSAF and
22% better than GeoEpidemic. In Fig. 5, it is observed that in terms of average
latency performance, CGOPP is 47% better than GSAF and 41% better than
GeoEpidemic.
Third, the number of nodes and message lifetime remains unchanged to their
default values. The TTL is increased and the impact of this increase on the
message delivery probability (DP ), number of messages forwarded, and average
latency are investigated. The results are captured in Figs. 3, 6 and 7. In Fig. 6, it
can be observed that in terms of message delivery ratio performance, CGOPP is
 Centrality Based Geocasting for Opportunistic Networks 711

Delivery Ra o vs. TTL

0.35
0.3
Delovery Ra o 0.25
0.2
0.15
0.1
0.05
0
30 60 90 120 150 180 210 240
TTL ( Minutes)

CGOPP GSAF GeoEpidemic

Fig. 6. Message delivery ratio vs. TTL.

Average Latency vs. TTL

6000
5000
Average latency

4000
3000
2000
1000
0
30 60 90 120 150 180 210 240
TTL (Minutes)

CGOPP GSAF GeoEpidemic

Fig. 7. Average latency vs. TTL

40% better than GSAF and 31% better than GeoEpidemic. In Fig. 3, it is found
that in terms of number of forwarded messages, CGOPP is 32% better than
GSAF and 30% better than GeoEpidemic. Finally in Fig. 7, It can be inferred
that in terms of average latency, CGOPP is 24.50% better than GSAF and
18% better than GeoEpidemic. From the above set of experiments, it can be
concluded that CGOPP outperforms GSAF and GeoEpidemic for the studied
performance metrics.

5 Conclusion
This paper has proposed a novel routing protocol that supports geo-casting in
OppNets (so-called CGOPP). Simulations results using real mobility traces have
shown that CGOPP significantly outperforms GSAF and GeoEpidemic in terms
712 J. Singh et al.

of average latency, delivery ratio, and number of messages forwarded, under

varying number of nodes, buffer sizes, and TTL, chosen as performance metrics.
As future work, we intend to investigate the security of the proposed routing
scheme, for instance against blackhole or wormhole attacks.

References
1. Wakeman, I., Naicken, S., Rimmer, J., Chalmers, D., Fisher, C.: The fans united
will always be connected: building a practical DTN in a football stadium. In:
International Conference on Ad Hoc Networks, pp. 162–177. Springer (2013)
2. Lilien, L., Kamal, Z.H., Bhuse, V., Gupta, A.: Opportunistic networks: the concept
and research challenges in privacy and security. In: Proceedings of the WSPWN,
pp. 134–147 (2006)
3. Newman, M.E.J.: The structure and function of complex networks. SIAM Rev. 45,
167–256 (2003)
4. Boldrini, C., Conti, M., Passarella, A.: Credit based social-cooperation enforcement
autonomic routing in opportunistic networks. In: Autonomic Communication, pp.
31–67. Springer (2009)
5. Pirozmand, P., Wu, G., Jedari, B., Xia, F.: Human mobility in opportunistic net-
works: characteristics, models and prediction methods. J. Netw. Comput. Appl.
42, 45–58 (2014)
6. Spyropoulos, T., Psounis, K., Raghavendra, C.S.: Spray and wait: an efficient rout-
ing scheme for intermittently connected mobile networks. In: Proceedings of the
2005 ACM SIGCOMM Workshop on Delay-Tolerant Networking, pp. 252–259.
ACM (2005)
7. Leguay, J., Friedman, T., Conan, V.: Evaluating MobySpace-based routing strate-
gies in delay-tolerant networks. Wirel. Commun. Mob. Comput. 7, 1171–1182
(2007)
8. Boldrini, C., Conti, M., Passarella, A.: Social-based autonomic routing in oppor-
tunistic networks. In: Autonomic Communication, pp. 31–67. Springer (2009)
9. Lu, S., Liu, Y.: Geoopp: Geocasting for opportunistic networks. In: Wireless Com-
munications and Networking Conference (WCNC), pp. 2582–2587. IEEE (2014)
10. Lu, S., Liu, Y., Liu, Y., Kumar, M.: Loop: a location based routing scheme for
opportunistic networks. In: IEEE 9th International Conference on Mobile Adhoc
and Sensor Systems (MASS), pp. 118–126 (2012)
11. Ott, J., Hyytiä, E., Lassila, P., Vaegs, T., Kangasharju, J.: Floating content: infor-
mation sharing in urban areas. In: IEEE International Conference on Pervasive
Computing and Communications (PerCom), pp. 136–146. IEEE (2011)
12. Ma, Y., Jamalipour, A.: Opportunistic geocast in disruption-tolerant networks.
In: Global Telecommunications Conference (GLOBECOM 2011), pp. 1–5. IEEE
(2011)
13. Haines, E.: Point in polygon strategies. In: Graphics Gems IV, pp. 24–26 (1994)
14. Keränen, A., Ott, J., Kärkkäinen, T.: The ONE simulator for DTN protocol eval-
uation. In: Proceedings of the 2nd International Conference on Simulation Tools
and Techniques, p. 55 (2009)
15. Rajaei, A., Chalmers, D., Wakeman, I., Parisis, G.: GSAF: efficient and flexible
geocasting for opportunistic networks. In: 2016 17th International Symposium on A
World of Wireless, Mobile and Multimedia Networks (WoWMoM), pp. 1–9. IEEE
(2016)
 An Efficient Data Transmission Technique
for Big Video Files over HetNet
in Emerging 5G Networks

Richa Siddavaatam1 , Isaac Woungang1(B) , and Sanjay Kumar Dhurandher2

1
Department of Computer Science, Ryerson University, Toronto, ON, Canada
[email protected], [email protected]
2
Department of Information Technology, Netaji Subhas University of Technology,
New Delhi, India
[email protected]

Abstract. This paper proposes an efficient data transfer scheme for

mobile big video files over HetNet in emerging 5G networks, along with
a data correction technique to deal with the problem of failure to retrieve
the stored data. The proposed scheme exploits the possibility of overlap-
ping coverage received by a UE from multiple base stations (BSs) in
HetNets to expedite the mobile big data transfer between the UE and
the cloud storage center. This is achieved by compressing a big video file
into smaller sized file, then splitting the compressed file into a number of
smaller chunks, which are transmitted simultaneously over the multiple
data links. Simulation results demonstrate that our proposed method
outperforms the baseline data storage method.

Keywords: Big video data · Data storage · Mobile cloud computing ·

HetNet · Error correction code · H.264 ·
High efficiency video coding (HEVC)

1 Introduction

Nowadays, the demand for increasing the storage capacity of cloud systems has
exploded due to explosive usage of smart mobile devices. Moreover, the traffic
generated by video applications executing on mobile devices and the transmis-
sion requirements for video-on-demand services demand for a better network
performance in terms of high data capacity, low data latency. MCC arises as an
instrumental technology to satisfy those requirements by breaking the device’
physical boundaries and enabling the data storage to occur in the remote cloud
data centers side. However, the increased data transmission latency and unre-
liability associated with the wireless channels make the efficient data transfer
over a standalone wireless network a challenging task. To address this prob-
lem, emerging 5G wireless standards have adopted the HetNet and Mobile Edge
Computing (MEC) technologies. HetNet is a hierarchical architecture of macro
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 713–724, 2020.
https://doi.org/10.1007/978-3-030-15032-7_60
714 R. Siddavaatam et al.

cells and small cells networks, designed to improve the indoor and outdoor net-
work coverage and the data transmission rates. On the other hand, the MEC
technology consists of a small cloud data center deployed on the BS site. The
adoption of MEC and HetNet deployment in MCC-based networks is expected to
boost the storage capability of the UE/mobile device by providing a single-hop
storage/computational center.
In the recent literature, there have been a growing interest in investigating the
integration of HetNet and MCC technologies as a promising solution to increase
the overall network capacity by achieving a flexible functionality assignment in
terms of radio access networks. In The MCC domain, most of the breakthroughs
on literature have focused on efficient offloading techniques of computational
intensive tasks from UE to the cloud system. However, there is a clear demand
for ubiquitous storage solutions for mobile big data that can efficiently exploit
the features of 5G networks. As an effort toward this direction, in [6,7], we
proposed algorithms for expediting the transmission of big data files in emerging
5G systems which exploit the overlapping coverage of HetNet. In this paper,
we are proposing an enhanced version of such algorithm, specifically designed
to transmit big video files from UE to the cloud data centers. The proposed
algorithm applies the HEVC compression algorithm to the big video file in order
to significantly reduce its size. Thereafter, the compressed video file is split
into multiple data chunks and simultaneously transmits them over the multiple
available data-paths. It uses the wireless resources efficiently by transmitting the
data chunks such that the size of each chunk is proportional to the instantaneous
uplink data rate of the data-path it will be transmitted on. An error correction
scheme is also embedded in the proposed algorithm to help protecting against a
failure to retrieve up to two data chunks.
The rest of the paper is organized as follows. In Sect. 2, some related work are
described. In Sect. 3, the considered system model is presented. In Sect. 4, the
proposed technique for efficient mobile big video files transmission over HetNet in
emerging 5G networks is described, along with the associated Lost Data Chunk
Recovery Method. In Sect. 5, simulation results are presented. Finally, Sect. 6
concludes the paper.

2 Related Work
In the literature, various techniques have been proposed with an attempt to
meet high data storage and computational capability demands of mobile devices
through the integration of MCC and HetNet technologies. Few of these methods
are described as follows. In [3], Kosta et al. proposed a framework for the offload-
ing of computational intensive tasks to the cloud centers through resource allo-
cation and parallel execution in the MCC network. In [4], Zhang et al. proposed
a cloud-based approach meant to manage the network resources by coordinating
the transmissions of small cells. In [5], Shiraz et al. studied the problem of high
resource demanding computational offloading for mobile cloud computing and
reported on several aspects that should be accounted for when designing efficient
big data transmission techniques in MCC networks.
 Data Transmission Technique for Big Video Files 715

Unlike previous methods that focus more on data security and data manage-
ment issues in cloud storage centers, our proposed method focuses on reducing
the data transmission delay while transmitting big video files to the cloud stor-
age centers through data compression technique and efficient utilization of the
enhanced infrastructure of 5G HetNet.

3 System Model

The considered network model consists of a HetNet comprised of two macro BSs
overlaid with multiple femto BSs, where each BS has a MEC server associated
with it as shown in Fig. 1.

Fig. 1. Considered network architecture

In this model, each BS supports a back-haul connection, which enables each

BS to access the public cloud data center. The considered network supports
NU Es = NUinE + NUout E number of mobile devices (UEs). The most of the UEs in
the network have access to multiple BSs due to the overlapping coverage areas of
the BSs. As per the 5G standards [8], the network is designed by considering the
system parameters given in Fig. 3. In this work, the primary network performance
parameter is data transmission delay (τ ). The total delay incurred by the data
chunk when transmitted on the data path between the ith UE and the j th data-
total
path (τi,j ) is calculated using Eq. (1). If the j th data path connects theith UE
to the associated BS’s MEC server, then the total delay is the delay incurred
on the wireless link, else, if it connects to the public cloud data center, then the
total delay is the sum of delays incurred on the wireless and back-haul links.
We assume that the back-haul link connecting the BS to the public cloud data
center is characterized by a high throughput, hence, the delay incurred while
transmitting the data onto it is negligible.
716 R. Siddavaatam et al.

⎧ wireless
⎪
⎨τi,j - If the data path is between the UE and
total
τi,j = its associated BS’s MEC server. (1)
⎪ wireless
⎩τi,j + τibackhaul - If the data path is between the UE and
the public cloud data center.

The delay incurred on the wireless link is calculated using Eq. (2), where
di,j is the size of data chunk to be transmitted on ij data-path and βi,j is
the maximum possible data rate on the ij data-path calculated using Eq. (3)
following the Shannon-Hartley theorem [12].

wireless di,j
τi,j = (2)
βi,j
βi,j = Wi,j · log2 (γi ) (3)

where the SINR γi experienced by the ith UE on the jth data path is obtained
as:

P̂ij
γi =   , (4)
N
total
BS
P̂ik × [1 + (η ÷ 100)]
k=1,k=j

η is the noise level as a percent of the total interference experienced by the UE

and P̂ij is the power received by that UE from that BS of the j th data-path, Wi,j
is the amount of bandwidth available to the ij th data-path, which itself relies
on the number of physical resource blocks (PRBs) that are available to that
particular data-path. Here, Wi,j = NPijRB × NSC total
, where NPijRB is the number
th total
of PRBs allocated to the ij data-path and NSC is the bandwidth per PRB.
Besides, the wireless path loss model proposed in [9] is adopted to simulate
the up-link and down-link data transmission. The power received by a UE in
case of the downlink channel simulation is calculated as:
m
P̂ = Ptx − PL (dB) (for macro BS) (5)
f
P̂ = Ptx − PL (dB) (for femto BS) (6)
f m
where Ptx (resp. Ptx ) represents the transmit power of the femto BS (resp. macro
BS) and P̂ is the received power by a UE. On the other hand, for the uplink
transmission simulation, the power received by a BS is calculated as:
UE
P̂ = Ptx − PL (dB) (for macro and femto BSs) (7)

UE
where Ptx is the transmit power of the UE and P̂ is the received power by a
BS.
The path loss P Lindoor
f (in dB) between the indoor users and the femtocell
BS is calculated as:
d2 +3df +1.54
P Lindoor
f = 37 + αin × 10 × log10 (d) + 18.3dff (8)
 Data Transmission Technique for Big Video Files 717

The path loss P Lindoor

m between the indoor users and the macrocell BS is calcu-
lated as:

P Lindoor
m = 40 + αout × 10 × log10 (d) + plbp (9)

The path loss P Loutdoor

m between the outdoor users and macrocell BS is calcu-
lated as:

P Loutdoor
m = 40 + αout × 10 × log10 (d) (10)

The path loss P Loutdoor

f between outdoor users and femto BS is calculated as:

P Loutdoor
f = 37 + αout × 10 × log10 (d) + plbp (11)

where d is the distance between the UE and the BS and the rest of the parameters
are listed in Fig. 3.
As per the 5G new radio (5G NR) with a subcarrier spacing of 15 KHz and
a transmission bandwidth of 20 MHz, the number of available PRBs should be
100 [11]. We are assuming that the frequency reuse factor is 1. In this paper, we
have considered that each macro and femto BS has 100 PRBs available for data
transmission, each of which is composed of 12 consecutive subcarriers for one
time slot of 0.5 ms duration. Thus, considering a sub-carrier spacing of 15 KHz,
the bandwidth available to each PRB is 180 KHz. Now, assuming that each BS
equally allocates its available PRBs to its associated UEs, the number PRBs
available to a UE associated with that BS is obtained as 100 ÷ N U Es , from
which the value of Wi,j can be deducted. The intercell interference is controlled
by using the fractional power control algorithm [14].
In the HetNet deployment, small cell BSs transmit at lower power compared
to macro BSs. It is likely that the small cell would be underutilized since the
majority of the UEs would be associated to the macro BSs due to the stronger
signal strength received from the macro BS compared to the signal strength
received from the femto BS. For this reason, a cell range expansion mechanism
such as the one proposed in [10] - which exploits the cell selection bias - is
implemented in this work to extend the usability of the small cell BSs. With the
implementation of this method, the BS association decision with a UE depends
on the signal to interference and noise ratio (SINR) experienced by the UE on
the downlink transmission link.
In this work, we are exploiting the fact that in HetNet, a UE can be associated
to more than one BSs. Typically, a UE will be associated to only those BSs from
which it has experienced a SINR higher than a predetermined SINR-threshold
value set by the network designer. Precisely, as per the LTE standard [10], the
SINR threshold γsmall for the association decision with a small cell BS is obtained
as γsmall = γmacro − α, where α is the cell selection bias and γmacro is the SINR
threshold for the BS association decision with a macro BS. Therefore, at any
time t, a UE is associated with multiple BSs, which provides some performance
whenever a SINR experienced is higher than the respective γf emto /γmacro value.
Since a UE may be associated with multiple BSs, multiple data paths exist
718 R. Siddavaatam et al.

between a UE and the cloud storage system which may be utilized for data
transfer purpose. For simplicity purpose, it is assumed that the network condition
remains unchanged during the transmission of a single video file.
In the HetNet deployment such as the one depicted in Fig. 1, the small cell
BSs such as femto BSs transmit at a lower power compared to macro BSs (A
typical transmit power of macrocell BS is 45 dBm while that of femtocell BS
is 20 dBm). As stated earlier, in this paper, a cell range expansion technique is
adopted for cell association decision. With the implementation of this technique,
a UE can be associated with multiple BSs, hence multiple paths (data links)
exist between a UE and the cloud storage system for data transfer purpose. For
example, in the considered network model shown in Fig. 1, the data from UE-1
can follow 8 different paths to reach the cloud storage system. These data links
are:

• UE-1 =⇒ SC-1 =⇒ MEC server

• UE-1 =⇒ SC-1 =⇒ Public cloud data center
• UE-1 =⇒ M-2 =⇒ MEC server
• UE-1 =⇒ M-2 =⇒ Public cloud data center
• UE-1 =⇒ M-1 =⇒ MEC server
• UE-1 =⇒ M-1 =⇒ Public cloud data center
• UE-1 =⇒ M-3 =⇒ MEC server
• UE-1 =⇒ M-3 =⇒ Public cloud data center.

5G Uplink Physical Layer: In LTE specifications, a PRB is the smallest

element of the resource allocation assigned to the UE/BS by the RAN [11]. As
per the LTE up-link specifications, the number of PRBs available for the UE’s
data transmission with the total transmission bandwidth of 20 MHz is 100 [11].
The task of our proposed radio resource management algorithm is to allocate
these PRBs to all the active UEs in the network sagaciously so as to maximize
the network performance.
Big Video File Preprocessing: For the considered network design model,
each BS is equipped with a MEC server and it has two data paths to the cloud
data centers, namely one path going through the backhaul to the public cloud
data center and the other path going through the MEC server associated with
it. Thus, the number of paths Nipaths that the ith UE can use to reach the cloud
data centres is twice the number NiU Es of BSs associated with that particular
UE. A big video data file to be transferred to the cloud storage is processed
using the well-known H.265 video coding algorithm [1], which compresses the
file to smaller sized file.
HEVC Compression Algorithm: In this work, we use the High Efficiency
Video Coding (HEVC) (or H.265) algorithm [2] for DATA video coding due to
its silent features. The compressed video file is split into Nichunks , where the size
dji of the ith chunk to be transmitted on the data link passing through the j th
BS can be calculated as follows:
 Data Transmission Technique for Big Video Files 719

Nipaths = 2 × NiU Es (12)

Nichunks = Nipaths −2 (13)
D×(βij − βim in)
dji = βisum (βimax − βimin )
, (14)

The right hand size of Eq. (13) is justified by the fact that in prevision to the
design of a data correction technique that can protect against a failure to retrieve
the data chunks in the cloud. Two of the total available paths are reserved for
the transmission of the associated syndromes.

4 Proposed Data Transmission Technique

Once the pre-processed compressed video file is ready for transmission to the
cloud data centers through available multiple data paths. Each of these paths
has a varied maximum data rate calculated using Eq. (3) following the Shannon-
Hartley Theorem [12]. The compressed file is split into Nchunks = Ndata−paths −2
number of data chunks. These data chucks along with the data recovery syn-
dromes φ and ϕ are transmitted to the cloud data centers through the available
data paths. There exists NU E active UEs in the network and each ith UE has
Ji set of data paths available for data transmission. The goal of our proposed
algorithm is to determine the size of the data chunks to be transmitted on each
data path D = {d1 , . . . , dNchunks } and the number of PRBs allocated to each
data path Z = {ζ1 , . . . , ζNchunks }. Our algorithm is meant to choose the data
chunck sizes and PRB allocation such that the average of the delays experienced
by all active UEs in the network is minimized. The delay experienced by a UE
is the maximum of the delays experienced by all data paths. This problem can
be formalized as:

1

NU E
{D, Z} = arg min max τ total
(15)
Ď,Ž NU E i=1 j∈Ji i,j

The major tasks of the proposed algorithm are as follows:

• Decision on splitting of data file into multiple data chunks
• Allocation of the up-link PRBs.

All available up-link PRBs are allocated among all the active UEs in the
network with a reuse factor of 1. The number of PRBs allocated to a UE are
equally split among all the associated BSs. Besides, a data file is split into a
number of data chunks equal to the number of data paths available to the UE.
The size of each data chunk is directly proportional to the maximum data rate
possible on the data-path on which the corresponding data chunk is supposed to
be transmitted. The flowchart of the proposed algorithm is presented in Fig. 2.
720 R. Siddavaatam et al.

Fig. 2. Flow chart of the algorithm for efficient big data transfer.

Lost Data Chunk Recovery Method: When several devices are expected to
share a distributed storage, fault-recovery technique needs to be incorporated
into the storage mechanism since it is likely that any of these devices may fail.
Moreover, in our considered system, the data are transmitted via unreliable
wireless links, hence the probability of the retrieval of a data chunk fail is quite
high. In our proposed algorithm, if the retrieval of any data chunk fails, the
entire video file will be corrupted. In order to provide immunity from failure
to retrieve a data chunk, a RAID-based data correction method adopted from
[15] and [13] is proposed. In the proposed algorithm, the size of the data chunk
is proportional to the uplink capacity of the data link on which it is to be
transmitted. For the calculation of the data recovery syndromes φ and ϕ, it is
considered that the size of φ (resp. ϕ) is that of the largest data chunk (resp.
the smaller data chunks), padded with zeros using Eq. (17). These syndromes
are obtained using Eqs. (18–19) and are saved to the cloud storage, along with
the data chunks of the file. Two of the available data paths are reserved for the
transmission of these syndromes.

S = max size(Di ) (16)

Di (k) for k ≤ size(Di )
P Di (k) = (17)
0 for size(Di ) < k ≤ S
 Data Transmission Technique for Big Video Files 721

φ1 = P D1 where, Di is ith data chunk.

φi = φi−1 ⊕ P Di , for 2 ≤ i ≤ Nichunks
φ = φi where i = Nichunks (18)
ϕ1 = g · P D1
1

ϕi = ϕi−1 ⊕ g i · P Di where i = Nichunks

ϕ = ϕi where i = Nichunks (19)

In (19), the algebra of Galois field GF(28 ) is used, where g denotes the Galois
field generator. A linear feedback shift register is implemented as field generator;
and it is assumed that in case a data chunk is loss, the UE has the knowledge of
the original size of this lost item. Therefore, the zero padded calculated chunk
can be truncated in order to recover the original lost data chunk. This method
provides protection against up to the loss of two data chunks. The calculation
of the lost data chunks can be found in [13].

5 Simulations
In the simulated network environment (Fig. 1), the locations of the femto cells
and BSs are pre-determined. On the other hand, the locations of the UEs are ran-
domly selected, and the Monte Carlo simulation technique is utilized to eliminate
the impact of this randomness on the algorithm’s performance. The algorithm
runs for Nite number of iterations, where for each iteration, the locations of
the UEs are set randomly. The performance measures experienced during each
iteration are averaged over the number of iterations. For comparison purpose,
we have implemented another data transmission strategy called conventional
method, where a UE associates itself only to one BS with the highest SINR
value and a data file is transferred to the cloud storage via the associated macro
BS and the back-haul connection.
A typical simulated network scenario is depicted in Fig. 4, where the coverage
areas of the macro BS M1 and M2 are indicated by the blue circles and the
coverage areas of femto cell BSs are indicated by green circles. The red rhombuses
indicate the locations of multiple UEs in the network.
In order to analyze the performance of the proposed algorithm, a number of
multiple video files are ceded using the HEVC algorithm and the compressed
video files are transmitted on the simulated network. The graph from Fig. 5
displays the performance of the HEVC algorithm considering the size of the
input and output files. It can be observed that with the application of the HEVC
algorithm, the file size reduces significantly with a varying factor ranging form 40
to 50 depending on the content of the video file. It can noted that three different
input files with size around 20 Gbits have generated output files with varying
sizes. The reason is video files with highly correlated frames will be compressed
efficiently compared to the ones with low correlation among the frames.
First, we study the impact of the variation in file size on the transmission
delay, i.e. the time (in seconds) required to transfer the corresponding files to
722 R. Siddavaatam et al.

500

450

U13
400 F2 F3
U20
350 U9
U18
U19 U2
300
U7
U15
250 M1 M2
U5
200

U6
150 U8
U3 U1 U10
F1 U16
U17
100
U4
U14
50
U11 U12

0
0 100 200 300 400 500 600 700 800 900

Fig. 3. System parameters. Fig. 4. Simulated locations of the

network elements.
10 8 performance of H.265 video compression method
9

8 70
By proposed method
By transferring whole file to single CCC
7
60
Size of compressed video file in bits

Average time required to store a file by UE in sec

6
50

40

30
3

20
2

1 10

0 0
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Size of original video file in bits 10 10 Size of original data file 10 10

Fig. 5. Performance of video compres- Fig. 6. Impact of the variation in data

sion algorithm. file size on the data transmission delay.

the cloud storage system while maintaining the same number of UEs and BSs.
The results are captured in Fig. 6. As expected, when the size of the video
file increases, the time required to transfer the file also increases linearly for
both the proposed and conventional methods. The slope of the response line for
the proposed method is 13.15e−10 compared to 3.17e−10 for the conventional
method, proving that the proposed algorithm outperforms the conventional one,
especially in the transmission of big video files.
Second, the number of UEs in the network is varied from 11 to 20, and the
impact of this variation on the transmission delay during the transmission of
1.92 Gbits data file size to the cloud storage while maintaining the same number
and locations of macro/femto BSs in the network is studied. The results are
captured in Fig. 7. It can be observed that under the conventional method, the
time required to transfer the file increases linearly with the number of UEs.
However, the transmission delay under our proposed algorithm is insensitive to
the number of UEs active in the network, but slightly varies with the locations
of the UEs.
Third, the number of femto BSs in the network is varied from 3 to 10 and
the impact of this variation on the transfer delay of a 1.92 Gbits data file size to
 Data Transmission Technique for Big Video Files 723

Performance of proposed method for fixed size of 1.92 Gbits Performance of proposed method for fixed size of 1.92 Gbits
250 300

By proposed method
By transferring whole file to single CCC
By proposed method
By transferring whole file to single CCC 250
Average time required to store a file by UE in sec

Average time required to store a file by UE in sec

200

200

150

150

100

100

50
50

0 0
31 32 33 34 35 36 37 38 39 40 3 4 5 6 7 8 9 10
Number of UEs in the network Number of femto BSs in the network

Fig. 7. Impact of the variation in the Fig. 8. Impact of variation in number

number of active UEs in the network of small BSs on the data transmission
on the data transmission delay. delay.

the cloud storage while maintaining the same aforementioned HetNet setting is
studied. The results are captured in Fig. 8. It can be observed that our proposed
algorithm significantly improves when the number of femto BS in the network
is increased. On the other hand, the transmission delay performance of the con-
ventional method does not yield any improvement. When the number of femto
BSs is increased from 3 to 5, the time required by our proposed algorithm to
transfer the data file is reduced by about 30 s compared to that taken by the
conventional method. This proves that the proposed algorithm efficiently utilizes
the enhanced infrastructure provided by the HetNet.

6 Conclusion
In this paper, a novel method for efficient transfer of mobile big video files to
the cloud storage is proposed, which takes advantage of the multiple paths that
are available between a user and the cloud computing centers in a MCC-HetNet
environment, where each BS is equipped with a MEC server. Simulation results
have shown that the proposed method outperforms the conventional method
in terms of transmission delay under varying file sizes, number of UEs in the
network, and number of femto BSs in the network, respectively. As future work,
we intend to formulate this problem as a combinatorial optimization problem
that can be solved using some novel optimization techniques.

References
1. Sullivan, G.J., Ohm, J.R., Han, W.J., Wiegand, T.: Overview of the high efficiency
video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. 22(12),
1649–1668 (2012)
2. Sze, V., Budagavi, M., Sullivan, G.J.: High Efficiency Video Coding (HEVC).
Integrated Circuit and Systems, Algorithms and Architectures, 39, p. 40. Springer
(2014)
3. Kosta, S., Aucinas, A., Hui, P., Mortier, R., Zhang, X.: ThinkAir: dynamic
resource allocation and parallel execution in the cloud for mobile code offload-
ing. In: Proceedings of IEEE INFOCOM, Orlando, FL, USA 2012, pp. 945–953
(2012)
724 R. Siddavaatam et al.

4. Zhang, N., Cheng, N., Gamage, A.T., Zhang, K., Mark, J.W., Shen, X.: Cloud
assisted HetNets toward 5G wireless networks. IEEE Commun. Mag. 53(6), 59–65
(2015)
5. Shiraz, M., Sookhak, M., Gani, A., Shah, S.A.A.: A study on the critical analysis
of computational offloading frameworks for mobile cloud computing. J. Netw.
Comput. Appl. 47, 47–60 (2015)
6. Siddavaatam, R., Woungang, I., Carvalho, G., Anpalagan, A.: Efficient ubiqui-
tous big data storage strategy for mobile cloud computing over HetNet. In: IEEE
Global Communications Conference (GLOBECOM), Washington, DC 2016, pp.
1–6 (2016)
7. Siddavaatam, R., Woungang, I., Carvalho, G., Anpalagan, A.: An efficient method
for mobile big data transfer over HetNet in emerging 5G systems. In: IEEE 21st
International Workshop on Computer Aided Modelling and Design of Communi-
cation Links and Networks (CAMAD), Toronto, ON 2016, pp. 59–64 (2016)
8. Lien, S.Y., Shieh, S.L., Huang, Y., Su, B., Hsu, Y.L., Wei, H.Y.: 5G new radio:
waveform, frame structure, multiple access, and initial access. IEEE Commun.
Mag. 55(6), 64–71 (2017)
9. Siddavaatam, R., Anpalagan, A., Woungang, I., Misra, S.: Ant colony optimiza-
tion based sub-channel allocation algorithm for small cell HetNets. Wirel. Pers.
Commun. 77(1), 411–432 (2014)
10. Deb, S., Monogioudis, P., Miernik, J., Seymour, J.P.: Algorithms for enhanced
inter-cell interference coordination (eICIC) in LTE HetNets. IEEE ACM Trans.
Netw. 22(1), 137–150 (2013)
11. Zyren, J., McCoy, W.: Overview of the 3GPP Long Term Evolution Physical
Layer, white paper. Freescale Semiconductor, Inc., 22 2007
12. Shannon, C.E.: Communication in the presence of noise. Proc. IRE 37(1), 10–21
(1949)
13. Anvin, H.P.: The mathematics of RAID-6 (2007)
14. Castellanos, C.U., Villa, D.L., Rosa, C., Pedersen, K.I., Calabrese, F.D.,
Michaelsen, P.-H., Michel, J.: Performance of uplink fractional power control in
UTRAN LTE. In: Proceedings of IEEE Vehicular Technology Conference (VTC),
Singapore, 11–14 May 2008, pp. 2517–2521 (2008)
15. Plank, J.S.: A tutorial on Reed–Solomon coding for fault tolerance in RAID like
systems. Softw. Pract. Exp. 27(9), 995–1012 (1997)
 Backhaul-Based Cooperative Caching
in Small Cell Network

Yu-Ting Wang(B) , Yun-Zhan Cai, Lo-An Chen, Sian-Jhe Lin,

and Meng-Hsun Tsai

Department of Computer Science and Information Engineering,

National Cheng Kung University, Tainan, Taiwan, R.O.C.
[email protected], [email protected], [email protected],
[email protected], [email protected]

Abstract. As mobile devices become more and more ubiquitous, mobile

traffic demands are increasing exponentially. To deal with such challenge,
caching is considered as an effective way to reduce redundant transmis-
sions. Cooperative caching allows the cache storage to be used more
efficiently and increases cache hit rate to further reduce the downloading
latency. Recent researches assume that a mobile user can be served by
multiple small cell base stations (SBSs) simultaneously, yet this hypoth-
esis requires costly re-design of the infrastructure’s physical layer. In this
paper, we propose a backhaul-based cooperative caching scheme which
groups several SBSs into a cluster and the SBSs cooperate through wired
interfaces. We first formulate a delay minimization problem for coopera-
tive caching, and then propose a greedy algorithm to obtain the content
placement with maximum reduced delay. We also explore the impact on
end-to-end latency under different simulation parameters, such as cache
size and the skewness of the popularity of requested contents.

1 Introduction
With global growth in the amount of mobile devices and connections, wireless
data traffic has increased tremendously. Mobile video traffic alone accounts for
more than half of all mobile data traffic worldwide, and is expected to reach over
three quarters by 2021 [5]. To cope with the ever growing demands for mobile
traffic, small cell base stations (SBSs) are anticipated to provide more capacity
by reusing the spectrum locally in the 5th generation (5G) networks [4].
Cooperative caching, which enlarges cached contents set, is considered a
promising solution. Recent research works assume that a mobile user is served by
multiple SBSs through radio simultaneously [7,9]. This hypothesis poses issues
on interference. Although the issues can possibly be mitigated by adopting inno-
vative interference management techniques (such as interference alignment and
coordinated multipoint), such solutions are costly and/or demand re-design of
the physical layer. Nonetheless, degradation in spectral efficiency is inevitable
due to increased connection distances between users and SBSs.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 725–736, 2020.
https://doi.org/10.1007/978-3-030-15032-7_61
726 Y.-T. Wang et al.

A good cooperative caching strategy depends mainly on high hit rate and low
cooperative overhead. In this paper, we propose a backhaul-based cooperative
caching (BCC) scheme which achieves high hit rate as well as low cooperative
overhead.
To further improve end-to-end latency, We formulate an optimization prob-
lem subject to constraints on cache capacity aiming to minimize the average
downloading delay for each user by determining optimal content placement in
the cooperative architecture. Then we propose a segment fill with maximum
reduced delay algorithm to solve this problem efficiently.
This paper is organized as follows. The related works on cooperative caching
are reviewed in Sect. 2. We present our architecture and the BCC scheme in
Sect. 3. Section 4 presents performance evaluation through simulation experi-
ments. Finally, our conclusions are provided in Sect. 5.

2 Related Works
Cooperative caching is a well investigated topic in recent years. In [3], Chen et al.
have investigated cooperative caching in cluster-centric small cell networks. Stor-
ing different contents at SBSs in the same cluster increases the cache hit rate
due to the enlarged set of cached contents. In this case, a mobile user may be
served by farther SBSs. Higher distances and path loss bring on the degradation
in spectrum efficiency. However, the tradeoff between caching content diver-
sity and spectrum efficiency is not well captured. Such tradeoff relationship is
addressed in [8] where each SBS can store only one file. Both [3] and [8] neglect
the restraint of radio resources for merely considering single user case. In [9],
Zhang et al. obtain minimized average file transmission delay by optimizing
content placement and cluster size based on stochastic topology information,
channel quality and file popularity. The studies in [3,7–9] all assume that a
mobile user is able to be served by multiple SBSs simultaneously which requires
costly re-design of the physical layer to deal with the challenges on subsequent
interference.
In this paper, we present a backhaul-based cooperative caching scheme in
SBS networks, which does not need to re-design the physical layer for dealing
with interference. For network topology, the distribution of mobile users and
SBSs are modeled as 2-D homogeneous Poisson Point Process (PPP).
To allow each file to be downloaded in segments from different SBSs, we
assume that files are encoded into independent segments by rate-less fountain
coding [6]. To formulate the content placement problem, we consider cooperative
cache hit rate based on content popularity, channel quality, and the overhead in
SBSs cooperation.

3 Backhaul-Based Cooperative Caching

In this section, we first describe the system model in Subsect. 3.1 and formu-
late the content placement problem in Subsect. 3.2. Then we derive hit rate
 Backhaul-Based Cooperative Caching in Small Cell Network 727

and wireless transmission delay in Subsect. 3.3. Finally, we propose segment fill
with maximum reduced delay algorithm for content placement problem in Sub-
sect. 3.4.

3.1 System Model

Figure 1 illustrates the architecture for our BCC scheme, and the important
notations are listed in Table 1. SBSs are randomly located and modeled as inde-
pendent PPP with density λb , and grouped into K-sized clusters based on their
cell identities. For example, when cluster size K = 3, SBS-1, SBS-2 and SBS-3
form a cluster, while SBS-4, SBS-5 and SBS-6 form another one, and so forth.
SBSs in the same cluster are called neighbors although they are not necessar-
ily adjacent to each other since they are interconnected via backhaul links. For
instance, the next neighbor of SBS-2 is SBS-3, and the next neighbor of SBS-3
is SBS-1 (not SBS-5 or SBS-6).

Fig. 1. BCC architecture

We use F = {f1 , f2 , ..., fF } to present file library in descending order of

popularity. Let P = {pf1 , pf2 , ..., pfF } denote the corresponding popularity. A
file content is encoded into independent segments of equal size L bits by rate-
less fountain coding and can be decoded when the number of segments of file-f
collected by a user reaches sf so that the file size sf · L is equal to the size of
the original file [6].
We assume that the SBSs are homogeneous with the same capabil-
ity and system parameters such as cache capacity, bandwidth, transmit
power, etc. Each SBS can store C coded segments at most. Denote Bu =
[bu,1 , bu,2 , ..., bu,k , ..., bu,K ] as the sequence of SBSs in a cluster which user-u
728 Y.-T. Wang et al.

Table 1. Notations

Notation Semantic
K Cluster size
λb Density of SBSs
λu Density of users
F File library; in terms of a sequence of files in descending
order of popularity
F Number of files in the file library
P Sequence of popularity for corresponding files in F
pf Popularity of file-f (f1 ≤ f ≤ fF )
L Length of each coded segments in bits
sf Sufficient number of segments for decoding file-f
cf,m Number of segments of file-f cached at SBS-m
C Cache capacity in terms of number of segments
Bu Sequence of SBSs in a cluster which user-u belongs to
bu,k The kth SBS in the cluster which user-u belongs to
(1 ≤ k ≤ K)
D̄nb Neighbor backhaul transmission delay
D̄re Remote backhaul fetching delay
D̄ Average downloading delay
S̄ Average number of segments of a file
Ru,f,k Ratio of file-f segments that user-u gets from bu,k
R̄f,k Ratio of file-f segments that all users get from bu,k
A System area
H̄k Average hit rate at bu,k for all users
T̄ Average transmission rate
Tu File transmission rate for user-u
β SBS radio bandwidth
Nu The number of users under bu,1
Pt SBS transmit power
du Distance between user-u and its bu,1
α Path loss exponent
γ Additive Gaussian noise
I Inter-cell interference
J Content placement matrix

belongs to. From a user’s aspect, bu,1 is user-u’s serving SBS and bu,2 to bu,K
are user-u’s neighbor SBSs. For example, in Fig. 1, since user-3 attaches to SBS-5,
the sequence of SBSs [b3,1 , b3,2 , b3,3 ] is equal to [SBS-5, SBS-6, SBS-4].
 Backhaul-Based Cooperative Caching in Small Cell Network 729

The file segments fetching process flowchart for user-u requesting file-f is
shown in Fig. 2. To request file-f , the user tries to fetch the segments directly
from bu,1 ’s cache. First, we initialize k to 1 (step 1 in Fig. 2). We use cf,m to
denote the number of segments of file-f cached on SBS-m. Then bu,1 checks
whether the retrieved segments are sufficient or not (step 2 in Fig. 2). If the
retrieved segments are insufficient for decoding, the request is forwarded to its
neighbors successively, and the segments from these neighbor caches are trans-
mitted to bu,1 via wired backhaul links (steps 3–5 in Fig. 2). The neighbor caches
are not delivered to user-u through bu,1 until total number of segments reaches
K
sf (i.e., i=1 cf,bu,i ≥ sf ). If the sum of cf,m in the cluster is less than sf , SBS
bu,1 fetches the remaining segments from remote servers via backhaul (step 6 in
Fig. 2). The whole file-f is delivered to user-u while the retrieved segments are
sufficient for decoding (step 7 in Fig. 2).

User-u
requests yes
file- f

2
1 no 3
k=1 ∑ki=1 c f ,bu,i ≥ s f ? k = k+1

7
deliver the whole
file- f to user-u End
5 4
retrieve c f ,bu,k
segments from bu,k k > K?
no

yes

6
retrieve remaining
segments from
remote servers

Fig. 2. File segments fetching process at bu,1 (for user-u requesting file-f )

3.2 Problem Formulation for Content Placement

We consider two types of backhaul delays: file transmission from neighbor SBSs
delay and file fetching from remote servers delay. Both delays are exponentially
distributed with the mean D̄nb and D̄re respectively. The average downloading
delay D̄ consists of radio transmission delay, neighbor backhaul transmission
delay and remote backhaul fetching delay. These delays are mainly affected by
average content hit rate H̄k . Let S̄ present the average number of segments of a
730 Y.-T. Wang et al.

file, and thus S̄L is the average file size in bits. Denote by S̄LT̄H̄1 the radio delay
of file segments delivery from bu,1 to user. Let H̄k D̄nb ∀k ∈ [2..K] express the
cooperative overhead and S̄LT̄H̄k ∀k ∈ [2..K] present the radio delivery delay of
the segments obtained from neighbor SBSs. Last, caused by content miss, the
backhaul delay of fetching file segments from remote servers and the radio delay
of transmitting the remaining segments are given as H̄K+1 D̄re and S̄LH̄T̄K+1 .
Hence, the average delay is estimated as:
 
S̄LH̄1 S̄L
D̄ = + + D̄nb (1 − H̄1 − H̄K+1 )
T̄ T̄
 
S̄L
+ + D̄re H̄K+1
T̄
S̄L
= + D̄nb (1 − H̄1 − H̄K+1 ) + D̄re H̄K+1 , (1)
T̄
and the optimal caching problem can be formulated as:

minimize D̄
{cf,bu,k }


F
s.t. cf,bu,k ≤ C
f =1

cf,bu,k = [0..sf ], ∀f ∈ F, k ∈ [1..K]. (2)

The above problem is similar to a multiple knapsack problem, a well-known NP-

complete problem, where each SBS represents a knapsack, and each file segment
represents an item.

3.3 Derivation for Hit Rate

In the above fetching process, the ratio of file segments of file-f that user-u can
get from bu,k is denoted by Ru,f,k , which can be expressed as
   
1 
k−1
Ru,f,k = min cf,bu,k , max sf − cf,bu,i , 0 . (3)
sf i=1

We denote Ru,f,K+1 as the ratio of remaining segments of file-f fetched from

remote servers for user-u, and R̄f,k as the average ratio of file-f segments obtain-
ing from bu,k for all users:
K
Ru,f,K+1 = 1 − k=1 Ru,f,k , (4)
λu A
R̄f,k = 1
λu A i=1 Ri,f,k , (5)

where λu is the mobile user density modeled by PPP, A is the system area and
λu A is the number of users. On the other hand, the hit rate of file-f for user-u
 Backhaul-Based Cooperative Caching in Small Cell Network 731

at bu,k can be expressed as pf Ru,f,k . Therefore, the average content hit rate at
the bu,k for each user is given by:


F
Hu,k = pf Ru,f,k . (6)
f =1

And denote by HK+1 the miss rate of content requests of each user:


K
Hu,K+1 = 1 − Hu,k . (7)
k=1

Hence, the average hit rate at bu,k and miss rate of a request in the system are
respectively given by:
F
H̄k = f =1 pf R̄f,k , (8)
K
H̄K+1 = 1 − k=1 H̄k . (9)

When user-u is served by bu,1 , the file transmission rate is given as Tu , where
β is available radio bandwidth for each SBS. In the signal-to-interference-plus-
noise ratio, Pt is the SBS transmit power in mW. Assuming a user attaches to
the nearest SBS, du is the distance between user-u and bu,1 in meter. The pass
loss exponent is expressed as α. Let γ denotes the additive Gaussian noise and
I denotes the inter-cell interference both in mW. Finally, average transmission
rate T̄ is derived as:

 −α 
Pt du
βE log2 γ+I
T̄ = E[Tu ] =
E[Nu ]
   
β log2 γ+I Pt
− αE[log2 du ]
≈ λu
λb
      
λb β Pt α 1 4−π
= log2 − log √ −
λu γ+I log2 2 λb 2π
⎛ ⎛  −α ⎞ ⎞
⎜ Pt 2 λb
√1
λb β ⎜ ⎟ α(4 − π) ⎟
= ⎝ 2⎝
log ⎠+ ⎠. (10)
λu γ+I 2πlog2

3.4 Segment Fill with Maximum Reduced Delay Algorithm

Equation (1) suggests that the average downloading latency are mostly influ-
enced by bu,k ’s hit rate. And the average hit rate rises as a segment is added
into a SBS’s cache. Consequently, the content placement affects the average
file downloading delay. Denote J as the content placement matrix, which is
expressed by
732 Y.-T. Wang et al.
⎡ ⎤
c1,1 c1,2 . . . c1,K
⎢ c2,1 c2,2 . . . c2,K ⎥
⎢ ⎥
J =⎢ . .. .. . ⎥, (11)
⎣ .. . . .. ⎦
cF,1 cF,2 . . . cF,K

where K is the number of SBSs in a cluster and each element in J is the number
of segments of cached content. When we add a segment of file-f ∗ into the cache
of SBS-m∗ , the updated content placement matrix is denoted by J  , which is
expressed as
⎡  ⎤
c1,1 c1,2 . . . c1,K
⎢ c2,1 c2,2 . . . c2,K ⎥
⎢ ⎥
J = ⎢ . .. . . . ⎥, (12)
⎣ .. . . .. ⎦
cF,1 cF,2 . . . cF,K

where

cf,m + 1 if f = f ∗ and m = m∗
cf,m = . (13)
cf,m otherwise

The marginal gain G(J , f ∗ , m∗ ) is defined as the decrement of average delay

after the content placement matrix is updated. That is,

G(J , f ∗ , m∗ ) =D̄(J ) − D̄(J  )

=D̄nb (1 − H̄1 − H̄K+1 ) + D̄re H̄K+1
− (D̄nb (1 − H̄1 − H̄K+1
 
) + D̄re H̄K+1 )

=D̄nb (H̄1 − H̄1 )

+ (D̄re − D̄nb )(H̄K+1 − H̄K+1 ). (14)

Algorithm 1 illustrates the proposed Segment Fill with Maximum Reduced

Delay (SFMRD) algorithm.
Initially, T̄ is calculated based on Eq. (10) (line 1). The SFMRD algorithm
starts with empty cache (line 2). In each iteration, we try to add one segment
for a file and an SBS, and then calculate the marginal gain G(J , f, m) based
on Eq. (14) (lines 4 and 5). The segment cached at certain SBS that minimizes
the average delay is added into content placement matrix in each iteration until
the cache storage in individual SBS is filled up (where the condition is checked
at line 6). In practical system, file popularity can be obtained from the request
history in a period of time. For example, if the last 24-h file popularity data is
adopted, the SFMRD algorithm should be executed once a day and the content
placement is renewed every day. The time complexity of Algorithm 1 is provided
in Theorem 1.
Theorem 1. The time complexity of Algorithm 1 is O(C · F · K 2 ), where C is
the cache capacity at each SBS, F is total number of files in file library, and K
is the number of SBSs in a cluster.
 Backhaul-Based Cooperative Caching in Small Cell Network 733

Algorithm 1. Segment Fill with Maximum Reduced Delay (SFMRD) Algorithm

Input: λb : SBS density; λu : user density; P: file popularity distribution; {sf }: file size dis-
tribution; K: cluster size; C: cache capacity; D̄nb : average backhaul neighbor delay; D̄re :
average backhaul remote delay;
Output: J : content placement
1: calculate T̄ based on Eq. (10)
2: J = 0F,K
3: repeat
4: f ∗ , m∗ = arg max G(J , f, m) based on Eq. (14)
5: J ← J

6: until F f =1 cf,m = C ∀m ∈ SBS set
7: return J

Proof. In each iteration, F files and K SBSs are selectable for the added segment,
such that the time complexity is O(F · K) for the iteration. The total cache
capacity at K SBSs is C · K segments. Therefore, the time complexity to fill up
all cache storage is O(C · F · K 2 ).


4 Performance Evaluation
In this section, we evaluate the efficiency of the proposed BCC scheme by com-
paring with two existing schemes using simulations developed in C++. These
simulations are performed in a square area of 1 km2 where SBSs and users are
Poisson distributed with the density λb = 5 × 10−5 /m2 and λu = 5 × 10−4 /m2
independently. The backhaul delays of fetching files from neighbor SBSs and from
remote servers are exponentially distributed with the mean of 75 ms and 200 ms
respectively. We assume the additive Gaussian noise power is −105 dBm and the
inter-cell interference power is −75 dBm. Note that the signal-to-interference-
plus-noise term in Shannon’s capacity formula should be expressed as a linear
power ratio instead of logarithmic decibels. The mean of request intervals per
user is exponentially distributed with mean 500 ms.
The file popularity input is based on real-world data trace [2]. These data
we use include all the videos in YouTube entertainment category collected at
December 2006 and their view counts are updated at March 2018. We take
the top 10% popular videos and proportionally select 1000 videos with their
view amounts normalized as file popularity. The most popular video has over
14 million view counts while the least popular one has been watched about 20
thousand times in this data set.
We compare our performance with two other approaches: non-cooperative
(NC) caching scheme and Zhang’s wireless cooperative caching scheme [9]. NC
scheme caches the most popular file at each SBS and a user is served by the
nearest SBS. In Zhang’s scheme, a user is allowed to attach to multiple SBSs
and receive encoded file segments from their cache. If the obtained segments are
insufficient to be decoded, the nearest SBS fetches the remaining segments from
remote server via backhaul. Besides, Zhang et al. assume that each SBS caches
the same number of file-f segments, i.e., cf,m = cf,n for all m and n. In Zhang’s
734 Y.-T. Wang et al.

scheme, a user belongs to group-k in the k-th nearest SBS it attaches to, e.g.,
user-u belongs to group-1 in the nearest SBS, and group-2 in second nearest SBS.
In addition, the authors derive the optimal wireless bandwidth ratio allocated
to different groups of each SBS. However, the bandwidth allocation does not
perform well in our simulation as we implement such scheme. Therefore, the
results of Zhang’s approach in our simulation, the radio bandwidth is equally
allocated to active users transmitting files at each SBS without applying the
group-based bandwidth allocation.
Zipf distribution is a widely adopted model to measure file popularity distri-
bution [1], given as:
1
fz
pf =  F 1
, (15)
i=1 iz
F
where f =1 pf = 1 and the exponent z reflects the skewness of popularity
distribution. When z = 0, the popularity of the contents is uniform, while larger
z represents that the user requests get more concentrated on a few certain files.
We use top 10% popular YouTube data trace and Zipf distribution with z = 0.5
and z = 1. Figure 3(a) illustrates the impacts of cache size and file popularity
skewness on the hit rate of NC, Zhang, and BCC schemes. Higher popularity
skewness leads to higher hit rate since some very popular videos account for
majority of the view counts.
This figure shows that the result of top 10% popular YouTube data is closer to
Zipf distribution when z = 1 than that when z = 0.5. When the normalized cache
size is 0.2 in the real data set, BCC increases the hit rate by 23.7% compared to
NC because cooperative caching enlarges the cache capacity. BCC’s hit rate is
7.7% better than Zhang’s approach due to the different assumptions on caching
scheme.
The average downloading delay with respect to cache size is demonstrated
in Fig. 3(b). Under the real data set, the average delay can be reduced by 16.2%
and 9.6% in the proposed scheme compared with NC and Zhang respectively
when the normalized cache size equals to 0.2. The increased hit rate reduces
the probability to fetch file segments from remote servers, which cause long
latency. Furthermore, the cooperative overhead in BCC is lower than that of
Zhang’s scheme since the wired transmission does not degrade as much as radio
transmission with larger connection distances.
As shown in Fig. 4(a) depicts that the BCC’s cooperative overhead reduces
as the average backhaul delay of fetching files from neighbor SBSs decreases.
When the mean of backhaul remote delay is 200 ms and the cache capacity is
0.025, BCC’s benefit increases from 12.2% to 24.7% and from 7.2% to 20.4%
compared with NC and Zhang as the backhaul neighbor delay decreases from
75 ms to 1 ms. In Fig. 4(b), when the backhaul remote delay rises from 100 ms
to 1000 ms, the average delay can be reduced from 5.5% to 29.8% compared to
NC, while maintaining around 7% compared with Zhang.
 Backhaul-Based Cooperative Caching in Small Cell Network 735

(a) content hit rate

(b) downloading delay

Fig. 3. Effects of cache size and file popularity skewness on (a) content hit rate; (b)
downloading delay

(a) backhaul neighbor delay (b) backhaul remote delay

Fig. 4. Performance comparison of our BCC scheme with other cache replacement
algorithms

5 Conclusions
In this paper, we designed a backhaul-based cooperative caching (BCC) scheme
in SBSs and presented a polynomial-time greedy algorithm to achieve low-
736 Y.-T. Wang et al.

est average downloading delay through the content placement with maximum
reduced delay. Simulation results show that our BCC scheme outperforms NC
up to 29.8% and Zhang up to 20.4%. In the conducted simulation experiments,
BCC always outperforms NC, and outperforms Zhang in more than 90% cases.

Acknowledgements. This work was financially supported by the Center for Open
Intelligent Connectivity from The Featured Areas Research Center Program within
the framework of the Higher Education Sprout Project by the Ministry of Education
(MOE) in Taiwan. Tsai’s work was supported in part by the Ministry of Science and
Technology (MOST) under Grant 106-2221-E-006-007- and Grant MOST 107-2221-E-
006-062-, and Industrial Technology Research Institute (ITRI).

References
1. Breslau, L., Cao, P., Fan, L., Phillips, G., Shenker, S.: Web caching and Zipf-like
distributions: evidence and implications. In: IEEE Conference on Computer Com-
munications (INFOCOM 1999), vol. 1, pp. 126–134. IEEE (1999)
2. Cha, M., Kwak, H., Rodriguez, P., Ahn, Y.Y., Moon, S.: I tube, you tube, every-
body tubes: analyzing the world’s largest user generated content video system. In:
Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement, pp.
1–14. ACM (2007)
3. Chen, Z., Lee, J., Quek, T.Q., Kountouris, M.: Cooperative caching and transmission
design in cluster-centric small cell networks. IEEE Trans. Wirel. Commun. 16(5),
3401–3415 (2017)
4. Ge, X., Tu, S., Mao, G., Wang, C.X., Han, T.: 5G ultra-dense cellular networks.
IEEE Wirel. Commun. 23(1), 72–79 (2016)
5. Index, C.V.N.: Global mobile data traffic forecast update. 2016–2021 white paper.
Accessed 2 May 2017
6. MacKay, D.J.: Fountain codes. IEE Proc. Commun. 152(6), 1062–1068 (2005)
7. Shanmugam, K., Golrezaei, N., Dimakis, A.G., Molisch, A.F., Caire, G.: Femto-
caching: wireless content delivery through distributed caching helpers. IEEE Trans.
Inf. Theory 59(12), 8402–8413 (2013)
8. Song, J., Song, H., Choi, W.: Optimal caching placement of caching system with
helpers. In: 2015 IEEE International Conference on Communications (ICC), pp.
1825–1830. IEEE (2015)
9. Zhang, S., He, P., Suto, K., Yang, P., Zhao, L., Shen, X.S.: Cooperative edge caching
in usercentric clustered mobile networks. IEEE Trans. Mob. Comput. (2017)
 A MEC-Assisted Method for Early Handover
Using the Fully Distributed Mobility
Management (MEC-F-DMM) Architecture

Chung-Ming Huang1(&), Duy-Tuan Dao1, and Meng-Shu Chiang2

1
Department of Computer Science and Information Engineering,
National Cheng Kung University, Tainan, Taiwan
{huangcm,daotd}@locust.csie.ncku.edu.tw
2
Department of Computer Science and Information Engineering,
Far East University, Tainan, Taiwan
[email protected]

Abstract. Distributed mobility management (DMM) is a promising network-

based mobility management protocol for mobile Internet. However, in case of an
increasing number of MNs are associating and attaching to an AP of the DMM’s
Mobility Anchor and Access Router (MAAR), the AP/MAAR may be easily
overloaded. The emergence of Mobile Edge Computing (MEC) can allow
operators to get better benefit in monitoring network’s situation. In this paper, A
MEC-assisted method for early handover using the Fully Distributed Mobility
Management (MEC-F-DMM) architecture was proposed. MEC-assisted DMM
can (i) define the trigger-timing to force MNs to do early handover to the better
target mobile network, (ii) decide which MNs are suitable to do early handover
to avoid the network unbalance. The performance analysis has shown that the
proposed MEC-F-DMM method has better network performance than the tra-
ditional fully DMM method in terms of average queuing delay, the packet
delivery time, and throughput.

1 Introduction

The mobile Internet has been posing numerous new challenges related to various
extensions of the base mobility management protocols in wireless networks. The key
idea of the Distributing Mobility Management (DMM) approach, which was explored
and standardized by the Internet Engineering Task Force (IETF), is based on the
flattened architecture that can employ mobility anchors much closer to MNs [1, 2].
In DMM, the control and data planes are distributed among network components and
managed near the network edge, i.e., MN’s default gateway. By flattening architecture,
DMM can improve its scalability and integrate with other ubiquitous communication
services, e.g., mobile-edge clouds. DMM can be partially distributed (P-DMM), i.e.,
only the data plane is distributed, or fully distributed (F-DMM), i.e., both the data plane
and the control plane are distributed. P-DMM is modeled by retaining a centralized
control plane, which may cause some problems that exist in the centralized mobility
management (CMM) protocols, such as single point of failure and scalability. Whereas,
the control function is distributed without depending on a centralized element in the
© Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 737–750, 2020.
https://doi.org/10.1007/978-3-030-15032-7_62
738 C.-M. Huang et al.

F-DMM approach [3, 4]. Most of these researches focused on the handover scenario for
which the timing of triggering the handover is when the sensed signal strength of the
currently attached AP connected with the associated MAAR becomes weak, i.e., when
a Link Going Down event happens.

Fig. 1. The illustrated configuration for having early handover.

Nevertheless, the other concern for triggering handover is load balance. Referring
to Fig. 1, let (i) MNx, which is having some high bandwidth’s application, e.g.,
streaming, be entering into the signal coverage of MAAR1’s APA, i.e., MNx is having
handoff from its previously attached AP to APA, (ii) there be many MNs that are
currently attached with APA and not a lot of MNs that are attached with MAAR2’s
APB, and (iii) there be some MNs that are in the overlapped region of APA and APB, in
which MN1–MN4 are attached with APA and MN5 is attached with APB. Then, it can
have some MNs that are currently attached with APA and can receive APB’s signal,
e.g., (a) MN3 and MN4 or even (b) MN1 and MN2, to have early handover to APB. In
 A MEC-Assisted Method for Early Handover Using the Fully DMM 739

this way, APA’s load can be lowered to accommodate MNx’s required bandwidth and
loads of APA and APB can be balanced.

Fig. 2. The illustrated scenario of the proposed MEC-F-DMM.

To have the early handover, the key point is to how to have the monitor service to
overlook related MAARs’ Wi Fi APs’ situations. In this work, we proposed to use the
Mobile Edge Computing (MEC) mechanism to assist early handover. MEC is a new
and emerging technology that is currently being standardized by ETSI [5–7]. MEC
allows to expose real-time signal and network information (such as users location,
network load, etc.) to enhance the capabilities of the broadband mobile network [9].
MEC can be used to deploy a number of computing resource elements, i.e., the MEC
servers, which are more proximate with mobile users. Thus, the computing capacity of
MEC servers can help estimate the network situation of mobile users for supporting the
real-time-required handover decision. In this work, a MEC-assisted method for early
handover using the F-DMM (MEC-F-DMM) architecture is proposed, for which the
abstract’s configuration is depicted in Fig. 2. Referring to Fig. 2, it is assumed that APA
is associated with the MEC Server1, which serves MAAR1’s domain, and APB is
associated with MEC Server2, which serves MAAR2’s domain. Additionally, two
neighboring MEC Servers, e.g., MEC Server 1 and MEC Server 2, exchange their
contexts, periodically or aperiodically, i.e., event-driven. Thus, MEC server 1/2
740 C.-M. Huang et al.

monitors the network connectivity in real time and delivers its own network condition
to MEC server 2/1 periodically and/or aperiodically. Two main issues that are to be
tackled in the proposed MEC-F-DMM architecture are as follows:
(1) When to trigger the early handover such that some MNs can connect to the new
APB/MAAR? At the link layer, the mobility management concentrates on an
alteration of connection in the access point and a handover is triggered to execute
the detachment and attachment with different access points. In case of an
increasing number of MNs are associating and attaching with APA/MAAR1, APA/
MAAR1 may be easily overloaded. To relieve the load of APA/MAAR1, it can
switch some MNs, which are (i) currently attached with APA/MAAR1 and (ii) are
in the overlapped region of APA/MAAR1 and APB/MAAR2, from APA/MAAR1
to APB/MAAR2, even if it is not the time to do handover for these MNs using the
traditional handoff criteria, e.g., the sensed signal strength of APA/MAAR is not
low, to keep the load balance of the mobile network. That is, when the load of
APA/MAAR1 exceeds a predetermined threshold, some MNs that are currently
attached with APA/MAAR need to do the early handover to APB/MAAR such that
APB/MAAR can accommodate some MNs that are currently attaching with APA/
MAAR. Let all of the MNs and all of the APs report their contexts, for which
(i) MNs need to report their contexts including the consumed bandwidth, IDs and
RSSIs of the sensed APs, and (ii) APs need to report their context including the
available bandwidth and the IDs of attached MNs, to the associated MEC server.
The early handover can be done through the help of MEC Servers based on the
received contexts. MEC server can calculates the timing of having some MNs to
do the early handover from their currently attached APA/MAAR to the new APB/
MAAR based on the current situation of the mobile network.
(2) Which MNs are the suitable ones that can be selected for the early handover to the
other AP for keeping the load balance between APs and keep or even improve the
performance of the F-DMM? When an MN that has the high requested bandwidth
is entering into an AP, it may need to have some MNs in the overlapped area to
handover to the other AP to maintain the load balance. Depending on the
requested bandwidth of MN that is entering into APA/MAAR1, MEC Server
selects some MNs for which the sensed signal strength of the previous AP (PAP),
i.e., APA/MAAR1 depicted in Fig. 2 is getting low and the sensed signal strength
of the new AP (NAP), i.e., APB/MAAR2, is getting high and the available
bandwidth of NAP, i.e., APB/MAAR2, is still good, to have the early handover
from PAP (APA/MAAR1) to NAP (APB/MAAR2).
The contributions of the proposed MEC-F-DMM are as follows: (1) Using the
MEC Server to monitor the network quality, i.e. the overloaded issue, and then have
some MNs to perform early handover to the better target network, and (2) selecting
suitable MNs to do early handover for maintaining the load balance between APs.
Hence, the situations of mobile network, e.g., average queuing delay, the packet
delivery time, and throughput are improved. Based on the performance analysis, the
proposed MEC-F-DMM method has the better performance than the traditional F-
DMM method in terms of average queuing delay, the packet delivery time, and
throughput.
 A MEC-Assisted Method for Early Handover Using the Fully DMM 741

The rest of this paper is organized as follows: Sect. 2 provides related works.
Section 3 presents the proposed MEC-F-DMM method in details. Section 4 presents
analytic network model and the performance analysis of the proposed MEC-F-DMM
method. Finally, Sect. 5 provides the conclusion remarks and future work.

2 Related Work

In this Section, related work of the handover mobility using MEC paradigm is pre-
sented and discussed. Mobile Edge Computing (MEC) plays a fundamental role in
improving the performance of the next-generation mobile Internet. The MEC frame-
work for supporting mobility can support the delay-sensitive applications and services.
Several ongoing research topics that investigate MEC-based mobility models have
been done in recent years [7–9]. The authors in [8] proposed the ETSI MEC-based
solution to support user mobility for maintaining efficient service repositioning in
mobile edge networks for stateless applications. When the user departs from the
coverage range of a specific cloudlet, the MEC system provides the prompt detection of
the user’s association. Relying on maintaining the lightweight service replication of
specific applications and enabling fast relocation of stateless micro-services for end
users, the method can support to (1) decrease the migration times based on MNs’
movements and (2) increase application service resiliency in case of link failure. The
authors in [9] investigated the use of mobility awareness hierarchical MEC servers for
serving mobile smart devices, i.e. IoTs. When mobile users move along, their handover
tasks may be accomplished by MEC servers. To support identifying the network
context awareness, MEC servers collect the received information from mobile users
within the local access network. Thus, MEC Servers help estimate (1) states of the
network based on the local real-time information and (2) network applications running
on the devices. However, similar to [8], service replication that is needed in the work of
[9] results in the complexity of operations in terms of the high processing power and
storage resources.
The aforementioned studies have considered using MEC to improve the network
performance in terms of achieving low-delay and supporting the mobility awareness for
the interactive communication of mobile users. The monitoring capacity of MEC has
the advantages of assisting the real-time network information. Therefore, it is important
to add more efforts to study the MEC paradigm that can efficiently support the network-
based mobility, especially the integration to the F-DMM.

3 The Proposed MEC-F-DMM Method

In this Section, details of the fully DMM based on Mobile Edge Computing (MEC-F-
DMM) are presented. In the proposed MEC-F-DMM, each MEC server plays the role
of monitoring and calculating the network quality. Table 1 depicts the possible situ-
ations of the PAP (APA/MAAR1) and NAP (APB/MAAR2), for which the early
handover is triggered when the loading of PAP is high and NAP’s load is low.
742 C.-M. Huang et al.

Table 1. Possible situations of the PAP and NAP for triggering the early handover.
PAP (APA/MAAR1) NAP (APB/MAAR2) MEC trigger (early handover)
Loading High High No
Loading High Low Yes
Loading Low High No
Loading Low Low No

The key idea of the proposed MEC-F-DMM method is to have the MEC Server to
monitor the networking situation based on the reported contexts from its administered
APs and MNs. When the number of MNs that are attached with APA/MAAR1 is
increased, it makes the networking situation of APA/MAAR1 be bad, i.e., it may become
overloaded. The 1st issue, i.e., when to trigger the early handover, is tackled as follows.
If the required bandwidth of MN, which is entering into APA/MAAR1, is higher than the
available bandwidth of APA/MAAR1, then MEC Server 1 can trigger the early han-
dover. The 2nd issue, i.e., which MNs need to have early handover, is tackled as follow.
Based on the received contexts, e.g., (i) the consumed bandwidth of each MN, (ii) IDs
and RSSIs of the sensed APs of each MN, and (iii) the available bandwidth of each AP
that is administered by MEC Server, the MEC server can compute the score of each MN
associating with each of its sensed APs. After that, the MEC server creates a score list
containing those MNs (MNi, i = 1..n,) that are in the overlapped area of PAP (APA) to
NAP (APB). The number of MNs that need to do early handover is based on the
requested bandwidth of MNx that is entering into APA/MAAR1. The MEC server
selects those MNs having the higher scores in the list, for which the sum of these MN’s
consumed bandwidth is equal to or greater than the required bandwidth that the entering
MN needs, and then have these MNs to do the early handover to the PAP (APA) to NAP
(APB). The MEC Server calculates a score for each MN associating with each X of the
APs for which the MN can receive X’s signal, i.e., MN is inside the signal coverage of
X. In contrast with the regular fully DMM handover processing, i.e., the handover
preparation messages are started when the Link Going Down event is triggered, the
proposed MEC-F-DMM method can select the suitable MNs to do the early handover to
have the better load balance and relieve the potential overloaded situation.

Fig. 3. The message flow chart of the early handover using the proposed MEC-F-DMM scheme.

The operational and messages flowchart of the proposed MEC-F-DMM scheme is

depicted in Fig. 3. Two phases in the proposed MEC-F-DMM control scheme are
 A MEC-Assisted Method for Early Handover Using the Fully DMM 743

(1) the handover preparation and (2) the handover phase. The handover procedure of
the proposed MEC-F-DMM is explained as follows:
Phase 1: Early Handover Preparation
Step 1: When to trigger the early handover?
Let MNx, which is using the high requested bandwidth application service, i.e., VoIP or
streaming video, be entering into APA/MAAR1’s domain from its previously attached
AP/MAAR. MEC Server1 can figure out whether the early handover processing needs
to be enabled or not using the following way. Since all of the administered MNs and APs
of MEC Server1 need to report their contexts to MEC Server1, MEC server1 can
calculate the loading index of APA/MAAR1 based on the requested bandwidth of MNx
and the available bandwidth of APA/MAAR1. The early handover will be triggered if
the requested bandwidth of MNx, which is denoted as the RBW (MNx), is higher than
the available bandwidth of APA/MAAR1, which is denoted as ABW (APA/MAAR1).
Step2: Which MNs are selected to have the early handover?
MEC Server1 and MEC Server2 can exchange their received contexts periodically.
The MEC Server uses formula (1) to determine which ones of the MNs, i.e., MNi,
i = 1..n, that are currently in the overlapped region of PAP (APA/MAAR1) and NAP
(APB/MAAR2), can be selected to do early handover from PAP (APA/MAAR1) to
NAP (APB/MAAR2).
8
> ScoreMNi ¼ PCBW  ðP RSSIApa-MNi  P RSSIApb-MNi Þ
RSSI RSSI
>
i
ðaÞ
< CBWi Apa-nMNj Apb-nMNj

P1::n 1::n 1::n

ð1Þ
>
>
k
i¼1 CBWMNi [ RBWðMNxÞ ðbÞ
:
ABWðNAPÞ  ABWðPAPÞ ðcÞ

Part (a) of Formula (1) is used to calculate the score of each MNi. If a MN
consumes the higher bandwidth, it can contribute the higher bandwidth to PAP (APA/
MAAR1) when it has the handover from PAP (APA/MAAR1) to NAP (APB/MAAR2).
Thus, a MN is assigned the higher score if consumes the higher bandwidth, which can
be derived based on the ratio of the current consumed bandwidth and the total con-
sumed bandwidth. Since an RSSI value is defined in a negative form (e.g., −80), the
closer the value is to 0, the stronger signal it has been received. MN can have the early
handover if the signal strength of the new AP is strong. Therefore, a MN is assigned the
higher score if the RSSI, which can be derived based on (i) the ratio of the sensed
APA’s RSSI of MNi and the sum of the sensed APA’s RSSI of all MNi, i = 1..n, minus
(ii) the ratio of the sensed APB’s RSSI of MNi and the sum of the sensed APB’s RSSI of
all MNi, i = 1..n. The parameters in the part (a) of Formula (1) is explained as follows:
CBWi, i = 1..n, denotes the consumed bandwidth of MNi, i = 1..n, that are in the
overlapped region of PAP (APA/MAAR1) and P NAP (APB/MAAR2); RSSIAPa-MNi
denotes the sensed APA’s RSSI of MNi, i = 1..n; ð RSSIAPa-nMNj Þ denotes the sum of
1::n
the sensed APA’s RSSI of MNi, i = 1..n, that are currently attached with
P PAP (APA);
RSSIAPb-MNi denotes the sensed APB’s RSSI of MNi, i = 1..n; ð RSSIAPb-nMNj Þ
1::n
denotes the sum of RSSI of MNi, i = 1..n, that are currently can sense NAP (APA). Part
744 C.-M. Huang et al.

(b) of Formula (1) denotes that the sum of the consumed bandwidth of those k MNs,
which are selected to do early handover from PAP (APA/MAAR1) to NAP (APB/
MAAR2) should be higher than the requested bandwidth of the MNx, which is entering
into APA/MAAR1. Part (c) of Formula (1) denotes the constraint that the available
bandwidth of NAP (APB/MAAR2) after k EHMNj, j = 1..k, having the early handover
from PAP (APA/MAAR1) to NAP (APB/MAAR2) is still higher than the available
bandwidth of PAP (APA/MAAR1) to avoid the NAP (APB/MAAR2) becoming
overloaded when the proposed method is applying. After getting the score of each MN,
MEC1 Server1 creates a score list, which ranks the scores from high to low. Then,
MEC Server1 selects the top k EHMNj, j = 1..k, from the score list, for which the sum
of the consumed bandwidth of these k MNs satisfies the condition of part (b) and part
(c) of Formula (1). It means that MEC Server1 can decide k MNs that have to do early
handover for keeping the load balance based on the requested bandwidth of MNx.
Phase 2: The Handover Phase
Step 3: The handover initiated (HI) message is sent by MAAR1 to MAAR2 for routing
state’s registration. Upon receiving the HI message, MAAR2 replies the Handover
acknowledge message (Hack) message to confirm the updated routing state.
MAAR1 and MAAR2 establish a bidirectional tunnel between them. Then,
MAAR1 (1) informs k of those MNi, i = 1..n, whose scores are the higher k ones
and are denoted as EHMNj, j = 1..k, through APA for preparing the handover
processing, (2) APA informs these k EHMNj, j = 1..k, to disconnect with APA and
attach to NAP (APB/MAAR2).
Step 4: When each one of EHMNj, j = 1..k, sends the L2 report to NAP (APB/MAAR2)
and connects to NAP, NAP notifies MAAR2 about the attachment of EHMNj,
j = 1..k. Since the tunnel is ready, the on-going packets of the peered EHCN11,
EHCN12…, EHCN1k will be transmitted through the established tunnel between
MAAR1, which becomes the mobility anchor for those packets, and MAAR2, for
the prevention of data loss.
After the handover processing of EHMNj, j = 1..k, from MAAR1 to MAAR2 is
finished, the early handover is done.

4 Analytic Network Model and Performance Analysis

This Section presents and discusses the analytical models to evaluate and compare the
network performance of the traditional fully DMM and the proposed MEC-F-DMM. In
this work, the analytical models are based on the fluid-flow models, for which the
handover rate i.e., the cross-boundary rate, is derived from moving speed, perimeter
and node density of the area of MAAR’s domain [10]. In addition, several significant
and valuable metrics, including the average queuing delay, throughput and packet loss
rate are derived in detail. The parameters of the analytic network models are given in
Table 2. Each MAAR’s administration domain has the same circular shaped cell. It is
assumed that CNs are functional to be non-mobile for the sake of simplicity. An MN is
considered to experience an Internet IP-handover when it crosses from one AP to other
AP. Therefore, the IP-handover rate is equal to the cell crossing rate.
 A MEC-Assisted Method for Early Handover Using the Fully DMM 745

Table 2. The parameters of the analytic network

Parameters Meaning Value
R Radius 100 m
v Average velocity of MN 10–60 m/s
Sa The average number of transmitted packets of a session 1000 bytes
hopMAAR-MAAR The average number of hops between MAA and MAAR 5 hops
hopMN-MAAR The average number of hops between MAAR and MN 2 hops
Sc The average control packet size 80 bytes
Lw The propagation latency of wired link 2 ms
Ld The propagation latency of wireless link 10 ms
BWd Bandwidth of the wireless link 10 Mbps
BWc Bandwidth of the wired link 100 Mbps
k The number of MN have to early handover 20% total MNs

4.1 Analytical Model for Performance Evaluation

In this sub-Section, the analytical network model for determining and evaluating the
handover performance in terms of packet delivery time, average queuing delay,
throughput, and packet loss rate is formulated.

4.1.1 Average Queuing Delay

The queuing delay, which is usually higher when the overload status occurs, is defined
as the summation of the waiting time of each packet in the queue of an AP. Let n
denote as the node density in one coverage area, i.e., the average number of mobile
users per square meter (mobile/m2). Then, the handover rate of regular fully DMM is
qr = (n * v * L)/p, whereas v denotes velocity of MN, L is the diameter of cell area.
Since APA that has k MNs to do early handoff using the proposed MEC-DMM method,
the handover rate of APA is formulated as qm = ((n − h) * v * L)/p, where as h
denotes the node density of k MNs having the early handover to APB. Referring to
Little’s theory [11] and the M/M/1 queuing model for the process of transmitting
packets [12], the average queueing delay time of MNs that still stay in APA is formed
as follows:

1
Wp-Fully-DMM ¼ ð2Þ
l  qr

1
Wp-MEC-F -DMM ¼ ð3Þ
l  qm

Whereas l denotes the average service rate [13].

4.1.2 Packet Delivery Time

The packet delivery time is defined as the total time elapsed when a packet of a session
is delivered from CN to MN through the tunnel. To obtain the packet delivery time, the
746 C.-M. Huang et al.

packet transmission delay and the average queuing delay should be considered. The
packet transmission delay is proportional to the hop-count distance from the source
network to the destination network [14]. It is assumed that wired links are stable and
reliable. Then, the delay time of a packet of size p sent from x to y is represented as
follows:

Sc
Tx;y ¼ hop  ð þ Lw þ W p Þ ð4Þ
BW
where hop, BW, Sc, Lw, and Wg are hop-count, the available bandwidth, the average
control message size, propagation latency and the average queuing delay at each router
in wired links respectively.
(a) The packet delivery time using the traditional fully DMM

PDTFully-DMM ¼ TL2 þ TMN-MAAR þ THI-fully-DMM þ maxfTHI ; TMAAR1-MAAR2 g

ð5Þ

Sc
TMN -MAAR ¼ hopMN -MAAR ð þ Ld Þ ð6Þ
BWd

Sc
THI -fully-DMM ¼ TMAAR1-MAAR2 ¼ hopMAAR-MAAR  ð þ Lw þ Wp-fully-DMM Þ ð7Þ
BWc

whereas Wp-fully-DMM denotes the wireless queuing delay of the traditional fully DMM,
and Lw, Ld denote the propagation latency of wire link and wireless link, respectively.
(b) The packet delivery time using the proposed MEC-F-DMM method
The average packet delivery time of using the proposed MEC-F-DMM method, i.e.,
after k MNs having finished the early handover, is as follows:

PDTMEC-F-DMM ¼ TL2 þ TMN-MAAR þ THI-MEC-F-DMM þ maxfTHI ; TMAAR1-MAAR2 g

ð8Þ

Sc
TMN -MAAR ¼ hopMN -MAAR ð þ Ld Þ ð9Þ
BWd

Sc
THI -MEC-F-DMM ¼ TMAAR1-MAAR2 ¼ hopMAAR-MAAR  ð þ Lw þ Wp-MEC-F-DMM Þ
BWc
ð10Þ

4.1.3 Throughput
Throughput (Tpt) is defined as the total volume of transmitted data packets in a session,
which is Sad * Lu, over the session delivery time [14].
 A MEC-Assisted Method for Early Handover Using the Fully DMM 747

TptFully-DMM ¼ Sad  Lu =½ðPDTFully-DMM þ ðSad  1Þ  qÞ ð11Þ

TptMEC-F-DMM ¼ Sad  Lu =½ðPDTMEC-F-DMM þ ðSad  1Þ  qÞ ð12Þ

whereas Sad is the average number of transmitted packets in a session, Lu is the average
packet size from CN to MN, and q is the packet-to-packet interval time. PDTFully-DMM
and PDTMEC-F-DMM is the packet delivery time for transmitting Lu packets using the
traditional fully DMM and using the proposed MEC-F-DMM method, respectively.
PDT is the packet delivery time from CN to MN, which is calculated as the total delay
time of transmitted packets given in formulas (5) and (8).

4.2 Performance Analysis

In this Sub-section, the performance analysis is investigated and presented. The con-
sidered performance metrics are mobility signaling cost, average handover latency and
packet loss. The default values of these parameters for performance analysis are given
in Table 2, which are based on the ones given in [13, 14].

4.2.1 The Average Queuing Delay

Figure 4 shows the comparison of average queuing delay between the proposed MEC-
F-DMM method and the traditional fully DMM method in terms of velocity of MNs.
When the velocity of MNs is increased, the average queuing delay of the proposed
MEC-F-DMM method is lower than that of the traditional fully DMM method. From
the result, when the velocity of MNs is around 10 ms, the average queuing delay of the
proposed MEC-F-DMM method is 60 ms; however, the average queuing delay of the
traditional fully DMM is 170 ms. Furthermore, when the velocity of MNs is around
40 m/s to 50 m/s, the average queuing delay time of the proposed MEC-F-DMM
method is lower than that of using the traditional fully DMM method for around 35% to
50%. The reason is that the proposed MEC-F-DMM method can have some MNs to do
the early handover through the help of MEC Servers, and thus the unbalanced load was
alleviated. As a result, the average queuing delay of the proposed MEC-F-DMM
method can be reduced.

Fig. 4. The average queuing delay vs. velocity of MNs.

748 C.-M. Huang et al.

4.2.2 Packet Delivery Time

Figure 5(a) shows the comparison of packet delivery time vs. number of hop count
between the proposed MEC-F-DMM method and the traditional fully DMM method.
When the number of hop count is changed from 2 to 10, the value of the packet
delivery time of the proposed MEC-F-DMM method is lower than that of the tradi-
tional fully DMM method. Figure 5(b) shows the comparison of packet delivery time
vs. wireless link delay between two methods. From Fig. 5, the proposed MEC-F-DMM
method has generates the better performance in terms of the packet delivery time,
which results from the reduction of the average queuing delay of the proposed MEC-F-
DMM method.

(a) (b)

Fig. 5. (a) The packet delivery time vs. number of hop count and (b) the packet delivery time vs.
wireless link delay.

4.2.3 Throughput
Figure 6(a) and (b) shows the comparison of throughput performance between the
proposed MEC-F-DMM method and the traditional fully DMM method based on the
number of packets in a session and session arrival rate, respectively. Figure 6(a) shows
that the throughput of the proposed MEC-F-DMM method is higher than that of the
traditional fully DMM method when the average number of packets in a session is
increased from 10 to 100 packets. At the average number of transmitted packets is 50,

(a) (b)

Fig. 6. (a) The throughput vs. the average number of packets in a session and (b) the throughput
vs. session arrival rate.
 A MEC-Assisted Method for Early Handover Using the Fully DMM 749

the throughput of the proposed MEC-F-DMM method is about 7800 Kbyte/s and is
higher than that of the traditional fully DMM method for about 6000 Kbyte/s. Figure 6
(b) shows that the throughput of the proposed MEC-F-DMM method is higher than that
of the traditional fully DMM method when session arrival rate is increased from 1 to
10. The proposed MEC-F-DMM method has the better throughput because of reducing
the packets delivery time by shortening the average queuing delay.

5 Conclusions

In this paper, a MEC-assisted method for early handover using the fully distributed
mobility management (MEC-F-DMM) architecture was proposed to improve the net-
work performance. The main idea of the proposed MEC-F-DMM method is to use
MEC Server to decide the timing to trigger the early handover and decide which MNs
to have the early handover for relieving the overloaded situation and thus resolve the
problem of overload in DMM. To evaluate the performance, we have developed an
analytic model for analyzing the average queuing delay, the packet delivery time and
throughput. Through performance analysis, it is has shown via that the proposed MEC-
F-DMM mechanism has the better performance when comparing with the traditional
fully DMM in terms of the average queuing delay, the packet delivery time and
throughput. For the future work, it can apply the proposed MEC-F-DMM method in
the vehicular network environment, i.e., the vehicle-to-everything (V2X) architecture.

Acknowledgement. This research was supported by the National Science Council of the
Republic of China, Taiwan, under the contract number MOST 107-2221-E-006-139.

References
1. Das, S.K.: Mobility management—a personal perspective. Comput. Commun. 131, 26–31
(2018)
2. Biswash, S.K., Jayakody, D.N.K.: Performance based user-centric dynamic mode switching
and mobility management scheme for 5G networks. J. Netw. Comput. Appl. 116, 24–34
(2018)
3. Carmona-Murillo, J., Friderikos, V., González-Sánchez, J.L.: A hybrid DMM solution and
trade-off analysis for future wireless networks. Comput. Netw. 133, 17–32 (2018)
4. Bernardos, C.J., De la Oliva, A., Giust, F.: A PMIPv6-based solution for distributed mobility
management. Internet-Draft (Work in Progress), draft-bernardos-dmm-pmip (2017)
5. ETSI MEC ISG: Mobile edge computing (MEC); framework and reference architecture.
ETSI MEC GS 003 (2016)
6. Ko, S.W., Han, K., Huang, K.: Wireless networks for mobile edge computing: spatial
modelling and latency analysis. IEEE Trans. Wirel. Commun. 17(8), 5225–5240 (2018)
7. Mao, Y., You, C., Zhang, J., Huang, K., Letaief, K.B.: A survey on mobile edge computing:
the communication perspective. IEEE Commun. Surv. Tutor. 19(4), 2322–2358 (2017)
8. Farris, I., Taleb, T., Flinck, H., Iera, A.: Providing ultra-short latency to user-centric 5G
applications at the mobile network edge. Trans. Emerg. Telecommun. Technol. 29(4), 1–14
(2018)
750 C.-M. Huang et al.

9. Zhang, K., Leng, S., He, Y., Maharjan, S., Zhang, Y.: Mobile edge computing and
networking for green and low-latency Internet of Things. IEEE Commun. Mag. 56(5), 39–45
(2018)
10. Lee, J.H., Bonnin, J.M., You, I., Chung, T.M.: Comparative handover performance analysis
of IPv6 mobility management protocols. IEEE Trans. Ind. Electron. 60(3), 1077–1088
(2013)
11. Gross, D., Shortle, J.F.: Fundamentals of Queueing Theory. Wiley, London (2008)
12. Murtadha, M.K., Noordin, N.K., Ali, B.M., Hashim, F.: Design and evaluation of distributed
and dynamic mobility management approach based on PMIPv6 and MIH protocols. Wirel.
Netw. 21(8), 2747–2763 (2015)
13. Rasem, A., St-Hilaire, M., Makaya, C.: Efficient handover with optimized localized routing
for proxy mobile IPv6. Telecommun. Syst. 62(4), 675–693 (2016)
14. Munir, K., Lagrange, X., Bertin, P., Guillouard, K., Ouzzif, M.: Performance analysis of
mobility management architectures in cellular networks. Telecommun. Syst. 59(2), 211–227
(2015)
 Link Level Capacity Analysis of Precoded
FBMC/OQAM Systems

Jihed Ghodhbane1,2(B) , Noureddine Boujnah3,4(B) , and Ridha Bouallegue1(B)

1
Innov’COM Laboratory, Sup’Com, Carthage University, Carthage, Tunisia
[email protected], [email protected]
2
National engineering school of Gabes, ENIG, Gabes, Tunisia
3
Faculty of science of Gabes, FSG, Gabes, Tunisia
4
Waterford Institute of Technology, TSSG, Waterford, Ireland
[email protected]

Abstract. This paper deals with capacity enhancement of the Fil-

ter bank based multicarrier with Offset-QAM systems (FBMC/OQAM)
using precoding technique. FBMC/OQAM is one of the candidate wave-
form for future fifth generation (5G) mobile communication systems.
Its advantages are twofold: Using orthogonal carriers and higher data
rate compared to OFDM system. By introducing filter bank techniques,
no cyclic prefix (CP) is required in FBMC/OQAM therefore capacity
increase. In this paper we insert a precoder to cope with channel effect
at each subcarrier. Optimizing system capacity using precoder is equiv-
alent to constrained convex optimization problem.

1 Introduction
Mobile communications systems are constantly evolving in terms of traffic vol-
ume and number of connected devices, creating the need for new physical layers
to meet these new challenges. In particular, network densification and increasing
throughput requirements necessitate even more efficient use of the very limited
spectral resource [1].
It is in this context that multi-carrier waveforms have emerged as part of the
4th generation of mobile communications.
Indeed, the standardization of the OFDM waveform allows the implementa-
tion of very high speed transmissions. It also translates this special interest to
multicarrier modulations, which are seen as solutions to boost system capacity
and reduce interferences [1,2].
Although, OFDM is characterized by a good spectral effeciency, issues related
to frequency localization still exists. In other hand, the OFDM does not allow the
implementation of asynchronous tolerant communications within the network
and, on the contrary, imposes strict synchronization constraints in time and
frequency.
To cope with this issues, a technique based on OFDM and filter bank was
recently proposed using different shape filter [2,3].
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 751–761, 2020.
https://doi.org/10.1007/978-3-030-15032-7_63
752 J. Ghodhbane et al.

The new waveforms are proposed to fifth generation cellular radio applica-
tions such as FBMC/OQAM. In fact, FBMC/OQAM can deal with the chan-
nel frequency selectivity without any guard interval requirement. Furthermore,
FBMC uses a frequency well-localized pulse shaping which reduces significantly
the out-of-band (OOB). However, FBMC introdues a shift of half the symbol
period between the in-phase and quadrature components of QAM symbols.
This arrangement along with a correctly designed prototype filter leads to a
transceiver system in which perfect data recovery at the receiver involves the use
of an analysis filter bank, removal of the phase shifts from the analyzed signal
samples, and taking the real parts. The imaginary parts are simply ignored.
However, one may realize that these imaginary parts, that are referred to as
intrinsic interference, are also dependent on the transmitted real-valued data
symbols and thus may carry some information beyond what exists in the real
parts [3,4].
A way to overcome this limitation is to linearly precode the signal before
transmission. Besides, this technique can be used to reduce the effect of inter-
ference that maximizes the capacity of the system as well [5,6].
It has been shown in [7] that they have made an attempt to take advantage of
the Tomlinson Harashima precoder (THP) in improving the system performance.
In [8] a Tomlinson Harashima precoder is proposed for frequency flat channels
in each subcarrier. In [9] they proposed FBMC/OQAM precoders based on the
signal-to-leakage plus noise ratio and on the signal to-interference plus noise
ratio. In [10] three precoders are inserted in each subcarrier to combat ICI in the
FBMC system. In [11] linear precoders with the linear equalizer were extended
for the FBMC/OQAM system.
This paper aims to precisely answer the question: how much the precoder
can increase the capacity of FBMC/OQAM system.
In this contribution, we study the effect of the precoding technique on the
capacity of the FBMC/OQAM systems.
For this reason, the precoding technical coefficients are formulated as an
optimization problem with convex constraints in order to minimize interference
and therefore maximize capacity.
The rest of the paper is organized as follows, in the subsequent section, we
present the system model. Then, we focus on useful power, interference and noise
in Sect. 2 and derive their expressions, in Sect. 3 the analysis of the capacity is
conducted in. Section 4 presents the numerical results. Finally, Sect. 5 concludes
the paper and suggests some ideas of future works.

2 System Model
A block diagram of a precoded FBMC/OQAM system is presented in Fig. 1.
In this system model, the symbols to be transmitted are precoded by a vector
B. The symbols are firstly set into parallel streams by using the serial to parallel
converter (S/P). Afterward, these symbols are multiplied by a precoder vector B.
The output symbols are mapped into M-QAM, and will be sent to the synthesis
 Link Level Capacity Analysis of Precoded FBMC/OQAM Systems 753

filter bank block transmitter; the output signal obtained is filtered by the channel
response and corrupted by an Additive White Gaussian Noise.
At the receiver level, the signal will be set into parallel streams, then it will
pass in the analysis filter before being equalized and decoded and subsequently
reconstructed by the OQAM demodulator. Finally, the signal will be transformed
by the Parallel to Serial converter (P/S) in order to make the M-QAM decision
and recover the estimated bits [4–6].

Fig. 1. A precoded FBMC/OQAM system

Where SFB is a synthesis filter bank, AFB is a filter of analysis, B is the

2
precoding vector, η is the additional white Gaussian noise with variance σkn .
The input symbols are multiplied by a preceding vector B. The precoded
symbols transmitted on each k-th subcarrier are written as follow [11]:

+∞
uk (t) = bk akn δ(t − nT ) (1)
n=0

where δ is the kronecker function, akn is a complex symbol of the k-th subcarrier
and time of index n, bk denotes the k-th element of the precoding vector and T
denoting the period of akn symbols.
Then, the precoded symboles are processed by an OQAM modulation block;
this block is composed by two operations [8,11]: The first operation is a complex
to real conversion: the real and imaginary parts of a complex input symbol are
separated from tow pure real and imaginary symbols, the second operation of
OQAM modulation is the multiplication of OQAM symbols by θkn [12,13]:
θkn = j (n+k) (2)
where j = exp ( πj
2 ).
This mapping is realized in such a way that two symbols on the same car-
rier are successively real and imaginary pure. Moreover, the adjacent symbols
between two subcarriers are also real or imaginary pure. The OQAM symbols
are formed from uk (t) where the real and imaginary parts are time staggered by
half a symbol period.
Thus, for the even sub-carriers, OQAM signals are written:

+∞
ck (t) = j n skn δ(t − nT /2) (3)
n=0
754 J. Ghodhbane et al.

In the same way, for odd k, we have [11]:


+∞
ck (t) = j (n+1) skn δ(t − nT /2) (4)
n=0

where skn = (bk akn ) for n even and skn = (bk akn ) for n odd.
Whatever the value of k, we can write [15,16]:

+∞
ck (t) = j (n+k) skn δ(t − nT /2)
n=0

+∞ (5)
= θkn skn δ(t − nT /2)
n=0

where K is the number of subcarriers,  and  are the real and imaginary parts
of the nth symbol and the kth sub-carrier, respectively.
The ckn symbols are processed by an SFB block.
The base-band continuous time model of the FBMC/OQAM transmitted
signal can be written as follows [11–13,15]:

K−1 +∞
nT 2jktπ
x(t) = skn exp (jφkn )f (t − ) exp ( )
2 T
k=0 n=0

K−1 +∞ (6)
= skn fkn (t)
k=0 n=0

fkn (t) is defined by:

nT 2jktπ
fkn (t) = f (t − ) exp (jφkn ) exp ( ) (7)
2 T
where: f (t) is the response of the transmitter prototype prototype filter , fkn is
the shifted versions of f (t) in time and frequency and φkn = π2 (n+k)−πkn is the
phase shift between adjacent transmitted symbols along the time and frequency
axes.
After passing through the multipath channel, the baseband version of the
received signal, noise taken apart, is given by [12,13]:

K−1 +∞
y(t) = skn f˜kn (t) + η(t) (8)
k=0 n=0
 +∞
where: f˜kn (t) = −∞ h(τ, t)fkn (t − τ )dτ is the channel distorted version of fkn ,
h(τ, t) is the temporel response of the channel, the h(τ, t) depend on the chan-
nel delay spread, and the maximum Doppler frequency, and η(t)is a base-band
complex additive white Gaussian noise with zero mean and variance σ 2 .
 Link Level Capacity Analysis of Precoded FBMC/OQAM Systems 755

The demodulation of the received signal denoted by zkn are obtained by pro-
jecting the received signal y(t) into the shifted version of the receiver prototype
filter gkn (t) [5,11].
zkn =< exp (jωkn )gkn , y >R
(9)

gkn (t) is defined by:

nT 2jktπ
gkn (t) = g(t − ) exp (jφkn ) exp ( ) (10)
2 T
where <, >R is the real scalar product, g(t) is the response of the receiver proto-
type filter, gkn is the shifted versions of g(t) in time and frequency and ωkn is the
phase used to compensate the phase shift by the channel at the time- frequency
position.
The prototype filter was designed to satisfy the orthogonality condition in
the real field [6]:

< fkn (t)gk n (t) >R = δkk δnn (11)

If we assume that zkn are invariant by frequencyand time translations within
the time-frequency lattice, keeping the time-frequency lattice unchanged and
according to the real orthogonality condition then zkn can be can be expressed,
as [6,7]:

zkn = skn < exp (jωkn )gkn , f˜kn >R + sk n < exp (jωkn )gkn , f˜k n >R
(k n )=(kn)

+ < exp (jωkn )gkn , η >R

= Ukn + Ikn + Nkn
(12)
where Ukn is the useful information, Ikn is the interference term, and Nkn is a
noise term at the demodulator output, it’s a additive gaussian noise with 0 mean
2
and σkn variance.
After demodulating FBMC, the symbol zkn is equalized, two methods of
equalization to cope with channel impairment, ZF and MMSE can be applied
[7].
Finally, the equalized symbols are processed by an OQAM demodulator.
The obtained symbols will be transformed by the Parallel to Serial converter
in order to make the M-QAM decision and recover the estimated. That is the
real part of this symbol particularly an estimate of the data vector [12,13].

2.1 Useful Power

In this subsection, we will derive the useful power expression of the
FBMC/OQAM system. The average power corresponding to the detected sym-
bol can be expressed as [16,17]:
PUc = E[Ukn
2
]
(13)
= E[s2kn ]({exp (−jωkn ) < gkn , f˜kn >})2
756 J. Ghodhbane et al.

Let note by Es the avreage transmitted energy symbols skn then:

Es = E[(skn f 2 )]
= E[skn ]f 2 (14)

From (13) and (14), the useful power of the OQAM-FBMC signal can be calcu-
lated as follows:
PU = E[PUc ]
Es (15)
= E[([exp (−jωkn ) < gkn , f˜kn >])2 ]
f 2
L−1
Let Ẽs = Es l=0 el is the average energy received per complex symbol, L is
the number of paths channel, and ē is the normalized multipath power profile of
the channel. Therefore, the useful power can be derived as [16–18]:
E˜s   +∞  +∞
L−1
∗ ∗
PU = 2
[ e˜l gkn (t)gkn (s)f˜kn (t − τl )f˜kn (s − τl ) exp (j2πvl (s − t))dsdt]
2f  −∞ −∞
l=0
L−1
  +∞  +∞
∗ ∗ ∗
+ [ e˜l e˜k gkn (t)gkn (s)f˜kn (t − τl )gkn (s)f˜kn (s − τk ) exp (j2π(vk s + vl t))dsdt]
l,k=0 −∞ −∞

(16)

2.2 Interference and Noise Powers

Again and as before for useful power, we will evaluate, without loss of generality,
the interference and noise powers for symbol skn . The avreage power of the
Interference terme is expressed as [15,18–20]:

PIc = E[Ikn
2
]

= E[s2k n ] [exp (−jωkn ) < gkn , f˜k n >]2 (17)
(k ,n )=(k,n)

Using the same notations, Ẽs and ẽ, as in useful power, the interference power
can be written as [15,18–20]:
PI = E[PIc ]
  L−1

Ẽs Ẽs
= E[[exp (−jωkn ) < gkn , f˜k n >]2 [ e˜l
f 2 2f 2
(k ,n )=(k,n) (k ,n )=(k,n) l=0
 +∞  +∞
∗
gkn (t)gkn (s)f˜k∗ n (t − τl )f˜k n (s − τl ) exp (j2πvl (s − t))dsdt]+
−∞ −∞
L−1
  +∞  +∞
[ e˜l e˜k ∗
gkn ∗
(t)gkn (s)f˜kn (t − τl )f˜kn (s − τk ) exp (j2π(vk s + vl t))dsdt]
l,k=0 −∞ −∞

(18)
 Link Level Capacity Analysis of Precoded FBMC/OQAM Systems 757

The noise power given by [15,18–20]:

Pσ = E[[exp (−jωkn ) < gkn , σ >]2 ]

1
= E[| < gkn , σ > |2 ]
2
 +∞  +∞
∗
= E[( gkn (t)η(t)dt)∗ ( ∗
(gkn (s)η(s)ds)] (19)
−∞ −∞
 +∞  +∞
∗
= E[η(t)η(s)]gkn (t)gkn (s)dtds
−∞ −∞
= σ g
2 2

3 Capacity Analysis
In this section, we propose a precoding technique in order to increase the capac-
ity, for this, to determine the optimal precoding technical coefficients, the prob-
lem of rendering optimization as a convex optimization problem with linear
constraints, which can be solved effectively using Lagrangian methods [9].
This section evaluates the capacity building upon the model in the in Sect. 2.
The Total achievable capacity of a FBMC/OQAM system can be calculated
as [7]: 
C= log2 (1 + SN R(f )) df (20)
W
In our system, we assume that transmitted power is constant at each carrier and
equal to pk = b2k PU , then the equation became:


K
C= Wk log2 (1 + SN Rk ) (21)
k=1

Where SN Rk is the signal to noise ratio of the k-th subcarrie and W is the
Bandwidth
The spectral efficiency is given by:

C  Wk
K
= log2 (1 + SN Rk )
W W
k=1
(22)
1 
K
= log2 (1 + SN Rk )
K
k=1

The spectral efficiency of FBMC/OQAM systems is analyzed into account

the intrinsic interferences. The additional information extracted from the intrin-
sic interference would increase the maximum achievable rate, due to the added
power of the desired signal. In order to consider the intrinsic interferences
effect we follow the procedure provided in Eq. 22. The spectral efficiency can
be expressed:
758 J. Ghodhbane et al.

1 
K
C (b2 PU + PI )hk 2
= log2 (1 + k ) (23)
W K Pσ
k=1

Maximizing system capacity under power constraint is equivalent to a con-

strained convex problem [14]:
⎧ K (b2k PU +PI )hk 2
⎪ 1
⎨min{− K k=1 log2 (1 + Pσ )}
(24)
⎪
⎩ K 2
Subject to: k=1 bk = 1
Using Lagrangian optimization method to determine bk coefficients:
−1
Pσ
+λ=0 (25)
(b2k PU + PI ) + hk 2

We can use water filling algorithm to solve the previous optimization problem
[15]:
1 Pσ
(b2k PU + PI ) = ( − )+ (26)
λ hk 2
We replace Eq. 26 we obtain:
1 Pσ2
b2k PU = ( − )+ − PI (27)
λ hk 2
Thus the coefficients of the precoding vector are written as follows:
1 Pσ2 PI
b2k = ( − )+ − (28)
λPU PU hk 2 PU

4 Simulation Results
In this section, we present simulation results in terms of FBMC/OQAM system
capacity with proposed method of a precoder. Comparing the results from the
capacity expressions derived in Sect. 3 and the provided by the system using
Tomlinson Harashima precoding and by the system without precoding. To eval-
uate performances of our proposed precoder we consider these parameters shown
in Table 1:
In Figs. 2 and 3, we plot the capacity obtained of the system as a function
of the SNR using expression (23) (proposed model) and the capacity obtained
for a system without preconding and this obtained by the precoded THP using
both ZF and MMSE equalization. Capacity may increase when deploying linear
precoder at transmitter. In our work we assume a non-equal power distribution
among subcarrier; the total power remains unchanged comparing to the case
without precoding. Precoder coefficients are time independent, as the channel
statistics are time dependant, it is possible to implement a precoder with channel
state information (CSI) to tackle channel changes.
 Link Level Capacity Analysis of Precoded FBMC/OQAM Systems 759

Table 1. Simulation parameters

Classes Subclass
Number of subcarriers 128
Subcarrier spacing [kHz] 15
Modulation order 16
Bandwidth [MHZ] 1.4
Overlapping factor L 4
Prototype filter PHYDYAS [14]
Channel delays [s] 10−9 [0, 110, 190, 410]
Channel powers [db] [0, −9.7, −19.2, −22.8]
Noise power per subcarrier −134.10

Fig. 2. Comparison between the Capacity of FBMC/OQAM systems as a function of

SNR with and without precoding using for MMSE equalizers

Fig. 3. Comparison between the Capacity of FBMC/OQAM systems as a function of

SNR with and without precoding using for ZF equalizers
760 J. Ghodhbane et al.

5 Conclusion
In this paper, we have derived an analytical expression of the optimized capacity
of the FBMC/OQAM system with precoding at transmitter side. Resolution of
the optimization problem is based on water filling algorithm, powers are allocated
based on channel conditions, hence, more power is allocated to carrier with
higher channel attenuation. Simulations results show that the proposed solution
achieve the maximum of the capacity compared to TH precoding technique and
transmission without precoder. As a future work, we will focus on multi user
capacity maximization and FBMC-OQAM massive MIMO.

References
1. https://www.metis2020.com/. Accessed 18 Dec 2018
2. http://5gnow.eu/. Accessed 18 Dec 2018
3. Häring, L.: Intrinsic interference-aware adaptive FBMC transmission with auto-
matic modulation classification. In: ISCIT (2017)
4. Caus, M., Perez, I., Kliks, A., Bodinier, Q., Bader, F.: Capacity analysis of WCC-
FBMC/OQAM systems. In: CTTC, Castelldefels (2016)
5. Razavi, R., Xiao, P., Tafazolli, R.: Information theoretic analysis of OFDM/OQAM
with utilized intrinsic interference. IEEE Signal Process. Lett. 22(5), 618–622
(2015)
6. Chang, B.S., da Rocha, C.A.F., Le Ruyet, D., Roviras, D.: On the effect of ISI in the
error performance of precoded FBMC/OQAM systems. In: The 18th Asia-Pacific
Communications Conference (APCC 2012), Jeju Island, South Korea (2012)
7. Chang, B.S., da Rocha, C.A.F., Le Ruyet, D., Roviras, D.: On the use of pre-
coding in FBMC/OQAM systems. In: The 7th International Telecommunications
Symposium (ITS 2010), Manaus, Brazil (2010)
8. Baltar, L.G., Waldhauser, D.S., Nossek, J.A.: MMSE subchannel decision feed-
back equalization for filter bank based multicarrier systems. In: IEEE International
Symposium on Circuits and Systems, pp. 2802–2805 (2009)
9. Caus, M., Perez-Neira, A.I.: Transmitter-receiver designs for highly frequency selec-
tive channels in MIMO FBMC systems. IEEE Trans. Signal Process. 60(12), 6519–
6532 (2012)
10. Chauvat, R.: Étude de liaisons SISO, SIMO, MISO et MIMO à base de formes
d’ondes FBMC-OQAM et de récepteurs Widely Linear. Ph.D. Thesis, Le CNAM
(2017)
11. Bulusu, C.: Performance analysis and PAPR reduction techniques for filter-bank
based multi-carrier systems with non-linear power amplifier. Ph.D. Thesis, Le
CNAM (2016)
12. Medjahdi, Y.: Modelisation des interfrence et analyse des performances des systmes
OFDM/FBMC pour les communications sans fil asynchrones. Ph.D. Thesis, Le
CNAM (2012)
13. De Candido, O., Baltar, L.G., Mezghani, A., Nossek, J.A.: SIMO/MISO MSE-
duality for multi-user FBMC for highly frequency selective channels. In: Proceed-
ings of the International ITG Workshop on Smart Antennas (2015)
14. Bellanger, M.: Specification and design of a prototype filter for filter bank based
multicarrier transmission. In: ICASSP, pp. 2417–2420 (2001)
 Link Level Capacity Analysis of Precoded FBMC/OQAM Systems 761

15. Khrouf, W., Siala, M., Abdelkefi, F.: How much FBMC/OQAM is better than
FBMC/QAM? A tentative response using the POPS paradigm. Wirel. Commun.
Mob. Comput. (2018)
16. Hraiech, Z., Abdelkefi, F., Siala, M.: POPS-OFDM: ping-pong optimized pulse
shaping-OFDM for 5G systems. In: Proceedings of the IEEE International Con-
ference on Communications, ICC 2015, pp. 4781–4786, June 2015
17. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press,
Cambridge (2004)
18. Cover, T.M., Thomas, J.A.: Elements of Information Theory, 2nd edn. Wiley,
Hoboken (2006)
19. Horlin, F., et al.: Interference-free SDMA for FBMC/OQAM. EURASIP J. Adv.
Signal Process. 46 (2013)
20. Caus, M., Perez-Neira, A.I.: SDMA for filterbank with Tomlinson Harashima pre-
coding. In: Proceedings of the IEEE International Conference on Communications,
June 2013
 A QoS-Based Flow Assignment for Traffic
Engineering in Software-Defined
Networks

Lakshmi Priya Thiruvasakan1 , Quoc-Tuan Vien1(B) , Jonathan Loo2 ,

and Glenford Mapp1
1
Middlesex University, London, UK
[email protected], {q.vien,g.mapp}@mdx.ac.uk
2
University of West London, London, UK
[email protected]

Abstract. In order to meet a tremendous amount of data storage

requirement in next-generation wireless networks, an increasing number
of cloud data centers has been deployed around the world. The underly-
ing core networks are expected to provide the ability to store data in a
dynamic and scalable computing environment. The traditional Internet
Protocol (IP) has shown to be restricted due to its static architecture,
which accordingly motivates the development of Software-Defined Net-
works (SDNs). In the SDNs, Traffic Engineering (TE) is simpler and pro-
grammable with a controller without the requirement of reconfiguration
for all network devices. However, the existing TE algorithm of the SDNs
rejects a number of requested flows caused by their undetermined routing
paths where only flow bandwidth is considered in path determination.
This paper proposes a Quality-of-Service (QoS) based Flow Assignment
algorithm which enables the computation of end-to-end path for traffic
flows guaranteeing the QoS requirements including bandwidth, end-to-
end delay and packet loss probability. Through the Open Source Hybrid
IP/SDNs platform, the proposed algorithm is validated and shown to
significantly reduce flow rejection rate of up to 50% compared to the
conventional approach, and therefore can be used to implement an effec-
tive DiffServ mechanism for flow allocation in the SDNs.

1 Introduction
Software-Defined Network (SDN) has recently emerged to overcome the limita-
tions of traditional Internet Protocol (IP) networks and Multi-Protocol Label-
Switching (MPLS) systems, by enabling dynamic and scalable computing [1].
In SDN, control plane is decoupled from data plane and placed in a centralized
SDN controller [2,3]. The network intelligence and state are logically central-
ized in a single place, allowing network administrators to easily apply network
wide policy at the controller. Cisco defined a new way of Traffic Engineering
(TE) called Segment Routing (SR) [4] to be used with SDN. SR implemented
in hybrid MPLS/SDN network can benefit from centralization logic of SDN,
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 762–774, 2020.
https://doi.org/10.1007/978-3-030-15032-7_64
 A QoS-Based Flow Assignment for Traffic Engineering 763

where routing paths can be computed at SDN controller using flow assignment
algorithm.
As an SDN emulator, Open Source Hybrid IP/SDN (OSHI) [5], a part of
DREAMER project, was designed and developed by Davoli et al. [6]. OSHI plat-
form is a whole package which provides tools to build network topology, develop
a flow assignment algorithm namely TE/SR algorithm, and test the algorithm
using emulated network [6]. There is an inherent TE/SR or flow assignment algo-
rithm within OSHI [7], which computes routing paths based only on bandwidth
requirement of the flow. In the path computation of traffic flows in real-time sys-
tems, the end-to-end delay and packet loss probability are also important met-
rics; however, they have received less interests in the literature. To this extent,
this paper aims at addressing these metrics in the flow assignment. The main
contributions of this paper can be summarized as follows:
1. Development of queuing models for network links and nodes: To enable OSHI
to use queuing models M/M/1 and M/M/1/K to simulate QoS metrics like
delay and packet loss probability.
2. A novel QoS based Flow Assignment (QFA) algorithm: The novelty of the
proposed QFA algorithm lies in enhancing the existing OSHI algorithm [7]
by considering delay and packet loss probability in addition to the already
supported bandwidth while computing routing path. The proposed algorithm
is shown to reduce significantly the flow rejection rate compared to the exist-
ing algorithm.

2 Related Works
With rapid growth of real time Internet applications, packets are expected to
reach destination on time, and also with a less packet loss. This can be achieved
when routing algorithm considers critical flow parameters like bandwidth, delay
and packet loss probability. Very few research efforts done in the past [8,9]
proposed an SDN routing algorithm, which considers bandwidth and end-to-end
delay metrics.
One approach to meet delay requirement of a flow is to map traffic flows to
a statically configured output queue of a switch based on their priorities. This
allows managing both bandwidth and delay properties of the flow at each SDN
switch. Kim et al., in 2010 [8] proposed an algorithm for automatic QoS control
over OpenFlow protocol (OF) [10]. The algorithm guarantees flow’s bandwidth
by generating an optimal routing path, where the links in the path have enough
capacity to support new flow. Delay requirement is then satisfied by configur-
ing multiple priority based output queues to which the flows are mapped. Flow
paths generated by SDN algorithm are placed in flow tables in the OF switch
using OF protocol [10]. Any packet arriving at the switch is matched against
the flow table entries to determine its routing path. A drawback in setting up
bandwidth-delay guaranteed tunnel in a large-scale network is that it requires a
highly computational complex algorithm [9]. To reduce complexity of TE algo-
rithms, Tomovic and Radusinovic proposed a simple QoS provisioning algorithm
764 L. P. Thiruvasakan et al.

[9], where control plane classified incoming QoS requests into finite number of
categories based on delay sensitivity level. A QoS request gets rejected when
algorithm fails to calculate shortest routing path. Their experiment proved that
simple algorithm can also perform better and lead to smaller rejection of QoS
requests than complex solutions. This paper aims to develop such a simple QoS
based routing algorithm.
For timely delivery of packets in real-time systems, it is essential for packets
to follow a routing path, where total delay encountered by packet is less than or
equal to delay requirement of flow. Implementing delay based flow optimization
in traditional networks involves expensive custom-built hardware and software
[11]. On other hand, with SDN the controller is aware of global view of network
and statistics, and thus delay based routing algorithm can be implemented in
a cheaper and efficient way. Recently, Kumar et al., in 2017 [11] designed an
algorithm which considers only high criticality flows in safety-critical systems,
whose properties like delay and bandwidth are known to algorithm designers
well ahead of time, before actual data transmission starts. The algorithm calcu-
lates end-to-end delay value by summing up various delay elements like nodal
processing delay, link transmission delay, link propagation delay and queuing
delay.
Most useful research is based on OSHI [6] (see Sect. 3). Our paper is different
from aforementioned works in two key ways in terms of optimization goals and
the considered use case. Furthermore, most of the papers only considered band-
width and/or end-to-end delay, while our research designs a QFA algorithm to
identify routing path that guarantees specified bandwidth, delay and packet loss
probability metrics.
Although there exist a variety of simulation platform for SDN such as MAT-
LAB and NS3 [12], OSHI is selected as a base for validating proposed algorithm.
This is due to the fact that it can provide a simple and efficient virtual envi-
ronment to develop and test flow assignment algorithm using simulated SDN
network.

3 OSHI System Model

OSHI encompasses various software components like Network Topology Parser,

Random Demand Generator, Flow Assignment or TE algorithm, SR algorithm
etc. [6]. Network can be graphically designed for small-scale topology and
deployed using Mininet [13] extensions. Alternatively, large-scale topology can
also be downloaded from Topology Zoo [14]. These graphs are converted to JSON
files and fed to a random demand generator.
Random demand generator [15] uses network topology as input, and gener-
ates a set of flow catalogues. TE algorithm [7] accepts network topology and flow
catalogue as inputs and computes optimal routing path for each flow in flow cat-
alogue file. This section describes various phases of existing TE algorithm [7]. TE
algorithm is a classical flow assignment problem which computes optimal path
between source and destination, given flow’s expected bandwidth (Megabits per
 A QoS-Based Flow Assignment for Traffic Engineering 765

Second or Mbps). Path computed by TE algorithm is further shortened by SR

algorithm. SR pusher then deploys SR paths into the network SDN nodes via
Ryu controller [6].
As mentioned in OSHI Quarterly report [16], TE algorithm implements below
3 phases in sequence.
1. Initialization phase: Network topology and flow catalogue JSON files are read
and converted to in-memory data structures. A network graph is created using
topology files.
2. Constrained Shortest Path First (CSPF) phase: Shortest routing path is cal-
culated for each flow using Dijkstra’s algorithm. Weight of any link ‘i’ is
calculated as,
BIGK
Wi = (1)
Ci − λi
where Wi is link weight, BIGK is biggest capacity present among all links,
Ci is capacity of link and λi is load of link. Flows for which shortest path
cannot be computed gets rejected in this phase. Only accepted flows continue
to next phase. At end of this phase, average crossing time of network [16] is
calculated as shown below.
1  λi
T = (2)
γ (Ci − λi )
i=links

where γ is total load introduced in network.

3. Heuristic reassignment phase: This phase aims at reducing T (average crossing
time) value, calculated in end of previous phase. During this phase, edge
weights Wi are calculated using (3), which is derived by differentiating T
with respect to λi . The derived value is increase of delay on link ‘i’ brought
on by a very small amount of traffic on the link.
1 Ci
Wi = (3)
γ (Ci − λi )2
Thus, shortest path calculated in this phase reduces overall delay in network
compared to previous phase.

4 Problem Definition and Proposed Solutions

Below are the enhancements added to OSHI platform [6] and its TE algorithm
[7].
1. Improvise OSHI platform to support M/M/1 and M/M/1/K queuing models.
2. Reduce flow rejection ratio by sorting flows. Existing algorithm [7] has a
very high flow rejection ratio i.e., 1470 flows out of 2198 requested flows gets
rejected.
3. Modify TE algorithm design such as to calculate routing paths that guarantee
QoS metrics bandwidth, packet loss and delay.
The new QFA algorithm is an offline and static algorithm which can be
scheduled to run periodically, say during night or off-peak times. The algorithm
can be used to handle predefined traffic flows in an organization or data centres.
766 L. P. Thiruvasakan et al.

4.1 OSHI Platform Changes to Support Queuing Models

4.1.1 Guaranteed Packet Loss Probability
Modelling each router as M/M/1/K system [17], packet loss probability or block-
ing probability is given by formula [17],
ρK
PK = (1 − ρ) (4)
1 − ρ( K + 1)
where ρ = λ/μ and is system utilization , λ is arrival rate (Packets per Second),
μ is service rate (Packets per Second), K is maximum system capacity which is
total number of packets present in ingress buffers plus one packet which is being
serviced.
For the experiments, packet loss probability is considered as 0.0001% [18],
which signifies that maximum packet loss allowed in any node is 1 in 106 packets.
With K and μ known for a node (which will be explained in Sect. 4.1.3), λ is
calculated such that PK < 0.0001%. This λ value denotes maximum packet
arrival rate allowed in a node to keep packet loss probability less than 0.0001%.
When calculating the shortest path between end points, the algorithm makes
sure that the packet arrival rate at any node is kept within this calculated λ
value.

4.1.2 Guaranteed End-to-End Delay

End-to-end delay (TEndtoEnd ) of a packet is summation of below components
[19]
 
TEndtoEnd = (Ttrans + Tqueue + Tpropagation ) + TnodalP roc (5)
links nodes

where Ttrans is link transmission delay, Tqueue is queuing delay, Tpropagation is

link propagation delay and TnodalProc is nodal processing delay.
When each link in routing path is modelled as M/M/1 system [19] total time
spent by packet waiting in outgoing queue and time for transmission over link
‘i’ is given by [19],
1
Ttrans + Tqueue = (6)
μCi − λi
where λi is load of ith link (Packets Per Second), Ci is capacity of ith link (Bits
Per Second) and μ1 is average length of data packet (Bits per Packet).
Propagation speed in physical media (‘v’) is usually a fraction of speed of
light i.e., 0.59c to 0.77c where c is speed of light in vacuum [20]. Propagation
delay of link ‘i’ can be expressed as [19],
li 1
Tpropagation = + (7)
v μCi
where li is length of ith link (Metres) and is assumed 100 m [11], v is propagation
speed of link, Ci is capacity of ith link (Bits Per Second) and μ1 is average length
of data packet (Bits per Packet).
 A QoS-Based Flow Assignment for Traffic Engineering 767

When modelling network node as M/M/1/K system, total time spent by

packet in the system is given by [21],
(
ρ
( 1−ρ ) − ( (K+1)ρ K+1)
1−ρ( K+1)
)
TnodalP roc = (8)
(1 − PK )λ

where ρ is system utilization, λ is packet arrival rate (Packets per Second), K

is maximum system capacity and PK is probability of having K packets in the
system as mentioned in (4).

4.1.3 OSHI Input Parameters to QFA Algorithm

QoS metric values for flows are taken from Colt CPE Solutions Service Level
Agreement [18]. The input data to algorithm are json files like nodes.json and
links.json which are created based on network topology, and flow catalogue.json
which is generated by random demand generator. Section 5 mentions the steps
to generate these JSON files. Table 1 shows various input parameters to QFA
algorithm along with their values.
Kumar et al., studied end-to-end delay constraints using SDN switches [11]
and they measured average packet processing time in a SDN switch to be in range
of 3.2 µs to 4.1 µs [11], assuming average packet size as 1600 bytes [11]. Same
values are considered for our research testing. Above mentioned µs values leads
to packet service rate values ranging from 243902 to 312500 packets/s. Buffer size
at the node can be found using B = RT T × √Cn , where RTT is average round
trip delay of the flow passing through the link, C is capacity of link and n is
number of flows multiplexed on same link. In OSHI experiments, ‘n’ was found
to be 29 with Colt Telecom topology. Assuming RTT as 250 milliseconds [23],
value of B is 2,785,430 bits. With packet size as 1600 bytes [11], B was found to
be around 218 packets.

Table 1. Simulation parameters and their values

Parameter File where the parameter Range of values

is defined
Bandwidth flow catalogue.json 0.0014 to 12.2 Mbps [22]
End-to-end delay flow catalogue.json 50 to 75 milliseconds [18]
Packet Loss Hardcoded in source code 0.0001% [18]
Link Capacity links.json 60 Mbps [22]
Average Packet Size Hardcoded in source code 1600 bytes [11]
Packet Service Rate nodes.json 243902 to 312500 packets/s
Node Buffer Size nodes.json 220 to 230 packets
768 L. P. Thiruvasakan et al.

4.2 Reducing Flow Rejection Ratio

In CSPF phase of existing Flow Assignment algorithm [7], a huge amount of

flows gets rejected when their shortest path cannot be computed. This is men-
tioned in Sect. 3. Nearly two-thirds of requested flows i.e., 1470 out of 2198 flows
gets rejected. When a flow is processed in CSPF phase, a temporary network
graph is created in such a way that the network links whose residual capacity is
not sufficient enough to support new flow’s bandwidth (or) the links which are
overloaded are pruned from temporary network graph, before invoking Dijkstra
API on it. When Dijkstra API could not compute path between source and des-
tination nodes, such flows are rejected. This is because links present in potential
routing path might have been overloaded and removed from temporary graph.
One way to solve link overload issue is to redistribute traffic across multiple
links. Traffic redistribution can be easily achieved by considering first, the flows
with smaller bandwidth values [24]. Hence we sort flows in ascending order as
per their bandwidth value during initialization phase.

4.3 QoS Based Flow Assignment Flowchart

The proposed QFA algorithm has five phases like Initialization, CSPF, Heuristic
Reassignment, FlowLatencyCalculation and FlowReallocation. The first three
phases are already present in existing design [7], while the last two phases are
newly added in this work to handle delay logic. As shown in Fig. 1, the design
of these two phases are as follows:

• FlowLatencyCalculation Phase: In this phase, end-to-end delay value is calcu-

lated for of all accepted flows as per formulae mentioned in Sect. 4.1.2 (Defined
as ‘Step a’ in Fig. 1). Flows whose calculated delay is greater than expected
delay are added to a list called ‘partialRejectedFlows’.
• FlowReallocation Phase: Flows added to ‘partialRejectedFlows’ list are pro-
cessed again to recompute their routing path with minimal delay value. Delay
value of links itself is used as link weights to calculate shortest path. Delay
components like Ttrans , Tqueue , Tpropagation and TnodalProc are calculated
using formulae mentioned in Sect. 4.1.2. Summation of these delay values
is used as link weights in network graph (Defined as ‘Step b’ in Fig. 1). It is
guaranteed that if shortest path computed at this stage does not meet delay
requirement of flow, no optimal path exists for this flow and so it can be
rejected. This phase is repeated for each flow in ‘partialRejectedFlows’ list.
Once all flows of ‘partialRejectedFlows’ list are processed, execution returns
to FlowLatencyCalculation phase. This is because, due to new routing paths
and new loads added on the network links in FlowReallocation phase, its prob-
able that end-to-end delay of few of already accepted flows might have crossed
their limit. FlowLatencyCalculation and FlowReallocation phases runs in a
loop until the list ‘partialRejectedFlows’ is empty which means all accepted-
Flows meet their Delay requirements (or) for hard-set rule of 3 times, which
was found to be sufficient enough to optimize all flows in experiment.
 A QoS-Based Flow Assignment for Traffic Engineering 769

Inputs to TE
algorithm:
nodes.json,
links.json

Initialization Phase

CSPF Phase

Heuristic Reassignment
Phase

FlowLatencyCalculation
Phase

Iterate through all

and calculate end

to end delay
value (Step a)
FlowReallocation
Phase
End of
Iterate through
list?
Yes partialRejected
Flows list
No
End of
Is Yes
calculated list?
Delay <
No
estimated
delay Create a temporary graph
which is a copy of original
network graph. Prune links
No No from temporary graph whose
Yes
available capacity( bit rate in

whose delay bit rate. Also prune the links

values are not whose source and destination
met to a new node cannot support the
list, say
partialRejected
Flows maximum allowed value.

Calculate the link weights

based on various delay
components and invoke
Is Dijkstra API to calculate the
partialRejectedFl shortest path to destination
ows empty or (Step b)
FlowReAllocatio
nPh-ase run 3
times?
Is delay of
computed
No
estimated
Yes delay
Reject
End the
TE and add
Yes it to
algorithm
rejected
Consider the list
new shortest
Output of TE path for the
algorithm : List
of it to accepted
ArrayElements
containing
acceptedFlows
and their
corresponding
routing path

Fig. 1. FlowLatencyCalculation and FlowReallocation Phase

770 L. P. Thiruvasakan et al.

5 Simulation Results
This section compares QFA algorithm against existing OSHI algorithm [7] in
various aspects like Global Network Crossing Time, Number of rejected flows and
link loads which were measured from the algorithm’s output. Both the algorithm
were evaluated in OSHI platform using Colt Telecom topology (large-scale) [14].
Main steps to test the QFA algorithm are [15]

1. Download Colt Telecom topology from Topology Zoo [14].

2. Topology Parser transforms network topology to JSON files (nodes.json,
links.json).
3. Random Demand Generator generates set of flow catalogues (flow
catalogue.json) using topology files.
4. Execute QFA algorithm from eclipse using nodes.json, links.json and
flow catalogue.json as input. Output is array elements containing accepted
flows and their routing path.

5.1 Flow Rejection Ratio

Figure 2 shows number of rejected flows measured with existing algorithm [7]
and proposed algorithm for same number of requested flows. As seen, existing
algorithm [7] rejects 1470 flows among 2198 flows that were requested, whereas
QFA algorithm rejects 880 flows for same number of requested flows. Sorting logic
added has reduced flow rejection ratio to around half its size. Section 4.2 explains
in detail the reason behind this reduction. Assuming algorithm is scheduled for
nightly periodic runs, with a single run the proposed design enables 1318 (2198
minus 880) traffic flows at a time, when existing algorithm [7] enables only 728
(2198 minus 1470) flows in network.

Fig. 2. Number of rejected flows in existing vs proposed solution (Colt Telecom)

 A QoS-Based Flow Assignment for Traffic Engineering 771

5.2 Global Network Crossing Time

The proposed design sorts the flows in ascending order as per their bandwidth
during Initialization phase. Sorting logic along with delay based optimization,
allows algorithm to calculate routing paths with minimal delay value. This leads
to reduction in overall network crossing time. Graph in Fig. 3 shows global net-
work crossing time value measured by old and new algorithm for various num-
ber of requested flows. Global network crossing time which is referred as ‘T’ is
weighted average of delays seen on the links and is measured using (2). It can
be clearly seen in Fig. 3 that there is a drastic drop in T value measured with
proposed solution than existing algorithm [7]. For instance, value of T measured
with existing algorithm is 172.58 s and with QFA algorithm is 0.8686 s, when
number of requested flows is 2198. This reduction is due to reason that the traf-
fic gets evenly distributed across links with new design compared to existing one
[7], which is explained in detail in following section.

Fig. 3. Global network crossing time in existing vs proposed solution (Colt Telecom)

Impact of link load on its delay value:

As mentioned in (6), when each network link is considered as M/M/1 system

with packet service rate following an exponential distribution, transmission delay
encountered in link can be generalized and given by,
1
Ttrans = (9)
LinkCapacity − LinkLoad
From (9), it is obvious that with link capacity being a constant value, trans-
mission delay value increases with increase in link load value and even approaches
infinity when link load is same as link capacity. This rapid increase of delay value
as load approaches capacity of link [25] is illustrated in Fig. 4. In Fig. 4 increase
in transmission delay value with respect to link load is very less until the link
is loaded to 99.6% of its capacity. When link is overloaded and crosses 99.95%
which can be referred as alarming limit, there is a rapid rise in delay value even
for a small amount of increase in link load. The existing algorithm [7] overloads
772 L. P. Thiruvasakan et al.

few links in this alarming limit i.e., beyond 99.95%, thereby resulting in a huge
global network crossing time value as shown in Fig. 3. On other hand, QFA algo-
rithm prevents links getting loaded more than alarming limit. This is illustrated
in Fig. 5. When number of requested flows is 2198, proposed design does not load
any links beyond 99.95% (alarming limit), whereas existing algorithm loads 8
links beyond this point (Fig. 5). Although 8 is very minimal number compared
to total number of 354 links, these 8 links have a very higher delay values, which
is more predominant than other link’s delay values (as in Fig. 4). These critical 8
links are reason for global networking crossing time being very high with existing
FlowAssignment algorithm [7], as shown in Fig. 3.

Fig. 4. Effect of link load % on its transmission delay value (assuming link capacity
60 Mbps)

Fig. 5. Number of links and their load % (Colt Telecom, requested flows = 2198)

Above mentioned fact was proved to be right by extending our testing to mon-
itor link load % values for different number of requested flows. Results shown
in Fig. 6 confirms the existing design [7] overloads links beyond alarming limit
(99.95%), which is avoided with proposed solution. QFA algorithm prevents link
overload issue by redistributing traffic across links. Flows with smaller band-
widths are found to be best suited for fair distribution of traffic across links
 A QoS-Based Flow Assignment for Traffic Engineering 773

[24] and thus sorting flows in ascending order of their bandwidth value (pro-
posed design) helped with this redistribution. Furthermore, delay based routing
optimization also contributed to reduction in global network crossing time.

Fig. 6. Number of links and their load % (Colt Telecom, requested flows = 1502)

6 Conclusion
In this paper, a new QoS based flow optimization algorithm for TE in SDN
was designed and implemented using OSHI platform. The experiment provides
promising results and proves new algorithm’s efficiency compared to the exist-
ing algorithm. Our study used M/M/1 and M/M/1/K queuing systems to model
SDN network, whereas real traffic flows can be more bursty than traffic repre-
sented by these queuing models. Various other queuing distributions can be
validated in future to match more closer to the real time traffic and algorithm
can be further tested in a small-scale network using real SDN switches.

References
1. Kreutz, D., Ramos, F.M., Verissimo, P.E., Rothenberg, C.E., Azodolmolky, S.,
Uhlig, S.: Software-defined networking: a comprehensive survey. Proc. IEEE
103(1), 14–76 (2015)
2. Hassan, M.A., Vien, Q.-T., Aiash, M.: Software defined networking for wireless
sensor networks: a survey. Adv. Wirel. Commun. Netw. 3(2), 10–22 (2017)
3. Ramirez, R.C., Vien, Q.-T., Trestian, R., Mostarda, L., Shah, P.: Multi-path rout-
ing for mission critical applications in software-defined networks. In: Proceedings
of the EAI INISCOM 2018, Da Nang, Vietnam, August 2018
4. Cisco: Introduction to segment routing. https://www.cisco.com/c/en/us/td/docs/
ios-xml/ios/seg routing/configuration/xe-3s/segrt-xe-3s-book/intro-seg-routing.
pdf
5. Davoli, L., Veltri, L., Ventre, P.L., Siracusano, G., Salsano, S.: OSHI homepage.
http://netgroup.uniroma2.it/twiki/bin/view/Oshi
774 L. P. Thiruvasakan et al.

6. Davoli, L., Veltri, L., Ventre, P.L., Siracusano, G., Salsano, S.: Traffic engineering
with segment routing: SDN-based architectural design and open source implemen-
tation. Extended version of poster paper accepted for EWSDN 2015, December
2015. https://arxiv.org/abs/1506.05941v4
7. Ventre, P.L.: Existing flow assignment algorithm in OSHI, June 2015. https://
github.com/netgroup/SDN-TE-SR-tools/blob/master/java-te-sr/src/it/unipr/net
sec/sdn/algorithm/FlowAssignmentAlgorithm.java
8. Kim, W., Sharma, P., Lee, J., Banerjee, S., Tourrilhes, J., Lee, S.J., Yalagandula,
P.: Automated and scalable QoS control for network convergence. In: Proceed-
ings of the Internet Network Management Workshop/Workshop on Research on
Enterprise Networking (INM/WREN), April 2010
9. Tomovic, S., Radusinovic, I.: Fast and efficient bandwidth-delay constrained rout-
ing algorithm for SDN networks. In: 2016 IEEE NetSoft Conference and Workshops
(NetSoft), Seoul, South Korea, pp. 303–311, June 2016
10. Open Networking Foundation: Openflow switch specification, October 2013.
https://3vf60mmveq1g8vzn48q2o71a-wpengine.netdna-ssl.com/wp-content/uploa
ds/2014/10/openflow-spec-v1.4.0.pdf
11. Kumar, R., Hasan, M., Padhy, S., Evchenko, K., Piramanayagam, L., Mohan, S.,
Bobba, R.B.: Dependable end-to-end delay constraints for real-time systems using
sdns. In: 15th International Workshop on Real-Time Networks, Dubrovnik, Croatia
(2017)
12. Hamood, A.S.: Simulator.ppt (2016). https://www.researchgate.net/publication/
301887282 most Simulator used in Software Defined Networking SDN and Cogni
tive Radio Network CRN
13. Mininet. http://mininet.org/
14. The University of ADELAIDE: Topology zoo. http://www.topology-zoo.org/
dataset.html
15. Davoli, L., Veltri, L., Ventre, P.L., Siracusano, G., Salsano, S.: OSHI github (2015).
https://github.com/netgroup/SDN-TE-SR-tools
16. Ventre, P.L., Salsano, S.: OSHI Quaterly Reports QR.2 (2014). http://netgroup.
uniroma2.it/twiki/pub/Oshi/WebHome/qr2 2 ventre.pdf
17. Kleinrock, L.: Queueing Systems: Theory, vol. 1. Wiley-Interscience, Hoboken
(1975)
18. Service Level Agreement: COLT Telecommunications (2001). https://www.rtr.at/
uploads/media/24875 SLA CPE.pdf
19. Kleinrock, L.: Queueing Systems: Computer Applications, vol. 2. Wiley-
Interscience, Hoboken (1976)
20. Coiner, P.: Calculating the propogation delay of coaxial cable. GPS Source,
January 2011. https://cdn.shopify.com/s/files/1/0986/4308/files/Cable-Delay-
FAQ.pdf
21. Dattatreya, G.R.: Performance Analysis of Queuing and Computer Networks. CRC
Press, Boca Raton (2008)
22. Davoli, L., Veltri, L., Ventre, P.L., Siracusano, G., Salsano, S.: OSHI virtual image
OSHI VM7b, April 2016. http://netgroup.uniroma2.it/twiki/bin/view/Oshi
23. Appenzeller, G., Keslassy, I., McKeown, N.: Sizing router buffers. In: SIGCOMM
2004 Proceedings of the 2004 Conference on Applications, Technologies, Architec-
tures, and Protocols for Computer Communications, Portland, Oregon, USA, pp.
281–292, September 2004
24. Köhler, S., Binzenhöfer, A.: Providing Quality of Service in Heterogeneous Envi-
ronments, vol. 5, pp. 21–30. Elsevier, Berlin (2003)
25. Weinstein, S.: The Multimedia Internet. Springer, Heidelberg (2005)
 A UAV-Collaborative Sensing Method
for Efficient Monitoring of Disaster Sites

Akimitsu Kanzaki1(B) and Hideyuki Akagi2

1
Institute of Science and Engineering, Academic Assembly, Shimane University,
1060 Nishikawatsu-cho, Matsue, Shimane 690-8504, Japan
[email protected]
2
People Software Corporation, 1-7-2 Achi, Kurashiki, Okayama 710-0055, Japan
[email protected]

Abstract. In this paper, we propose a method that achieves effi-

cient monitoring of the entire of a disaster site, using multiple UAVs
(Unmanned Aerial Vehicles) owned by different organizations and indi-
viduals. In our proposed method, UAVs collaboratively operates by shar-
ing information about UAVs operating in the target region via local
direct wireless communication. Using information shared with other
UAVs, each UAV sets its own moving path so as to move to areas where
no UAVs have visited for a long time. In doing so, our proposed method
enables to monitor the entire of a disaster site evenly and frequently. We
confirmed the effectiveness of our proposed method through simulation
experiments.

1 Introduction
Recently, UAV s (Unmanned Aerial Vehicles), commonly known as drones, have
been widely used for many purposes such as aerial shooting. In addition, there
have been many research and development activities for sensor data gathering
at a certain target region using UAVs which are equipped with several sensor
devices such as temperature and humidity sensors [1,2,5,7]. Especially, the ini-
tial response to heavy disasters like great earthquake or floods is one of the
suitable applications that can make use of flyability of UAVs. For example, a
prefectural or city government can rapidly grasp information required for res-
cue efforts (e.g. existences/locations of collapsed buildings and injured victims)
by gathering several sensor data like pictures and thermal images from its own
UAV. In addition, when each UAV has a functionality to communicate with
other wireless devices (e.g. smartphones held by victims) via local direct com-
munications such as Wi-Fi Direct and LTE Direct, it will be possible to inform
victims about the nearest evacuation site and safe routes to the site.
In a situation immediately after a heavy disaster, status at each location in
the disaster site may change with every moment due to afterquakes and move-
ments of victims. Thus, in order to achieve efficient information collection and
diffusion, it is desirable to monitor the entire disaster site as frequently as possi-
ble. In other words, it is desirable for UAVs to visit every location in the entire
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 775–786, 2020.
https://doi.org/10.1007/978-3-030-15032-7_65
776 A. Kanzaki and H. Akagi

site as frequently as possible. Until now, there have been many studies on UAV
control mechanisms which aim to efficiently monitor a target region [3–6,8–11].
These studies basically assume that a single organization (e.g. prefectural or city
government) controls its own UAV(s). On the other hand, with a rapid popular-
ization of UAVs, many UAVs have been owned not only by government agencies
but also by academic institutions and individuals. When we can make use of
these ‘grassroots’ UAVs, it is possible to further improve the efficiencies of infor-
mation collection and diffusion. In such a situation, however, each UAV may
independently start its operation without any information of other UAVs. Thus,
when starting its operation, each UAV knows neither how many nor where other
UAVs are operating in the region. In order to improve efficiencies of information
collection and diffusion in such a situation, it is effective to enable each UAV
to wirelessly communicate with other UAVs and to share information about
UAVs (e.g. existence, location and trajectory) with each other. We call such a
collaborative work of UAVs the UAV-collaborative sensing.
In this paper, we propose an effective UAV-collaborative sensing method
named RCS/MR (UAV-Collaborative Sensing with Mobility Reenactment) that
achieves efficient monitoring of the entire target region, especially assuming infor-
mation diffusion in a disaster site, using ‘grassroots’ UAVs each of which inde-
pendently starts its operation. Our proposed method introduces the notion of
time-attenuating coverage, that indicates the frequency of monitoring at each
location in the target region. Each UAV shares information on the coverage
with wirelessly connected UAVs. In addition, each UAV adjusts its own moving
path so as to move to areas where no UAVs have visited for a long time. In
doing so, every location in the entire target region can be frequently monitored
by UAVs.
The contributions of this paper are summarized as follows:

• We propose a novel method for frequent monitoring of the entire of a disaster

site using grassroots UAVs.
• In order to achieve efficient monitoring, our proposed method introduces the
coverage which attenuates over time. By sharing such a time-attenuating
coverage among UAVs, our proposed method achieves frequent monitoring of
the entire region.
• We confirmed the effectiveness of our proposed method through extensive
simulation experiments.

The remainder of this paper is organized as follows: In Sect. 2, we describe

the assumptions in this paper, In Sect. 3, we briefly introduce related work. In
Sect. 4, we present the details of our proposed method, and in Sect. 5, we show
the results of the simulation experiments. Finally, we summarize this paper in
Sect. 6.
 A UAV-Collaborative Sensing Method for Efficient Monitoring 777

1 2 3 4 5

Sensing area s Cell

2

6 7 8 9 10

Cell ID
c
11 12 13 14 15

16 17 18 19 20
UAV

Communication area 3

Fig. 1. Assumed environment (N = 3, Mc = 5, Nc = 4).

2 Assumption

2.1 UAVs

We assume that there are N UAVs ui (i = 1, . . . , N ) operating in the target

region. Each UAV is equipped with a sensor device which has a circular sensing
area with a radius of rs . For simplicity, we assume that all UAVs are equipped
with the identical sensor device. In addition, each UAV can directly communicate
with UAVs located in the circle with radius rc , centered on itself.
As presented in Sect. 1, we assume that each UAV is owned by different
organizations and individuals and starts its operation independently. Thus, each
UAV has no information on other UAVs when starting its operation.

2.2 Target Region and Application Requirement

We assume a two-dimensional plane as the target region. The region is divided

into hexagonal cells each of which is inscribed in the sensing area of UAVs. In
this paper, we assume that there are Mc × Nc cells in the target region and that
each cell is assigned a unique identifier, as shown in Fig. 1.
In the assumed applications, it is required for each cell to be monitored by
UAVs as frequently as possible. In order to express such a requirement, our
proposed method defines the coverage for each cell. The coverage of a cell is
defined as 1.0 when a UAV exists at the cell. Otherwise, the coverage of a cell
attenuates at the rate of δ every time UAVs move from a cell to its adjacent cell
(we call the time the unit time) until the coverage becomes 0.0. Such a time-
attenuating coverage indicates the elapsed time since each cell was monitored by
any of operating UAVs.
In the above assumptions, the application requires to keep coverages for all
cells in the target region as higher as possible.
778 A. Kanzaki and H. Akagi

3 Related Work
In [3], the authors have proposed a method that determines the moving path
of a UAV for monitoring the entire target region. Based on the experiments
using actual UAVs, this method constructs an energy consumption model of
UAV. Using the constructed model, this method sets the moving path of a UAV
with smaller energy consumption that can monitor the entire target region. This
method aims to determine an efficient moving path of a single UAV. In contrast,
our proposed method aims to utilize multiple UAVs for monitoring a target
region.
In [5], the authors have proposed a method that controls multiple UAVs for
efficient monitoring of the entire target region. This method divides the target
region into multiple areas and assigns a UAV to each divided area. Each UAV
moves only in its assigned area. In doing so, this method reduces the size of
the area where each UAV has to monitor, and improves the efficiency of mon-
itoring. In [4], the authors have proposed a method that periodically monitors
a certain set of locations in a target region using multiple UAVs. This method
assigns a part of locations to each UAV and sets the moving path of each UAV
in which the UAV visits all of its assigned locations. By setting the moving path
of each UAV in such a way that every location is monitored by a UAV within
a predetermined time interval, this method achieves the periodical monitoring.
These methods assume that the existences of all UAVs are known in advance
of starting the monitoring operation. In contrast, our proposed method utilizes
‘grassroots’ UAVs each of which independently starts its operation without infor-
mation about other UAVs.

4 UCS/MR: Our Proposed Method

As described in Sect. 1, our proposed method, UCS/MR (UAV-Collaborative

Sensing with Mobility Reenactment), shares information on coverage among wire-
lessly connected UAVs. Based on the shared information, UCS/MR improves the
efficiency of monitoring in the entire region.
When starting its operation, each UAV does not know the existence of other
UAVs. Thus, each UAV first sets its own moving path to monitor the entire region
only by itself. When two UAVs establish a direct wireless connection, these UAVs
recognize the existence of each other and share information on coverage. By doing
so, each UAV can grasp the coverages of cells monitored by other UAVs. After
sharing information, each UAV adjusts its own moving path to preferentially
visits cells with lower coverage.
In addition, each UAV in UCS/MR reenacts the movement of UAVs which
have shared information in the past. Based on the reenacted movement, each
UAV estimates the coverages of cells which is expected to be monitored by other
UAVs, and sets its own moving path in order to avoid visiting cells that are
expected to be monitored by other UAVs.
 A UAV-Collaborative Sensing Method for Efficient Monitoring 779

Furthermore, when sharing information with another UAV, each UAV sends
not only information of itself but those of other UAVs which have shared infor-
mation in the past. By doing so, UCS/MR aims to further improve the efficiency
of monitoring according to the movement reenactment described above.
In the followings, we first present the details of information on coverage that
each UAV holds. Then, we present the basic strategy to determine the moving
path of each UAV and the detailed procedures to share information among UAVs
which are directly connected with each other.

4.1 Coverage Information

In order to determine its own moving path, each UAV holds the coverage map,
which records the coverages of all cells that the UAV itself recognizes. In addi-
tion, as the information of other UAVs which shared information via wireless
communication, each UAV holds information on the identifier and current (esti-
mated) location of each of those UAVs for a predetermined period τ .
When starting operation, each UAV sets the coverage of the cell where it
currently exists to 1.0, and those of all other cells to 0.0.

4.2 Strategy to Determine Moving Path

As described above, each UAV sets its own moving path in order to preferentially
move to cells with lower coverage. Specifically, each UAV determines a cell to
move among all adjacent cells every unit time. Basically, each UAV chooses a
cell in the current row (i.e. adjacent left or right cell) as its destination. When
it determines that the coverages of the cells in the current row are sufficiently
higher, the UAV moves to another row (i.e. adjacent upper or lower row).
In the followings, we present the detailed procedure to determine a cell to
move using examples shown in Fig. 2:

1. The UAV first calculates the average of coverages of all its left-hand (right-
hand) cells in its current row. Then, the UAV checks whether the lower value
between calculated two averages is smaller than the predetermined threshold
α or not.
2. When the lower value is smaller than α, the UAV sets the direction (left
or right) with the lower average value as its moving path, and moves to the
adjacent cell. Let us show an example of this procedure using Fig. 2a, in which
the value in the bottom of each cell denotes the coverage of the corresponding
cell that UAV u1 recognizes, and the threshold α is set to 0.5. In this example,
u1 derives the average of coverages all its left-hand and right-hand cells (0.05
and 0.85), respectively. Since the lower value between the derived two averages
(0.05) is smaller than α = 0.5, u1 moves to adjacent left cell.
3. When the lower value is equal to or larger than α, the UAV moves to a cell
at either of the upper or lower row. Specifically, the UAV first calculates the
average of coverages of all its upper (lower) cells from its current row. Then, it
sets the direction (upper or lower) with the lower average value as its moving
780 A. Kanzaki and H. Akagi

1 2 3 4 5 1 2 3 4 5
avg: 0.30
0.2 0.7 0.5 0.1 0.6 0.0 0.5 0.3 0.0 0.4
6 7 8 9 10 6 7 8 9 10

0.8 0.5 0.6 0.7 0.1 0.6 0.3 0.4 0.5 0.0
11 12 13 14 15 11 12 13 14 15

avg: 0.05 1
avg: 0.85 1 avg: 0.75
0.1 0.0 1.0 0.9 0.8 1.0 0.9 0.8 0.7 0.6
16 17 18 19 20 16 17 18 19 20

0.2 0.3 0.4 0.3 0.3 0.0 0.1 0.2 0.1 0.1
21 22 23 24 25 21 22 23 24 25
avg: 0.14
0.1 0.2 0.6 0.0 0.7 0.0 0.0 0.4 0.0 0.5

(a) Move in the current row (left or right). (b) Move to adjacent row (upper or lower).

Fig. 2. Strategy to determine moving path (α = 0.5).

path, and moves to the adjacent cell. For example in Fig. 2b, since the average
of coverages of all its right-hand cells (0.75) becomes larger than α = 0.5, u1
determines to move to its adjacent upper or lower cell. In this example, since
the average of coverages of all its lower cells (0.14) is smaller than that of all
its upper cells (0.30), u1 moves to adjacent lower cell.

4.3 Information Sharing via Wireless Communication

When a UAV recognizes another UAV which can directly communicate with each
other like Fig. 3, it sends its own identifier, current location and its holding cov-
erage map, to the connected UAV. On receiving this information, the connected
UAV records information on the received identifier and location to its memory
space, together with the current time as the timestamp. This information is used
in the movement reenact procedure described in Sect. 4.4. After that, the UAV
updates its own coverage map. Specifically, the UAV compares coverage of each
cell between its holding coverage map and the received map. When the coverage
of a cell in the received map is higher than that in its own map, the UAV updates
the coverage of the corresponding cell to that in the received map. For example,
when u1 in Fig. 3, which holds its own coverage map shown in Fig. 4a, received
the coverage map from u2 shown in Fig. 4b, u1 updates coverages of colored cells
in Fig. 4c in its own map since the coverages of these cells in the received map
is higher than those in its own map. Note that this procedure is also conducted
by u2 using the coverage map received from u1 . Thus, the coverage maps of
connected UAVs are synchronized after the information sharing process.
In addition, when a UAV holds information (identifier, estimated location
and timestamp) of other UAVs in its memory space, i.e. the UAV had shared
information with other UAVs in the past, it also sends the stored information to
the connected UAV. On receiving this information, the connected UAV checks
 A UAV-Collaborative Sensing Method for Efficient Monitoring 781

1 2 3 4 5

6 7 8 9 10

11 12 13 14
connected 15

16 17 18 19 20

Fig. 3. Direct connection between u1 and u2 .

1 2 3 4 5 1 2 3 4 5

0.0 0.0 0.0 0.6 0.7 0.0 0.0 0.0 0.0 0.0
6 7 8 9 10 6 7 8 9 10

1
0.0 0.0 1.0 0.9 0.8 0.0 0.0 0.0 0.0 0.0
11 12 13 14 15 11 12 13 14 15

2
0.0 0.0 0.0 0.0 0.0 0.8 0.9 1.0 0.0 0.0
16 17 18 19 20 16 17 18 19 20

0.0 0.0 0.0 0.0 0.0 0.7 0.6 0.0 0.0 0.0

(a) Coverage map held by u1 . (b) Coverage map held by u2 .

1 2 3 4 5

0.0 0.0 0.0 0.6 0.7

6 7 8 9 10
update
1
0.0 0.0 1.0 0.9 0.8
11 12 13 14 15

2
0.8 0.9 1.0 0.0 0.0
16 17 18 19 20

0.7 0.6 0.0 0.0 0.0

(c) Coverage map after sharing.

Fig. 4. Coverage map sharing between UAVs.

whether it already holds information on the same UAV. If so, it updates infor-
mation of the corresponding UAV based on the received information only when
the timestamp in the received information is newer than that in its own record;
Otherwise, it simply adds the received information to its own memory space. For
example in Fig. 5a, u1 and u2 in Fig. 3 send information on UAVs {u3 , u4 }, which
had shared information in the past. In this case, u1 newly adds information on
782 A. Kanzaki and H. Akagi

1 2 3 4 5 1 2 3 4 5

ID Est. Location Timestamp ID Est. Location Timestamp

2 13
6 715 8 9 10 2 13
6 715 8 9 10

3 6 10 1 3 6 10 1
4 20 10
11 12 14 15 11 12 14 15

2 2
ID Est. Location Timestamp ID Est. Location Timestamp
16 17 18 1 819 1520 16 17 18 1 819 1520
3 2 5 3 6 10
4 20 10 4 20 10

(a) Sharing with each other. (b) After sharing.

Fig. 5. Sharing information of other UAVs.

1 2 3 4 5 1 2 3 4 5

ID Est. Location Timestamp ID Est. Location Timestamp

0.0 0.0
13 0.0
15 8 0.6 0.7 20.0 14 0.0 15 0.0 0.5 0.6
26 7 9 10 6 7 8 9 10

1 1
0.0 0.0 1.0 0.9 0.8 0.0 1.0 0.9 0.8 0.7
11 12 13 14 15 11 12 13 14 15

2 2
0.8 0.9 1.0 0.0 0.0 0.7 0.8 0.9 1.0 0.0
16 17 18 19 20 16 17 18 19 20

0.7 0.6 0.0 0.0 0.0 0.6 0.5 0.0 0.0 0.0

(a) u1 reenacts the movement of u2 . (b) Updated coverage map held by u1 .

Fig. 6. Mobility Reenactment of other UAVs (δ = 0.1).

u4 included in the information received from u2 . On the other hand, u2 updates

information on u3 to that included in the information received from u1 since the
received information is newer than that in its own record. As a result, u1 and
u2 hold information on other UAVs as shown in Fig. 5b.

4.4 Mobility Reenactment of Other UAVs

Every time each UAV moves to the adjacent cell according to the procedure
described in Sect. 4.2, it updates information on other UAVs obtained in the
procedure in Sect. 4.3. Specifically, the UAV updates the estimated location of
each UAV by applying the procedure in Sect. 4.2, based on its own coverage map.
In addition, it updates its own coverage map based on the estimated locations
of UAVs.
For example, u1 in Fig. 6a reenacts movements of u2 and updates its own
coverage map based on its own location and the estimated location of u2 as
shown in Fig. 6b.
 A UAV-Collaborative Sensing Method for Efficient Monitoring 783

5 Performance Evaluation
In order to verify the effectiveness of our proposed method, we have conducted
the simulation experiments, assuming information diffusion in a wide disaster
site. In the simulation experiments, we compare the performances of UCS/MR
with the following two methods:

Naive Method : In this method, each UAV independently moves in the region
without any collaboration. Specifically, each UAV just moves in the region
based on the strategy described in Sect. 4.2. Even when another UAV comes
within its wireless communication range, no wireless communication is con-
ducted. Thus, every UAV recognizes neither the existence of other UAVs nor
the change in coverages of cells caused by the movement of other UAVs.
UCS (w/o Mobility Reenactment): This method activates the information shar-
ing among connected UAVs described in Sect. 4.3, but does not apply the
mobility reenactment of other UAVs described in Sect. 4.4. Specifically, each
UAV updates its coverage map only according to its own mobility.

5.1 Experimental Settings

5 UAVs are randomly deployed (N = 5) in a two-dimensional region consisting of

10 × 10 cells (Mc = Nc = 10). In the experiments, all UAVs simultaneously start
their operations at the starting time of the simulation. The coverage attenuation
rate δ is set to 0.03.
The communication range of UAV rc is set such that each UAV can directly
communicate with others located adjacent cells. In the moving strategy, the
threshold α to determine the moving direction is set to 0.5. In UCS/MR, each
UAV holds information on other UAVs for 10 unit times (τ = 10). In other
words, each UAV reenacts the movement of another UAV for 10 unit times since
it shared information with the corresponding UAV.
In the above settings, we ran 100 trials each of whose duration is 60 unit
times.

5.2 Metrics

In the experiments, we have measured the following metrics:

Average coverage: The average of coverages of all cells at every unit time. The
larger average coverage indicates that the target region is monitored by UAVs
more frequently.
Variance in coverage: The variance in coverages of all cells at every unit time.
The smaller variance indicates that all cells in the region are uniformly mon-
itored by UAVs.
784 A. Kanzaki and H. Akagi

0.7 0.2
0.6

Variance in coverage
Average coverage

0.5 0.15
0.4
0.1
0.3
UCS/MR UCS/MR
0.2 0.05
UCS UCS
0.1
Naive Naive
0 0
0 20 40 60 0 20 40 60
Elapsed time [unit time] Elapsed time [unit time]

Fig. 7. Average coverage. Fig. 8. Variance in coverage.

0.0 (low) 1.0 (high)
Coverage:

(a) UCS/MR (b) UCS (c) Naive Method

Fig. 9. Distribution of coverages at 60 unit times after a trial starts.

5.3 Experimenal Results

Figures 7 and 8 respectively show the results of the average coverage and the
variance in coverage. The horizontal axis denotes the time elapsed since the
simulation starts.
From the result in Fig. 7, we can see that the coverages in UCS/MR and UCS
become larger than the Naive method. This indicates that the information shar-
ing among connected UAVs is effective to improve the efficiency of monitoring
in the entire region. In addition, UCS/MR achieves larger coverage than UCS.
This indicates that the mobility reenactment is effective to further improve the
efficiency of monitoring.
From the result in Fig. 8, we can see that the variances in UCS/MR and
UCS become smaller than that in Naive method except in the beginning part
of the simulation. This indicates that the information sharing among UAVs is
also effective to monitor the entire region uniformly. Also, we can see that the
mobility reenactment in MCS/MR further suppresses the variance in coverage.
In order to confirm the effectiveness of our proposed method, we show the
distribution of coverages at 60 unit times after a trial starts in Fig. 9. From
Fig. 9c, we can see that the upper area in the region is not monitored by any
 A UAV-Collaborative Sensing Method for Efficient Monitoring 785

UAV, although the coverages of cells in the lower area become higher. This
is because all UAVs moves only in the lower area for a long time, and cells
in the area are redundantly monitored by multiple UAVs. On the other hand,
from Figs. 9a and b, the number of cells with the coverage of 0.0 (black-colored
cells) become smaller by applying information sharing among UAVs. Especially,
from Fig. 9a, we can see that almost all cells are monitored by UAVs before
the coverage becomes zero by applying both information sharing and mobility
reenactment.

6 Conclusion

In this paper, we have proposed UCS/MR, which is a method for monitoring the
entire of a disaster site evenly and frequently. This method keeps coverages of all
cells higher by introducing two key techniques: (i) information sharing among
wirelessly connected UAVs and (ii) mobility reenactment of UAVs. Through
simulation experiments, we have confirmed that our proposed method achieves
efficient monitoring of the target region.
Our proposed method introduces several parameters such as α and τ which
affect the mobility of UAVs. Since these parameters may strongly have an impact
to the performance of monitoring, we plan to evaluate the impacts of those
parameters to the performance through extensive simulation experiments. In
addition, we currently propose a method assuming that all UAVs just monitor
the target region. In the future, we plan to design the method that efficiently
collects data obtained by UAVs from the entire target region, assuming the
existences of some base stations. Furthermore, we plan to extend our method
considering the battery constraint of UAVs.

Acknowledgements. This research is supported by the Grants-in-Aid for Young Sci-

entists (B)(17K12673) of Japan Society for the Promotion of Science, Japan, and the
Cooperative Research Project of the RIEC, Tohoku University.

References
1. Cesare, K., Skeele, R., Yoo, S.H., Zhang, Y., Hollinger, G.: Multi-UAV explaration
with limited communication and battery. In: Proceedings of IEEE International
Conference on Robotics and Automation (ICRA 2015), pp. 2230–2235 (2015)
2. Chen, Y., Zhang, H., Xu, M.: The coverage problem in UAV network: a survey. In:
Proceedings of the International Conference on Computing Communications and
Networking Technologies (ICCCNT 2014), pp. 1–5 (2014)
3. Franco, C.D., Buttazzo, G.: Energy-aware coverage path planning of UAVs. In:
Proceedings of the IEEE International Conference Autonomous Robot Systems
and Competitions (ICARSC 2015), pp. 111–117 (2015)
4. Ghaffarkhah, A., Mostofi, Y.: Dynamic networked coverage of time-varying envi-
ronments in the presence of fading communication channels. ACM Trans. Sens.
Netw. 10(3), Article no. 45 (2014)
786 A. Kanzaki and H. Akagi

5. Maza, I., Ollero, A.: Multiple UAV cooperative searching operation using polygon
area decomposition and efficient coverage algorithms. Distrub. Auton. Rob. Syst.
6, 221–230 (2007)
6. Mirzaei, M., Sharifi, F., Gordon, B.W., Rabbath, C.A., Zhang, Y.M.: Cooperative
multi-vehicle search and coverage problem in uncertain environments. In: Pro-
ceedings of the IEEE Conference on Decision and Control and European Control
Conference (CDC-ECC 2011), pp. 4140–4145 (2011)
7. Rashed, S., Soyturk, M.: Analyzing the effects of UAV mobility patterns on data
collection in wireless sensor networks. Sensors 17(2), 413 (2017)
8. Sanchez-Garcia, J., Reina, D.G., Toral, S.L.: A distributed PSO-based exploration
algorithm for a UAV network assisting a disaster scenario. Future Gener. Comput.
Syst. 90, 129–148 (2019)
9. Sharma, V., Kumar, R.: A cooperative network framework for multi-UAV guided
ground ad hoc networks. J. Intell. Rob. Syst. 77(3–4), 629–652 (2015)
10. Sujit, P.B., Lucani, D.E., Sousa, J.B.: Joint route planning for UAV and sensor
network for data retrieval. In: Proceedings of the IEEE International Systems Con-
ference (SysCon 2013), pp. 681–687 (2013)
11. Yin, C., Xiao, Z., Cao, X., Xi, X., Wang, P., Wu, D.: Offline and online search: UAV
multi-objective path planning under dynamic urban environment. IEEE Internet
of Things J. 99, 1 (2017)
 A Scheme to Improve Stream Transaction
Rates for Real-Time IoT Applications

Chaxiong Yukonhiatou1(B) , Tomoki Yoshihisa2 , Tomoya Kawakami3 ,

Yuuichi Teranishi2,4 , and Shinji Shimojo2
1
Graduate School of Information Science and Technology, Osaka University,
Suita, Japan
[email protected]
2
Cybermedia Center, Osaka University, Suita, Japan
{yoshihisa,shimojo}@cmc.osaka-u.ac.jp
3
Nara Institute of Science and Technology, Ikoma, Japan
[email protected]
4
Network System Research Institute,
National Institute of Information and Communications Technology, Tokyo, Japan
[email protected]

Abstract. Due to the recent proliferation of IoT (Internet of Things)

devices, a large amount of stream data such as video data or sensor data is
transmitted to remote processing computers. The stream transaction rate
is one of the main factors to improve the performance of some IoT appli-
cations. For instance, in surveillance systems, the probability to catch
a moving person increases as the processing computer analyzes video
with a higher transaction rate. To improve stream transaction rates,
some schemes reduce communication time between a processing com-
puter and stream data sources. They target periodic stream transactions
and assume static transmission intervals. However, the communication
and transaction time changes dynamically. Therefore, stream transaction
rates can be further improved by changing transmission intervals dynam-
ically depending on it. In this paper, we propose a scheme to improve
stream transaction rates by changing transmission intervals dynamically.
In our proposed scheme, a processing computer sometimes changes trans-
mission intervals to be the same length as the average transaction time.
Moreover, our proposed scheme adopts a progressive quality improve-
ment (PQI) approach to reduce communication and transaction time.
Our evaluation results revealed that the proposed scheme can achieve a
higher stream transaction rate than a conventional scheme with the PQI
approach.

1 Introduction
Due to the recent prevalence of IoT (Internet of Things) technologies, many
IoT devices such as cameras or sensors connect to the Internet. They are gener-
ally small size and their processing powers are also low. Therefore, in most IoT
applications, stream data generated by these devices are transmitted to remote
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 787–798, 2020.
https://doi.org/10.1007/978-3-030-15032-7_66
788 C. Yukonhiatou et al.

processing computers. The processing computers process stream data contin-

uously and get various useful results. For example, in surveillance systems, a
processing computer receives video data continuously from surveillance cameras
and analyzes image data of each frame of the video to identify recorded people.
In IoT applications, a higher stream transaction rate leads a more frequent
data analyses and enables performance improvements. In the above example,
the number of the people identified increases as the transaction rate increases
since they are moving and the probabilities to record them in the video increase.
Therefore, stream transaction rate is one of the main factors to improve perfor-
mances of IoT applications. We distinguish transactions and processes in this
paper. A transaction consists of some processes (the detail is in Subsect. 4.1).
To improve stream transaction rates, some schemes reduce communication time
between a processing computer and stream data sources. A shorter communi-
cation time enables a shorter transmission interval of stream data and thus the
stream transaction rate increases. They target periodic stream transactions and
assume static transmission intervals.
However, communication time and transaction time dynamically change
depending on the amounts of transmitted data and the number of them in each
transmission. Although a shorter communication time or a shorter transaction
time enables a shorter transmission interval, the conventional schemes assume
static intervals. Stream transaction rate can be further improved by changing
transmission intervals dynamically depending on them.
In this paper, we propose a scheme to improve stream transaction rate by
changing transmission intervals dynamically. In our proposed scheme, a process-
ing computer sometimes changes transmission intervals to be the same length
as the average transaction time. Transaction time is time from starting get-
ting data from a data source to finishing processing all the transmitted data
at a processing computer. Frequent changes of transmission intervals cause the
inconsistency of the times to get data. To keep the consistency as possible, in
our proposed scheme, the processing computer changes the interval every a fixed
number of transactions. Moreover, our proposed scheme adopts a progressive
quality improvement (PQI) approach to reduce communication time and trans-
action time. In the approach, the data amount of transmitted data is reduced
by transmitting the data that have necessary quality.
The rest of this paper is organized as follows. In Sect. 2, we introduce some
related work. In Sect. 3, we will explain our assumed system environments. Our
proposed scheme is explained in Sect. 4, and the performance evaluation results
are shown in Sect. 5. Finally, we will conclude the paper in Sect. 6.

2 Related Work
Some methods to improve transaction rate for IoT applications have been pro-
posed.
A method to reduce communication delays by controlling the number of the
data packets for transmission was proposed in [1]. A method to control trans-
mission buffer to keep transmission rates was proposed in [2]. These methods
 A Scheme to Improve Stream Transaction Rates 789

are similar with our proposed method that data is divided into some parts, but
our method progressively collects the divided data considering their necessities.
A method to reduce communication traffic between a video stream data
source and a processing computer was proposed in [3]. A method to reduce
communication traffic by compressing data was proposed in [4]. Different from
these methods, in our proposed method, the processing rate is improved while
the qualities of stream data progressively improve.
To reduce the communication delay, many schemes have been proposed [5–
8]. These schemes degrade qualities of data such as resolutions for image data,
to reduce data to be collected and achieve a shorter transaction time. These
methods result in performance degradations of IoT applications. In our pro-
posed method, applications can improve their performances by changing the
transmission intervals dynamically.
A model to reduce the delay for starting data processing were proposed in [9].
In the model, the processing computer prepares separated data queues for each
process and selects the data to process so as to reduce the processing delays. In
our proposed method, we can adopt this method in the processing computer. Our
proposed method is different from this in the point that we improve transaction
rates by managing how to process data.
Some methods to control data generation timings to reduce communication
traffic were proposed. The method proposed in [10] considers communication
distance between the data sources and the processing computer. The method
proposed in [11] considers the communication channels such as wireless or wired.

3 Assumed System
3.1 System Architecture

Figure 1 shows our assumed system architecture. Some IoT devices such as
surveillance cameras continuously get data about their observations such as video
data and act as stream data sources. They can generate some data items which
have levels from their observed data. These IoT devices connect to a computer

Fig. 1. Our assumed system architecture

790 C. Yukonhiatou et al.

network such as the Internet and communicate with a processing computer. The
processing computer executes designated processes every data reception from
the stream data sources. Such a type of processes is called stream processing.
The users designate the processes for the stream data to a processing computer.
The processing computer has a buffer for storing received data and executes
processes for the data.

3.2 An Application Scenario

Suppose an area in that some surveillance cameras are deployed and a processing
computer gathers their recorded video data. They connect to a designated com-
puter network and communicate with each other similar to our assumed system
architecture.
As an example application scenario, we assume a person identification system
by a face recognition. For this, the application designates the process that notifies
to the user when the processing computer identifies person’ face and determines
whether the person is registered or not in the video data got from surveillance
cameras. To detect faces, the user submits the face images of registered persons
to the processing computer beforehand. The processing computer continuously
analyzes image data got from surveillance cameras and identifies faces in received
image data. When the processing computer finds faces in an image data, it checks
whether the found faces are those of registered persons or not. If the processing
computer detects the faces that are not registered, it sends a notification to the
user by e-mail or other messaging services.

3.3 Research Objective

One of the main factors to improve performances of IoT applications is stream

transaction rate. Also, It is better to give a consistency to data transmission
intervals for transactions. For example, in the above scenario, a higher and con-
sistent transaction rates, i.e., a more frequent and periodic data gathering from
the cameras, increases the chance to identify people since they are moving and
the probabilities to record them in the video increase.

Fig. 2. Stream data collection of an conventional approach and of the PQI approach
 A Scheme to Improve Stream Transaction Rates 791

Conventional methods improve transaction rate by reducing communication

time between a processing computer and stream data sources. They target peri-
odic stream processing and assume static transmission intervals. However, com-
munication time and transaction time dynamically change depending on the
amount of transmitted data and the number of them in each transmission.
Stream transaction rate can be further improved by changing transmission inter-
vals dynamically depending on them.
Our research objective in this paper is improving stream transaction rates
by changing transmission intervals dynamically.

4 Proposed Method
In this section, we explain our proposed method. In our proposed method
called PQI-CDI (Progressive Quality Improvement approach with Cycle-based
Dynamic Interval), the system changes transmission intervals dynamically and
adopts PQI approach to reduce communication and processing time.

4.1 PQI Approach

Generally, data have some qualities, e.g., resolution of image data. Data analy-
ses can be applied for each quality and data with the highest quality often gives
the best performance for analyses. If processing computers analyze data sequen-
tially in the order of quality from the lowest to the highest, they can stop data
analyses when the subsequent analyses for higher quality data are meaningless.
For example, in the above scenario, the processing computer first receives the
lowest quality image data of a frame and analyzes the difference from the previ-
ous frame. In case that the difference are small, the processing computer skips
the analyses of higher quality image data since new humans do not appear in
the frame because of small difference. In cases that the probability to proceed
to higher quality data analyses is small, the total amount of received data is
reduced, compared with the case that all quality data are received. Therefore,
the data amount to be received is reduced when the probability is small com-
pared with the processing computer receives the highest quality data without
consideration of data qualities. Thus, communication time is reduced keeping the
application performance. We call this approach progressive quality improvement
approach.
Figure 2 shows a timing chart for stream processing under the conventional
approach and the PQI-CDI approach. In the PQI-CDI approach, the data Dd,a (t)
(d = 1, 2, t = 1, · · · ) is divided into some levels. Each transaction includes some
processes for each divided data. In the figure, the number of the levels is 2 and
the transaction consists of two processes for the divided data Dd,1 (t) and Dd,2(t) .
d is the stream number, t is the cycles for data collections, and l is the level.
In the cycle 1, the transaction finishes at the first level in both streams. In the
cycle 2, the processing computer requires D1,2 (2) when it finishes the process
for D1,1 (2). The camera 1 transmit the required D1,2 (2) and the processing
792 C. Yukonhiatou et al.

computer starts the process for D1,2 (2). In this case, the transaction finishes
when the processing computer finishes the process for D1,2 (2) since the number
of the levels are 2. The transaction time in this case is reduced compared with
that under the conventional approach as shown in the figure.

4.2 Cycle-Based Dynamic Interval

For changing the intervals, the PQI-CDI method determines when and how long
the processing computer changes the intervals.

4.2.1 Timings to Change Intervals

A more frequent change of intervals causes a less consistency. On the other hand,
a less change of intervals cannot improve the transaction rate further since it
takes a longer time to change intervals based on the communication time and the
processing time. However, it is difficult to find the appropriate timing to change
intervals depending on the communication time and processing time since these
times change dynamically.
One of the solution for such a dynamic situation is fixed period. Therefore,
in the PQI-CDI method, the processing computer changes the intervals of each
stream every finishing Cn transactions with the data source n.

Start of the Receive the Receive a request

t th cycle request of Dn,q(t) of changing
interval to i
Get and
store Dn,a(t)
Generate Dn,q(t) Set
from Dn,a(t) In=i
Generate Dn,1(t)
from Dn,a(t)
Send Dn,q(t) Rearrange the start
to the processing of the next cycle
Send Dn,1(t) computer
to the processing
computer

Fig. 3. Data sources generation

4.2.2 How to Determine Intervals

A longer interval causes a less transaction rate. On the other hand, a shorter
interval than a transaction time causes a longer transaction time because the
processing computer receives the next data before it finishes the current process.
Therefore, the interval that is the same length as the transaction time is the most
appropriate value. However, it is difficult to grasp the transaction time since it
depends on the communication time and the transaction time.
The PQI-CDI method has a cycle to change intervals Cn . Therefore, we adopt
the average transaction time for the previous cycle as the new interval.
 A Scheme to Improve Stream Transaction Rates 793

4.2.3 Algorithms
Figure 3 shows the flow chart of data sources. When the t th cycle starts, each
data source n gets Dn,a (t) from their sensors and stores it to their storages
temporary. Here, Dn,a (t) is the observed original data of the data source n at
the cycle t. Dn,l (t) (l = 1, · · · , L) is the generated data from Dn,a (t) of that level
is l. First, they generate Dn,1 (t) from Dn,a (t) and send Dn,1 (t) to the processing
computer. When the data source n receives the request of Dn,q (t), it generates
Dn,q (t) from stored Dn,a (t) and sends Dn,q (t) to the processing computer. When
the data source n receives the request of changing interval to i, it changes its
interval to i and rearranges the start of the next cycle.
Figure 4 shows the flow chart of the processing computer. When the process-
ing computer receives Dn,l (t), it processes Dn,l (t). When l = L and Dn,l (t) is the
final level data, the process of t th cycle finishes. Otherwise, the processing com-
puter judges the necessity of Dn,l+1 (t). In case that Dn,l+1 (t) is needed for the
process execution, the processing computer requets Dn,l (t) to the data source
n, otherwise, the process finishes. When the process finishes, in the PQI-CDI
method, the processing computer checks whether cn reaches Cn or not. cn is the
variable to count the number of transactions for the data source n. Here, again,
Cn is the interval of transactions to change the interval of the data transmission
of the data source n. In case that cn reaches to Cn , the processing computer
calculates the new interval and sends the request for changing the interval to
the data source n. Then, initialize cn . Here, T Tn (t) is the transaction time of the
t th cycle of the data source n, i.e., the time to get the original data at the data
source n to the time to finish the process of the data at the processing computer.

Receive Dn,l(t)

Process Dn,l(t)

Yes Finish the process

l=L
of t th cycle

No
Judge the No Send a request for
Check No
necessity changing interval to
cn<Cn
of Dn,l+1(t) i=Average TTn(t)
t=t-c+1, ,t
Yes Yes

Request Dn,l+1(t) cn++ cn=1

Fig. 4. Data processing in the processing computer

Table 1. Parameter values

Input bandwidth 10 [Mbps]

Output bandwidth 10 [Mbps]
Total data amount 12.5 [Kbytes]
Processing time ratio 10−6
794 C. Yukonhiatou et al.

5 Evaluation
5.1 Evaluation Setup
5.1.1 Evaluation Parameters
In this evaluation, we assume the application explained in Subsect. 3.2 and use
the parameters shown in Table 1. Input Bandwidth is the input communication
bandwidth for the processing computer. When the processing computer com-
municates with some data sources, the input bandwidth is fairly shared among
data sources. Output Bandwidth is the output communication bandwidth of each
data source. Total data amount is the data amount of Dn,a (t) (n = 1, · · · , N ,
t = 1, · · · , T ). To make the evaluation results easily understandable, we set the
same data amount for all data items. Processing Time Ratio is the value of the
transaction time divided by the amount of the data item for the process. We
set these parameters considering practical situations. We use the same values
for P P robn,p (t) (p = 1, · · · , Q − 1). P P robn,p (t) is the probability to the next
level from the level p. For this, we set the final probability F P rob for processes
to proceed to the final level. P P robn,p (t) = F P rob1/N . We simulate the stream
processing system for 60 s.

5.1.2 Evaluation Items

The main evaluation item is the transaction rates. The transaction rates are the
number of the transactions that the processing computer finishes in a second.
One of the performance for stream processing is the transaction times. We calcu-
late the average values since we confirmed that the transaction times converge.
One of the demerits of dynamic interval is the inconsistency of the intervals.
To investigate this, we calculate the fairnesses of the intervals. We adopt the
Jain’s coefficient for the fairness. A smaller fairness indicates a more inconsistent
interval.

5.2 Influence of Number of Streams

We measure the performances changing the number of the streams. In this exper-
iment, the number of the transactions that the processing computer changes the
interval Cn (n = 1, · · · , N ) is 2 since this value gives a higher transaction rate
and a higher fairness as shown in the next subsection. N is the number of the
streams. We set the number of the levels to 5 as an example value. The initial
interval is 100 [msec.].
Figure 5 shows the average transaction rate. The horizontal axis is the num-
ber of the streams and the vertical axis is the average transaction rate. Our
proposed PQI-CDI method gives a higher average transaction rate than that for
the case of static interval (100 [msec.]) when the number of the streams is less
than 9. This is because the network and the processing computer have an extra
capacity to improve the transaction rate compared when the intervals are 100
[msec.] and the PQI-CDI method exploits this extra capacity by changing the
interval dynamically. The average transaction rate increases as the final proba-
bility increases since the average amount of the data to be transmitted from the
 A Scheme to Improve Stream Transaction Rates 795

data sources decreases as the final probability decreases and the extra capacity
increases. The line for the static interval stops at the point that the number
of the streams is 8 since the transaction times diverge in the cases where the
number of the streams is larger than 8.
Figure 6 shows the average transaction time. The horizontal axis is the num-
ber of the streams and the vertical axis is the average transaction time. Our
proposed PQI-CDI method gives a shorter average transaction time than that
for the case of static interval (100 [msec.]). This is because the network and
the processing computer have an extra capacity to improve the transaction time
compared when the intervals are 100 [msec.] as the same reason as the average
transaction rate.
Figure 7 shows the fairness of the intervals. Our proposed PQI-CDI method
gives a lower fairness than that for the case of static interval (100 [msec.]) since
the intervals dynamically change under the PQI-CDI method and the intervals
have various values. The fairness decreases slightly as the number of the streams
increases since the range of the intervals increases as the number of the streams
increases.

300
Average Transaction Time [msec.]
Average Transaction Rate [num/sec.]

Static Interval (100.0 [msec.]) Static Interval (100.0 [msec.])

80
Final Probability=0.1 Final Probability=0.1
250
Final Probability=0.3 Final Probability=0.3
Final Probability=0.5 Final Probability=0.5
Final Probability=0.7 Final Probability=0.7
60 200
Final Probability=1.0 Final Probability=1.0

150
40

100

20
50

0 0
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
Number of Streams Number of Streams

Fig. 5. Average transaction rates Fig. 6. Average transaction times

under different final probabilities under different final probabilities
1 35
Final Probability=0.1
Average Transaction Rate [num/sec.]

0.9
30 Final Probability=0.3
0.8 Final Probability=0.5
Final Probability=0.7
Fairness of Intervals

0.7 25
Final Probability=1.0

0.6
20
0.5
Static Interval (100.0 [msec.]) 15
0.4 Final Probability=0.1
Final Probability=0.3
0.3 10
Final Probability=0.5
0.2 Final Probability=0.7
Final Probability=1.0
5
0.1

0 0
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
Number of Streams Cycle Length

Fig. 7. Fairness of intervals under dif- Fig. 8. Average transaction rates

ferent final probabilities under different final probabilities
changing cycle length
796 C. Yukonhiatou et al.

5.3 Influence of Cycles

In our PQI-CDI method, the cycles for changing intervals influence the perfor-
mances. We investigate the influence changing the cycle length Cn (n = 1, · · · , N )
under different final probabilities. The cycle lengths for all streams are the same.
The final probabilities are 0.1, 0.3, 0.5, 0.7, 1.0. The initial interval is 100 [msec.]
and the number of the level is 5 as an example value.
Figure 8 shows the average transaction rate. The horizontal axis is the number
of the cycle length and the vertical axis is the average transaction rate. We can
see that a shorter cycle length causes a higher average transaction rate since the
interval is adjusted to the transaction time frequently by changing the interval.
Therefore, the processing computer can collection more data by changing the
transmission interval dynamically under the PQI-CDI method than that under
the conventional method.

170 1
160
Average Transaction Time [msec.]

0.9
150
140 0.8
130
Fairness of Intervals

120 0.7
110
100 0.6
90
0.5
80
70 0.4
60 Final Probability=0.1
50 0.3 Final Probability=0.3
Final Probability=0.1
40 Final Probability=0.3 Final Probability=0.5
0.2
30 Final Probability=0.5 Final Probability=0.7
20 Final Probability=0.7 0.1 Final Probability=1.0
10 Final Probability=1.0
0 0
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
Cycle Length Cycle Length

Fig. 9. Average transaction times Fig. 10. Fairness of intervals under dif-
under different final probabilities ferent final probabilities changing cycle
changing cycle length length

Figure 9 shows the average transaction time. The horizontal axis is the num-
ber of the cycle length and the vertical axis is the average transaction time. In
the PQI-CDI method, the average transaction time for a higher final probability
gives a longer average transaction time since a higher final probability enables
more transactions to proceed to the final level. We can see some average trans-
action times with a lower final probability gives a shorter average transaction
time.
Figure 10 shows the fairness of the intervals. The horizontal axis is the num-
ber of cycle length and the vertical axis is the fairness of intervals values. The
fairness decreases as the cycle length increases. This is because under our pro-
posed method the intervals dynamically change.

6 Conclusion
The stream transaction rate is one of the main factors to improve the perfor-
mance of IoT applications. In this paper, we proposed the PQI-CDI method in
 A Scheme to Improve Stream Transaction Rates 797

order to improve the transaction rate. In the PQI-CDI method, the processing
computer changes transmission intervals to be the same length as the average
transaction time every a fixed number of transactions. Moreover, the PQI-CDI
method adopts the PQI approach to reduce the communication and the transac-
tion time. Our evaluation results revealed that the PQI-CDI method can achieve
a higher stream transaction rate than a conventional scheme with the PQI app-
roach.
In the future, we will plan to evaluate our proposed method in practical
situations and consider other method to change intervals.

Acknowledgement. This work was supported in part by JSPS KAKENHI Grant

Numbers JP17K00146 and JP18K11316 and by Research Grant of Kayamori Founda-
tion of Informational Science Advancement. Also this work include the result of NICT
· Osaka University joint research “research and development of advanced network plat-
form technology for large scale distributed computing”.

References
1. Yu, S., Tian, T., Zhou, J., Guo, H.: An adaptive packet transmission model for
real-time embedded network streaming server. In: Proceedings of the IEEE Inter-
national Conference on Audio, Language and Image Processing, Shanghai, pp.
848–853 (2008)
2. Zhao, P., Yu, W., Yang, X., Meng, D., Wang, L.: Buffer data-driven adaptation
of mobile video streaming over heterogeneous wireless networks. IEEE Internet of
Things J. 5(5), 3430–3441 (2018)
3. Kanzaki, H., Schubert, K., Bambos, N.: Video streaming schemes for industrial
IoT. In: Proceedings of the IEEE International Conference on Computer Commu-
nication and Networks (ICCCN), Vancouver, pp. 1–7 (2017)
4. Agrawal, U.A., Jani, P.V.: Performance analysis of real time object tracking system
based on compressive sensing. In: Proceedings of the IEEE International Confer-
ence on Signal Processing, Computing and Control (ISPCC), Solan, pp. 187–193
(2017)
5. Ortega, A., Khansari, M.: Rate control for video coding over variable bit rate
channels with applications to wireless transmission. In: Proceedings of the IEEE
International Conference on Image Processing, Washington, pp. 388–391 (1995)
6. Incel, O.D., Krishnamachari, B.: Enhancing the data collection rate of tree-based
aggregation in wireless sensor networks. In: Proceedings of the IEEE Communi-
cations Society Conference on Sensor, Mesh and Ad Hoc Communications and
Networks, San Francisco, pp. 569–577 (2008)
7. Xhafa, F., Naranjo, V., Caballé, S., Barolli, L.: A software chain approach to big
data stream processing and analytics. In: Proceedings of the IEEE International
Conference on Complex, Intelligent, and Software Intensive Systems, Blumenauv,
pp. 179–186 (2015)
8. Papadopoulos, G.Z., Pappas, N., Gallais, A., Noel, T., Angelakis, V.: Distributed
adaptive scheme for reliable data collection in fault tolerant WSNs. In: Proceedings
of the IEEE 2nd World Forum on Internet of Things (WF-IoT), Milan, pp. 116–121
(2015)
798 C. Yukonhiatou et al.

9. Beard, J.C., Chamberlain, R.D.: Analysis of a simple approach to modeling per-

formance for streaming data applications. In: Proceedings of the IEEE 21st Inter-
national Symposium on Modelling, Analysis and Simulation of Computer and
Telecommunication Systems, San Francisco, pp. 345–349 (2013)
10. Zhu, X., Huang, P., Han, S., Mok, A.K., Chen, D., Nixon, M.: MinMax: a sampling
interval control algorithm for process control systems. In: Proceedings of the IEEE
International Conference on Embedded and Real-Time Computing Systems and
Applications, Seoul, pp. 68–77 (2012)
11. Dias, G.M., Nurchis, M., Bellalta, B.: Adapting sampling interval of sensor net-
works using on-line reinforcement learning. In: Proceedings of the IEEE 3rd World
Forum on Internet of Things (WF-IoT), Reston, pp. 460–465 (2016)
 A Video Data Distribution Method for Flexible
Bandwidth Allocation in Hybrid Broadcasting
Environments

Satoru Matsumoto(&) and Tomoki Yoshihisa

Cybermedia Center, Osaka University,

Mihogaoka 5-1, Osaka, Ibaraki 567-0047, Japan
{smatsumoto,yoshihisa}@cmc.osaka-u.ac.jp

Abstract. Due to the recent development of wireless broadcasting technolo-

gies, video data distribution systems on hybrid broadcasting environments has
attracted great attention. In some video distribution methods for hybrid broad-
casting environments, the server broadcasts some pieces via the broadcasting
channel and delivers them to all the clients to effectively reduce the interruptions
times. They fix the bandwidth allocation to predict when the clients finish
receiving pieces. However, the server can change the bandwidth allocation
flexibly in recent hybrid broadcasting environments such as MBMS (multimedia
broadcast and multicast services) mode of 5G networks. In this paper, we
propose a video data distribution method for flexible bandwidth allocation in
hybrid broadcasting environments. In our proposed method, the server broad-
casts the requested pieces from the clients only when the transmission time via
the broadcasting channel is earlier than that via the communication channel. The
server allocates the bandwidths so as to reduce the interruption times further.
Our simulation results revealed that our proposed method achieves the shortest
interruption time by appropriately allocating bandwidths.

1 Introduction

Due to the recent development of wireless broadcasting technologies, streaming video

delivery on hybrid broadcasting environments has attracted great attention. One of the
pioneer points of hybrid broadcasting environments is that the clients of a system can
receive data both from the broadcasting channel and the communication channel.
MBMS (multimedia broadcast and multicast services) mode of 5G networks or the use
of unicast and broadcast addresses in WiFi networks can provide such hybrid broad-
casting environments. In streaming video delivery, the clients encounter playback
interruptions when they cannot receive a part of the video data (a piece) until the time
to play it. For example, a user of a video delivery service plays a video on his
smartphone. When the smartphone has finished the first piece reception, it starts
playing the video while it keeps the receptions of subsequent pieces. In the cases that
the playback of the video reaches to the time position of that piece the smartphone has
not yet received, the playback interrupts. Long interruption time annoys the users and a
shorter interruption time is required for streaming video delivery services. Therefore,

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 799–809, 2020.
https://doi.org/10.1007/978-3-030-15032-7_67
800 S. Matsumoto and T. Yoshihisa

interruption time reduction for streaming video delivery on hybrid broadcasting

environments is one of the hot topic in video delivery research fields.
To reduce the interruption time, various video data distribution methods have been
proposed [1–9]. Most of them adopt the division-based broadcasting technique for the
broadcasting channel. In the technique, the video data are separated into some pieces
and the server broadcasts the pieces that a larger number of the clients does not have.
The total interruption time can be reduced since the probability that the clients receive
each piece before they start playing the video contained in it increases. For the com-
munication channel, the clients request the pieces that they can receive earlier than the
reception via the broadcasting channel and receive them. In the conventional methods,
the systems can predict when the clients finish receiving pieces since they assume that
the allocation of the bandwidths for the broadcasting channel and the communication
channel is fixed.
However, in most hybrid broadcasting environments, the system can flexibly
change the bandwidth allocation. For example, in the MBMS mode of 5G networks,
although the total bandwidth is fixed, the system can control the bandwidths for the
broadcasting channel and that for the communication channel. In the systems with
flexible bandwidth allocation in hybrid broadcasting environments, interruption time
can be further reduced by appropriately allocating bandwidths. When the number of the
clients is large, a larger broadcasting bandwidth has a larger possibility to reduce the
interruption time since all the clients can receive the broadcasted pieces. Otherwise, a
larger communication bandwidth has a larger possibility to reduce it since the clients
can receive their requested pieces earlier.
In this paper, we propose a video data distribution method for flexible bandwidth
allocation in hybrid broadcasting environments. In our proposed method, each client
requests the pieces of that time to start playing is the earliest in all the pieces that it does
not have. The broadcasting server broadcasts the requested pieces only when the
transmission time via the broadcasting channel is predicted to be earlier than that via
the communication channel. Otherwise, the server returns the requested pieces to the
client via the communication channel. The server allocates the bandwidths so as to
reduce the interruption times further. Our simulation results revealed that our proposed
method achieves the shortest interruption time by appropriately allocating the band-
widths than the conventional methods.
In the future, by applying the proposed RDB method, it is possible to deliver video
with reduced interruption time, while providing adaptive contents adapting to viewer
and terminal requests using the communication channel side.
The paper is organized as follows: we introduce some related work in Sect. 2. We
explain our assumed environments and our proposed system in Sects. 3 and 4. Our
experimental results are presented and discussed in Sect. 5. Finally, we conclude the
paper in Sect. 6.

2 Related Work

Various methods to reduce interruption time for video delivery have been proposed.
 A Video Data Distribution Method 801

Fig. 1. An image of our assumed hybrid broadcasting environments

A method to reduce interruption time using peer-to-peer (P2P) communication is

proposed in [10]. In P2P communication, the clients communicate with the server and
also other clients. By exchanging received pieces with other clients, the communication
load of the server is reduced and thus the interruption time is reduced. To earlier find
the clients that do not have the received pieces, a method to exchange the information
of received pieces was proposed in [11]. These methods do not target hybrid broad-
casting environments. In hybrid broadcasting environments, interruption time is
reduced further by delivering pieces also from the broadcasting channel.
One of the attractive features of the next generation cellular networks (5G) is the
MBMS mode. In the MBMS mode, the server can transmit data to some clients
concurrently. Recently, many researches in the streaming video delivery field focus on
the feature and propose schemes to reduce the communication traffic by exploiting
multicasting [1–3]. However, these methods assume that the allocation of the band-
widths for the broadcasting channel and the communication channel is fixed.
Various video broadcasting methods have been proposed [4–6, 8, 9]. In these
methods, the server broadcasts all pieces via the broadcasting channel although the
clients can receive some pieces via the communication channel. Hence, we have
proposed a data piece elimination technique for hybrid broadcasting environments in
[7]. However, these also assume fixed bandwidth allocation.
Some systems for hybrid broadcasting environments have been proposed [12–14].
These systems transmit video data via the broadcasting channel and other contents
related to the broadcasted video via the communication channel such as the Internet. In
our proposed system, different from these systems, the video data are transmitted by
using both of the broadcasting channel and the communication channel.
There are some other researches related to hybrid broadcasting environments.
A system to deliver 3-D videos over the environments is proposed in [15]. A hybrid
system of CDN (Contents Delivery Networks) and P2P (Peer-to-Peer) is proposed in
[16]. Our assumed environments are different from these systems in that the server can
change the bandwidth allocation flexibly.
802 S. Matsumoto and T. Yoshihisa

Fig. 2. An image of pieces

3 Assumed Environments

In this section, we explain our assumed environments.

3.1 Hybrid Broadcasting Environments

Figure 1 shows an image of our assumed system architecture for the hybrid broad-
casting environments. The hybrid broadcasting environments haves two data trans-
mission channels, the broadcasting channel and the communication channel. The
clients such as smartphones or laptops can receive data from both channels. The server
has video data. It can broadcast data to all the clients via the broadcasting channel and
can transmit them to each client based on their request via the communication channel.
The server transmit data using the base station. The total bandwidth of the base station
is fixed, but the system can control the allocation of the bandwidths for the broad-
casting channel and the communication channel.
A practical example of the hybrid broad casting environments is the MBMS mode
of 5G networks. In the case of 5G, the broadcasting channel is a communication
channel between the server and the clients provided by the MBMS mode. The com-
munication channel is a ordinary communication channel. The system can flexibly
control the allocation of the bandwidth for the MBMS mode and for the ordinary mode
under the condition that the total bandwidth is fixed.

3.2 Video Delivery

The data for a video is divided into some pieces such as GoP (Group of Pictures) of
MPEG (Moving Picture Experts Group). Figure 2 shows the data structure of our
assumed video data. A general data amount for a GoP is the data amount of the video
for 0.5 s. For example, suppose the case that a video for 30 min is encoded by MPEG
and the bitrate is 5 Mbps, a general video bitrate of television services. The data
amount for the video is 1.125 Gbytes and that for a piece is 312.5 Kbytes.
The users operate the clients to play the video at arbitrary timings. The clients play
the video to the end without skips once they start playing it. In the cases that the clients
do not have a piece at the time to play it, they encounter interruptions and wait for the
reception of the piece.
 A Video Data Distribution Method 803

4 Proposed Method

In this section, we explain our proposed system. We call our proposed method the
Request-based Division Broadcasting (RDB) method. In the RDB method, the server
broadcasts the requested pieces only when the transmission time via the broadcasting
channel is predicted to be earlier than that via the communication channel. The detail
follows.

4.1 Data Transmission via Communication Channel

The clients encounter interruptions when they do not have the piece even when the time
to play it comes. The clients play the video data from the beginning to the end without
skips. Therefore, the probability to encounter interruptions can be further reduced by
receiving the pieces of that time to start playing is the earliest in all the pieces that each
client does not have. Moreover, simultaneously multiple (parallel) piece receptions
cause long time to finish the receptions of each piece.
Hence, in the RDB method, the clients request the piece of that time to start playing
is the earliest in all the pieces that each client does not have. When they finish the
reception of the requested piece, they request the next piece. Also, the server returns the
requested pieces one by one to each client to make the time to finish the transmission of
each piece earlier.

4.2 Data Transmission via Broadcasting Channel

Some conventional methods adopt static broadcasting, i.e., the server creates a
broadcasting schedule in advance and broadcasts the pieces in the order of the
schedule. However, in the hybrid broadcasting environments, the server can commu-
nicate with the clients and can broadcast the pieces considering the pieces that they do
not have. In the cases that the server broadcasts a piece before the clients receive it via
the communication channel, they can receive the piece earlier. Moreover, all the clients
can receive the broadcasted pieces.

Table 1. Simulation parameters

Item Value
Bit rate 5 Mbps
Video duration 30 [min.]
Piece size 0.5 s
Simulation time 5 [h]

Hence, in the RDB method, the server broadcasts the requested pieces only when the
transmission time via the broadcasting channel is predicted to be earlier than that via the
communication channel. The server broadcasts the requested pieces one by one to make
the time to finish the transmission earlier, as the same as the strategy for the
804 S. Matsumoto and T. Yoshihisa

communication channel. Therefore, the server can predict when it finishes transmitting
each requested piece via the broadcasting channel. For the prediction of the time to finish
transmitting each requested piece via the communication channel, the sever uses the
average communication speed between it and the requested client and calculates the time.

5 Evaluation Results

In this section, we will show some experimental results for evaluating the RDB
method.

5.1 Experimental Setting

We simulate the interruptions for video delivery on hybrid broadcasting environments
and compare them under the RDB method and other conventional methods. We
measure interruption time and the number of interruptions. Here, interruption time
means total time during the video playback is interrupted in each client.
Table 1 shows the experimental parameters. We set these values assuming the
hybrid broadcasting environment on the MBMS mode of 5G networks. We give the
times to start playing the video on the clients by a Poisson distribution since the users
operate the client to ply the video at arbitrary timings as explained in Subsect. 3.2. We
perform the simulation changing the average arrival rate of the timing to start playing
the video. The video duration is 30 min and the bitrate is 5 Mbps assuming that the
video data is MPEG2 encoded drama data. The duration for each piece is 0.5 s.
Assuming that there is one bottleneck link between the server and the clients, we set the
communication bandwidth is shared among the clients. The simulation time is 5 h since
the interruption time converges sufficiently 5 h after the simulation starts.

Fig. 3. Average interruption time and the average arrival interval

 A Video Data Distribution Method 805

We compare the RDB method with the DBSC-SM method proposed in [8] and the
carousel method. In the DBSC-SM method, the server broadcasts the pieces sequen-
tially from the requested piece while the number of the clients that communicate with
the server (R) exceeds Rth. We set Rth by 0. This means that the server broadcasts the
pieces sequentially to the end once a piece is requested by a client. In the carousel
method, the server broadcasts the pieces sequentially from the beginning to the end
cyclically when a client starts playing the video.

5.2 Average Arrival Interval

We found the bandwidth allocation that gives the shortest average interruption time for
each method and plot the shortest value. The detail of how to find the appropriate
bandwidth allocation is written in Subsect. 5.3. The total bandwidth is 20 Mbps or
30 Mbps. The result is shown in Fig. 3. The horizontal axis is the average arrival rate of
the time to start playing the video and the vertical axis is the average interruption time.

Fig. 4. Average interruption time and the broadcasting bandwidth

We can see that our proposed RDB method always gives the shortest average
interruption time. The DBSC-SM method givers close average interruption times to
those of the RDB method. This is because the server under the DBSC-SM method
broadcasts the almost similar pieces to those under the RDB method since the client by
that requested piece is broadcast often requests the next piece continuously in the RDB
method and the requested piece is broadcast. Therefore, the broadcasting pieces are
often the same as those under the DBSC-SM method, in that the server broadcasts the
subsequent pieces. Only in the case that the client has already received the next piece,
the piece to be broadcast differs among the DBSC-SM method and the RDB method.
The average interruption time decreases as the average arrival interval or the total
bandwidth increases since the clients can receives the pieces faster.
806 S. Matsumoto and T. Yoshihisa

For example, in the case that the average arrival interval is 10 s and the total
bandwidth is 30 Mbps, the average interruption time of our proposed RDB method is
6.4 s and that of the DBSC-SM method and of the Carrousel method are 8.8 s and 61 s
respectively.

5.3 Bandwidth Allocation

In the hybrid broadcasting environments, the bandwidth allocation influences the
average interruption time. The total bandwidth of the broadcasting bandwidth and the
communication bandwidth is fixed. A larger broadcasting bandwidth causes a smaller
communication bandwidth and it is difficult to find the appropriate bandwidth alloca-
tion in hybrid broadcasting environments. Therefore, we investigate the influence of the
bandwidth allocation in this subsection.
Figure 4 shows the result for the average interruption time under different broad-
casting bandwidth. The total bandwidth is 20 Mbps. Lambda means the average arrival
rate of the clients. The horizontal axis is the broadcasting bandwidth and the vertical
axis is the average interruption time.
The average interruption time tends to decrease when the broadcasting bandwidth is
small and tends to increase when large for all the methods. In the case that the
broadcasting bandwidth is 20 Mbps, the average interruption time diverges since the
clients have no way to request pieces and the server does not send any piece. For
example, in the case where the lambda is 25 s, the average interruption times of all the
methods are the shortest when the broadcasting bandwidth is 10 Mbps. In the case
where the lambda is 5 s, the average interruption times of the RDB or the DBSC-SM
methods are the shortest when the broadcasting bandwidth is 15 Mbps. That of the
Carousel method is the shortest when the broadcasting bandwidth is 18 Mbps. It is
difficult to find the appropriate broadcasting bandwidth by formulae since it depends on
some factors such as the method, average arrival interval (lambda), the total bandwidth,
and so on. Figure 3 shows the shortest values given by changing the broadcasting
bandwidth like in Fig. 4.

5.4 Broadcasting Bandwidth

To investigate the influence of the broadcasting bandwidth, we simulate the interrup-
tions changing the broadcasting bandwidth. The results are shown in Figs. 5 and 6. The
broadcasting bandwidth for the case of lambda is 5 s is 6 Mbps and that for 25 s is
9 Mbps. These values are the broadcasting bandwidth that gives the shortest average
interruption time when the total bandwidth is 20 Mbps.
In Fig. 5, the average interruption time sometimes increases even when the
broadcasting bandwidth increases since the simulation results fluctuate. It is natural that
the average interruption time tend to decrease as the broadcasting bandwidth increases
since the total bandwidth increases. Figure 6 shows the number of the interruptions.
The number of the interruptions tend to decrease as the c broadcasting bandwidth
increases since the total bandwidth increases. Although the number of the interruptions
is large when the broadcasting bandwidth is small, the number of the interruptions can
 A Video Data Distribution Method 807

400 400
RDB (Lambda=5 [sec]) RDB (Lambda=5 [sec])
DBSC-SM (Lambda=5 [sec]) DBSC-SM (Lambda=5 [sec])
Average Interruption Time [sec]
350 350
Carousel (Lambda=5 [sec]) Carousel (Lambda=5 [sec])

Number of Interruptions
300 RDB (Lambda=25 [sec]) 300 RDB (Lambda=25 [sec])
DBSC-SM (Lambda=25 [sec]) DBSC-SM (Lambda=25 [sec])
Carousel (Lambda=25 [sec]) Carousel (Lambda=25 [sec])
250 250

200 200

150 150

100 100

50 50

0 0
0 5 10 15 20 25 30 0 5 10 15 20 25 30
Broadcasting Bandwidth [Mbps] Broadcasting Bandwidth [Mbps]

Fig. 5. Average interruption time and the Fig. 6. Number of interruptions and the broad-
broadcasting bandwidth casting bandwidth

400 400
RDB (Lambda=5 [sec]) RDB (Lambda=5 [sec])
DBSC-SM (Lambda=5 [sec]) 350 DBSC-SM (Lambda=5 [sec])
Average Interruption Time [sec]

350
Carousel (Lambda=5 [sec]) Carousel (Lambda=5 [sec])
Number of Interruptions

RDB (Lambda=25 [sec]) 300 RDB (Lambda=25 [sec])

300
DBSC-SM (Lambda=25 [sec]) DBSC-SM (Lambda=25 [sec])
Carousel (Lambda=25 [sec]) Carousel (Lambda=25 [sec])
250 250

200 200

150 150

100 100

50 50

0 0
0 5 10 15 20 25 30 0 5 10 15 20 25 30
Communication Bandwidth [Mbps] Communication Bandwidth [Mbps]

Fig. 7. Average interruption time and the Fig. 8. Number of interruptions and the com-
communication bandwidth munication bandwidth

be reduced by waiting the time to start playing each piece when an interruption occurs
even when it is received.

5.5 Communication Bandwidth

To investigate the influence of the communication bandwidth, we simulate the inter-
ruptions changing the communication bandwidth. The results are shown in Figs. 7 and
8. The broadcasting bandwidth for the case of lambda is 5 s is 14 Mbps and that for
25 [s] is 19 Mbps. These values are the broadcasting bandwidth that gives the shortest
average interruption time when the total bandwidth is 20 Mbps.
As similar to the results of changing the broadcasting bandwidth, as shown in these
figures, the average interruption time and the number of the interruptions tend to
decrease as the communication bandwidth increases since the total bandwidth
increases.
808 S. Matsumoto and T. Yoshihisa

6 Conclusion

Recently, streaming video delivery on hybrid broadcasting environments has attracted

great attention. One of the research topics in video delivery research fields is inter-
ruption time reduction for the environments, in that the clients can receive data both
from the broadcasting channel and the communication channel. In this paper, we
proposed a video data distribution method for flexible bandwidth allocation. In our
proposed method, different from conventional methods, the server allocates the
bandwidths so as to reduce the interruption times further. Our simulation results
revealed that our proposed method achieved the shortest interruption time by appro-
priately allocating bandwidths than the conventional methods.
In the future, we will give a formula to calculate the bandwidth allocation that gives
the shortest average interruption time. Also, we will propose a method for multiple
video delivery and will adopt P2P communication techniques in the communication
channel.

Acknowledgements. A part of this work was supported by JSPS KAKENHI (Grant Number
JP17K00146, and JP18K11316) and by Research Grant of Kayamori Foundation of Informa-
tional Science Advancement.

References
1. Araniti, G., Scopelliti, P., Muntean, G.-M., Lera, A.: A hybrid unicast-multicast network
selection for video deliveries in dense heterogeneous network environments. IEEE Trans.
Broadcast. 11 (2018). Early Access
2. Guo, J., Gong, X., Liang, J., Wang, W., Que, X.: An optimized hybrid unicast/multicast
adaptive video streaming scheme over MBMS-enabled wireless networks. IEEE Trans.
Broadcast. 64, 791–802 (2018)
3. Tian, C., Sun, J., Wu, W., Luo, Y.: Optimal bandwidth allocation for hybrid video-on
demand streaming with a distributed max flow algorithm. ACM J. Comput. Netw. 91(C),
483–494 (2015)
4. Fratini, R., Savi, M., Verticale, G., Tornatore, M.: Using replicated video servers for VoD
traffic offloading in integrated metro/access networks. In: Proceedings IEEE International
Conference on Communications (ICC), pp. 3438–3443 (2014)
5. Mattoussi, F., Zaharia, G., Crusiere, M., Helard, J.-F.: Analytical modeling of losses in FDP
protocol of HbbTV based push-VOD services over DVB networks. In: Proceedings IEEE
Wireless Communications and Networking Conference (WCNC), pp. 1–6 (2017)
6. Gotoh, Y., Yoshihisa, T., Kanazawa, M., Takahashi, Y.: A broadcasting protocol for
selective contents considering available bandwidth. IEEE Trans. Broadcast. 55(2), 460–467
(2009). Special Issue on IPTV
7. Yoshihisa, T.: Data piece elimination technique for interruption time reduction on hybrid
broadcasting environments. In: Proceedings of IEEE Pacific Rim Conference Communica-
tions, Computers and Signal Processing (PACRIM), p. 6 (2017)
8. Yoshihisa, T.: Dynamic data delivery for video interruption time reduction in hybrid
broadcasting environments. In: Proceedings of International Conference on Network-Based
Information Systems (NBiS), pp. 621–626 (2014)
 A Video Data Distribution Method 809

9. Yoshihisa, T., Tsukamoto, M., Nishio, S.: A scheduling scheme for continuous MediaData
broadcasting with a single channel. IEEE Trans. Broadcast. 52(1), 1–10 (2006)
10. Zhang, G., Liu, W., Hei, X., Cheng, W.: Unreeling Xunlei Kankan: understanding hybrid
CDN-P2P video-on-demand streaming. IEEE Trans. Multimedia 17(2), 229–242 (2015)
11. Sheshjavani, A.G., Akbari, B., Ghaeini, H.R.: An adaptive buffer-map exchange mechanism
for pull-based peer-to-peer video-on-demand streaming systems. Int. J. Multimedia Appl. 76
(5), 7535–7561 (2016)
12. Boronat, F., Montagud, M., Marfil, D., Luzon, C.: Hybrid broadcast/broadband TV services
and media synchronization: demands, preferences and expectations of Spanish consumers.
IEEE Trans. Broadcast. 64(1), 52–69 (2018)
13. Boronat, F., Marfil, D., Montagud, M., Pastor, J.: HbbTV-compliant platform for hybrid
media delivery and synchronization on single- and multi-device scenarios. IEEE Trans.
Broadcast. 64(3), 6 (2017)
14. Christodoulou, L., Abdul-Hameed, O., Kondoz, A.-M.: Toward an LTE hybrid unicast
broadcast content delivery framework. IEEE Trans. Broadcast. 63(4), 656–672 (2017)
15. da Silva Cruz, L.A., Cordina, M., Debono, C.-J., Assuncao, P.-A.A.: Quality monitor for 3-
D video over hybrid broadcast networks. IEEE Trans. Broadcast. 62(4), 785–799 (2016)
16. Zhang, Y., Gao, C., Guo, Y., Bian, K., Jin, X., Yang, Z., Song, L., Cheng, J., Tuo, H., Li, X.
M.: Proactive Video push for optimizing bandwidth consumption in hybrid CDN-P2P VoD
systems. In: Proceedings INFOCOM, p. 6 (2018)
 Multilingual Entity Matching

Ilgiz Mustafin1(B) , Marius-Cristian Frunza2,3 , and JooYoung Lee1

1
Innopolis University, Innopolis, Republic of Tatarstan, Russia
{i.mustafin,j.lee}@innopolis.ru
2
Schwarzthal Kapital, 176 av Charles de Gaulle, 92200 Neuilly sur Seine, France
[email protected]
3
LABEX ReFi, 79 av de la Republique, 75011 Paris, France

Abstract. The aim of this paper is to explore methods of multilingual

entity matching. Name matching is currently the main technique used
for entity resolution. When dealing with entities having features recorded
in different languages and with different alphabets the basic approaches
have serious limitation. The basic name matching approach using string
comparison metrics is enriched with phonetic rules and with relational
information. The results show that the approach using transliteration
enhanced by phonetic matching provides with the best performance.

1 Introduction
Entity resolution (also known as record linkage [23], reference reconciliation [21],
object matching) is the task of finding records from one or multiple databases,
referring to the same real-world entity [22]. Entity resolution in a single database
case is sometimes called duplicate detection or deduplication [6].
As the amount of online data increases and duplicate messages are flooded,
entity resolution needs more attention. Practically, information seekers distin-
guish entities by themselves when there is confusion. But it is important to link
entities across platforms to provide users with advanced services.
Name matching techniques are essential for joining data from different
sources. Using exact string matching is not enough because one person can be
referred to in multiple ways: “Bill Clinton”,“William Jefferson Clinton”, or even
in Russian texts [5]. Matching these possible forms of the name
is the task of name matching.
Applying name matching techniques has some difficulties. Usually, name sim-
ilarity functions are designed to measure similarity between two words (two first
names, two last names, etc.) and not between full names, thus names should be
separated into parts and the corresponding parts should be found. Splitting full
names into parts without knowing the context (language, naming customs of the
person, etc.) without building a dictionary of all names is a hard problem.
Using name similarities is insufficient to distinguish two different real-world
people having the same name. Other attributes can be used to solve this problem
and to increase the accuracy of the matching overall.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 810–820, 2020.
https://doi.org/10.1007/978-3-030-15032-7_68
 Multilingual Entity Matching 811

In this paper, we explore the current approaches and examine their validity.
Then we discuss limitations of such approaches and introduce several techniques
which can be used to overcome the limitations in the multilingual name match-
ing. Later we show how an existing model can be improved with the new proposed
techniques to do the multilingual entity resolution and present the performance.

2 Background
Entity resolution has been studied for a long time. Entity resolution methods
can be categorized by the level of using relational information in matching, as
described in [4]:
• Attribute-only Entity Resolution. Record similarity depends only on the sim-
ilarities of the attributes.
• Naive Relational Entity Resolution. Record similarity depends on the simi-
larities of the attributes of the two records (as in the previous case) and the
similarities of the attributes of the records related to the two records being
matched.
• Collective Relational Entity Resolution. Record similarity depends on the
similarities of the attributes of the two records (as in the attribute-only case)
and the similarity of the records related to the two records being matched.
All of the categories from above can use different similarity functions depend-
ing on the domain of the data and the structure of the data. For example, Adafre
and Rijke [1] use sets of incoming links to measure the similarity of Wikipedia
articles, Han et al. [10] match in-text references and use the similarity of texts
near the references as one of the similarity criteria, Kouki et al. [12] use name
similarities of people names to match references in patient reports. Addition-
ally, collective matching criteria can be used [4]: matching of two references can
depend on decision about other references.
While there is already some research on collective entity resolution [4,8],
there is still a possibility to improve the performance of matching by using
better attribute similarity measures. Improving this way can be favorable when
the computational power is limited. In this paper we want to focus on the name
matching aspect of entity resolution.
Name matching is also a part of entity resolution where name strings are
linked if they are instances of the same name [17,18]. Given a name represented
by the string A in one language and a name represented by the string B in
possibly other language, a name matching algorithm should tell if A and B
represent the same person or give a probability of this.
One alternative to exact string matching is to convert name strings to some
common phonetic representation of the names and then to compare the phonetic
representations. Some of the possible phonetic representations are Soundex [20],
Match Rating Approach [16], Daitch-Mokotoff Soundex [15], Beider-Morse Pho-
netic Matching [3], Double Metaphone [19]. We evaluate the applicability of the
latter in this paper.
812 I. Mustafin et al.

• Soundex algorithm [20] is the ancestor of phonetic name matching algorithms.

Soundex maps names to a special code consisting of a letter and three digits.
The letter is the first letter of the name and the digits describes approx-
imately the consonants of the name. The initial aim of Soundex was to be
easily computed manually by human and was designed to be applied to paper
documents. Its performance is relatively poor compared to more recent devel-
opment.
• Match Rating Approach (MRA) [16] employs a very basic but straightforward
process that transforms the string by deleting the vowels if the names does
not start with a vowel and by deleting the second consonants in the pairs of
double consonants, thereby reducing the name to a maximum of six characters
by retaining the first and the last three characters. After the two names are
encoded the MRA gives the matching rating of the two names based on the
similarity of the two strings.
• Daitch-Mokotoff Soundex [15] is an alternation of the original Soundex for
Yiddish and Slavic languages. In this algorithm names are given six digit
codes. The first letter is coded too (contrast to original Soundex where the
first letter was retained as it was). Names can have multiple codes which
is different from the original Soundex where names are mapped to only one
code.
• Beider-Morse Phonetic Matching (BMPM) [3] was designed to decrease the
number of false hits produced by Soundex-like algorithms. BMPM incorpo-
rates more than 300 common rules and has a number of language-specific
rules to support 10 languages. The first step of BMPM is to identify the lan-
guage and only then the approximate phonetic value is calculated based on
the language detected.

After the names were converted to their phonetic representations or if no

conversion was done and the names remained as they were given the two strings
should be compared with a string distance metric like Levenshtein distance,
Guth algorithm, Jaro [7], Jaro-Winkler [24], etc.

• Levenshtein distance [13] between two strings is a metric representing the

number of one character changes (substitute a character by another charac-
ter, remove a character, insert a character) needed to change one word into
another.
• Guth algorithm was specifically designed to compare names. It takes two
names as an input and as an output provides with the probability of the two
names are variants of spelling of one name. Guth’s algorithm compares two
strings character by character, sometimes skipping or backtracking one or two
characters.

3 Problem Setting
Kouki et al. [12] presented an entity resolution system which constructs familial
networks based on healthcare records of patients from one family. Such health-
 Multilingual Entity Matching 813

care records can contain one patient having a 15-year daughter named Lisa suf-
fering from a high blood pressure and contain another patient having a 16-year
old daughter named Liza with a type 1 diabetes. Identifying if both 15 years
old Liza and 16 years old Lisa are actually the same person, assuming the data
(names, ages, etc.) can contain errors, is the task of entity resolution.
In this paper we add an additional source of uncertainty. The model will be
required to do entity resolution between records with names written in Russian
and names written in English.

4 Original Model Description

The solution proposed by Kouki et al. [12] is based on Probabilistic Soft
Logic (PSL) [2] framework. PSL uses soft truth values ∈ [0, 1] and relaxation
rules to encode logical models. The relaxation rules are [11]:
p ∧ q = max(0, p + q − 1)
p ∨ q = min(1, p + q)
¬p = 1 − p
As a result of training, the model will be able to give probability of two
mentions a and b referring to the same real-world entity: Same(a, b).
The proposed model consists of a number of PSL rules. The used rules can be
separated into several categories: attribute similarity rules, relational-attribute
rules, prior rules.
Attribute similarity rules state that if some attribute is similar in two refer-
ences, then the references should be matched, and if some attribute is not similar
in two references, then the references should not be matched. Some examples:
SimNameJW (a, b) =⇒ Same(a, b)
¬ SimNameJW (a, b) =⇒ ¬ Same(a, b)
SimAge(a, b) =⇒ Same(a, b)
¬ SimAge(a, b) =⇒ ¬ Same(a, b)

Where SimNameJW is the Jaro-Winkler name similarity of the two records,

SimAge is defined as maximum of the ages divided by the minimum of the two
ages.
Kouki et al. note that some attributes like gender should not be an evidence
for matching, but can be an evidence against matching:
SameGender(a, b) =⇒ Same(a, b)
¬ SameGender(a, b) =⇒ ¬ Same(a, b)
Relational-attribute rules can be used together with the attribute similarity
rules. Relational-attribute rules state that if two references mention similar peo-
ple (e.g. both have mothers with similar names), then the two references should
be matched. Some of the employed relational-attributes rules are:
814 I. Mustafin et al.

HasMother(a) ∧ HasMother(b) ∧ SimMother(a, b) =⇒ Same(a, b)

HasSisters(a) ∧ HasSisters(b) ∧ SimSisters(a, b) =⇒ Same(a, b)

Where SimMother is defined as the maximum of Levenshtein and Jaro-Winkler

similarities of the first names. SimSisters and other multi-valued relationships
are defined differently: for a relationship type t (is a sister of for SimSisters) and
two mentions a, b we find two sets A = {x : t(a, x)} and B = {y : t(b, y)} which
are the sets of mentions related to a and b respectively with the relationship t
(for SimSisters A and B will be the sets of all sisters of a and b respectively).
The similarity between sets A and B will be the relational-attribute similarity
of a and b and is defined as:
1 
Sim(A, B) = max SimName(x, y)
|A| y∈B
x∈A

Assuming relationship is symmetric it is assumed, without loss of generality, that

|A| ≤ |B|.
The prior rule introduce the belief that most of the references do not match:

¬ Same(a, b)

5 Computing Name Similarities Across Languages

In this section, we specifically discuss multilingual entity resolution problem with
Russian and English names. It should be noted that we specifically chose Russian
and English languages because of several reasons. The main reason is that the
Russian language is highly phonetic (spelling represents pronunciation) and the
English language is not so phonetic (spelling represents pronunciation poorly)
and that the names from other languages (especially with Latin alphabet) are
used in English in their original form (Sigmund Freud) rather than being trans-
formed to better represent the original pronunciation (possibly Zikmont Froyd).
As a consequence of this, we are actually having not only English-Russian name
pairs, but also many other name pairs.

5.1 Transliteration
Russian and English languages use different alphabets (Cyrillic and Latin respec-
tively) which makes string similarity functions (like Jaro-Winkler, Levenshtein)
unsuitable for comparing names from these languages. To overcome this issue,
we convert Russian names to Latin script.
In general, for a name there can be several possible valid equivalents in other
language. Such conversion can be done via transliteration, transcription or trans-
lation. Transliteration is a more systematic and reversible procedure (
to Elcin), transcription is a more phonetic conversion focused on preserving the
pronunciation of the name ( to Yeltsin). translation is done via mapping
a name from the first language to a traditional equivalent name of the second
 Multilingual Entity Matching 815

Fig. 1. Cyrillic-Latin and Latin-Cyrillic name variations

language ( to Nathaly), if there is one. All of these conversions have

different rules and customs for different language pairs [14], see Fig. 1.
Different domains can contain names produced by different conversion meth-
ods: Russian international passports have names converted from Russian to Latin
script using a strict set of transliteration rules which are sometimes different from
the transcription rules used in less formal contexts.
Even though transcription can be used more widely, we will use the current
Russian transliteration rules for the international passports because they are
well defined.

5.2 Aligning Names

Names in different languages and cultures can have different structures: Rus-
sian names (e.g. Vladimir Vladimirovich Putin) have a given name (Vladimir),
patronymic (Vladimirovich) and a family name (Putin), Spanish names (e.g.
Pedro Sánchez Pérez-Castejón) have a given name (Pedro) and two family
names(Sánchez and Pérez-Castejón), English names (e.g. Theresa Mary May)
consist of the first name (Theresa), the last name (May) and optional middle
names (Mary). Name variation can be entailed also by deviations of the same
name in different languages.
Moreover, a name can appear in different forms: name parts can be omitted
(Vladimir Vladimirovich Putin, Vladimir Putin), name parts can be reduced
to initials (V. Putin), grammatical transformations like inflection (Vladimira
Putina) and so on.
This variety of forms makes extracting name parts and computing name sim-
ilarities a hard task. To partially overcome this issue, we introduce the alignment
procedure as a part of computing name similarity.
816 I. Mustafin et al.

Full-name similarity of two names with alignment is defined as the maximum

similarity of all part permutations (V. Putin, Putin V.) with the second name.
This way, similarity between Vladimir Putin and Putina Lyudmila will be high,
but not maximum. It is not needed to compute permutations of the second name,
assuming the similarity function is symmetric.

5.3 Phonetic Transformations

Converting names between languages can make names less recognizable, espe-
cially so when translations or historical transliteration customs are used. For
example, historically, letter H (sound /h/) is transcribed from German to Rus-
sian as Γ (sound /g/). Consequently, converting Hitler to Russian and then to
English might result in Gitler. This is not the only example of such inconsisten-
cies. Another example is the Persian name Rostam which is known as Rüstem
in Turkish, Röstäm in Tatar, and in Kazakh languages. Converting to
English by dropping diacritics [14] will result in several different names: Rostam,
Rustem, Rustam.
Sometimes names have equivalents for names in other languages. For exam-
ple, Russian name (Mikhail) has a Ukrainian counterpart
(Mykhailo). Both of these names would be printed in the passport in Soviet
Union and the official transliteration would be Mikhail, but nowadays the name
is spelled as Mykhailo [25].
In this paper we use Double Metaphone [19] to encode names. Double Meta-
phone was designed to account for the differences in writings of names in different
languages. For example, Double Metaphone codes of both Michael and Maykl
(from ) will be equal to MKL.

6 Model Discussion and Improvements Motivation

We can see that the SimName predicate is used in many rules of the original
model, thus the performance of matching depends heavily on the name similarity
metrics.
However, the original definition of the SimName metric has some limitations
for applying it as-is to other datasets.
The first problem is that SimName is defined as a similarity of the first names
and SimNameJW computes similarities between first names, middle names, last
names. Both of these functions are left undefined for the cases where names are
represented as one string and it is unknown how to separate names into parts
(first name, last name, etc.).
The second problem is that SimName and SimNameJW use Jaro-Winkler and
Levenshtein distances which will give maximum distances for strings written in
two different alphabets (e.g. Latin and Cyrillic). This also limits the applicability
of the approach in multi-lingual context.
 Multilingual Entity Matching 817

7 Experiments
The original paper [12] describes the performance of the approach on two
datasets, a dataset from the National Institutes of Health (NIH) [9] and a dataset
crawled from Wikidata1 . We generate a similar dataset from Wikidata with peo-
ple from different countries to focus on multi-lingual matching.
Matching performances are measured with the feature sets used in the orig-
inal paper for Wikidata: Names, Names + Personal Info, Names + Personal
Info + Relational Info (1st degree). As in the original paper, Gaussian noise
generated from a Gaussian distribution (N (0, 0.16)) is added to features. Full
names (e.g. first name and last name in one string) are passed to name similarity
functions.
Names feature set: only name similarities with SimNameJW are used for
matching.
Names + Personal Info feature set: same as Names but with age similarity
and gender features.
Names + Personal Info + Relational Info (1st degree) feature set: same
as Names + Personal Info but with mother, father, daughters, sons, sisters,
brothers, spouses similarity features.
Each feature set was evaluated using two name sets: Original and Translit.
In Original matching, name similarity functions were computing similarities
between English versions of the names and N (0, 0.16) noise was added as in
the original paper. In Translit matching, name similarities were computed as
similarities between English and Russian versions of the names as described in
Sect. 5.1. Translit matching was refined by the alignment procedure described
in Sect. 5.2 and the results are reported as Translit Align. Translit Align was
further enhanced by phonetic matching techniques described in Sect. 5.3 and the
results are reported as Translit Align Phonetic.
Threshold selection technique from the original paper was also implemented:
the dataset is split not only to train and test parts, but a third subset is intro-
duced to find the threshold of the resulting truth value which maximizes the F1
score of the matching.
The crawled dataset consists of 470 people from 64 families. Each person
is tested for matching against each person. The evaluation (test) dataset split
contains 143 people, so there are 143 matching pairs and 67067 non-matching
pairs, probability of a match is .002. The performance of the model on different
data with different name matching techniques is shown in Table 1.
Precision is the ratio of the true matches found to all found matches (i.e.
what part of the found matches are true matches). Recall is the ratio of the true
matches found to all true matches (i.e. what part of all matches was found).
F1 score is the harmonic mean of precision and recall. 0 is the worst possible
precision, recall, F1 score and 1 is the best possible.

1
https://www.wikidata.org/.
818 I. Mustafin et al.

Table 1. PSL model matching performance

Feature Set Precision Recall F1 Score

Original Names .22 .52 .31

Translit Names .67 .58 .62
Translit Names Phonetic .93 .59 .73
Translit Names Align .54 .72 .61
Translit Names Align Phonetic .93 .59 .73

Original Names + PI .55 .49 .52

Translit Names + PI .79 .69 .74
Translit Names Align + PI .63 .81 .71
Translit Names Align Phonetic + PI .85 .71 .77

Original Names + PI + R1 .61 .49 .54

Translit Names + PI + R1 .77 .71 .74
Translit Names Align + PI + R1 .69 .78 .73
Translit Names Align Phonetic + PI + R1 .72 .76 .73

7.1 Discussion
The results show that the approach using transliteration enhanced by phonetic
matching techniques provides with best results. Several insights can be derived
from these results.
Transliteration from Russian Makes Less “Noise” than N (0, 0.16). Original
Names and Translit Names experiments show that the model doing pure name
matching of names with N (0, 0.16) noise was less efficient both precision- and
recall-wise than the same model working with matching English names and
transliterated Russian names.
Aligning Improves Recall. Aligning procedure described in Sect. 5.2 improves
recall of matching by reordering words in names to increase the similarity. How-
ever, aligning can increase the similarity of actually non-matching names thus
decreasing the precision of matching.
Phonetic Transformations Improve Precision. Using Double Metaphone codes
for computing similarities (described in Sect. 5.3) improved the precision of
matching by putting several different transliterations of one name into same
bucket e.g. Michael and Maikl (a transliteration of both have the same
Double Metaphone key MKL which results in the maximum name similarity.

8 Conclusion
This paper explores methods of multilingual entity matching. The basic name
matching approach using string comparison metrics is enriched with phonetics
rules and with relational information. The results show how different techniques
affect precision and recall. Nevertheless, the presented approaches do not address
 Multilingual Entity Matching 819

the issues of name variability in the transliteration/transcription/translation

process between languages with different alphabets. This aspect will be explored
in a further research.

References
1. Adafre, S.F., de Rijke, M.: Discovering missing links in Wikipedia. In: Proceedings
of the 3rd international workshop on Link discovery, pp. 90–97. ACM (2005)
2. Bach, S.H., Broecheler, M., Huang, B., Getoor, L.: Hinge-loss markov random fields
and probabilistic soft logic. arXiv preprint arXiv:1505.04406 (2015)
3. Beider, A.: Beider-morse phonetic matching: an alternative to Soundex with fewer
false hits. Avotaynu: Int. Rev. Jewish Geneal. (2008)
4. Bhattacharya, I., Getoor, L.: Collective entity resolution in relational data. ACM
Trans. Knowl. Discov. Data (TKDD) 1(1), 5 (2007)
5. Bilenko, M., Mooney, R., Cohen, W., Ravikumar, P., Fienberg, S.: Adaptive name
matching in information integration. IEEE Intell. Syst. 18(5), 16–23 (2003)
6. Christen, P.: Data Matching: Concepts and Techniques for Record Linkage, Entity
Resolution, and Duplicate Detection. Springer, Heidelberg (2012)
7. Cohen, W., Ravikumar, P., Fienberg, S.: A comparison of string metrics for match-
ing names and records. In: KDD Workshop on Data Cleaning and Object Consol-
idation, vol. 3, pp. 73–78 (2003)
8. Dong, X., Halevy, A., Madhavan, J.: Reference reconciliation in complex informa-
tion spaces. In: Proceedings of the 2005 ACM SIGMOD international conference
on Management of data, pp. 85–96. ACM (2005)
9. Goergen, A., Ashida, S., Skapinsky, K., De Heer, H., Wilkinson, A., Koehly,
L.: Knowledge is power: improving family health history knowledge of diabetes
and heart disease among multigenerational mexican origin families. Public Health
Genomics 19(2), 93–101 (2016)
10. Han, X., Sun, L., Zhao, J.: Collective entity linking in web text: a graph-based
method. In: Proceedings of the 34th International ACM SIGIR Conference on
Research and Development in Information Retrieval, pp. 765–774. ACM (2011)
11. Kimmig, A., Bach, S., Broecheler, M., Huang, B., Getoor, L.: A short introduction
to probabilistic soft logic. In: Proceedings of the NIPS Workshop on Probabilistic
Programming: Foundations and Applications, pp. 1–4 (2012)
12. Kouki, P., Pujara, J., Marcum, C., Koehly, L., Getoor, L.: Collective entity resolu-
tion in familial networks. In: 2017 IEEE International Conference on Data Mining
(ICDM), pp. 227–236. IEEE (2017)
13. Levenshtein, V.: Binary codes capable of correcting spurious insertions and deletion
of ones. Probl. Inf. Transm. 1(1), 8–17 (1965)
14. Li, C.W.C.: Foreign names into native tongues: how to transfer sound between
languages-transliteration, phonological translation, nativization, and implications
for translation theory. Target. Int. J. Transl. Stud. 19(1), 45–68 (2007)
15. Mokotoff, G.: Soundexing and genealogy (2007). http://www.avotaynu.com/
soundex.html
16. Moore, G.B.: Accessing Individual Records from Personal Data Files Using Non-
unique Identifiers, vol. 13. US Department of Commerce, National Bureau of Stan-
dards (1977)
17. Patman, F., Thompson, P.: Names: a new frontier in text mining. In: International
Conference on Intelligence and Security Informatics, pp. 27–38. Springer (2003)
820 I. Mustafin et al.

18. Peng, T., Li, L., Kennedy, J.: A comparison of techniques for name matching.
GSTF J. Comput. (JoC) 2(1), 55–61 (2012)
19. Philips, L.: The double metaphone search algorithm. C/C++ Users J. 18(6), 38–43
(2000)
20. Russell, R.: Index. US Patent 1,261,167 (1918). https://www.google.com/patents/
US1261167
21. Saıs, F., Pernelle, N., Rousset, M.C.: L2R: a logical method for reference reconcil-
iation. In: Proceedings of the AAAI, pp. 329–334 (2007)
22. Singla, P., Domingos, P.: Entity resolution with markov logic. In: Sixth Interna-
tional Conference on Data Mining, ICDM 2006, pp. 572–582. IEEE (2006)
23. Winkler, W.E.: The state of record linkage and current research problems. In:
Statistical Research Division, US Census Bureau. Citeseer (1999)
24. Winkler, W.E.: Overview of record linkage and current research directions. In:
Bureau of the Census. Citeseer (2006)
25. Yermolovich, D.: Imena sobstvennyye na styke yazykov i kultur [proper names
across languages and cultures]. R. Valent, Moscow (2001)
 Evaluating DASH Player QoE with
MPTCP in Presence of Packet Loss

Sunit Kumar Nandi1,3(B) , Pranav Kumar Singh2,3 , and Sukumar Nandi3

1
Department of CSE, National Institute of Technology,
Papum Pare 791112, Arunachal Pradesh, India
[email protected]
2
Department of CSE, Central Institute of Technology Kokrajhar,
Kokrajhar 783370, Assam, India
3
Department of CSE, Indian Institute of Technology Guwahati,
Guwahati 781039, India

Abstract. MPTCP aggregates multiple network interfaces for a single

TCP session for enhanced transfer rates and improved reliability to appli-
cations. Due to this property, MPTCP has the potential to provide better
end-user viewing experience (QoE) for DASH video when used instead
of TCP on a multi-homed device. This paper evaluates the performance
of DASH on MPTCP and its usability in topologies with independent
links between two end-hosts having different packet loss rates and varying
DASH video segment sizes. We found that MPTCP provides better play-
back bitrate than TCP in most cases, at the cost of lower utilization of
available bandwidth. MPTCP is also resistant to playback interruptions.
Moreoever, the optimal video segment size for QoE is a tradeoff between
lesser number of down switches and higher average playback bitrate. So,
with the proper choice of video segment size, MPTCP is preferable over
TCP for DASH video QoE on a multi-homed wireless mobile device.

Keywords: MPTCP · DASH · Performance evaluation ·

Quality of experience · Transport protocol · Packet loss

1 Introduction

Improving video streaming Quality of Experience (QoE) has been an open chal-
lenge for two decades now due to increasing popularity and consumption of video
content by mobile device users. Old video streaming technologies have been
superseded by Dynamic Adaptive Streaming over HTTP (DASH) on mobile
devices as it can adjust the playback bitrate to match that of the available
throughput, to avoid interruptions and rebuffering events that can severely
impact viewer QoE on wireless networks. Also, mobile devices today have mul-
tiple interfaces, with a minimum of one LTE and one Wi-Fi interface. Using
Multipath TCP (MPTCP) as a transport layer for DASH has been proposed
as a solution to improve video streaming on mobile devices, as it uses all the
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 821–833, 2020.
https://doi.org/10.1007/978-3-030-15032-7_69
822 S. K. Nandi et al.

available interfaces for the TCP session to provide aggregated throughput and
improved reliability for the video stream. A lot of investigation has gone into the
usability of MPTCP for DASH video streaming, but they are mostly focused on
meeting Quality of Service (QoS) guarantees. However, QoS alone is not suffi-
cient for MPTCP to be considered a candidate for transport layer for DASH.
The end-user’s viewing experience determines how well a change to the network
stack is likely to be accepted. Hence, QoE is a more important factor to jus-
tify the usage of MPTCP. QoE issues in this direction have been explored only
partially in the current literature. Moreover, performance evaluations done so
far have not yet taken into account the degradation of packet loss on end-user
experience.
There are two types of QoE parameters, subjective and objective. Subjective
parameters are determined according to viewer’s perspective. They include Peak
Signal to Noise Ratio (PSNR), Structural Similarity (SSIM), Video Quality Met-
ric (VQM) and Mean Opinion Score (MOS). To calculate these parameters, we
need both the original video frames and the frames received by the end-user from
the network. Also, there is no direct relationship between subjective parameters
and network metrics. So, we cannot study the improvement or degradation of
QoE according to subjective parameters. Objective QoE parameters like start-
up delay, response time, delivery synchronization, freshness, playback bitrate,
player buffer level and number of streaming interruptions provide significant
detail to correlate QoE with network conditions.
We present a detailed performance evaluation of DASH using MPTCP as
transport protocol in the presence of different levels of packet loss. With this we
make the following contributions:

– We have made a emulated framework for testing DASH on MPTCP having

3 independent links with different packet loss rates which can be generalized
to any multi-homed wireless mobile device like a smartphone or tablet.
– We have presented a comparative study between DASH on MPTCP and
DASH on TCP in situations with different packet loss rates to determine the
desirability on MPTCP in terms of performance gains over TCP.
– We have run DASH video tests with different segment sizes and determined
the best performing segment size in terms of end-user QoE.

This study shows that MPTCP can offer significant QoE improvement over
TCP when used as transport layer for DASH on a mobile device with wireless
interfaces.

2 Background and Related Work

Multi-path video streaming has shortcomings due to quality of selected paths,

scheduling, out-of-order packets and huge variation in end-to-end delay. It is still
in its infancy and research is ongoing in this direction in two major perspectives.
 Evaluating DASH Player QoE with MPTCP in Presence of Packet Loss 823

2.1 Network Perspective

In May 2016, Corbillon et al. [4] proposed a cross-layer scheduler on top of
MPTCP leveraging information from both the application layer and transport
layer to reorder the transmission of data and prioritize the most important por-
tions of the video. Their proposed scheduler retains advantages of MPTCP as
well as offers better QoE compared to the traditional scheduler.
In September 2016, Wu et al. [9] proposed the quAlity-Driven MultIpath TCP
(ADMIT) scheme that uses utility maximization based Forward Error Correc-
tion (FEC) coding and rate allocation to minimize end-to-end video distortion.
ADMIT outperforms the reference transport protocols in terms of video PSNR
(Peak Signal-to-Noise Ratio), end-to-end delay, and goodput.
In December 2016, Han et al. [5] proposed a new multipath framework called
MP-DASH that can accept and use network interface preferences from the user.
Using this preference information, MP-DASH strategically schedules video seg-
ment delivery across the interfaces and thus satisfy user preferences, providing
reduction in cellular usage by up to 99% and radio energy expenditure by up to
85% compared to regular MPTCP with negligible effect on QoE.

2.2 Player Perspective

In 2014, Chen et al. [3] proposed a new client-based video streaming solution
named MSPlayer utilizing multiple TCP connections instead of MPTCP to con-
nect to multiple video sources as well as over multiple interfaces. Thus, MSPlayer
makes it possible to leverage both path diversity and source diversity simultane-
ously reducing start-up delay as well as providing robust and high quality video
streaming on mobile devices.
In 2016, James et al. [6] evaluated DASH performance on MPTCP on mobile
devices. They created a simple testbed with two paths between the video server
and client. One path emulates a WiFi link and the other an LTE link. They
compared the performance of TCP and MPTCP on performance metrics such
as segment quality, network throughput and number of bitrate switches. They
found that MPTCP is beneficial under ample and stable bandwidth, but does
not offer any advantage in certain network conditions. This study, however, does
not take into account effect of different video segment sizes. Moreover, the results
do not take into account changes in the packet loss rate on the wireless network
interfaces.
Our work, on the other hand, considers both network and player (end-user)
perspectives. We have exhaustively tested DASH on both TCP and MPTCP
and evaluated their performance with respect to QoE metrics in topologies with
links having both low and high packet loss as well as with different video segment
sizes.

3 Performance Evaluation Setup

We have used the following experimental framework to evaluate the performance
of DASH on MPTCP in presence of links having different packet loss rates.
824 S. K. Nandi et al.

3.1 Tools and Platform Used

The experiments are performed on a computer running Ubuntu (Linux) 16.04

LTS running version 0.92 of the MPTCP Linux kernel implementation [2]. We
use Mininet as our Software Defined Networking emulator, in which switches
are instances of Open vSwitch. Mininet handles all instantiation of end hosts,
switches and controllers. We use Python 2.7 to script the configuration and pro-
grammatically generate the test cases and experimental conditions in Mininet. To
create the environment in Mininet, we use Nginx as our web server to serve DASH
description files (MPDs) and video and audio segments. GPAC MP4Client [1]
is used as a DASH client. The log files generated by MP4Client are then parsed
using Python to retrieve player statistics, which are then plotted into graphs
using GNUPlot.
We use the Ultra High Definition HEVC DASH data set by Feuvre et al.
[7] with content bitrates ranging from 2 Mbps (720 p at 30 fps) all the way
to 20 Mbps (2160 p at 60 fps). We have chosen the ‘Live profile with bitstream
switching’ as it is the most appropriate profile to study rate adaptation according
to network conditions in DASH (client) players. For MPTCP configuration, we
left it to default: the shortest RTT first scheduler, BALIA congestion control
and ‘fullmesh’ path manager.

3.2 Topology

We take a topology with link bandwidths and latencies as shown in Fig. 1.

Path 1 along pl1-s1-s2-sr1 is on subnet 10.0.0.0/24, path 2 pl1-s3-s4-s5-sr1 is
on 10.0.1.0/24 and path 3 pl1-s6-s7-s8-sr1 is on 10.0.2.0/24. The IP addresses of
‘sr1’ are 10.0.0.1, 10.0.1.1 and 10.0.2.1, while that of ‘pl1’ are 10.0.0.2, 10.0.1.2
and 10.0.2.2. The web server hosting the DASH description file and video seg-
ments is run on host ‘sr1’ while the player resides on host ‘pl1’.

Fig. 1. Topology used in DASH streaming for 3 independent paths.

 Evaluating DASH Player QoE with MPTCP in Presence of Packet Loss 825

3.3 Experiments

We use the player on host ‘pl1’ to stream a two minute video from the web
server on ‘sr1’ and play it simultaneously. The player initiates the connection
over pl1-s1-s2-sr1 and starts playing the video, while logging statistics to a file.
In TCP, only path 1 (pl1-s1-s2-sr1) is used while in MPTCP that path is the
primary path. The experiment is performed for both TCP and MPTCP with
varying video segment sizes of 2 s, 4 s, 6 s, 10 s and 20 s. 3 sets of tests are run:
first having no packet loss, second having path 1 with 2% and path 2 with 1%
packet loss respectively (i.e. minimal packet loss) and third having path 1 with
10%, path 2 with 5% and path 3 with 1% loss respectively (i.e. high packet loss).

4 Results and Analysis

The results obtained from the experiments are presented in this section in terms
of graphs and tables comparing TCP and MPTCP in varying video segment sizes
and packet loss rates. The QoE metrics used for performance evaluation are (1)
network throughput, (2) playback bitrate, (3) player buffer level, (4) start-up
delay and (5) number of bitrate switches and playback interruptions.

4.1 Result with No Packet Loss

The comparison of network throughput, playback bitrate and buffer level among
TCP and MPTCP at 4 s segment size are given in Figs. 2, 3 and 4 respectively.
Results for other segment sizes are available in [8].

(a) Segment size 4s, TCP (b) Segment size 4s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 2. Comparison of network throughput of TCP and MPTCP for DASH streaming
for 3 independent paths with no packet loss.

Additional statistics like start-up delay, average throughput, average bitrate,

number of bitrate switches and number of interruptions for audio and video are
given in Table 1.
826 S. K. Nandi et al.

(a) Segment size 4s, TCP (b) Segment size 4s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 3. Comparison of playback bitrate of TCP and MPTCP for DASH streaming for
3 independent paths with no packet loss.

(a) Segment size 4s, TCP (b) Segment size 4s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 4. Comparison of buffer level of TCP and MPTCP for DASH streaming for 3
independent paths with no packet loss.

From the network throughput data collected, we found that the DASH client
using TCP is able to use approximately 80% of the first link’s bandwidth on aver-
age (i.e. 8 Mbps out of 10 Mbps), irrespective of segment size. On MPTCP, how-
ever, it has varying throughput according to segment size. The average through-
put for 2 s segment size is close to 12.6 Mbps, for 4 s 15.5 Mbps, for 6 s 16.9
Mbps, for 10 s 17.4 Mbps and for 20 s 17.2 Mbps. MPTCP overall allows higher
network throughput but has lower efficiency as it only uses 71% of the aggregate
link bandwidths. This inefficiency could be attributed to additional overhead
and scheduling and handling out-of-order packets in the MPTCP layer. At 10 s
segment size, peak network throughput is achieved.
Coming to the playback bitrate, we observe that the DASH client using TCP
is able to play at a higher bitrate, the smaller the segment size is. The DASH
player only fetches bitrate of content lesser or equal to the network throughput
(and consequently bandwidth). The player has the opportunity to switch to a
higher bitrate just after the current segment has finished playing. At smaller
segment sizes, it has more opportunities to switch to a higher quality (like for
2 s segment size, every 2 s). At larger segment sizes, the player is locked to the
given bitrate for a significant period of time. As a result, it does not switch to
 Evaluating DASH Player QoE with MPTCP in Presence of Packet Loss 827

Table 1. Statistics of TCP and MPTCP for DASH streaming on three independent
paths with reliable links.

Tranport TCP MPTCP

protocol
Segment size 2 4 6 10 20 2 4 6 10 20
(secs)
Startup 50 91 56 42 155 41 156 9 89 51
delay (ms)
Av. vid. nw. 7735 7810 7827 7814 8016 12649 15577 16984 17412 17176
thrpt. (kb/s)
Av. aud. nw. 6521 7170 7572 7667 7936 7969 12338 14042 14670 17281
thrpt. (kb/s)
Av. vid. cnt. 6765 6425 5993 4752 2702 6680 10552 9291 6441 2702
b.r. (kb/s)
Av. aud. cnt. 130 131 131 135 128 130 131 131 135 128
b.r. (kb/s)
# video b.r. 18 18 11 12 7 45 20 16 13 7
switches
# audio b.r. 1 1 1 1 1 1 1 1 1 1
switches
# video 0 0 0 0 0 0 0 0 0 0
interruptions
# audio 0 0 0 0 0 0 0 0 0 0
interruptions
Abbreviations used: Av. = Average, vid. = video, nw. = network, cnt. = content, b.r. =
bitrate, thrpt. = throughput

a better quality as quickly. On MPTCP, however, a balance has to be made for

segment size. For the smallest size 2 s, the DASH player makes more number
of network requests, which creates more overhead for the MPTCP layer. As a
result the playback bitrate first hits a peak and then wildly switches between
higher and lower bitrates. This can quickly become an annoyance for the person
viewing the video. For largest size 20 s, the playback bitrate stays locked for
a significant amount of time and the performance is no different from regular
TCP. For the remaining segment sizes 4 s, 6 s and 10 s, we see that larger segment
size means less number of bitrate switches and lesser average playback bitrate
while smaller segment sizes mean more number of switches and higher average
playback bitrate. At 4 s segment size, the user gets the highest video quality but
still sees huge variations in quality visible to naked eye. Whereas at 6 s, the user
gets second highest possible video quality but notices less quality variations.
The observations so far correspond with the readings collected in the addi-
tional statistics table. The startup delay on average is slightly more for MPTCP
than TCP. We have no playback interruptions in both TCP and MPTCP as the
links are reliable.
The buffer levels show a similar trend for TCP and MPTCP. Depending
on the segment size, the player restricts the buffer between a maximum and
minimum level. The rise and fall of buffer levels is similar in TCP and MPTCP,
however, the average buffer level is slightly higher in MPTCP.
828 S. K. Nandi et al.

4.2 Result with Minimal Packet Loss

The comparison of network throughput, playback bitrate and buffer level among
TCP and MPTCP for 4 s segment size are given in Figs. 5, 6 and 7 respectively.
Results for other segment sizes are available in [8].

(a) Segment size 4s, TCP (b) Segment size 4s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 5. Comparison of network throughput of TCP and MPTCP for DASH streaming
for 3 independent paths with 2 having minimal packet loss.

(a) Segment size 4s, TCP (b) Segment size 4s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 6. Comparison of playback bitrate of TCP and MPTCP for DASH streaming for
3 independent paths with 2 having minimal packet loss.

Additional statistics like start-up delay, average throughput, average bitrate,

number of bitrate switches and number of interruptions for audio and video are
given in Table 2.
Looking at the network throughput, we observe that the DASH player using
TCP shows this trend: for minimally lossy links, the larger the segment size, the
more is the utilization of the link bandwidth. At 2 s segment size, the bandwidth
utilization is 70.8% which goes up to 75.3% at 20 s segment size. MPTCP also
shows a similar trend, with 41.2% at 2 s segment size till 60.9% at 20 s segment
 Evaluating DASH Player QoE with MPTCP in Presence of Packet Loss 829

(a) Segment size 4s, TCP (b) Segment size 4s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 7. Comparison of buffer level of TCP and MPTCP for DASH streaming for 3
independent paths with 2 having minimal packet loss.

Table 2. Statistics of TCP and MPTCP for DASH streaming on three independent
paths with two minimally lossy links.

Tranport TCP MPTCP

protocol
Segment size 2 4 6 10 20 2 4 6 10 20
(secs)
Startup 202 156 1118 146 48 193 40 154 151 187
delay (ms)
Av. vid. nw. 7081 7080 7160 7090 7529 9880 11953 14279 14396 14625
thrpt. (kb/s)
Av. aud. nw. 6081 6712 7032 7365 7460 8928 9488 12476 12648 13512
thrpt. (kb/s)
Av. vid. cnt. 6325 5046 4957 4565 2704 6878 6263 5610 5851 2705
b.r. (kb/s)
Av. aud. cnt. 130 131 131 135 128 130 131 131 135 128
b.r. (kb/s)
# video b.r. 17 18 13 9 7 21 23 22 14 7
switches
# audio b.r. 1 1 1 1 1 1 1 1 1 1
switches
# video 0 0 0 0 0 0 0 0 0 0
interruptions
# audio 0 0 0 0 0 0 0 0 0 0
interruptions
Abbreviations used: Av. = Average, vid. = video, nw. = network, cnt. = content, b.r. =
bitrate, thrpt. = throughput

size. Again, we see that MPTCP offers higher throughput, however, its overhead
brings down the efficiency by a significant margin.
Coming to playback bitrate, we see that for both TCP and MPTCP, the
average playback bitrates are similar, with higher segment sizes giving lower
average bitrates. However, MPTCP allows for much higher peak bitrates. The
average playback bitrate is the highest at 2 s video segment size. The number
of bitrate switches are a little higher in MPTCP than TCP. This means that
830 S. K. Nandi et al.

the player in MPTCP experiences sharp drops in playback bitrate, and therefore
drops in content quality after hitting the peak bitrate. From a QoE perspective,
MPTCP in default configuration is worse than TCP in this experiment as it
brings abrupt changes in content quality visible to naked eye.
The buffer level graphs show a similar trend for TCP and MPTCP.
The additional statistics table readings correspond with the observations
from the graphs. The startup delay is noticeably lower for MPTCP, than TCP.
This makes it more appealing to use MPTCP over TCP on wireless networks.

4.3 Result with High Packet Loss

The comparison of network throughput, playback bitrate and buffer level among
TCP and MPTCP with 6 s segment size are given in Figs. 8, 9 and 10 respectively.
Results for other segment sizes are available in [8].

(a) Segment size 6s, TCP (b) Segment size 6s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 8. Comparison of network throughput of TCP and MPTCP for DASH streaming
for 3 independent paths with high packet loss.

(a) Segment size 6s, TCP (b) Segment size 6s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 9. Comparison of playback bitrate of TCP and MPTCP for DASH streaming for
3 independent paths with high packet loss.
 Evaluating DASH Player QoE with MPTCP in Presence of Packet Loss 831

(a) Segment size 6s, TCP (b) Segment size 6s, MPTCP
Note: AS#1 is video and AS#2 is audio.

Fig. 10. Comparison of buffer level of TCP and MPTCP for DASH streaming for 3
independent paths with high packet loss.

Additional statistics like start-up delay, average throughput, average bitrate,

number of bitrate switches and number of interruptions for audio and video are
given in Table 3.
The observations change significantly when DASH is used on networks with
high packet loss.
When DASH client uses TCP, the average throughput is between 0.9% to
2.3% of the link bandwidth. This happens because three packet losses halve a
TCP window size, and the cumulative effect of this behaviour reduce the flow
of data significantly. On the other hand, when the client is run using MPTCP,
the average network throughput varies between 40.4% to 57%. For TCP, the
network throughput is highest when segment size is the smallest. For MPTCP,
the network throughput for video is highest when the segment size is largest,
while for audio, the throughput is highest, when the segment size is 6 s. These
graphs clearly show the advantage of MPTCP over TCP in extremely poor
wireless network conditions.
The average playback bitrate with MPTCP is far higher than that with TCP.
But it comes at the cost of very frequent increase and decrease in content bitrate.
At lower segment sizes switches happen very often while at higher sizes, the peak
bitrate is low. In any case, however, MPTCP presents a higher playback bitrate
than TCP. A good compromise between bitrate switches and average playback
bitrate is at 6 s segment size. The additional statistics table readings also reflect
this observation. This graph too, shows that MPTCP is clearly better than TCP
on links with high packet loss.
The buffer level graphs show an interesting trend. The DASH player, as
usual, sets minimum and maximum buffer levels at the start of the playback.
Due to high packet loss, the buffer quickly empties and the level stays at the
lowest possible value for majority of the playback duration. As a result, there are
interruptions which is confirmed by the readings in the statistics table. MPTCP
on the other hand, keeps the buffer level significantly above the minimum value
and there are no interruptions. The table confirms this observation too. Once
again, we notice that MPTCP is better for streaming in poor network conditions.
832 S. K. Nandi et al.

Table 3. Statistics of TCP and MPTCP for DASH streaming on three independent
paths with high packet loss.

Tranport TCP MPTCP

protocol
Segment size 2 4 6 10 20 2 4 6 10 20
(secs)
Startup 221 62 176 235 200 2153 150 40 1346 151
delay (ms)
Av. vid. nw. 1652 1437 1082 1125 993 8607 8297 8522 9064 9755
thrpt. (kb/s)
Av. aud. nw. 3409 2433 2422 1085 2346 5732 7998 13689 4933 4288
thrpt. (kb/s)
Av. vid. cnt. 1907 1909 1883 1961 2090 3556 4129 4319 2785 1938
b.r. (kb/s)
Av. aud. cnt. 130 131 131 135 135 130 131 131 135 128
b.r. (kb/s)
# video b.r. 1 1 1 1 1 57 31 22 14 7
switches
# audio b.r. 1 1 1 1 1 1 1 1 1 1
switches
# video 0 17 9 5 2 0 0 0 0 0
interruptions
# audio 0 17 9 5 1 0 0 0 0 0
interruptions
Abbreviations used: Av. = Average, vid. = video, nw. = network, cnt. = content, b.r. =
bitrate, thrpt. = throughput

The start up delay shows huge variations for MPTCP while they are low in
TCP. No conclusions could be drawn from the readings.

5 Conclusion

This work demonstrated the behaviour of MPTCP on a multi-homed end-host

in presence of packet loss. We found that MPTCP performs better than TCP
in most of the cases, providing higher network throughput and in turn higher
playback bitrate. However, MPTCP uses the available bandwidth less effectively
compared to TCP. At lower video segment sizes, DASH on MPTCP faces more
bitrate down-switches than TCP, whereas at higher bitrates, the playback quality
stays locked to low bitrate for longer period of time. So the video segment size is
a tradeoff between lesser number of down switches and higher average playback
bitrate. Also, DASH on MPTCP is resistant to playback interruptions unlike
DASH on TCP. Thus, with the proper choice of video segment size, MPTCP
is preferable over TCP as transport layer for DASH video streaming on mobile
devices with multiple wireless network interfaces.
 Evaluating DASH Player QoE with MPTCP in Presence of Packet Loss 833

References
1. GPAC Multimedia Open Source Project: MP4Box. https://gpac.wp.imt.fr/
mp4box/
2. MultiPath TCP - Linux Kernel implementation. http://www.multipath-tcp.org/
3. Chen, Y.C., Towsley, D., Khalili, R.: MSPlayer: multi-source and multi-path lever-
Aged YoutubER. In: Proceedings of the 10th ACM International on Conference on
emerging Networking Experiments and Technologies, pp. 263–270. ACM (2014)
4. Corbillon, X., Aparicio-Pardo, R., Kuhn, N., Texier, G., Simon, G.: Cross-layer
scheduler for video streaming over MPTCP. In: Proceedings of the 7th International
Conference on Multimedia Systems, p. 7. ACM (2016)
5. Han, B., Qian, F., Ji, L., Gopalakrishnan, V., Bedminster, N.: MP-DASH: adaptive
video streaming over preference-aware multipath. In: CoNEXT, pp. 129–143 (2016)
6. James, C., Halepovic, E., Wang, M., Jana, R., Shankaranarayanan, N.: Is multipath
TCP (MPTCP) beneficial for video streaming over DASH? In: 2016 IEEE 24th
International Symposium on Modeling, Analysis and Simulation of Computer and
Telecommunication Systems (MASCOTS), pp. 331–336. IEEE (2016)
7. Le Feuvre, J., Thiesse, J., Parmentier, M., Raulet, M., Daguet, C.: Ultra high defi-
nition HEVC DASH data set. In: Proceedings of the 5th ACM Multimedia Systems
Conference, pp. 7–12. ACM (2014)
8. Nandi, S.K., Singh, P.K., Nandi, S.: All plots for evaluating DASH player QoE with
MPTCP in presence of packet loss. http://pc.cd/9SJctalK
9. Wu, J., Yuen, C., Cheng, B., Wang, M., Chen, J.: Streaming high-quality mobile
video with multipath TCP in heterogeneous wireless networks. IEEE Trans. Mob.
Comput. 15(9), 2345–2361 (2016)
 VoIP Traffic Management Using a
Software-Defined Networking Approach

Paulo Vieira Jr.1(B) and Adriano Fiorese2(B)

1
Instituto Federal do Paraná, Curitiba, PR, Brazil
[email protected]
2
Santa Catarina State University (UDESC), Joinville, SC, Brazil
[email protected]

Abstract. Voice Over IP technology (VoIP) allows people to commu-

nicate voice using the Internet as a mean of transmission. VoIP calls
between clients using the same Codec consume communication and
computational resources from VoIP server. Software-Defined Network-
ing (SDN) has been disseminating the decoupling between control and
data plane in network technologies. SDN can be used in order to optimize
network traffic and consequently to reduce resources usage. Therefore,
this paper aims to present a management approach based on SDN to
reduce the use of resources in a VoIP server. Scenarios were emulated
with and without the use of SDN to assess CPU utilization and band-
width used in a VoIP server. Experimental results indicate considerable
reduction on the consumption of the mentioned resources.

Keywords: Management · VoIP · Traffic · SDN

1 Introduction
Packet-switched networks, initially designed to perform tasks without Quality-of-
Service (QoS) requirements, such as e-mail traffic and file transfer, were quickly
disseminated. A large number of switches, routers, firewalls, and other devices
have been and are still being deployed, manufactured by several vendors. Usu-
ally, each device must be configured by the administrator through low-level com-
mands in the embedded software in the device. This growth made the networks
proprietary, vertically integrated by ossifying the infrastructure [6] and limiting
the experimentation of new protocols and management solutions [11].
Protocols initially proposed fulfilled their role for the purpose outlined. How-
ever, in today’s context, computer networks play roles that go beyond what they
were initially designed and are critical part of network infrastructure for busi-
nesses, universities, research centers, government and daily activities. In addition
to the traffic initially designed, networks carry packets of data from a variety of
sources, such as video, voice and other applications that require Quality of Ser-
vice (QoS) during their execution. In response to the limitations of the physical
infrastructure of traditional networks, the Software-Defined Networking (SDN)
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 834–845, 2020.
https://doi.org/10.1007/978-3-030-15032-7_70
 VoIP Traffic Management Using a Software-Defined Networking Approach 835

paradigm emerged, breaking the vertical integration and separating the data
layer from the control layer.
At the same time, the telecommunications area has suffered important
changes related to the representation and transportation of voice data. Voice
over Internet Protocol (VoIP) provisioning has been leveraging the expansion
of packet-switched networks to allow voice traffic using the Internet as a mean
of transmission. Currently, VoIP is used by home users and geographically dis-
tributed companies, allowing communication through various devices (e.g., IP
phones, smartphones, cell phones, tablets, softphones).
Several factors influence the quality of a VoIP call, such as: the signaling
protocol, the selected codec, security attributes, the ability to cross Firewalls and
Network Address Translation (NAT) servers, among others. Therefore, factors
such VoIP server configuration or the location of the clients on the network can
make the voice packets routed through the VoIP server consuming CPU and
wasting bandwidth.
Figure 1 shows three ways in which data traffic can be routed in VoIP calls:
(a) In a call, data from both clients with the same audio codec (e.g., G711) are
transmitted from one client to another via VoIP server. All data are routed
passing through the VoIP server. In this case the server is in the media
path and two channels are created, one for each client. The VoIP server
reads media from one channel and writes to the other, increasing memory
and CPU usage while reading and writing channels. This configuration is
commonly used to monitor calls, collecting information related to the QoS
provided by sale centers.
(b) Clients with the same audio codec have their call data traffic made directly
between them. In this case, VoIP server is not in the media path and it is not
possible to record the call, add background music, transfer or put the call
on hold. All data flow directly between the clients. This approach reduces
CPU usage, allocated memory and bandwidth used by the server.
(c) Clients with different codecs have their call’s data always routed through the
VoIP server. Similar to the scenario presented in Fig. 1(a), VoIP server is in
the media path and two channels are created, one for each client. However,
VoIP server translates voice packets according to the codecs used by clients,
increasing consumption of server resources. In this way, clients can receive
and listen to the audio [5].
Clients located in administrative domains protected by firewall or using Net-
work Address Translation (NAT) increase the complexity of the problem. In this
case, VoIP server is not able to resolve the client’s private IP addresses in a call.
Thus, all voice traffic is based on the public IPs of the clients and in this case
the traffic is routed through the VoIP server, consuming CPU and bandwidth.
The NAT traversal problem is known and there are several solutions and studies
about it in [7,9]
Specifically, bandwidth consumption is a concern when discussing the adop-
tion of VoIP technology. Companies with headquarters and branch offices can
adopt VoIP technology to reduce costs with phone calls. However, many Internet
836 P. Vieira Jr. and A. Fiorese

Fig. 1. Examples of VoIP traffic with the same codec (a and b) and with different
codecs (c)

usage plans offer different bandwidth and often with different rates for down-
load and upload. The available bandwidth is important for service provisioning
by directly influencing the quantity and quality of concurrent calls that can be
supported.
In this context, the present work aims to develop a solution based on SDN
that allows to reduce the consumption of resources like CPU and bandwidth
of the VoIP server, applicable to corporate networks. The proposed solution is
based on SDN and makes the audio traffic shown in Fig. 1(a) to follow the pattern
as shown in Fig. 1(b), allowing calls between clients using the same Codec to be
performed with the voice traffic flowing directly between the clients. Therefore,
there is reduction on CPU and bandwidth consumption in the VoIP server, and
consequently in the network segment where the VoIP server is located.
The remainder of the paper is organized as follows: Sect. 2 presents theo-
retical basis for the development of the work. Section 3 discusses related works,
while Sect. 4 presents the proposed solution. Section 5 presents the experimental
analysis performed and Sect. 6 elaborates on final considerations.

2 General Background

VoIP is a technology that enables voice traffic over switched networks, including
the Internet [7]. Among the features offered by the technology are long-distance
and international calls, voicemail, caller ID, conference, call monitoring, call
waiting, call forwarding and integration with the Public Switched Telephone
Network (PSTN). It offers services with the efficiency of an IP network and the
quality of a traditional phone network. In addition to these features the main
reason for its adoption is the reduction of costs, since a single network supports
both services: voice and data.
The operation of VoIP calls is similar to a traditional telephone call and
involves signalling, coding and decoding. A call can be initiated on an IP tele-
phone, in which voice is digitized and transformed into packets through encoders.
These packets are routed through the IP network by routers and switches to the
VoIP server or directly to the destination (as represented in Fig. 1b).
 VoIP Traffic Management Using a Software-Defined Networking Approach 837

2.1 Establishing VoIP Sessions

The establishment of VoIP sessions is usually done by Session Initiation Protocol
(SIP) and the call’s data transported by the Real-time Transport Protocol (RTP)
and Real-Time Transport Control Protocol (RTCP). SIP is an application-layer
control protocol that can establish, modify, and terminate multimedia sessions
(e.g., conferences) such as Internet telephony calls with one or more partici-
pants [15].
In fact, any type of session can be established using the SIP protocol, such as
multicast conferences, as well VoIP calls. On the shoulders of UDP, SIP uses the
Session Description Protocol (SDP) protocol to represent the metadata needed
to establish sessions [3].
RTP is an application-layer protocol that provides end-to-end network trans-
port functions suitable for applications transmitting real-time data, such as
audio, video or simulation data, over multicast or unicast network services. RTP
carry, in addition to the data of its payload, the identification of the media type,
sequence number, date, time and delivery tracking information [16]. Using UDP,
RTP itself does not provide any mechanism to ensure timely delivery or provide
other QoS guarantees, but relies on lower-layer services to do so [16]. Thus a
sequence number is included in the RTP packet allowing the receiver to recon-
struct the sender’s packet sequence when needed.
For establishing an audio call, a pair of logic ports are used. Initially, one
port UPD is assigned to the RTP service to send and receive audio. A second
port, numbered in sequence to the port used by RTP, is required for control of
RTP packets, which is done by RTCP. RTCP packets are sent periodically to all
session participants, to monitor the quality of service and to convey information
about the participants in an on-going session.

2.2 Voice Encoding and Decoding

A Codec is an algorithm that allows transportation of an analog signal through
digital lines/channels. There are several Codecs that differ in complexity,
required bandwidth and voice quality. Usually, the more bandwidth a Codec
requires, the better the voice quality is [7]. Different codecs can be used in a
call. However, it is necessary to transcode the call, that is, the data needs to be
decoded and recoded so the audio can be listened at the terminals of the call. This
process consumes CPU on VoIP servers, affecting in several cases the voice qual-
ity [5]. The International Telecommunication Union-Telecommunication (ITU-
T) is the organization that controls the approval of a Codec. ITU-T evaluates
and assigns a score to the Codec after a testing process. The score is called Mean
Opinion Score (MOS) with values from 1 (bad) to 5 (excellent) [19].

2.3 VoIP and Software-Defined Networking

SDN is a networking paradigm that separates the network control plane from
the data forwarding plane with the promise to dramatically improve network
838 P. Vieira Jr. and A. Fiorese

resource utilization, simplify network management, reduce operating cost, and

promote innovation and evolution [1]. In an SDN architecture, the lowest level
represents the network infrastructure or data layer. This layer represents the
devices that route traffic based on the decisions made by the control layer. The
middle layer is the control layer and the key element is the controller, responsible
for managing all devices on the network, adding, removing, or changing entries
in compliant switches’ flow tables. In other words, the controller is responsible
to the intelligence at the network, making the decisions enforced on the data
layer.
Communication between the controller and the routing devices can be done
through several protocols. OpenFlow is an open protocol that standardizes com-
munication between control and data layers, defining how software can program
the flow tables on different switches in the SDN network [11]. According to the
specification of the OpenFlow protocol [12], the architecture of an SDN switch
must have the following components: one or more flow-tables, a group-table, a
secure communication channel with a controller and a metric table. Each SDN
switch’s flow table contains a set of flow entries and each flow consists of match-
ing rules, counters and a set of actions that must be applied to packets.1
Controller is responsible for adding, updating, and removing entries from
flow-tables and can be centralized or distributed in the network. In SDN, appli-
cations manage the network through the Application Programming Interfaces
(APIs) available by the controllers. These APIs provide a level of abstraction
that allows applications to do not rely on specific infrastructure details, making
it easier to use features provided by the controller and to perform their functions
properly. Examples of applications in this layer of the SDN architecture are: rout-
ing protocols, load balancing, monitoring, attack detection, policy enforcement
and firewall.
For the purpose of this work, a flow corresponds to the tuple of information
identifying the VoIP communication terminals (source and destination client),
and must be installed on the OpenFlow switches that are part of the path
between them. The information identifying the VoIP communication terminals
refers to the MAC addresses, IP addresses and logical ports used in the SIP ses-
sion, in voice data traffic carried by the RTP and, in control data traffic carried
by the RTCP protocols.

3 Related Works
The application of SDN to improve the QoS of communicating applications is a
recurring subject. [10] proposed a method of adapting Codecs in SDN (Codec
Adaptation over SDN - CAoS) networks. That work is based on the SDN archi-
tecture and according to the authors, implements an approach to avoid the
degradation of the voice in corporate networks. That proposed solution reduces
1
A series of bytes comprising a header, a payload, and a trailer, in that order, treated
as a unit for processing and routing purposes. The default packet type is an Ethernet
frame, but other packet types are also supported.
 VoIP Traffic Management Using a Software-Defined Networking Approach 839

the effects of network congestion on SDN by reducing the quality of voice in

calls to a less bandwidth-consuming Codec when congestion occurs. According
to the results obtained, CAoS allows a greater number of calls to be made in
periods of great use without degrading the voice quality excessively. Originally,
the authors did not quantify or consider the overhead of VoIP servers, a factor
that can degrade the QoS of calls.
In Lee et al. [8] is presented a framework called meSDN that enables
WLAN virtualization, application-oriented QoS and improved energy efficiency
in Android devices. The solution is based on SDN and uses the Open Virtual
Switch (OVS) [13] to monitor and manage mobile application traffic. Authors
argue that extending the capabilities of an SDN network to mobile devices it
can provide solutions to many network problems such as QoS, virtualization,
and fault diagnosis. The solution proposed in Sect. 4 reduces the bandwidth
consumption of VoIP servers, and can be combined with meSDN to reduce data
traffic.
FlowVista is proposed in [17]. It is a lightweight solution to identify flows
using the SDN Northbound interface providing an SDN bandwidth monitoring
tool driven by business application interaction. The authors used FlowVista to
monitor flow matching performance and bandwidth overhead in a VoIP instanti-
ation. In [17] controller plays an active rule since it needs make polling to obtain
crucial information regarding the application. Although objectives are different,
the proposed VoIP resource management solution is completely agnostic in this
regard, meaning the controller only process network incoming packets in order
to perform as planned.
Garg et al., presented a study on the limitations of 802.11 (a/b) wireless
networks in supporting VoIP calls [4]. Their study showed the number of simul-
taneous calls that can be supported in an 802.11 (a/b) network cell. For example,
using the codec G711 with 20 ms of audio, an 802.11 (a/b) cell can only support
3 to 12 calls. This limiting factor supports the development of this proposal: by
reducing data traffic to VoIP servers, communication in a cell can be used to the
maximum for performing call data traffic directly between clients.
In Costa et al. [2] the suitability of the Asterisk Private Branch Exchange
(PBX) server is investigated to provide VoIP communication capabilities with
an acceptable Mean Opinion Score (MOS) for a large number of calls. The call
blocking probability metric is used to measure the capacity of the PBX server
while MOS is used to evaluate the quality of voice calls. The results of that
work showed that the Asterisk PBX server can effectively handle more than 160
concurrent voice calls with a call blocking probability of less than 5%, providing
voice calls with MOS above the average 4. Also, in [2], MOS is measured by
a tool called VoIPMonitor, which observes VoIP traffic and computes the score
following the ITU-T [18] recommendation. The present proposal is agnostic to
the monitoring technologies used, being able to be used with VoIPMonitor.
840 P. Vieira Jr. and A. Fiorese

4 SDNVoIP Management Module

This section introduces SDNVoIP, our proposal for controlling VoIP traffic in
SDNs. The application manages VoIP traffic, reducing CPU and bandwidth
utilization in the VoIP server SDN segment. To reduce CPU usage on servers,
SDNVoIP reconfigures/redirects calls that use the same Codec to perform end-
to-end communication without call’s data need to be routed through the VoIP
server. Therefore, scenario in Fig. 1(a) is transformed into scenario in Fig. 1(b),
without the need to reconfigure VoIP servers.

4.1 SDNVoIP Implementation

SDNVoIP was developed as a Floodlight’s module [14]. Among the existing SDN
controllers, Floodlight was selected because it presents a mature stage of develop-
ment and it is adopted in several experimental and industrial scenarios. As for the
implementation, Floodlight is a centralized controller developed in Java, which
supports physical and virtual switches based on OpenFlow protocol. SDNVoIP
application keeps information about clients that are in on-going calls, allowing
the installation of flows in OpenFlow switches in the network path between them.
Thus, it is possible to install the flows at two points in time: when the destina-
tion client sends the SIP/SDP response message to the VoIP server, responding
to the call made by the source client; or when RTP traffic between source client
and destination client starts.
Four flows per call are installed in OpenFlow switches. Two flows regarding
the source-destination and destination-source paths for RTP packets; and two
flows for the source-destination and destination-source path for RTCP packets.
For each VoIP flow, seven actions [12] are applied to the RTP and RTCP packets
in the corresponding switch:
1. Change source MAC address to MAC address of the VoIP server;
2. Change destination MAC address to destination client’s MAC address;
3. Change source IP address to IP address of the VoIP server;
4. Change destination IP address to destination client IP address;
5. Change source RTP / RTCP port to RTP / RTCP port of the VoIP server;
6. Change destination RTP / RTCP port to destination client’s RTP / RTCP
port;
7. Output packet at a physical switch port.
The entries in the flow tables must be installed in the switches with an
expiration time. When the call is terminated by one of the clients, the entry is
removed. However, in case of abrupt interruption of the call, related entries are
automatically removed from the switches after the expiration time.

5 Experimental Analysis
This section presents the experimental analysis performed with SDNVoIP. Two
metrics are collected: CPU consumption on VoIP servers and the bandwidth
consumed for a VoIP call.
 VoIP Traffic Management Using a Software-Defined Networking Approach 841

5.1 Tests Environment

For the development of this work the following resources were used:
1. As controller it was used Floodlight in its version 1.2 and some modifica-
tions in its code were made, so the protocols SIP, RTP and RTCP can be
manipulated by the module SDNVoIP.
2. Asterisk version 1.13 was used as VoIP server, running on a computer with
3.2 GHz Intel QuadCore processor and 8 GB of RAM.
3. In order to generate the load on the VoIP server, was used the softphone
Linphone version 3.6.1.
4. A physical device used as a switch was a domestic wireless router TP-Link
model WR1043 version 3. It runs native embedded software for some experi-
ments and a customized firmware (Open Wireless Router (OpenWRT) version
15.01) and Open Virtual Switch (OVS) version 2.3, for others.
5. OpenFlow version 1.3 was used.

5.2 Experimental Scenarios

Two scenarios were used to perform the experiments, with and without the use
of SDN, both on LAN environment.

5.2.1 VoIP Calls Not Using SDN

Scenario not using SDN, comprehends the VoIP server, the TP-Link WR1043 as
a switch (with native firmware), and a PC where the instances of softphone Lin-
phone were started. A script initializes the softphone and receives as a parameter
the number of instances that must be executed, i.e., the number of users to the
experiment. Each instance of softphone has a client with username u followed
by a sequential number, so clients have a username like u1, u2, and so on up to
the limit provided as parameter to the shell script.
The calls are generated by another shell script, according to the following
logic: client u51 makes a call to client u1, client u52 makes a call to client
u2, and so on up to the final client. Thus, 50 simultaneous calls are generated
between 100 clients. The softphone has been configured to answer incoming calls
automatically and immediately after answer it plays an audio file. All calls used
the same Codec (G711) and all call’s data are routed through the VoIP server.
In this way, the load generated (CPU usage) on the VoIP server is established
with several simultaneous calls between two clients simulating a conversation in
which the two participants speak and listen.

5.2.2 VoIP Calls Using SDN

Scenario using SDN is similar to the scenario not using SDN. The only differences
are the addition of the Floodlight controller and reconfiguration of the TP-
Link WR1043 as switch with OpenWRT and OVS enabled and communicating
with the controller. Like the former scenario, all audio data exchanged between
842 P. Vieira Jr. and A. Fiorese

clients is routed through the VoIP server. However, after the controller obtains
all necessary information comprising VoIP communicating clients, four flows per
call are installed in the switch’s flow table, making audio from calls flowing
directly between clients.
Although the proposed solution operates in LAN, a realistic scenario like
companies with headquarters and branches (WAN environment), can benefit
from the solution presented in the present work. Figure 2 presents a network
topology of a company that has both, a Head Office and a Branch. A call between
a source client (SC) and a destination client (DC), both located in Branch, has
a route passing through switches C, A, and B. On a scenario not using SDN,
all voice traffic from SC to DC and vice-versa will be routed through the VoIP
server located in the Head Office, increasing CPU utilization and consuming
bandwidth.
With the deployment of OpenFlow switches and the use of the SDNVoIP
module, server’s CPU utilization and bandwidth consumption can be reduced
in the Internet link while maintaining a centralized architecture for the VoIP
server, facilitating it’s maintenance in terms of configuration. SDNVoIP module
detects the call made between SC and DC and installs the required flows in
switch C. Therefore, all voice traffic flows directly between clients in Branch. A
decrease in CPU utilization on the VoIP server and reduction in interconnection
bandwidth utilization is achieved, with the voice traffic occuring only on the
subnet where SC and DC are located.

Fig. 2. Scenario WAN with SDN: call between clients in branch 1

5.3 Result Analysis

The CPU utilization rate was collected with the mpstat tool at one second inter-
vals. Five minutes samples of conversation on simultaneous calls were made and
the experiments were repeated ten times, on which the arithmetic mean and
standard deviation were calculated. Table 1 shows the data obtained regarding
CPU utilization on a VoIP server (Asterisk) with the respective loads. It is
 VoIP Traffic Management Using a Software-Defined Networking Approach 843

Table 1. VoIP server’s CPU usage comparison

Simultaneous calls 50 100 150 200 250

Not using SDN
VoIP server CPU usage % 4.03 9.22 13.92 19.35 22.78
Variance 0.53 5.59 1.33 1.04 1.37
Std. deviation 0.13 0.37 0.21 0.17 0.21
Using SDN
VoIP server CPU usage % 0.75 0.73 0.71 0.67 0.71
Variance 0.01 0.00 0.00 0.00 0.00
Std. deviation 0.08 0.01 0.03 0.06 0.03

Table 2. VoIP server bandwidth comparison not using SDN and using SDN

50 100 150 200 250

Not using SDN
rx 1.03 Mbps 2.05 Mbps 3.07 Mbps 4.10 Mbps 5.23 Mbps
rxpps 5.10 Kbps 10.13 Kbps 15.19 Kbps 20.23 Kbps 25.84 Kbps
tx 1.03 Mbps 2.07 Mbps 3.10 Mbps 4.14 Mbps 5.28 Mbps
txpps 5.04 Kbps 10.08 Kbps 15.12 Kbps 20.16 Kbps 25.73 Kbps
Using SDN
rx 0.008 Kbps 0.008 Kbps 0.008 Kbps 0.008 Kbps 0.008 Kbps
rxpps 0.005 Kbps 0.01 Kbps 0.01 Kbps 0.005 Kbps 0.008 Kbps
tx 0.32 Kbps 0.70 Kbps 1.85 Kbps 2.94 Kbps 3.44 Kbps
txpps 1.43 Kbps 4.19 Kbps 11.28 Kbps 17.91 Kbps 21.08 Kbps

observed that 250 concurrent calls using the same Codec and all traffic routed
through the VoIP server, generates a VoiP server CPU utilization rate of approx-
imately 20%. With the use of SDN, all the traffic generated by the calls is routed
directly between the clients, making the VoIP server CPU approximately 99%
of idle time.
As can be seen from the results comprising scenario without using SDNVoIP,
the higher the number of calls the higher the CPU consumption. Therefore, the
solution using SDNVoIP reduces CPU utilization. This way the CPU remains
available for other tasks such as transcoding between client calls using different
codecs. The solution presented in this work using SDN can reduce CPU usage
for larger loads, as long the clients use the same Codec.
Table 2 presents the bandwidth metrics in a scenario not using SDN collected
on the Asterisk server. Thus, rx is the receive rate, rx pps the packet receive
rate in packets per second, tx the transmission rate and tx pps the packet trans-
mission rate per second. It is observed that for 50 simultaneous calls the typical
reception and transmission rate is 1.03 Mbps not using SDN. Therefore, using
844 P. Vieira Jr. and A. Fiorese

SDN this values decreases to 0.008 Kbps and 0.32 Kbps for reception and trans-
mission rates. In the largest scenario experimented, with 250 simultaneous calls,
it is observed that the typical reception and transmission rate not using SDN
is 5.23 Mbps and 5.28Mbps respectively. Meanwhile using SDNVoIP the typical
reception and transmission rate decrease to 0.008 Kbps and 3.44 Kbps, respec-
tively.

6 Final Considerations
The present work aggregates the advantages and benefits of the use of SDN in the
telecommunications area through the proposition of the SDNVoIP application
for the management of VoIP calls in packet-switched environments that support
OpenFlow. Results show that it is possible to reduce CPU utilization in a VoIP
server by creating/using flows that redirect Voice over IP traffic directly between
involved clients using the same Codec. Also, there is reduction in bandwidth
consumption in the network segment of the VoIP server.
Reduced CPU utilization allows greater processor’s availability for calls that
require codec translation, as well as reducing bandwidth consumption may allow
more calls to be completed. Finally, with the proposed solution it is possible to
resize the number of calls supported by existing VoIP servers in a network,
without having to purchase more servers or instanciate more virtual machines
in the cloud to support more calls.
As a future work, the SDNVoIP module can be enhanced to redirect VoIP
traffic from clients that use different Codecs and need to be transcoded to a server
with less CPU usage (considering an environment with several VoIP servers).
Finally, another approach is the redirection of VoIP calls’ traffic to the VoIP
server with a network path with less traffic, avoiding congestion.

References
1. Akyildiz, I.F., Lee, A., Wang, P., Luo, M., Chou, W.: A roadmap for traffic engi-
neering in SDN-openflow networks. Comput. Netw. 71, 1–30 (2014)
2. Costa, L.R., Nunes, L.S.N., Bordim, J.L., Nakano, K.: Asterisk PBX capacity
evaluation. In: 2015 IEEE International Parallel and Distributed Processing Sym-
posium Workshop (IPDPSW), pp. 519–524. IEEE (2015)
3. Camarillo, G.: Session description protocol (SDP) format for binary floor control
protocol (BFCP) streams. RFC 4583. IETF, RFC Editor, November 2006. https://
tools.ietf.org/html/rfc4583
4. Garg, S., Kappes, M.: Can i add a VoIP call? In: IEEE International Conference
on Communications, ICC 2003, vol. 2, pp. 779–783. IEEE (2003)
5. Goode, B.: Voice over internet protocol (VoIP). Proc. IEEE 90(9), 1495–1517
(2002)
6. Handley, M.: Why the internet only just works. BT Technol. J. 24(3), 119–129
(2006)
7. Karapantazis, S., Pavlidou, F.N.: VoIP: a comprehensive survey on a promising
technology. Comput. Netw. 53(12), 2050–2090 (2009). https://doi.org/10.1016/j.
comnet.2009.03.010
 VoIP Traffic Management Using a Software-Defined Networking Approach 845

8. Lee, J., Uddin, M., Tourrilhes, J., Sen, S., Banerjee, S., Arndt, M., Kim, K.H.,
Nadeem, T.: meSDN: mobile extension of SDN. In: Proceedings of The Fifth Inter-
national Workshop on Mobile Cloud Computing and Services, pp. 7–14. ACM
(2014)
9. Lin, Y.D., Tseng, C.C., Ho, C.Y., Wu, Y.H.: How nat-compatible are voip appli-
cations? IEEE Commun. Mag. 48(12), 58–65 (2010)
10. Maribondo, P.D., Fernandes, N.C.: Avoiding voice traffic degradation in IP enter-
prise networks using CAoS. In: Proceedings of the 2016 workshop on Fostering
Latin-American Research in Data Communication Networks, pp. 34–36. ACM
(2016)
11. McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford,
J., Shenker, S., Turner, J.: OpenFlow: enabling innovation in campus networks.
ACM SIGCOMM Comput. Commun. Rev. 38(2), 69–74 (2008)
12. Open Networking Foundation: OpenFlow Switch Specification (2012). http://
www.cs.yale.edu/homes/yu-minlan/teach/csci599-fall12/papers/openflow-spec-
v1.3.0.pdf
13. Pfaff, B., Pettit, J., Koponen, T., Jackson, E.J., Zhou, A., Rajahalme, J., Gross,
J., Wang, A., Stringer, J., Shelar, P., Amidon, K., Casado, M.: The design and
implementation of open vSwitch. In: Proceedings of the 12th USENIX Conference
on Networked Systems Design and Implementation, NSDI 2015, pp. 117–130 (2015)
14. Project Floodlight: Project floodlight: Open source software for building software-
defined networks (2016). http://www.projectfloodlight.org/floodlight
15. Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks,
R., Handley, M., Schooler, E.: SIP session initiation protocol. RFC 3261, RFC
Editor (2002). https://www.ietf.org/rfc/rfc3261.txt
16. Schulzrinne, H., Casner, R., Frederick, V.J.: RTP: A transport protocol for real-
time applications. RFC 3550, IETF (2003). https://www.ietf.org/rfc/rfc3550.txt
17. Siniarski, B., Murphy, J., Delaney, D.: FlowVista: low-bandwidth SDN monitoring
driven by business application interaction. In: 2017 25th International Conference
on Software, Telecommunications and Computer Networks (SoftCOM), pp. 1–6
(2017). https://doi.org/10.23919/SOFTCOM.2017.8115539
18. International Telecommunication Union: Methods for subjective determination of
transmission quality (2005). http://www.itu.int/rec/T-REC-P.800-199608-I/en
19. International Telecommunication Union: Testing requirements and method-
ology (2005). https://www.itu.int/en/ITU-T/C-I/Pages/HFT-mobile-tests/
Methodologies.aspx
 A Self Healing Microservices
Architecture: A Case Study in Docker
Swarm Cluster

Basel Magableh1(B) and Muder Almiani2

1
School of Computer Science, Technological University Dublin, Dublin, Ireland
[email protected]
2
Al-Hussein Bin Talal University, Ma’an, Jordan
[email protected]

Abstract. One desired aspect of a self-adapting microservices architec-

ture is the ability to continuously monitor the operational environment,
detect and observe anomalous behaviour as well as implement a rea-
sonable policy for self-scaling, self-healing, and self-tuning the computa-
tional resources in order to dynamically respond to a sudden change in its
operational environment. Often the behaviour of a microservices archi-
tecture continuously changes over time and the identification of both nor-
mal and abnormal behaviours of running services becomes a challenging
task. This paper proposes a self-healing Microservice architecture that
continuously monitors the operational environment, detects and observes
anomalous behaviours, and provides a reasonable adaptation policy using
a multi-dimensional utility-based model. This model preserves the cluster
state and prevents multiple actions to taking place at the same time. It
also guarantees that the executed adaptation action fits the current exe-
cution context and achieves the adaptation goals. The results show the
ability of this model to dynamically scale the architecture horizontally
or vertically in response to the context changes.

Keywords: Self healing · Microservices architecture ·

Anomaly detection · Run-time configuration

1 Introduction
A microservices architecture could be defined in the context of a service-oriented
architecture as a composition of tiny fine-grained distributed loosely coupled
building blocks of software components [27]. In a microservices cluster, the per-
formance of its nodes might fluctuate around demands to accommodate scala-
bility, orchestration and load balancing issued by the leader node. To achieve
an optimal level of performance, the architecture requires a model that (i) is
able to detect anomalies in real-time, (ii) with a high accuracy and (iii) leads
to a low rate of false alarms. In addition, a set of possible configurations should
be designed and incorporated in the architecture in order to adapt itself to the
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 846–858, 2020.
https://doi.org/10.1007/978-3-030-15032-7_71
 A Self Healing Microservices Architecture 847

changes in its operational environment. This adaptability requires a dynamic

decision making component that is capable of selecting a desired cluster state
according to a set of constraints. This research proposes a method to continu-
ously observe and monitor the Docker Swarm cluster state and detect anomalous
behaviour. This method also aims at equipping a microservices architecture with
adaptation strategies able to reason about detected anomalies detected able to
self-adjust its parameters and to verify its actions at runtime without human
intervention. In details, the proposed method offers microservices architecture
a self-adaptation property by following the MAPE-K (Monitor-Analyse-Plan-
Execute over a shared Knowledge) model. The main contribution of this work
is the employment of a utility function in the process of adaptation.
The remainder of the paper is structured as follows: Sect. 2 provides an
overview of self-healing architectures and surveys the approaches for anomaly
detection and run-time configuration. Section 3 presents a model that can con-
tinuously observe microservices architecture with self-healing capabilities. Adap-
tation planning and execution is discussed in Sect. 3.2. The implementation of
this model is discussed in Sect. 3.3. Section 3.4 is focused on presenting results
followed by a critical discussion of the effectiveness of this model. Section 4 sum-
marises this research, highlighting its contribution and setting future work.

2 Related Work
A microservices architecture is a composition of tiny fine-grained distributed
loosely coupled building blocks of software components [27]. A self-healing archi-
tecture refers to the capability of its software components to discover, diagnose
and react to disruptions. Such architecture can also anticipate potential problems
and, accordingly, take suitable actions to prevent a failure by self-adaptation [14].
In order to achieve this, a microservices architecture require a decision-making
strategy that can work in real-time and is able to reason about its own state
and its surrounding environment in a closed control loop and then to act accord-
ingly [4]. Typically, a self-adapting architecture should implement the MAPE-K
(Monitor-Analyse-Plan-Execute over a shared Knowledge) approach. This app-
roach includes: (i) gathering of data related to the surrounding context (context
sensing); (ii) context observation and detection; (iii) dynamic decision making;
(iv) execution of adaptation through actions to achieve a set of objectives defined
as QoS; (v) verification/validation of the applied adaptation actions in terms of
accuracy in meeting the adaptation objectives.
A number of approaches exist for achieving high degrees of self-adaptability.
For instance, in [26], self-adaptability involves context sensing and collection,
observation and detection of contextual changes in an operational environment.
Self-adaptation in an architecture and a dynamic adjustment of its behaviour can
be achieved using parameter-tuning [5], component-based composition [18], or
middleware-based approaches [6]. An important aspect of a self-adaptive system
is related to its ability to validate and verify the adaptation action at run-time.
This can be done by employing game theory [28], utility theory [15] or a model-
driven approach as in [24]. Context information refers to any information that
848 B. Magableh and M. Almiani

is computationally accessible and upon which behavioural variations depend

[13]. Context observation and detection approaches are used to detect abnor-
mal behaviour within the microservices architecture at run-time. Related work
in context modelling, context detection and engineering self-adaptive software
systems are discussed in [4,8,23,26]. In dynamic decision making and context
reasoning an architecture should be able to monitor and detect normal/abnormal
behaviour by continuously monitoring the contextual information found in the
microservices cluster. There are two phases for detecting anomalies in a software
system: a training phase which involves profiling the normal behaviour of the
system; and a phase aimed at testing the learned profile of the system with new
data and employing it to detect normal/abnormal behaviours [19].
Three major techniques for anomaly detection have emerged from the litera-
ture: (a) statistical anomaly detection, (b) data-mining and (c) machine learning-
based techniques. In the statistical methods, the anomaly detection algorithm
usually observes the activity of the software system and generates profiles with
system metrics such as CPU and memory to represent its behaviour. Various
statistical anomaly detection systems have been proposed as in [2,22]. They
provide accurate notifications of malicious attacks that occur over long peri-
ods of time and this class of systems performs better than the other classes in
detecting denial-of-service attacks [19]. In statistical anomaly detection a skilled
attacker might train a statistical anomaly detection system to accept the abnor-
mal behaviour as normal. It is difficult to determine the thresholds that make
a balance between the likelihood of a false negative – the system fails to iden-
tify an activity as an abnormal behaviour – and the likelihood of a false positive
(false alarms). Therefore, statistical anomaly detection systems need an accurate
model with precise distributions for each metric. In practice, the behaviour of
virtual machines/computers cannot be entirely modelled using solely statistical
methods. Data-mining anomaly detection techniques are about finding insights
which are statistically reliable, previously unknown, and actionable from data
[21]. The traditional data-mining process involves discovering a novel, distin-
guished and useful data pattern in large datasets to extract hidden relationships
and information. However, two issues exist in anomaly detection in microservices
architectures: a lack of a large dataset containing information about the archi-
tecture itself, and the small number of approaches applied to these architectures
[21]. Machine learning-based anomaly detection models are data-driven and are
mainly focused on learning exclusively from past data [19]. When additional
and new data becomes available, they can intrinsically influence the detection
strategy and classify significant deviations from the normal behaviour of an
underlying software programme. Therefore, they need to be often retrained to
be in line with current data. These approaches generally use a combination
of clustering and classification algorithms to detect anomalies. The former are
used to cluster the dataset and label observations. The latter algorithms, such
as decision trees can then be used for classification to distinguish between nor-
mal/abnormal behaviour [3]. Other applications and information can be found
in [3,10,11]. It is important to highlight that, due to the opening deployment
 A Self Healing Microservices Architecture 849

and limited resources generally found in a microservices cluster, a lightweight

approach to data clustering/classification should be used.
Other anomaly detection approaches exist in the literature. For example, the
Numenta Platform for Intelligent Computing (NUPIC) is based on the hierar-
chical temporal memory (HTM) model proposed in [12]. This has been exper-
imentally applied in real-time anomaly detection of streaming data [17] and it
is claimed to be efficient and tolerant to noisy data, capable to adapt to the
changes of data statistics. It can also detect extremely subtle anomalies with a
very minimal rate of false positives. In a similar study, Ahmad et al. [1] proposed
an updated version of this anomaly detection algorithm introducing the anomaly
likelihood concept. The anomaly score calculated by the NUPIC algorithm repre-
sents an immediate calculation of the predictability of the current input stream.
This approach works very well with predictable scenarios in many practical appli-
cations. As there is no noisy and unpredictable data found, the raw anomaly
score gives an accurate prediction of false negatives. However, the changes in
predictions would lead to revealing anomalies in the system’s behaviour. Thus,
instead of using the raw anomaly score, authors in [1] proposed a method for cal-
culating the anomaly likelihood by modelling the distribution of anomaly scores
and by using it to check the likelihood of the current state of the system to
identify anomalous behaviour. The anomaly likelihood is metric which defines
how anomalous the current state is based on the prediction history calculated by
the HTM model. The anomaly likelihood is calculated by maintaining a window
of the last raw anomaly scores and then calculating the normal distribution over
the last obtained/trained values. The most recent average of anomalies is then
calculated using the Gaussian tail probability function (Q-function) [7].

3 Design and Methodology

This research focuses on proposing a mechanism that can continuously observe

and monitor the microservices architecture and be able to detect anomalous
behaviour with high accuracy and generate low rate of false alarms. At the same
time, this mechanism should be able to respond to true positive alarms by sug-
gesting a set of adaptation policies (adaptation strategy), that can be deployed in
the cluster to achieve high level of self-healing in response to changes in its oper-
ating environment. The envisioned property of this mechanism is that it can be
easily deployed with fewer and smaller footprints on the limited resources found
in the tiny containers running in a microservices cluster.

3.1 Self-healing Microservices Architecture

One important aspect of a self-healing microservices architecture is the ability to

continuously monitor the operational environment, detect and observe anoma-
lous behaviour, and provide a reasonable policy for self-scaling, self-healing, and
self-tuning the computational resources to adapt a sudden changes in its opera-
tional environment dynamically at run-time. A typical microservices architecture
850 B. Magableh and M. Almiani

is shown in Fig. 1 and it was designed according to the MAPE-K model (Monitor-
Analyze-Plan-Execute over a shared Knowledge) [25].

Fig. 1. A microservices architecture implemented in Docker Swarm [25].

This model offer to the microservices architecture the following

functionalities:

• Metric collection: continuous collection of fine-grained metrics about clus-

ter nodes, services and containers such as CPU usage, Memory, Disk Reads
Bytes/sec, Network Read/s, network write/s and Disk Writes Bytes/sec)
which is streamed into the anomaly detection service at real-time;
• Model Training: the NUPIC anomaly detection service [1] continuously
runs over the streamed metrics stored in a database, enabling the training of
a model with the collected metrics;
• Anomaly Detection: collected real-time data is feed on the fly to the
NUPIC anomaly detection service, which provides two features: continuous
detection of anomalous behaviour with high accuracy and predictions about
the architecture performance based on historic data. This service can alert
the architecture about incoming spike on resources demand which can then be
used by the adaptation manager to schedule a proactive adaptation strategy
ahead of time. In addition, it is able to detect anomalies as early as possible
before the anomalous behaviour interrupts the functionality of the running
services in the cluster Ahmad et al. [1];
• Adaptation Election: once an anomalous behaviour is detected, anomaly
score and likelihood are calculated by the Anomaly Detection Service as in
Fig. 1. The alert manager services notifies the adaptation manager about the
anomaly detected and then selects the adaptation action(s) after calculat-
ing the utility value for each of the possible actions, as detailed in Sect. 3.2.
Subsequently, the Adaptation Manager uses the input of the anomaly likeli-
hood, architecture constraints (specified by the DevOp during deployment)
 A Self Healing Microservices Architecture 851

and desired/predicted QoS to calculate the best variation of the adaptation

that has the highest utility;
• Adaptation Execution: the adaptation manager executes the strategies
according to the aggregated value of the utility returned by the algorithm.
Once the adaptation action is completed, a set of adaptation actions are
deployed in the architecture. To avoid, conflicts between multiple adapta-
tion polices, the adapter allows the adaptation actions to be fully completed
and verified by the cluster leader according to the consensus performed by
the RAFT algorithm [20]. It then set a cool off timer before initiating new
adaptation actions. This technique is used to avoid resources thrashing and
preserving the cluster state for auto-recovery. The adaptation manager then
sends to the cluster leader a set of instructions that might involve tuning of
cluster parameters – horizontal scaling – adding/removing nodes or vertical
scaling of microservice’s containers like scaling a service in/out;
• Adaptation Verification: The cluster leader and all managers in the cluster
subsequently vote on the adaptation action based on the RAFT consensus
algorithm [20]. The results of the vote are used to validate and verify the
adaptation action. If the adaptation action passes the voting process, it will
be executed by the cluster leader and the adaptation manager records the
adaptation attempt as successful. Otherwise, the adaptation manager keeps
the current state of the cluster and records the adaptation attempt as failed.
In both cases, the adaptation manager records the number of attempts used
to complete the adaptation actions.

3.2 Adaptation Election

To provide the model, described in the previous section, with dynamic policy
election that guarantees high accuracy of selecting the best adaptation action
that fits in the current execution context, an extension of the adaptation manager
with a policy election process by employing a utility function is proposed. This
function is aimed at computing the probability of transition from one state
to another. In this process the anomaly detection service plays a significant
role. At each state s of the microservices architecture, there is a set of context
values CV: c1 , . . . , cm measuring the metrics in the operating environment such
as CPU, Memory, Disk I/O and Network. The anomaly detection service reads
the current values of all metrics in CV and NUPIC calculates the anomaly scores
AS: as1 , . . . , asm and anomaly likelihoods AL: al1 , . . . , alm in the current state
for each of them. The anomaly likelihood accurately defines how anomalous the
current metric is when compared to the distribution of the values learned by
the anomaly detection service. The anomaly score and the anomaly likelihood
are scalar values in [0..1]. For instance, if the alcpu is 1 and c2 is the CPU value
of 70%, then c2 can be associated with a high utility score and it might be
considered in the next adaptation action. In turn, the adaptation manager can
select an adaptation policy able to reason about the anomalous behaviour of the
CPU. In another scenario, if the anomaly likelihood is 0, then the metric can be
852 B. Magableh and M. Almiani

associated with a low utility score so it will not be considered in the subsequent
adaptation action.
m
W (alm , Cm ) = ali · ci (1)
i=1

20
U (cvx ) = max( cvxi × alx ) (2)
t=1

in Eq. 2 CV the vector contains the context value cvs of the context metric, AL
the vector containing all the anomaly likelihood al for each metric returned by
NUPIC. These are taken 20 times i in a time window that is set to 300 s.
From utility theory, a von Neumann-Morgenstern utility function Ui : Xi →
R assigns a real number to each quality dimension i, which we can normalize
to the range [0, 1] [9]. Across multiple dimensions of contextual changes Cm ,
we can attribute a percentage weight to each dimension to account for its rela-
tive importance compared to other dimensions. These weights form the utility
preferences. The overall utility is then given by the utility preference w(cm ) func-
tion calculated using Eq. 1. For example, if three objectives, u(cpu), u(memory),
u(disk), are given anomaly likelihood as follows: the first is twice as important
as the second, and the second is three times as important as the third. Then
the weight would be quantified as [w1 : 0.6, w2 : 0.3, w3 : 0.1], where the weight
is the Anomaly Likelihood of each metric. This gives the CPU metric higher
priority to be consider in the adaptation action.
In this paper. We argue that the use of anomaly likelihood to weight the
collected metrics provides an accurate calculation of the utility function and
provides the model with better estimation of the adaptation action. So the max-
imum metric is selected using the equation described in 2, which select the max-
imum W of specific metric value that has the highest Anomaly Score returned
by the Anomaly detection service.

(Current(cm ) − N U P ICP redicted(cm )) · (asm ) · alm , cm ))

Cost(um ) = (3)
U sageT ime ∗ InstanceM onthlyP rice

where Current(cm ) is the current value of the metric CM , N U P ICP redicted

(cm ) is the predicted value of CM computer by the NUPIC algorithm,
AnomScore(cm ) is the anomaly score of Cm at time ti calculated by NUPIC,
and Wi is the anomaly likelihood for metric CM as per Eq. 1. The U sageT ime)
refers to the total number of hours the node is expected to be used per day
(constant value). The InstanceM onthlyP rice is the price in $ (dollars) for pro-
visioning an instance per month. Normally this is a constant price specified by
the cloud infrastructure provider based on the instance type.

changem = (Wa (alm , Cm ) − Wb (alm , Cm )) (4)

where i is the current time stamp, m is a future time stamp

we calculate the weight of the current metric Cm and the previous value.
 A Self Healing Microservices Architecture 853

3.3 Experimental Setup and Evaluation Strategy

To validate the ideas presented in this paper, we design and develop a working
prototype of Microservice architecture in Docker swarm1 as shown in Fig. 1. The
cluster consisted of manager and worker nodes. Each cluster has one leader,
which maintains the cluster state and preserves the cluster logs. Also, the leader
node initializes the vote of Raft Consensus Algorithm [20] to agree/disagree on
specific value based on the consensus by all nodes in the cluster. Only the leader
node is allowed to commit and save the variable values or logs. To meet scalability
and availability, the leader node distributed the work load between the workers
based on Raft Consensus Algorithm [20]. This means that each service could be
executed by assigning multiple containers across the cluster’s nodes.
The main services implemented in this architecture are: Time series metrics
database for context collection, Nodes metrics used to collect metrics from all
nodes in the cluster, Alert and notification manager used to notify the adaptation
manager about contextual changes offered by Prometheus framework2 . Docker
containers metrics collector for collecting fine-grained metrics about all running
containers in all nodes3 . Reverse proxy for routing traffic between all services
in the cluster4 . Unsupervised Real-time Anomaly Detection based on NUPIC5 ,
Adaptation manager for executing, validating the adaptation actions developed
as a prototype of this research. Time series analytic and visualisation dashboard
for observing the behaviour of the Microservices cluster6 .
This live snapshot7 provides a full virtualisation of all services running in
the cluster. The evaluation of the effectiveness of this model will be based on
calculating a utility function for all metrics monitored, then it will calculate
the number of adaptation attempts, successful convergence of services/nodes, or
errors which leads to unstable state of the cluster.
The evaluation of the model is set over two stages: (i) assessing the con-
sistency of the behaviour of the cluster by evaluating the state of the swarm
after a sequence of adaptation actions. The idea is to start with no nodes and
the utility model should be able to create a new cluster and add the required
number of nodes/replicas until it reaches an stable state. In other words, this
when the cluster reaches a convergence and all the services in it are accessible
and available. The decision is then left for the adaptation manager to scale the
cluster horizontally or vertically until it reaches the sable state. (ii) evaluating
the accuracy of the model in electing the correct adaptation action by identify-
ing the highest metric value that need to be consider in the adaptation and the
selected adaptation action. In summary the evaluation criteria are: (a) criterion
1
https://docs.docker.com/engine/swarm/.
2
https://prometheus.io.
3
https://github.com/google/cadvisor.
4
https://caddyserver.com/docs/proxy.
5
http://nupic.docs.numenta.org/stable/index.html.
6
https://grafana.com.
7
https://snapshot.raintank.io/dashboard/snapshot/UJlrTzwubrRQjDwM1YFJle5zv
dK3Anr7?orgId=2.
854 B. Magableh and M. Almiani

Fig. 2. Dimensional analysis of variations of utility functions

1: the ability of the model to manage a microservices cluster and to scale it hor-
izontally or vertically until it reaches an stable state (when all the services are
available). (b) criterion 2: the ability of the model to handle dynamic changes
in the cluster and to dynamically adapt to sudden changes, such as simulated
stress test or Distributed Denial of Service attack (DDOS), without leading the
cluster to an unstable state; (c) criterion 3: the ability of the architecture to
meet demands dynamically and maintain a stable state for the cluster.

3.4 Discussion
To test the first evaluation criteria, a stress test has been executed in the cluster
manager until its CPU usage reaches 70%, which triggers an alert to the adapta-
tion manager. In turn, the adaptation manager collects the current reading of the
metrics, anomaly score and anomaly likelihood produced by NUPIC. Then, it
calculates the rate of changes for each metric in order to elect the metric that has
the highest utility. As example, Fig. 2 depicts the utility of CPU usage, memory
usage, disk reads (bytes/s), disk writes (bytes/s), docker network (sent/received
bytes/s). The CPU usage has the highest utility as confirmed by the utility
indifference indicator (curve in Fig. 2). Also, the memory usage of the service
shows slow rate of changes over time, which make it optional to be considered
in the adaptation action. With regard to the utility of disk read/write, there is
no divergence above the moving average (utility indifference curve) so it is not
considered in the subsequent adaptation action. The docker network shows no
changes over time as the load balancer and the reverse proxy manage to divert
the traffic to many containers distributed in the cluster. As the U (CP U ) has
the highest value of changes, this triggers an adaptation action and it allows the
model to reason about the high demand of CPU usage. As a consequence, the
 A Self Healing Microservices Architecture 855

Fig. 3. Utility(cpu) rate of changes and cost calculated based in Eqs. 3 and 4

adaptation manager creates additional nodes and add them to the swarm cluster
automatically. The number of nodes is equal to the utility calculated as in Eq. 2.
This lead to new nodes addition to the swarm as shown in the snapshot8 (a full
visualised and analytics dashboard of the swarm after the adaptation). Once the
CPU demand is reduced, the adaptation manager calculates the variations of
the utility and remove a number of nodes equal to the value returned by the
cost function of Eq. 3. A snapshot9 of the system after executing the adapta-
tion action to reason about the low level of the CPU usage. This satisfies the
evaluation criteria 2.
The accuracy of the utility cost, rate of changes, and the maximum utility
dimension are vital for the success of the adaptation process. So, Fig. 3 depicts
the calculation of the rate of changes and the utility cost to reach the desired
number of nodes/replicas needed. The calculation accurately satisfies the adapta-
tion objectives and provides the architecture with a suitable number of required
nodes/replicas. As shown in Fig. 3, this number increases at the right time when
the CPU demand spikes. In fact, the number of nodes/replicas reduces just
before the CPU demand declines significantly. The rate of changes in CPU usage

8
https://snapshot.raintank.io/dashboard/snapshot/sstuT2tuYkob8zjIbh1YXzBYxS
JDFd9z?orgId=2.
9
https://snapshot.raintank.io/dashboard/snapshot/UJlrTzwubrRQjDwM1YFJle5zv
dK3Anr7?orgId=2.
856 B. Magableh and M. Almiani

declined so the utility function returns a negative value for the required number
of nodes/replicas as long they are above the minimum amount specified by the
Dev-Ops. Also, as shown in Fig. 3 the utility cost normalizes and tunes the CPU
demand. This provides evidence that the adoption of the utility provides the
adaptation cycle with a dynamic variability over the needed/allocated resources
rather than scaling the architecture in/out according to a static threshold. This
satisfies the evaluation criteria 3.
In another scenario, a Distributed Denial of Service attack to a web service
running in the swarm was simulated. This was aimed at verifying that the adap-
tation manager can accommodate the DDOS attack by adding more replicas to
the service. As in the proposed model, if the anomaly detection service would
consider a specific record as anomalous and this was an actual anomaly, then
this attempt is classified as a True Positive. If the anomaly detection service
consider the data as normal behaviour, and it is actually normal data, then this
attempt is classified as True Negative. If it classifies an anomalous behaviour
as normal behaviour, then it means that NUPIC fails to detect the anomaly and
this attempt is classified as False Negative. Eventually, if the service classifies
the data as anomalous behaviour but the data actually corresponds to a nor-
mal behaviour, then this attempt is considered a False Positive (false alarm).
Both True Positive and False Positive are important benchmarks to measure the
accuracy of the intrusion detection mechanism. Table 1 summarizes the confu-
sion matrix emerged during our experiment and it provides a better pictures of
the accuracy of the anomaly detection algorithm. The confusion matrix will be
used to calculate the model detection rate, false positive, and accuracy, which
achieves the second objective of the evaluation (criterion 3).

Table 1. Results of the proposed anomalies detection model on confusion matrix

X = 1528 Predicted anomalies Predicted normality

Actual anomalies (TP + FN) (55) TP = 49 FN = 6
Actual normalies (FP + TN) (1473) FP/ False Alarm = 38 TN = 1435

The true positive rate (TPR), sensitivity or recall is 49/(49 + 6) = 89%, the
false positive rate (FPR) is 38/1473 = 2.5%, the false negative rate (FNR) is
6/(49 + 6) = 11% and the true negative rate (TNR), specificity or selectivity
is 1435/1473 = 97.5%. The precision (PPV) of the algorithm is 49/(49 + 38)
= 56.3% while its false omission rate (FOR) is 6/6 + 1435 = 4.4%. The false
discovery rate (FDR) is 38/49 + 38 = 43.7% while its negative predicted value
(NPV) is 1435/6 + 1435 = 99.5%. Thus the overall accuracy of the algorithm is
(49 + 1435)/(49 + 38 + 6 + 1435) = 97.1%. Since the actual tests is unbalanced,
having more normalies than anomalies, then the balanced accuracy is computed:
(0.89 + 0.975)/2 = 93.25%. Overall, the model has a low precision score (PPV)
and an high recall score (TPR). This means that the model has a moderate
degree of exactness but a high completeness.
 A Self Healing Microservices Architecture 857

4 Conclusions and Future Work

This model manages to offer the Microservices architecture with continuous mon-
itoring, continuous detection of anomalous behaviour, and provides the archi-
tecture with dynamic decision making based on the employment of multidimen-
sional utility-based model. The results in above, shows high accuracy in detecting
the anomaly and an accurate calculation of variant adaptation actions. It Also
shows high success rate in performing horizontal and vertical scaling in response
to contextual changes. The uses of utility-based model enables the architecture
to dynamically elect a reasoning approach based on the highest utility dimension
of context changes in the operational environment.

References
1. Ahmad, S., Lavin, A., Purdy, S., Agha, Z.: Unsupervised real-time anomaly detec-
tion for streaming data. Neurocomputing 262(Suppl. C), 134–147 (2017)
2. Anderson, D., Frivold, T., Valdes, A.: Next-generation intrusion detection expert
system (NIDES): a summary. SRI International, Computer Science Laboratory
Menio Park, CA (1995)
3. Buczak, A.L., Guven, E.: A survey of data mining and machine learning methods
for cyber security intrusion detection. IEEE Commun. Surv. Tutor. 18(2), 1153–
1176 (2016)
4. Cheng, B., de Lemos, R., Giese, H., Inverardi, P., Magee, J., Malek, R.M., Müller,
H., Park, S., Shaw, M., Tichy, M.: Software engineering for self-adaptive systems: a
research road map (draft version). In: Dagstuhl Seminar Proceedings 08031 (2008)
5. Cheng, S.W., Garlan, D., Schmerl, B.: Evaluating the effectiveness of the rainbow
self-adaptive system. In: ICSE Workshop on Software Engineering for Adaptive
and Self-Managing Systems, SEAMS 2009, pp. 132–141. IEEE (2009)
6. Cheung-Foo-Wo, D., Tigli, J.Y., Lavirotte, S., Riveill, M.: Self-adaptation of event-
driven component-oriented middleware using aspects of assembly. In: Proceedings
of the 5th International Workshop on Middleware for Pervasive and Ad-Hoc Com-
puting: Held at the ACM/IFIP/USENIX 8th International Middleware Confer-
ence, pp. 31–36 (2007)
7. Craig, J.W.: A new, simple and exact result for calculating the probability of error
for two-dimensional signal constellations. In: Conference Record, Military Commu-
nications in a Changing World, Military Communications Conference, MILCOM
1991, pp. 571–575. IEEE (1991)
8. De Lemos, R., Giese, H., Müller, H.A., Shaw, M., Andersson, J., Litoiu, M.,
Schmerl, B., Tamura, G., Villegas, N.M., Vogel, T., et al.: Software engineering
for self-adaptive systems: a second research roadmap. In: Software Engineering for
Self-Adaptive Systems II, pp. 1–32. Springer (2013)
9. Fishburn, P.C., Kochenberger, G.A.: Two-piece von Neumann-Morgenstern utility
functions. Decis. Sci. 10(4), 503–518 (1979)
10. Golmah, V.: An efficient hybrid intrusion detection system based on C5.0 and
SVM. Int. J. Database Theory Appl. 7(2), 59–70 (2014)
11. Haq, N.F., Onik, A.R., Hridoy, M.A.K., Rafni, M., Shah, F.M., Farid, D.M.: Appli-
cation of machine learning approaches in intrusion detection system: a survey.
IJARAI-Int. J. Adv. Res. Artif. Intell. 4(3), 9–18 (2015)
858 B. Magableh and M. Almiani

12. Hawkins, J., Blakeslee, S.: On Intelligence. Macmillan, London (2007)

13. Hirschfeld, R., Costanza, P., Nierstrasz, O.M.: Context-oriented programming. J.
Object Technol. 7(3), 125–151 (2008)
14. Horn, P.: Autonomic computing: IBM’s perspective on the state of information
technology. Technical report (2001)
15. Kakousis, K., Paspallis, N., Papadopoulos, G.A.: Optimizing the utility function-
based self-adaptive behavior of context-aware systems using user feedback. In:
OTM Confederated International Conferences “On the Move to Meaningful Inter-
net Systems”, pp. 657–674. Springer (2008)
16. Kohavi, R., Provost, F.: Confusion matrix. Mach. Learn. 30(2–3), 271–274 (1998)
17. Lavin, A., Ahmad, S.: Evaluating real-time anomaly detection algorithms–the
Numenta anomaly benchmark. In: 2015 IEEE 14th International Conference on
Machine Learning and Applications, ICMLA, pp. 38–44. IEEE (2015)
18. Mikalsen, M., Paspallis, N., Floch, J., Stav, E., Papadopoulos, G.A., Chimaris, A.:
Distributed context management in a mobility and adaptation enabling middleware
(MADAM). In: Proceedings of the 2006 ACM Symposium on Applied Computing,
pp. 733–734. ACM (2006)
19. Mishra, A., Nadkarni, K., Patcha, A.: Intrusion detection in wireless ad hoc net-
works. IEEE Wirel. Commun. 11(1), 48–60 (2004)
20. Ongaro, D., Ousterhout, J.K.: In search of an understandable consensus algorithm.
In: USENIX Annual Technical Conference, pp. 305–319 (2014)
21. Phua, C., Lee, V., Smith, K., Gayler, R.: A comprehensive survey of data mining-
based fraud detection research. arXiv preprint arXiv:1009.6119 (2010)
22. Roesch, M., et al.: Snort: lightweight intrusion detection for networks. In: LISA,
vol. 99, pp. 229–238 (1999)
23. Salehie, M., Tahvildari, L.: Self-adaptive software: landscape and research chal-
lenges. Trans. Auton. Adapt. Syst. (TAAS) 4(2), 14 (2009)
24. Sama, M., Rosenblum, D.S., Wang, Z., Elbaum, S.: Model-based fault detection in
context-aware adaptive applications. In: Proceedings of the 16th ACM SIGSOFT
International Symposium on Foundations of software engineering, pp. 261–271.
ACM (2008)
25. Sterritt, R., Bustard, D.: Towards an autonomic computing environment. In: Pro-
ceedings of the 14th International Workshop on Database and Expert Systems
Applications, pp. 694–698. IEEE (2003)
26. Strang, T., Linnhoff-Popien, C.: A context modeling survey. In: Workshop on
Advanced Context Modelling, Reasoning and Management, UbiComp, vol. 4, pp.
34–41 (2004)
27. Stubbs, J., Moreira, W., Dooley, R.: Distributed systems of microservices using
docker and serfnode. In: 2015 7th International Workshop on Science Gateways,
IWSG, pp. 34–39. IEEE (2015)
28. Wei, W., Fan, X., Song, H., Fan, X., Yang, J.: Imperfect information dynamic
stackelberg game based resource allocation using hidden Markov for cloud com-
puting. IEEE Trans. Serv. Comput. 11(1), 78–89 (2016)
 Transparent State Machine Replication
for Kubernetes

Felipe Borges, Luis Pacheco, Eduardo Alchieri(B) , Marcos F. Caetano,

and Priscila Solis

Department of Computer Science, University of Brasilia, Brası́lia, Brazil

[email protected], [email protected],
{alchieri,mfcaetano,pris}@unb.br

Abstract. State Machine Replication (SMR) is an approach widely

used to implement fault-tolerant systems. In this approach, servers are
replicated and client requests are deterministically executed in the same
order by all replicas. Virtualization can be seen as a technique that favor
development of fault-tolerant applications, since it provides an architec-
ture that isolate virtual machines or containers. In order to provide sup-
port to the development of fault-tolerant virtualized applications, this
work proposes an architecture to provide SMR for applications virtu-
alized in containers managed by Kubernetes. Transparency is the main
design principle addressed by the proposed architecture: applications are
still being developed as in the traditional non-replicated approach and
end users also access the system as in the traditional way. The open-
source Bft-SMaRt SMR library was used to implement a prototype
of the proposed architecture and a key-value store service. Experiments
conducted with this service show the practical behavior of the proposed
solutions.

1 Introduction
State Machine Replication (SMR) [20] is a well-known approach for implement-
ing fault-tolerant systems [5,16,20]. Basically, in this technique servers are repli-
cated and execute the same set of operations requested by clients determinis-
tically and at the same order, which provides a replication system with strong
consistency (linearizability) [12]. In order to keep the execution determinism,
operations are ordered and sequentially executed following the same order across
the replicas.
Virtualization emerged between 1960 and 1970, by IBM, which developed
operational systems with support for virtualization [9,10]. After that, hardware
became less expensive and virtualization became less important in the 80’s. In the
90’s, the utilization of virtualization returned mainly due the Java programming
language. Virtualization can be seen as a technique that favor development of
fault-tolerant applications, since virtual machines (or containers) are isolated
from each other, and thus are being used for this purpose [8,14,17,19]. However,
the main goal of these works is only the integration of consensus protocols (e.g.:
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 859–871, 2020.
https://doi.org/10.1007/978-3-030-15032-7_72
860 F. Borges et al.

Raft [13]) in a virtualized environment, without considering other aspects of the

SMR such as recovery and state transfer. Moreover, these proposed architectures
do not make the aspects of replication transparent in the system.
Aiming towards a comprehensive SMR solution for virtualized environments,
this work proposes a principled architecture for integrating SMR protocols into
the container orchestrator Kubernetes. Kubernetes [1] is an open-source con-
tainer orchestrator developed by Google, providing all the necessary functional-
ity to create and manage containers (as Docker containers [6]). The proposed
architecture aims to make this integration transparent both for users and appli-
cation developers, that is, applications are developed to execute in a container
as in the traditional non-replicated system, and all replication is hidden from
end users accessing the system.
More precisely, the paper makes the following contributions:

• It proposes a principled architecture for integrating SMR protocols in the

Kubernetes containers orchestration. The main design principle followed by
the proposed architecture is transparency.
• It uses the BFT-SMaRt [3] library to prototype the proposed architecture
and implement a key-value store service. BFT-SMaRt [3] is a complete SMR
library for the development of fault-tolerant applications, providing support
for recovery, reconfiguration, state transfer, among others SMR protocols.
• It presents and analyses several experiments, bringing better understanding
about the practical behavior of the proposed architecture.
The remaining of this paper is organized as follows. Section 2 presents the
theoretical basis for this work, addressing the concepts of virtual machines and
containers, as well as explaining the Kubernetes orchestrator and discussing the
concepts involving SMR and the BFT-SMaRt library. Section 3 presents the
proposed architecture for integrating SMR into Kubernetes. Section 4 presents
and analyzes some experiments. The conclusions of the paper are presented in
Sect. 5.

2 Theoretical Concepts and Background

This section presents some important concepts that forms the basis for this work.

2.1 Virtual Machines and Containers

A bare metal machine is made up of several physical components, which provide

an interface to the operating systems and their applications. A virtual machine is
nothing more than a layer of virtualization (software) built on this interface and
that provides a new interface for the other components of the system. A virtual
machine can also be defined as an isolated and protected copy of an bare metal
machine [10]. A virtual machine consists of three parts: the host system - which
encompasses real hardware and software resources; the virtualization layer, or
 Transparent State Machine Replication for Kubernetes 861

the supervisor/monitor system - which provides the virtual interfaces from the
real interface; and the guest system - which runs on the virtualized system.
Supervisor systems usually have the following properties, which can be used
by applications for security and fault tolerance: equivalence - the supervisor-
provided environment is almost identical to that of the bare metal machine;
resource control - the supervisor has complete control of the bare metal machine
resources; efficiency - most of the virtual processor instructions should run
directly on the actual processor, without the supervisor’s interference; isola-
tion - what runs in a virtual machine is isolated from the rest of the system;
inspection - the supervisor has access and control over the internal state of the
virtual machine; manageability - the supervisor manages resources used by guest
systems; encapsulation - the supervisor can make checkpoints of guest applica-
tions/systems to allow recovery, migration and inspection in case of failures;
recursion - it is possible to run a supervisor within a virtual machine, producing
a new virtualization layer.
Containers (operating-system-level virtualization or containerization) emer-
ged as an alternative to complete virtual machines. Containers is the name given
to an isolated, manageable, and with layers of file environment that can be
created over an operating system. Container creation is very fast and resource
provisioning is more efficient when compared to virtual machines. In this work
the container-based virtualization system known as Docker [6] is used.
There are fundamental differences in the architecture of a complete virtu-
alization system and a container-based virtualization system. In the complete
virtualization architecture, for each application to be executed in isolation, it
is necessary to install a complete operating system. In this way, more resources
are necessary when compared to the container approach. In this approach, the
container management system is the mediator in the use and access of the
computational resources, not requiring the installation of a complete operating
system [18].

2.2 Kubernetes
The growing demand for containerization solutions made it necessary to create
software that facilitates the management of containers. These software, called
container orchestrators, are intended to automate deployment, upgrade, access
control tasks and provide some features such as scalability, fault tolerance, and
load balancing.
Kubernetes [1] is a container orchestrator created from the experience
obtained by Google in other systems, such as Borg [4] and Omega [21]. Kuber-
netes is a software that manages distributed containers on a cluster of heteroge-
neous machines, which are divided between master and nodes. The master is the
machine that makes cluster services available through an API and is responsible
for coordinating its activities, for example performing tasks such as scheduling
and updates. On the other hand, nodes are the cluster workers and run the con-
tainers that provide their services. When a new service is created in the cluster,
862 F. Borges et al.

the master instantiates containers in the nodes, in units called PODs, taking
into account factors such as hardware requirements of the application and the
current load of each node.
PODs. PODs are composed by a set of highly coupled containers. A POD can
be seen as a recipe that holds groups of containers that must work together
to deliver a service. Considering a Web server that needs a database, a POD
may contain two containers: one for the Web server and one for the database
server. Containers inside a POD are viewed externally as a single entity. They
share the same network domain, having the same IP address and sharing the
entire set of ports, as if they were a single container or machine. This allows
containers of the same POD to access each other through the loopback interface
(localhost). Moreover, containers inside the same POD have access to shared
volumes associated to the POD and their processes can communicate through
Inter-Process Communication mechanisms.
Deployments. Through deployments it is possible to create, manage and mon-
itor identical POD sets to provide the same service. Using deployments, services
become elastic (being able to increase or decrease according to the demand),
updates can be applied on a large scale and the service load can be distributed
over multiple PODs. At the moment of creating a deployment, some deployment
parameters are defined, which include: number of PODs that must be created;
minimum number of PODs that must be available at a given moment (at running
time); number of PODs that can be instantiated beyond the defined maximum
number (used in special cases, such as during updates); and how many versions
should be retained to allow rollback.
Services. PODs created in a Kubernetes cluster only can be accessed by pro-
cesses inside the cluster itself. Services are used to make an application visible
externally. Although it seems like no purpose to make PODs visible only inside
the cluster, this allows changes to occur transparently to the users. For example,
the amount of PODs can be increased or decreased, load balancing can be done
or even updates can be performed without the user noticing any interference in
the service.
Services can be exposed in four different ways: ClusterIP – when PODs are
exposed in this way, they receive an internal IP of the cluster and are only
visible within it; NodePort – the service is exposed through a port available on
the master and on the cluster nodes (in this case, the service is accessed through
the IP of the node and its port); LoadBalancer – the service is exposed through
an external load balancer provided by cloud services; and ExternalName – maps
the service through a name, such as service.cluster.com.

2.3 State Machine Replication

State Machine Replication (SMR) consists of replicating the servers and coor-
dinate the interactions between clients and these replicas, with the objective
of having the same evolution in their states. The basic programming model of
 Transparent State Machine Replication for Kubernetes 863

an SMR involves two primitives: invoke(operation) - used by clients to invoke

operations on the service implemented by the SMR; and execute(operation) -
implemented by replicated servers, being used whenever an operation must be
executed by the SMR.
The invoke(operation) primitive is used to send an request (operation) to the
replicas, which execute the execute(operation), processing the request (operation)
and returning the response to be sent to the client. The implementation of these
primitives must meet the following requirements, which define the determinism
of the replicas:

1. Initial state: all correct replicas must start from the same state.
2. Determinism: all correct replicas, by executing the same operation on the
same state, produces the same change in their states and the same response
to the client.
3. Coordination: all correct replicas execute the same sequence of operations.

To guarantee item (1) it is enough to start all replicas with the same state
(e.g., start all variables that represent the state with the same values at all
replicas). To provide item (2), operations performed by the servers must be
deterministic. Finally, for the replicas to execute the same sequence of operations,
ensuring item (3), it is required the use of an atomic broadcast protocol [11].
Assuming the above three requirements are satisfied, an SMR implementation
must satisfy the following security and termination properties:

• Security: all correct replicas execute the same sequence of operations.

• Termination: all correct client operations are executed.

The security property enables the implementation of strong consistency,

known as linearizability [12]. The termination property ensures that the oper-
ations invoked by correct clients are executed and will finalize. The proposed
protocols for SMR limit the number of failures tolerated in the system, requir-
ing n ≥ 2f + 1 (crash model) [16] or n ≥ 3f + 1 (Byzantine model) [5] replicates
to tolerate up to f failures. Moreover, it is necessary a partially synchronous
system [7] for termination [5,15,20]. The idea behind these models is that
the system works asynchronously (not respecting any time limit) most of the
time. However, during periods of stability, the time for message transmissions is
limited.

2.3.1 BFT-SMART: A SMR Library

Bft-SMaRt [3] is a open-source library for implementation of fault-tolerant
applications following the SMR paradigm [20]. Initially developed to tolerate
Byzantine failures, the system can be configured to tolerate only crashes (as
used in this work). This replication library was developed in Java and implements
protocols for reconfiguration and state management (checkpoints, update, and
state transfer). The interested reader can find a more detailed description of
Bft-SMaRt in [3]. Briefly, each replica performs the following tasks:
864 F. Borges et al.

• Receiving requests. Clients send their requests to the replicas, which store
them in different queues for each client. The system can be configured to
ensure the authenticity of requests by using digital signatures (in this case
clients must sign their requests). In this way, any replica can verify the authen-
ticity of the requests and a proposal for ordering, which contains the request
to be ordered, is only accepted by a correct replica after the authenticity of
this request is verified.
• Ordering requests. Whenever there are requests to be executed, an atomic
broadcast protocol is executed to define a delivery order. Bft-SMaRt uses
batches of requests in the ordering protocols to improve system performance.
• Executing requests. When the execution order of a batch of requests is
defined, this batch is added in a queue and then delivered to the application.
After processing each request, a response is sent to the client.

Using BFT-SMaRt for programming a fault-tolerant application is quite

simple. The basic API for clients and servers provides methods for the client to
invoke the replicated service and a kind of callback for servers to execute these
requests/operations (representing the invoke and execute primitives previously
described). To access the replicated service, a Bft-SMaRt client should only
instantiate a ServiceProxy class by supplying its identifier (integer) and a con-
figuration file containing the address (IP and port) of each server. After this,
whenever the client wants to send a request to the replicas (servers), it must
invoke the invokeOrdered method specifying the request. On the other hand,
to implement the server, each replica must extend the Executable interface and
implement the abstract method executeOrdered, that is invoked when a request
must be executed. In addition, it is necessary to instantiate a ServiceReplica
that represents the replica, supplying the identifier of the replica (integer) that
is mapped to a port and IP address through a configuration file. A detailed
description of this API can be found at [3].

3 Proposed Architecture
The main principle followed by the proposed architecture is transparency: clients
and services must be implemented and executed as in a traditional system with-
out replication. Moreover, by using the facilities provided by Kubernetes (e.g.,
PODs), the architecture aims to automate the implementation of infrastructure
necessary to promote the use of SMR through the weak coupling between their
protocols and the user application. It is important to note that this separation of
SMR protocols logic and application code allows any application, written in any
language, without the need of any specific library, to use the solution without
any modification. In order to make this type of service available, the proposed
architecture benefits from the existing POD features in Kubernetes, which trans-
parently couple multiple containers. It is important also to notice that in this
work we are considering a system model subject only to crash failures. The SMR
library that we used to prototype the system (BFT-SMaRt) can be configured
to tolerate only crash failures.
 Transparent State Machine Replication for Kubernetes 865

Fig. 1. Proposed architecture.

Overview. The proposed architecture, depicted in Fig. 1, is composed by three

main components: Proxy, No SMaRtClient and No SMaRtServer. There is no
restriction on the place of execution of these components, e.g., they can be
running on the same or different servers and subnets. The green containers are
introduced in our architecture to intercept requests (SMaRtClients), execute
the SMR protocols (SMaRtServers) and forward the requests to the application
(SMaRtServers). The red containers are normal containers executing the service
application and are programmed as in the traditional system without replication.
The Proxy service runs on the Kubernetes master node and is responsible for
receiving external requests to the architecture (coming from application clients)
and forwarding them internally to the SMaRtClients (which are running on
nodes called No SMaRtClient). Distribution of requests is done in a balanced
way among all instantiated SMaRtClients. Upon receiving a request, the selected
SMaRtClient encapsulates this request and forwards it to all SMaRtServers using
SMR protocols. Each POD instantiated in a No SMaRtServer consists of two
containers: one running the SMaRtServer and one running an instance of the
virtualized application.
After executing the SMR ordering protocols (at the time of processing a
request), each SMaRtServer forwards the requests to its respective application
container (the other container at the same POD). The responses, in turn, follow
the inverse path: the application responds to SMaRtServer, which forwards to
866 F. Borges et al.

the SMaRtClient, which in turn will receive the response of all SMaRtServers
and will decide which message is correct through the SMR protocols, and then
forward it to the client via proxy. It is important to note that Kubernetes has
solutions to handle proxy failures. Moreover, to maintain a high degree of trans-
parency for applications, SMaRtServers open a connection with the respective
application containers for each request to be executed. Since the nature of the
transmitted data is unknown, SMaRtServers do not know when the complete
response has been received unless the application containers close the TCP con-
nection after the response has been sent.
Dealing with Faulty SMaRtClients. In this work we are only considering
crash failures. While the SMR protocols are responsible for masking faults occur-
ring on servers (No SMaRtServers), no solution is provided for No SMaRtClients
that could fail during execution and not forward requests or responses to
servers or client, respectively, leading to a incomplete request execution. Conse-
quently, the following solutions must be incorporated in the system to solve these
problems.

• Incomplete requests solution: to solve this problem, a timer is associated

with each request sent by the client. If a timeout occurs before the response
is obtained, the client sends the request again and resets the timeout, until
a response is received. By resubmitted requests in the system, Kubernetes
eventually removes the failed component and choose (through the load bal-
ancer) a correct SMaRtClient to execute the request. The complete solution
to this problem involves combining the mechanism described above with the
duplicate request removal described below.
• Duplicate requests solution: request retransmissions enables the servers
(SMaRtServers) to receive the same request more than once. In this case,
only one request must be executed and the other must be eliminated to keep
system consistency. For this to be possible, the client must identify each
request with a unique identifier, through its identity and a sequence number,
in a way that servers can verify if such request can be executed. To this end,
each SMaRtServers keeps a buffer where responses corresponding to the last
request of each client are stored. Thus, a request is forwarded to the appli-
cation container only if it has a sequence number one unit larger than the
request whose response is in the buffer. If the request received by the SMaRt-
Servers has the same identifier as the last one executed for this client, the
response stored in the buffer is sent to the requesting SMaRtClient. In any
other case the request can be discarded since the client already received a
response for it. Notice that in order to this mechanism work properly, a client
can not send a request without the previous one being fully completed. In
practice, this threshold can be relaxed to k requests if the server stores the
last k responses for each client.

Finally, it is worth notice that the state transfer and recovery protocols used
in the SMR [3] could be directly implemented in the proposed architecture by
 Transparent State Machine Replication for Kubernetes 867

taking the application container image as the current application state used by
these protocols.

4 Experiments

In order to evaluate the performance of the proposed architecture, as well as

the behavior of SMR running with Kubernetes, we performed some experiments
considering two different scenarios. In the first one, we evaluate the performance
impact caused by the number of SMaRtClients, since this intermediary layer
can be a bottleneck because SMaRtClients execute the Bft-SMaRt protocols
to forward to the SMaRtServers the requests coming from the external clients.
For the execution of this scenario, the number of SMaRtServers was fixed in 3
(to tolerate one server failure since we consider only crash failures and in this
case n ≥ 2f + 1) and the number of SMaRtClients was varied with the following
values: 1, 2, 4, 8, 16 and 32.
The second scenario evaluates the system when there is variation in the num-
ber of PODs containing the two containers: the SMaRtServer and the application
(see Fig. 1). For simplicity, we call these PODs as SMaRtServers only. The num-
ber of SMaRtServers impacts the degree of fault tolerance of the system (more
servers means that more failures are tolerated), but increasing this number also
leads to an increase in messages exchanged in the system (the message com-
plexity for SMR ordering protocols is O(n2 )), which impacts the performance.
For this scenario, the number of SMaRtClients was fixed in 16 (the number
necessary to avoid that this layer becomes a bottleneck), while the number of
SMaRtServers was varied with the following values: 3, 5, 7, 9 and 11, to tolerate
1, 2, 3, 4 and 5 failures, respectively.
Application. We implemented a key-value store (KV-Store) application for
the experiments. This application supports operations such as [2]: put(K, V ) –
stores value V associated with key K; get(K) – gets the value associated with
K; and remove(K) – remove K and its associated value from the store. In the
experiments, we consider keys and values as Integer objects to determine the
latency and throughput for the operations. Latency was measured in the clients
and the reported values represent a 100k-execution average, excluding 5% with
higher deviation. The throughput shown is the peak reached by the servers,
measured at the server that was the consensus leader [3], at every 5k requests
execution. We show results only for put and get operations since they are similar
for all considered operations.
Experimental Setup. The validation of the proposed architecture was con-
ducted in an environment composed of three homogeneous servers used in an
isolated network. The servers have the following configuration: two Intel Xeon
E5620 processors, 6 GB RAM (1067 MHz), Ethernet 100 Mbps, Ubuntu Server
LTS 16.04.3, Kubernetes 1.8.4 and BFT- SMaRt v1.1-beta. One of the servers
was used to host the containers with No SMaRtServers, one for containers with
No SMaRtClients and the last one for the external clients.
868 F. Borges et al.

Results and Analysis. Figure 2 displays the values of throughput when vari-
ating the amount of SMaRtClients. Given the approximate computational cost
in performing all KV-Store operations, the graphs show the same behavior and
almost no variation between put and get operations. It is possible to verify that
the increase in the number of SMaRtClients has an impact on throughput, which
increases when the number of these nodes increases up to 16. From this point,
the increase in number of SMaRtClients does not translate into the increase
of throughput since this layer no longer represents the bottleneck of the sys-
tem. The number of parallel (external) clients is another point to be observed,
because the increase in their number reflects in the increase of load submitted
to the system. Saturation is achieved, in all operations, when the number of
external clients reaches approximately 20.
Figure 3 shows latency values for the first scenario. Increasing the number
of SMaRtClients reduces the latency. Similar to the results for throughput, this
improvement is achieved up to 16 SMaRtClients. The obtained results suggest
that when the number of SMaRtClients is 16, this layer ceases to be the bottle-
neck.

Fig. 2. Throughput for a system with 3 SMaRtServers.

Fig. 3. Latency for a system with 3 SMaRtServers.

 Transparent State Machine Replication for Kubernetes 869

Figure 4 shows throughput values for the second scenario, where we set the
number of SMaRtClients to 16, since the first scenario defined that this num-
ber os enough to avoid them to become a system bottleneck. As expected, the
increase in the number of SMaRtServers leads to a throughput decrease. This
behavior is related to the nature of SMR protocols: a larger number of SMaRt-
Servers implies a higher number of messages exchanged. However, it is worth
noting that with more servers, more failures are also tolerated. Finally, Fig. 5
shows latency values. It is possible to observe that the increase in the number
of SMaRtServers also entails in the increase of the latency, presenting a similar
behavior to throughput.

Fig. 4. Throughput for a system with 16 SMaRtClients.

Fig. 5. Latency for a system with 16 SMaRtClients.

5 Conclusions

This work presented our efforts to introduce fault tolerance support in virtu-
alized environments by proposing and architeture to integrate SMR protocols
into the Kubernetes container orchestrator. The resulting architecture provides
870 F. Borges et al.

a high degree of transparency for both application developers and end users.
This transparency is desired to prevent developers from having to understand
and spend time in aspects related to replication. Consequently, developers can
focus on the services inherent complexity. Moreover, the transparency provided
by the architecture also makes feasible its use to provide fault tolerance to legacy
systems.

Acknowledgements. This work was partially supported by RNP/CTIC (Brazil)

through projects ATMOSPHERE and P4Sec.

References
1. Bernstein, D.: Containers and cloud: from LXC to docker to kubernetes. IEEE
Cloud Comput. 1(3), 81–84 (2014)
2. Bessani, A., Santos, M., Felix, J., Neves, N., Correia, M.: On the efficiency of
durable state machine replication. In: Proceedings of the USENIX Annual Techni-
cal Conference (2013)
3. Bessani, A., Sousa, J., Alchieri, E.: State machine replication for the masses with
BFT-SMaRt. In: Proceedings of the International Conference on Dependable Sys-
tems and Networks (2014)
4. Burns, B., Grant, B., Oppeheimer, D., Brewer, E., Wilkes, J.: Lessons learned from
three container-management systems over a decade: borg, omega, and kubernetes.
ACM Queue Mag. 14, 70–93 (2016)
5. Castro, M., Liskov, B.: Practical Byzantine fault-tolerance and proactive recovery.
ACM Trans. Comput. Syst. 20(4), 398–461 (2002)
6. Docker: What is docker (2018). https://www.docker.com/what-docker. Accessed
March 2018
7. Dwork, C., Lynch, N.A., Stockmeyer, L.: Consensus in the presence of partial
synchrony. J. ACM 35(2), 288–322 (1988)
8. Garfinkel, T., Rosenblum, M.: A virtual machine introspection based architecture
for intrusion detection. In: Network and Distributed Systems Security Symposium
(2003)
9. Goldberg, R.P.: Architecture of virtual machines. In: Proceedings of the Workshop
on Virtual Computer Systems (1973)
10. Goldberg, R.P., Mager, P.S.: Virtual machine technology: a bridge from large main-
frames to networks of small computers. In: Proceedings of the Compcon Fall (1979)
11. Hadzilacos, V., Toueg, S.: A modular approach to the specification and imple-
mentation of fault-tolerant broadcasts. Technical report, Department of Computer
Science, Cornell (1994)
12. Herlihy, M., Wing, J.M.: Linearizability: a correctness condition for concurrent
objects. ACM Trans. Program. Lang. Syst. 12(3), 463–492 (1990)
13. Howard, H., Schwarzkopf, M., Madhavapeddy, A., Crowcroft, J.: Raft refloated: do
we have consensus? ACM SIGOPS Oper. Syst. Rev. 49(1), 12–21 (2015)
14. Jiang, X., Wang, X.: “out-of-the-box” monitoring of VM-based high-interaction
honeypots. In: 10th International Conference on Recent Advances in Intrusion
Detection (2007)
15. Lamport, L., Shostak, R., Pease, M.: The Byzantine generals problem. ACM Trans.
Program. Lang. Syst. 4(3), 382–401 (1982)
 Transparent State Machine Replication for Kubernetes 871

16. Lamport, L.: The part-time parliament. ACM Trans. Comput. Syst. 16(2), 133–169
(1998)
17. Laureano, M., Maziero, C., Jamhour, E.: Intrusion detection in virtual machine
environments. In: Proceedings of 30th Euromicro Conference (2004)
18. Merkel, D.: Docker: lightweight linux containers for consistent development and
deployment. ACM Linux J. 2014(239), 1–8 (2014)
19. Oliveira, C., Lung, L.C., Netto, H., Rech, L.: Evaluating raft in docker on kuber-
netes. In: Świa̧tek J., T. J. (eds.) International Conference on Systems Science
(ICSS). Advances in Intelligent Systems and Computing, vol. 539, pp. 123–130.
Springer (2016)
20. Schneider, F.B.: Implementing fault-tolerant service using the state machine
aproach: a tutorial. ACM Comput. Surv. 22(4), 299–319 (1990)
21. Schwarzkopf, M., Konwinski, A., Abd-El-Malek, M., Wilkes, J.: Omega: flexible,
scalable schedulers for large compute clusters. In: European Conference on Com-
puter Systems (2013)
 An Optimal Travel Route Recommendation
System for Tourists’ First Visit to Japan

Chen Yuan(&) and Minoru Uehara

Graduate School of Information and Science and Arts, Toyo University,

Kawagoe, Japan
{s3b10180023,Uehara}@toyo.jp

Abstract. As the Tokyo Olympics approach, increasing numbers of foreigners

are planning to travel to Japan. It is not easy for these tourists to make travel
plans by searching for information about unfamiliar Japanese sightseeing spots
on the Internet. In particular, it is difficult to search for the relevant tourism
information, such as names, categories, or open ratings. In this study, we aim to
facilitate recommendations of sightseeing routes with high satisfaction. Tourists’
satisfaction is judged by various restrictions, such as travelers’ multiple pur-
poses, staying time at the tourism site, travel time, and purpose. According to the
users’ requirements, the optimal sightseeing route is selected by fitness function
score. Finally, we will take the recommended route for the user.

1 Introduction

The Japanese population is currently declining amidst a stagnant economy because of

the birthrate decline and an aging population. In addition, tourism requirements are
diversifying because tourists’ sightseeing preferences have changed from traveling in
groups to personal travel. In particular, we anticipate seeing an increase in foreign
tourists visiting Japan. Thus, the quality of sightseeing information recommendations is
very important, and a recommendation system for sightseeing spots and tourist routes is
necessary. The Japan Tourism Agency [1] observed that “for our country to grow
economically, the number of foreign tourists visiting Japan must increase. Therefore, to
expand foreign tourists’ domestic consumption while they visit Japan, it is important to
capture their vigorous international demand.” According to an investigation of the
behavioral characteristics of foreign tourists in 2017 [2], the numbers of foreign tourists
visiting Japan are rapidly increasing. In addition, the proportion of individual travelers
is increasing annually. Because of the decision to host the Tokyo Olympic Games in
2020, more foreign tourists will travel to Japan as individuals. However, personal travel
may not be easy for some foreign travelers because they may need to prepare their
journey itinerary before they travel by surveying information about a variety of tourism
sites. Therefore, the travel industry should provide a service for foreign tourists to
obtain their sightseeing plans more conveniently using a travel-recommendation
program.
One of the first online technologies was website recommendations, e.g., Amazon.
com, which are now very convenient for everyone. Linden et al. [3] observed that many

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 872–882, 2020.
https://doi.org/10.1007/978-3-030-15032-7_73
 An Optimal Travel Route Recommendation System 873

procedures on Amazon.com require users to enter explicit items to represent their

interests and attributes. These items are then used in the algorithm to recommend
related items to the user. The premise of the recommendation algorithm is that it
requires a lot of information based on the evaluations of thousands of users. The
algorithm uses traditional collaborative filtering methods to search for similar items and
recommend them to users. There are three types of recommendation websites. These
websites need to collect and label feature information, such as the users’ age, sex, and
preferences. Searches are based on this information and use an algorithm to identify
and recommend the items that match the users’ characteristics. Another recommen-
dation method is based on collaborative filtering, which differs from content recom-
mendation. The core idea of collaborative filtering is to use the current user’s historical
behavior instead of their feature preferences to produce recommendations. Most rec-
ommendation systems today are hybrid content–collaborative recommendation systems
that use a variety of recommendation algorithms. The hybrid recommendation strategy
could use such methods as weighted fusion, scene switching, result mixing and
reordering, feature combination, algorithm cascading, and algorithm element-level
fusion. Thorat et al. [4] found that most travel recommendation systems are now hybrid
recommendation systems.
The United States was one of the first countries to propose the concept of smart
tourism. Researchers have explored positioning systems and route planning to better
meet the intelligent needs of self-help tourists. For example, some studies explored the
route-planning problem for the shortest route from the departure point to the destination
using an enumerated search algorithm. Route planning in scenic tourism is a relatively
new research field and some recent research has explored travel route planning.
However, most existing route-planning algorithms cannot provide satisfactory travel
routes for their customers because of their personalized special requirements and the
wealth of information available on travel websites. If we want to make a reasonable
travel plan, we need to know the time, costs, visitor evaluations, and other information
about the sightseeing spot. When we are unable to determine the destination, it is
difficult to make a travel plan that suits the visitors’ travel time and preferences. Thus,
we considered recommending routes with higher satisfaction for travelers who are
planning to travel to Japan for the first time. However, it is possible to obtain a suitable
travel plan even if the user does not know any information, such as the place name,
attributes, or distance to the tourism site. Therefore, this study considers an algorithm to
solve this problem with the aim to recommend many kinds of purposes and satisfactory
travel plans for first-time foreign visitors to Japan, even in circumstances when these
users do not know anything about Japan. When the algorithm considers the tourism
sites using the users’ preferences, it can recommend information about the tourism site
route using the users’ input, including travel time, time at the tourism site, other users’
evaluations, and the users’ preferences for different tourism categories to recommend a
route with high satisfaction for individual users.
In this study, we first obtained information about the most popular sightseeing spots
in Tokyo from Foursquare, a social network service (SNS) [5]. Second, the collected
information about the sightseeing spots was entered into the program and all the
corresponding routes were arranged. Then, differences in fitness function were used to
select each route. Each route receives its own score based on its fitness function.
874 C. Yuan and M. Uehara

Finally, the three routes with the highest fitness function were recommended to the
user. The route with the highest user satisfaction was found using the fitness function
for each different restriction. The rest of this article is organized as follows. In Sect. 2,
we describe the problem and the related research. In Sect. 3, we explain the main idea,
including an analysis of the problem, solving steps, and algorithms. Our algorithm was
evaluated through experiments and the obtained results are summarized in Sect. 4.
Finally, Sect. 5 concludes this study and discusses future work.

2 Related Work

2.1 Tourist Recommendation System

Tourist recommendation systems (TRS) are very useful for travelers to search for
relevant travel information. Originally derived from information extraction and search
technologies, the TRS were very effective in the development of the tourism industry.
In 2003, the development of mobile recommendation systems became increasingly
vigorous and many computer-based recommendation systems became more powerful
after 2005. Thus, TRS has developed to a degree that allows tourists to use their
computers to organize their entire trip instead of to just obtain information about the
tourism sites. According to Nakajima et al. [6], many users post travel information
using services such as SNS because of the spread of smartphones. Therefore, we
utilized the location information from a SNS (i.e., Foursquare) to extract tourist
information and sightseeing routes. To collect sightseeing tweets and extract tourist
routes, the locations were classified into three categories: “meal,” “landscape,” and
“action.” This system used tourist tweets classified as extracted tourist routes to rec-
ommend tourism routes; however, the system is limited to recommending the sight-
seeing order at the sightseeing spot using the type of sightseeing spot. Thus, the system
cannot judge whether the visiting time at the tourism site matches the users’ tastes.
According to Arts et al. [7], by assuming a case where the weather can be predicted
probabilistically, a schedule tree should be identified that maximizes the total expected
user satisfaction values by using an approximation algorithm based on the greedy and
local search methods. In addition, we recommended schedules to user groups even
during arbitrary weather by considering three kinds of weather. However, this study did
not consider the users’ travel purposes. Therefore, we considered an algorithm based on
that approximation algorithm to solve the problem that aims to recommend many kinds
of purposes and satisfactory travel plans for first-time foreign visitors to Japan even in
circumstances when they do not know anything about Japan. When considering
tourism sites according to the users’ preferences, the information about the tourism
routes with the highest satisfaction are recommended following the users’ input
according to their travel time, visiting time at the tourism site, evaluations of the
tourism site, and the users’ preferences for different categories of tourism sites. First,
the most popular tourism sites in Tokyo were collected using a SNS, Foursquare [5],
which is used by most Japanese tourists to write information about the sites. The full
array algorithm was then used to organize all the tourism sites in addition to the routes
for each situation. The differences in fitness function were used to select all routes.
 An Optimal Travel Route Recommendation System 875

Finally, the three routes with the highest fitness function were recommended to the
users to identify the route with the highest satisfaction as shown by the fitness function
for each different restriction.

2.2 Foursquare
Kurata [8] defined tourist information as “communicative knowledge to guide human
judgment and behavior on tourism.” The tourism data used in this study were obtained
from Foursquare: i.e., information about the name of the tourism site, its user ratings,
opening and closing times, among other attributes. The travel times and distances to the
tourism sites were also obtained from Foursquare. Information about these tourism sites
were stored in a list and used to calculate the fitness function of the route and rec-
ommend the best travel plan to the user. The limitation of Foursquare is that it can be
used only in 100 global metropolitan areas; however, it can be used to search for spots
by category, such as gourmet food and shopping. Foursquare includes functions where
the user can rate and review their preferred tourism sites from 0 to 10 points. Four-
square can also be used to obtain the distance and movement time between the tourism
sites. Foursquare is used by Tokyo metropolitan area users to upload their favorite sites.
This study used Foursquare’s features to obtain information about the 11 most famous
tourism sites in Tokyo, Japan. These 11 tourism sites were sorted into different cate-
gories. Five more tourism sites were added for the subsequent tests; thus, the total
number of tourism sites used in this study was 16. Foursquare was used to obtain the
average movement time between these 16 spots.

2.3 Recommendation Technology

A sightseeing plan is the reasonable organization of the order of visiting tourism sites
during the travel period, including the time at the site. Each knowledge-based rec-
ommendation method typically includes case- and constraint-based recommendations.
The case-based recommendation method uses the similar existing solutions to solve the
new recommendation problem. Central to successful recommendation systems is to
solve the calculation of similarity. Thus, the project score information with high
similarity of the target project is used to predict the users’ score on the target project.
The system generally uses the user to select a project that they like or a similar project.
The basic idea is to first find the nearest neighbor of the target project with high
similarity and establish a project neighbor matrix to predict the current user’s score on
the target recommendation object according to the neighbor matrix of the current user’s
unrated project (target object). Several items with a high ranking in the predicted score
were then recommended to the current user. Commonly used similarity algorithms
include Euclidean distance, cosine similarity, and Pearson similarity. Euclidean dis-
tances are suitable for denser matrices. Cosine similarity is more suitable for calculating
sparse data. Cosine similarity uses the cosine between the angles of two vectors in a
vector space as a measure of the difference between two individuals. The cosine value
is close to 1 and the angle tends to 0, which indicates that the two vectors are more
similar. The cosine value is close to 0, and the angle tends to 90°, which indicates that
the two vectors are not similar. A constrained TRS effectively integrates tourism
876 C. Yuan and M. Uehara

resources to support their users to overcome their information overload by recom-

mending the appropriate travel service information. Tourism resources include struc-
tured tourism information, such as the departure times for the tourist route, scenic spots,
routes, hotels, and transportation. Tourist attributes include a description of visitors’
preferences and needs. This collection of constraints includes the constraints between
visitor and travel resource attributes, including recommendation and personalization
rules, and logical representations. Thus, we can consider the problem of making rec-
ommendations based on the constraint-based recommendation method as a constraint
satisfaction problem (CSP). A typical CSP can be represented by the element ancestor
(V, D, C), where V is a set of variables, D is a set of nonempty finite field values for
variables in V, and C is a collection of constraints. The CSP describes the combination
of values that can be given simultaneously by the variables in V. Many recommen-
dation problems can be simplified to identify multiple congregations of different
assignments of variables in V under a specific user’s required C congregation. Each
congregation of V variables corresponds to a valid recommendation.

3 System Design

Some specific conditions were set in this study; e.g., travel plans were recommended in
the form of a day trip. Therefore, a total of 16 tourism sites were obtained from
Foursquare and the data were categorized according to four categories: shopping, food,
viewing, and experience. Only one category attribute was available for each site, which
requires the users’ input to be consistent with the site’s category attribute. The users’
range of purpose was also categorized into these four categories. To pursue a more
humane and comfortable journey plan, this study proposed a search method using the
degree of purpose. Many previous route-recommendation studies required users to
input the travel destinations. In our research, however, we save the users’ time
searching for sightseeing spots by removing their need to input destinations. Thus, the
goal of this study is to recommend sightseeing routes for foreigners who are visiting
Japan for the first time. To minimize the user-entered information, it is only necessary
for the user to enter their travel time, i.e., the departure and return times, and the degree
of purpose. We decided to use the degree of purpose from the users’ input to select the
routes so that the selected spot would match the users’ preferences. The sum of the total
degree of purpose for the category attributes satisfies 1. This study proposed to use the
full array of algorithms to obtain the entire route congregation of the number of spots.
Differences in fitness function were used to screen out the optimal route with the
highest evaluation. Table 1 shows the site information obtained from Foursquare.
Table 1 shows all the route situations for 16 sightseeing spots in Table 1. Each
route obtained, its travel time, total cost, return to hotel time, rating, and gourmet spot
time are different. Therefore, we define the fitness function by this difference to cal-
culate the score for each route. Then we need to solve the calculation problem for
 An Optimal Travel Route Recommendation System 877

Table 1. Details of sightseeing spots.

Site Name Category Rating
1 Tokyo Tower Scenery 9.3
2 Tokyo Sky Tree Scenery 9.2
3 Sensoji Temple Scenery 9.4
4 Isetan Shinjuku Store Shopping 8.9
5 Tsukiji Fish Market Gourmet 9.2
6 Ginza Mitsukoshi Shopping 8.4
7 Harajuku Omotesando Hills Shopping 7.9
8 Uba Gourmet 8.8
9 Half-A-Minute Gourmet 9.5
10 Tokyo Disneyland Experience 9.5
11 Oedo Onsen Monogatari Experience 8.7
12 Roppongi Hills Scenery 9.0
13 Ryogoku Kokugikan Experience 8.5
14 Meiji Shrine Scenery 9.5
15 Wako Main Building Shopping 8.1
16 Harmonica Alley Gourmet 8.7

issues such as the fitness function algorithm, the difference in the degree of purpose, the
time-limit problem. Thus, we solve this problem using the following equations:
Xn
f ðp ; P Þ:
i¼1 i i i
ð1Þ

If T  Return Time  Start Time 1, Else 0 ð2Þ

Pn
i¼1 ðvi yi Þ
simðvi ; yi Þ ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
Pn qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð3Þ
2 Pn 2
i¼1 i ðv Þ i¼1 ðyi Þ

1 av rate  r ðtÞ  HOUR ðmÞ ðbÞ

weightðfitÞ ¼  þ þ ðvÞ þ þ þ ð4Þ
c num spot num spot HOUR HOUR

In the above formulae, Eq. (1) is the total score for each route, which is obtained by
summing the fitness function of individual routes. Equation (2) is a constraint for time,
i.e., in all routes, this route is not recommended if it does not match the time range
entered by the user. Equation (3) is the cosine similarity algorithm is used to calculate
the degree of similarity between the rate of the degree of purpose input by the user and
the degree of purpose for each route. Thus, the difference between them can be known.
If the cosine value is closer to 1, the angle is closer to zero and the result is the more
similar value between the two. Equation (4) is the calculation of the weighted average;
i.e., we assign weights to each attribute and use them to calculate the fitness function.
In this study, the weight score is subjectively defined by the individual. The reason for
selecting routes in this study is to use the different ratios of the different degrees of
purpose for each route to facilitate multiple options for tourism purposes. Therefore, the
878 C. Yuan and M. Uehara

weight of degree of purpose is the highest score among the defined weights (i.e., 600).
The degree of purpose is compared with the users’ input and optional routes. If the
difference between them is smaller, the fitness function score will be higher. In my
case, one of the most important conditions for screening tourist spots is the sightseeing
spot’s rating. Thus, the weight of the evaluation of the spot is set as 500. We then
considered that if the total rating score is high, its fitness function score is also high.
After that, the smaller the total cost, the more the added fitness function, otherwise the
score is reduced. The weight of the cost is 100. The smaller the total movement time
between the sites, points are added by the fitness function. Otherwise, the score is
reduced and the weight is 10. When the hotel time is closer to 22:00, the fitness
function adds points and the weight is 5. The closer the meal time is to 12:00, 15:00,
and 19:00, the fitness function adds points and the weight is 9. Finally, the total fitness
function score constrained by each attribute is obtained. The six programming steps are
described in the following:
1. Enter site data (including name, category, rating, cost, duration, move time).
2. Obtain user input requirements (including expected category vector, start time, end
time, number of sites).
3. Fully arranged route according to the number of sites.
4. Calculate the fitness function of various attributes and perform operations on cor-
responding weights.
5. Calculate the fitness function score for each route and finally obtain the three routes
with the highest scores.
6. When the process is terminated, three optimal routes are output with their corre-
sponding scores.
The following section analyses the results of the test output.

4 Evaluations

The correctness of the test results was evaluated. The total time for the tourism route
was set at 24 h. Travel times were calculated based on the user input start time minus
the return time; e.g., the departure and return times defined in this study were 8:00 and
22:00, respectively. There were five sightseeing spots and the starting hotel was located
at Haneda Airport. The first test used a total of 11 collections of tourism sites. Different
levels of purpose were entered into the system. The main purpose of this study is to
avoid users entering their own destinations; therefore, the tourism sites were identified
by the degree of purpose. Thus, the setting of the degree of purpose is very important.
In our test, we set the degree of purpose in the combinations of recommended routes as
shown in Table 2 as combinations for identical situations. A total of 55 cases were
measured according to the above process. The route that best met the users’ expecta-
tions accounts for about 37% of the total. Therefore, we analyzed why the program-
recommended route had many elements that did not meet the users’ expectations. We
defined the travel time as 1 day; i.e., the tour time is 16 h if we leave the hotel at 8:00
and return at 24:00. However, the duration of each experience at the tourism sites is 3
to 4 h and they are far apart, therefore, the movement time between the sites is
 An Optimal Travel Route Recommendation System 879

relatively long. If three routes can be found to connect the tourism sites, then the route
is not reasonable. Therefore, we removed all expected routes with more than three
tourism sites. Thus, the expected route satisfaction increased to 82%. The above sit-
uation shows that this system will filter out routes that cannot be implemented and the
recommended route is well-chosen and reasonable.

Table 2. Recommended route test.

Expected Recommended Route score Satisfaction Number of
purpose level route tourism sites
10, 0, 0, 0 1, 4, 0, 8, 2 −73.79419703103932 60 11
8, 2, 0, 0 1, 4, 0, 8, 2 395.109244264507 75 11
6, 0, 4, 0 2, 0, 1, 5, 3 430.988866396761 80 11
6, 2, 2, 0 1, 8, 0, 2, 3 443.619163292847 80 11
4, 4, 2, 0 0, 8, 3, 2, 4 449.269163292847 85 11
4, 2, 2, 2 2, 9, 0, 8, 3 396.159244264507 75 11
10, 0, 0, 0 2, 13, 0, 1, 11 424.167110799439 85 16
8, 2, 0, 0 2, 1, 13, 0, 8 454.493875642823 85 16
6, 4, 0, 0 13, 8, 0, 4, 2 470.943875642823 90 16
4, 4, 2, 0 13, 8, 2, 4, 3 483.522463768115 90 16
4, 2, 2, 2 2, 9, 13, 8, 3 439.655189340813 80 16

Table 2 shows the results of the recommended route test. First, we tested a total of
11 spots. In the case of the scenery, shopping, and food categories, there are two
tourism sites. Suppose you want to go to five sites. If the degree of purpose for the
scenery is 1, the other is 0. The result is then all the tourism sites featuring a landscape.
Other sites are then required because the total number of scenery sites is insufficient.
The chosen route also included gourmet and shopping categories. However, there are
restrictions on the increase in the score of gourmet spots at 12:00 and 19:00. Therefore,
the program-recommended route is Tokyo Sky Tree, Tsukiji Fish Market, Tokyo
Tower, Uba, and Sensoji Temple. Food sites were recommended to the users at areas
on the route near 12:00 and 19:00 by the program. In this study, the degree of purpose
setting is the most important. This route has a low degree of purpose; therefore, the
fitness function score is negative.
To test the correctness of the program results more clearly, we added five sites,
including two site categories: i.e., one gourmet, one shopping, and one experience. The
degree of purpose is still that scenery is 1, while the others categories are 0. In the case
of five scenery sites, the recommended route should be all scenery sites, i.e., Sensoji
Temple, Meiji Shrine, Tokyo Tower, Sky Tree, and Roppongi Hills. The degree of
purpose is identical to the users’ input; however, the fitness function score is still not
high. Although the program satisfies its most important conditional purpose, it also
includes the reason for the route condition with high satisfaction. There is no gourmet
spot during meal times, which will reduce the fitness function score. Therefore, we
tested the degree of purpose of 0.6, 0.4, 0, 0 and 0.4, 0.4, 2, 0, respectively. In the first
purpose, the scenery and food accounts for 6 and 4, respectively. The total number
880 C. Yuan and M. Uehara

from the 16 attractions on this distribution route is five sites; i.e., Tokyo Sky Tree,
Tsukiji Market, Tokyo Tower, Tsuji Han, and Sensoji Temple. This route not only
satisfies the distribution of the purpose, but also satisfies the meal-time conditions by
visiting the gourmet food sites. Therefore, the fitness function score increased. How-
ever, if we do not have a high requirement for restrictions on the available meal-time
conditions for gourmet food, then its weight is not high. Although the fitness function
score improved, it was not very obvious. This is followed by an allocation of 0.4, 0.4,
0.2, 0. At this degree of purpose, two scenery sites, two gourmet sites, and one
shopping site can be chosen. The result of the program output was Meiji Shrine, Half-
A-Minute, Sensoji Temple, Tsukiji Fish Market, and Isetan Shinjuku Store. Because a
specific amount was not input in the attraction data for shopping, it was tentatively set
to 0. Therefore, the output route is consistent with the previous one and the purpose is
consistent. The meal time has a gourmet attraction with a smaller amount spent. The
route also meets the travel time requirements; therefore, this score is the highest for the
fitness function scores of all the above routes. Thus, a travel route that is more rea-
sonable in terms of spending and eating time is understood to be recommended to the
user under the constraints of meeting the time and degree of purpose requirements.
Through these tests, we can understand that the route recommended by the system to
the user is the route that best meets the user’s purpose: i.e., the travel time is short, the
food attraction time is reasonable, the total amount spent is the smallest, and the total
time is not exceeded. Thus, the user can specify the optimal route based on their travel
time (Fig. 1).

Fitness
600

400

200

0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55

Pepole-defined route score

Program-recommended route score

Fig. 1. The program-recommended route scores in this test were always higher than the people-
defined route scores. The X coordinate is the number of test routes and a total of 55 tests were
performed. The Y coordinate is the route score, which ranges from 0 to 500. The tree histogram is
the fraction of the people-defined route. The horizontal curve is the score of the program-
recommended route. The figure shows that the program-recommended route score in the 55
routes is always greater than or equal to the score of the people-defined route. We calculated that
the average score of the people-defined route is 321.6. The program-recommended route average
score is 340.3; therefore, the program-recommended route is more reasonable with higher
satisfaction.
 An Optimal Travel Route Recommendation System 881

5 Conclusions

This study proposed an improved algorithm for personalized route recommendations.

The main feature of the algorithm is to prevent foreign tourists from experiencing any
difficulties in finding their destination when they visit Tokyo for the first time. The
degree of purpose was used to find the best tourism routes. The cosine similarity
algorithm was used to find the routes with the same degree of purpose. After the same
rate of tourism sites was found, the scores for each route were increased or decreased
using the fitness function, which includes restrictions on time, meals, amount spent,
return to hotel, and travel time. Thus, our recommended tourism route is relatively high
in satisfaction and reasonable for the users’ experience. The results show that the users
are relatively satisfied with the recommended spots. The program-recommended route
is faster and more reasonable than the user-defined route because it meets the infor-
mation restrictions entered by the user. In addition, we also screened out many
unreasonable routes. Thus, the program-recommended route can be arranged humanely
in terms of meals, time, and money. We tried to reduce the required information to be
input by the user; i.e., the users only need to input the necessary conditions such as the
degree of purpose and time. However, there are still several shortcomings in the current
stage of the program. The efficiency time for the calculation of the recommended route
needs to be improved. In addition, we also need to consider whether the user-defined
weight is reasonable. We will also consider increasing the number of sites in future
work. Experienced tourists can also provide information about the tourism sites. The
arrangement of routes also needs to consider the users’ physical problems, such as the
time available for visiting the shopping attractions and for rest.

Acknowledgments. We thank Peter Fogarty, MA English 1st Class, from Edanz Group (www.
edanzediting.com/ac), for editing a draft of this manuscript).

References
1. Japan Tourism Agency: Ministry of Land, Infrastructure, Transport and Tourism. Revision of
“Basic Plan for Promotion of Tourism Nation”, 25 April 2017. (In Japanese). http://www.mlit.
go.jp/common/001182997.pdf
2. Tokyo Metropolitan Government, Result of Heisei 29-year Foreign Travel Agency Behavior
Characteristic Survey. (In Japanese). http://www.metro.tokyo.jp/tosei/hodohappyo/press/
2018/06/06/13.html
3. Linden, G., Smith, B., York, J.: Amazon.com recommendations item-to-item collaborative
filtering, January–February 2003. Published by the IEEE Computer Society 1089-
7801/03/2003 IEEE
4. Thorat, P.B., Goudar, R.M., Barve, S.: Survey on collaborative filtering, content-based
filtering and hybrid recommendation system. Int. J. Comput. Appl. 110(4), 0975–8887 (2015)
5. Foursquare. https://ja.foursquare.com/
6. Nakajima, Y., Shinkumi, H., Ota, O.: Travel route recommendation using tweets with location
information. Res. Rep. Database Syst. (DBS) 2013 158(28), 1–6 (2013). https://ci.nii.ac.jp/
naid/1100096323477. https://ja.foursquare.com/
882 C. Yuan and M. Uehara

7. Arts, M., Murata, Y., Shibata, N., Ito, M.: Algorithm for composing satisfactory tour
schedules for fickle weather. Res. Rep. Math. Model. Prob. Solving (MPS) 75(3), 1–6 (2009).
Information Processing Society of Japan
8. Kurata, Y.: Information design in tourism. Tokyo Metropolitan University (2013). http://
www.comp.tmu.ac.jp/kurata/class/tinfo2013-8.pdf
 Simulation of Secure Volunteer Computing
by Using Blockchain

Johjima Shota1(&), Kaneko Kosuke2, Subodh Sharma3,

and Sakurai Kouichi4
1
Graduate School of Information Science and Electrical Engineering,
Kyushu University, Fukuoka, Japan
[email protected]
2
Cybersecurity Center, Kyushu University, Fukuoka, Japan
[email protected]
3
Indian Institute of Technology, Delhi, India
[email protected]
4
Faculty of Information Science and Electrical Engineering,
Kyushu University Security Laboratories and Cybersecurity Center,
Kyushu University, Fukuoka, Japan
[email protected]

Abstract. Volunteer Computing (VC) is one of methods of distributed com-

puting. In VC process, several volunteers provide their computing resources for
some computational purposes. Because most VC is supposed to be executed
based on a situation that each volunteer is an honest one, in a case that some
malicious volunteers are inside the VC network, the VC contains potential risk
of causing to output an unreliable computational result. To improve the relia-
bility of the computational result, several existing VCs adopt a method of
majority voting using reliability parameters. However, this method contains
potential vulnerability against to “Sybil Attack”. Therefore, in this paper, we
propose our VC system using blockchain technology named “Hierarchy-
System” to solve these problems and also show the evaluation result executed
on a VC system simulated our proposed method.

1 Introduction

1.1 Background
In recent years, with the development of information technology, computers have dealt
with large-scale problems in a wide range of fields. In the case to deal with too large-
scale problem, not enough to deal with one computer, certain number of computers are
sometimes used to compensate the limited computational resources of one computer.
One of popular methods is that each computer deal with their own assigned task in
parallel and share a result of the task with each other among their connected network.
Such a method is called “Distributed Computing”. As an example of distributed
computing, Berkeley Open Infrastructure for Network Computing, known as BOINC,
is popular one. BOINC was developed at the University of California at Berkeley in
2002 and is a client/server type software developed as a distributed computing

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 883–894, 2020.
https://doi.org/10.1007/978-3-030-15032-7_74
884 J. Shota et al.

platform. SETI @ home/BOINC is one of a projects using BOINC. As a kind of

distributed computing, “Grid Computing” is also a popular keyword. Grid computing
also uses certain number of computers connected each other using computer network
and provide their computational resources as an integrated system. Because each
computer has their own environment such as their computer architecture or their
machine specification, it’s difficult to integrate them as one system. To support the
differences of their environments, a middleware is sometimes adopted between an
operating system and an application executing for grid computing. Grid computing is
used not only for parallel processing but also for storing data in computers participating
in the grid [1]. “Volunteer Computing” (VC) is another types of distributed computing.
VC recruits a volunteer who provide their computational resources for calculation of its
distributed computing. Our proposed system in this paper focuses on VC.

1.2 Motivation
Since VC uses computational resources of volunteer machines, one of the merits of VC
is to enable high-performance computation with low cost. Because most VC is sup-
posed based on a situation that each volunteer is an honest one, in a case that a
malicious volunteer is inside the VC network, the VC contains potential risk of causing
to output a unreliable computational result. To improve the reliability of the compu-
tational result, several existing VCs adopt a method of majority voting using reliability
parameters. However, this method contains potential vulnerability against to “Sybil
Attack”. Another problem is difficult to keep stable computation caused by shortage of
volunteers. The followings are problems VC potentially contains.
– Vulnerable to attack increasing number of nodes
– Lack of stability of calculation due to shortage of participants
– The reliability of the calculation result is low.
Therefore, propose our VC system using blockchain technology named “Hierarchy-
System” to solve these problems.

1.3 Our Contribution Comparing with Existing Results

In existing researches, each volunteer machine has their calculation tasks. In our
proposed system, each volunteer machine has a role besides the calculation task. the
roles are named as, Project Node, Calculation Node, Check Node and Ready Node.
The roles provide a process to verify a result of each calculation node and to keep high
reliability performance of VC. In addition, by introducing blockchain and Proof-of-
work (PoW), we aim to solve problems of existing research and increase participants
by cryptocurrency. We simulated in a case that several malicious nodes who output a
wrong result are in VC network. In the existing study of majority voting, the error rate
is about 2.2% in a case that three malicious nodes are in fifteen units. In our proposed
method, the error rate is 0.556% in the same environment. Moreover, we simulated
Sybil Attack, which is an effective attack against existing research. In majority voting,
in a case that 50% of volunteer nodes are malicious nodes, the error rate is 50%. In our
proposed method, the error rate can be reduced to 2.56% in the same environment.
 Simulation of Secure Volunteer Computing 885

2 Related Work
2.1 Volunteer Computing Overview
VC means a distributed processing using computing resources provided by general
participants called volunteers. It is a method of a distributed parallel processing with
unused calculation resources of volunteers. Here, volunteers basically provide com-
putational resources free of charge. However, depending on the project, some people
who contributed to the calculation can gain benefits such as digital contents and
rewards. Since VC uses computational resources provided by volunteers, it has the
merit of being able to calculate with low cost than cloud computing which purchases
and uses external computing resources [2]. On the other hand, there is a possibility
computer that may perform wrong calculations, to discover the problem and counter-
measures are not easy. Therefore, there is a possibility that a malicious volunteer may
provide a calculation resource that performs erroneous calculation, and there is a
problem that the reliability of the calculation result may be low [3]. The BOINC
mentioned above exists as a middleware widely used for VC projects [4]. BOINC aims
to disseminate VC by making computing resources easier to provide. In the project
which introduced BOINC, the client requests a divided problem on the server, and
when solving, it returns the calculation result to the server.
The following example shows as a project succeeded in VC.
• GIMPS: It is an initial project of VC launched in 1996 and stands for Great Internet
Mersenne Prime Search. It is a project aimed at discovering Mersenne prime
numbers, starting with the discovery in 1996 and achieving 16 results by 2018 [5].
• Distributed.net: This is a project launched in 1997. This project starts with analysis
of RC 5 cryptosystem, and performs DES cryptanalysis and Golomb ruler search
[6].
• SETI @ home: This is a research to receive and analyze radio waves from the
universe which was started in 1999 for the discovery of extraterrestrial life forms.
The University of California, Berkeley operated to over 5.2 million participants, and
the execution performance achieved 769 TFLOPS [7].
• FightAIDS @ home: A research that calculates and verifies combinations that are
likely to be applied to AIDS treatment in 2005. This project has shown to be
promising for drug resistant strains arising from HIV protease inhibitors [8].
• Folding @ home: A research that analyzes proteins made in 2007 and creates
therapeutic drugs for protein-derived diseases. It also has a mechanism that can
participate in calculations also in PlayStation 3 for home game machines. In 2007,
Supercomputer, BlueGene’s performance was 478 TFLOPS, while Folding @
home achieved 1000 TFLOPS and was registered as Guinness as the highest per-
forming distributed computer [9].
• Quake-Catcher: It is a project started in 2008. The information of the three-
dimensional acceleration sensor built in the participant’s laptop PC is aggregated in
the server. Then, it is a study to identify the epicenter of the earthquake from the
location information of the participants and sensor information [10].
886 J. Shota et al.

As mentioned in this chapter, VC can be implemented low cost and there is an

advantage that high-performance calculation can be realized. On the other hand, VC
has the problem that the reliability of the calculation result is low in order to use the
computing resources provided by the volunteer.

2.2 Related Research on VC and Problems

Since VC calculates using the volunteer’s computing resources, there is a concern that
a malicious participant intentionally gives erroneous calculation results. We describe
the traditional method to improve the reliability of VC calculation result and attack
against it as follows:
• Spot-checking
While multiple nodes are calculating in VC, correct answers cannot be obtained if
nodes give erroneous answers mix. To get a correct answer, if a small amount of error
nodes is included for all nodes, it is necessary to redo the calculation by all nodes.
Therefore, the generic name of an inspecting method whether a node is processing
correctly, known as a Spot-checking. For example, let a node to solve the problem
which correct answer is known beforehand. And another node will verify the answer. If
the answer is differed, the node will be discarded.
• Voting [11]
Voting is a method of judging the correctness of the calculation result without using
encryption or advanced calculation. In VC, the server duplicates the problem dis-
tributed to a node and distributes it to multiple nodes. The server waits until the
calculation result gathers from multiple nodes distributing the problem. When a certain
number of results gathers, the server side regards the answer that the most majority
group occupies as the correct answer. Redundancy, which distributes replication
problems to multiple nodes, is related to reliability with this method. When a malicious
node gives an incorrect answer, there is an advantage that even if a wrong answer is
given by one unit, the answer of the malicious node is not adopted by other nodes
returning correct answers. Distributing the problem to more nodes leads to improve
reliability. On the other hand, there is a trade-off relationship in which the number of
nodes that solve the same problem is increased while the processing performance is
decreased. Majority Voting adopting majority solutions is widely used in VC, and also
adopts BOINC. However, there are effective attacks in this method.
• Sybil Attack [12]
Sybil Attack is an attack that an attacker prepares multiple nodes and arbitrarily
manipulates the system. In case of a system that does not require registration or one
person can easily register multiple, you can let the system recognize the number of
participants or more. If this system gives equal rights to participants, the attacker who
registered multiple entries has stronger rights than normal participants. Sybil Attack
used this right. Since this attack tends to succeed especially for systems with few
participants, securing participants is a challenge in VC.
 Simulation of Secure Volunteer Computing 887

• Credibility-based Fault-Tolerance [13]

Credibility-based Fault-Tolerance is an idea based on the credibility threshold
principle. And it is a method which introduced credibility value to Majority Voting. In
Majority Voting, when multiple nodes solved the replication problem, which solution
is the correct answer was decided only for the number of nodes. In Credibility-based
Fault-Tolerance, the solution is determined by the sum of the Credibility Values set for
each node. In Majority Voting, Redundancy is high because it is necessary to replicate
and distribute the problem to more nodes in order to increase the reliability of the
result. On the other hand, in Credibility-based Fault-Tolerance, the number of distri-
butions is reduced and it is possible to improve the processing speed. With this method,
even if an attacker sends an incorrect answer with Sybil Attack, the attacker’s answer is
not adopted because the new node has a low Credibility Value. However, this method
is vulnerable to advanced Sybil Attack which gives an incorrect answer after raising
Credibility Value by issuing correct answer [14].
In this paper, we introduce the blockchain and consensus algorithm technology to
solve the problem of VC. These techniques will be described below.

2.3 Overview of Blockchain

Blockchain is the technology of distributed ledger [15], it is utilized for virtual cur-
rency. Blockchain is a technology that connects blocks with information and shares
them with participants. Blocks are linked by including the hash of the previous block.
Since anyone can obtain the block, it is possible to refer to and confirm past infor-
mation. Therefore, it is possible to confirm from the third party’s fair stand whether the
information contained in the newly generated block is correct. If a malicious participant
rewrites certain block information, the hash information included in the next block does
not match, so that it is not easily to rewrite. In the Blockchain, since it is necessary to
uniquely share information among participants, the block is single concatenated.
Therefore, when concatenating blocks, it is necessary to decide which blocks to link
and share. In this way the problem of not being able to share decisions among users is
referred to as Byzantine General problem [16]. It is said that it is necessary to solve the
Byzantine General problem as a necessary condition for the P2P network system to
operate normally [16]. To solve this problem, the Blockchain adopts a consensus
algorithm to select one from multiple concatenation candidate blocks. The consensus
algorithm is an algorithm that employs blocks of nodes that satisfy certain conditions.
For example, in Proof of Work (PoW) introduced by Bitcoin, the block of the node
which solved the hash calculation first was adopted. For this reason, nodes with higher
computing power are more likely to generate blocks. Other consensus algorithms
include Proof of Stake (PoS) [17] and Practical Byzantine Fault Tolerance (PBFT) [18]
which takes majority vote among nodes secured trust. For the reasons described above,
the block chain is resistant to falsification of information, and information can be
shared uniquely throughout the network.
888 J. Shota et al.

2.4 51% Attack

51% attack is an attack lacking fairness from the system, having 51% or more of the
conditions for obtaining the block creation right by the consensus algorithm. For
example, in Bitcoin using PoW, it is established when a specific group has 51% or
more of the computing power of nodes participating in the block chain network. As a
countermeasure to this attack, to make it difficult to intentionally obtain 51% or more.
Also, there is no advantage for attackers even if they hold 51% or more, there is a
system to eliminate incentives [19]. For example, PoW depends on the amount of
calculation, while PoS is affected by the number of owners of virtual currency. In other
words, if a malicious participant gains block creation rights with PoS, it will hold 51%
or more of the total virtual currency and attempt to establish an attack. However, if a
malicious participant succeeds in the attack, the reliability of the system falls, which
lowers the value of the attacker’s virtual currency of 51% or more. Thus PoS is
resistant to 51% attack.

3 Proposed Method

In this section, we explain about “Hierarchy-System” in detail. In our proposed

method, we assume that the Master/Slave method is used for general VC. The
master/slave system is a system consisting of a master node that manages and controls
multiple nodes, and a large number of slave nodes which perform processing. The
master node in VC refers to the node that provides the question to be solved, and this
method is executed by Project Node. In addition, the slave node refers to multiple
nodes that handle distributed computing, and in this method, slave nodes are divided
into two roles: Calculation Node and Check Node. Also, nodes which do not belong to
any of these classifications and do not process, are called Ready Nodes. The role of
each node is described as the followings.
Project Node: has role to provide the question a user wants to solve by using our VC
system. Project Node manages Calculation Node and Check Node. Project Node gives
cryptocurrency to Calculation Node as a reward for the computation through Block-
chain technology.
Calculation Node: outputs an answer for a question provided by Project Node. This
node obtains cryptocurrency as a reward for their contribution providing computational
resources to maintain VC system. These answers submitted from each calculation node
are validated by Check Node. In the case that Calculation Node outputs an incorrect
answer and Check Node find the answer is incorrect, then the node is expelled from
Calculation Node.
Check Node: has to validate whether an answer submitted by Calculation Node is
correct or not. This validation work is conducted as Spot-check. The Check Node
group obtains copies of the same question as the target Calculation Node from the
Project Node and calculate the question. After that, hash calculation is performed, and
the answer of the node that acquired the block creation right is taken as a correct
answer. If the answer matches the answer of Calculation Node, Check Node acquires
 Simulation of Secure Volunteer Computing 889

Reliability. If the answer does not match the Calculation Node, the Calculation Node is
expelled and the Reliability of the Check Node is reduced. Reliability is a parameter of
the Check Node. When the Calculation Node runs short, the node with the highest
Reliability among the Check Nodes is promoted to the Calculation Node.
Ready Node: waits for instructions from the Project Node. Ready Node is promoted to
Check Node if demand gets well.
The system is constructed by the above roles. Also, we explain the flow when
providing computing resources with a volunteer perspective.
1. Calculation resources provided by volunteers are registered as Ready Nodes.
2. As demand increases, it becomes Check Node from Ready Node.
3. Check Node Spot-checks the calculation performed by Calculation Node. At this
time, Check Node uses PoW to determine the correctness of the calculation. As a
result, the reliability of Check Node increases or decreases.
4. As demand for Calculation Node increases due to expulsion of Calculation Node,
Check Node is promoted to Calculation Node in descending order of reliability.
5. The node that has become a Calculation Node receives compensation from Project
Node and performs distributed processing.
From now, we consider the case where an attacker gives a wrong answer in this
mechanism. When an attacker gives an incorrect answer at Calculation Node, if it is
judged as an incorrect answer by Check Node, it will be expelled and no reward will be
obtained. In addition, if an attacker gives an incorrect answer at Check Node, it is
possible to expel the Calculation Node, but since the reliability of its own decreases,
the attacker can’t become a Calculation Node. In addition, Calculation Node is added
with Check Node with high Reliability, so calculation is no problem. If there are attack
nodes’ nodes in Check Node and Calculation Node, it is difficult because 51% attack
must be established to account for more than 51% of the computing power of the whole
system. The following Table 1 shows the results that are received when an incorrect
answer/correct answer is given.

Table 1. Result of taking action

Calculation Node
Incorrect Answer Correct Answer
Check Node
Need more 51% Expelled
Incorrect Answer calculation power
Lose reliability
Expelled Gain reward
Correct Answer
Lose reliability Gain reward
890 J. Shota et al.

4 Experiment

We simulated in a case that several malicious nodes are in our proposed system. In this
experiment, we prepared three Ready Nodes, eight Check Nodes, four Calculation
Nodes and one Project Node. Three malicious nodes are randomly generated in the
nodes. Project Node distribute for Check Nodes when four Check Nodes gather. The
Fig. 1 show the state of the node at the time of simulation start. And it illustrates
[Ready Node, Check Node, Calculate Node] from the left to the right. In addition, □
indicates a normal node, and ■ indicates a malicious node. In this experiment, PoW
was simulated as only a result of random parameters.

Fig. 1. The state of the node at simulation start

The number of questions was set to 60. When Calculate Node solves 60 questions,
the simulation completes. However, if it is judged as malicious from Check Node,
Calculate Nodes will rework the question. The Fig. 2 shows the number of unresolved
questions in the simulation. Although the questions are basically decreased, some parts
of them are increased such as [5] ! [5, 12] represents the case where it becomes
necessary to re-solve.

Fig. 2. Calculation progress

The result after the simulation is shown in the Fig. 3. As at the start, the top row
represents the role of the node at the end point. (0, 1, 2) of the malicious node contained
in (Ready, Check, Calculate) has changed to (2, 1, 0). This indicates that the malicious
nodes in the Calculation Node are expelled and become Ready Node. In addition, the
following □ represents a normal answer and ■ represents a malicious answer. “Not
 Simulation of Secure Volunteer Computing 891

check:” is an answer solved by Calculation Nodes, but not checked for correctness by
Check Nodes. “Check:” indicates what has been confirmed by Check Nodes. All is a
summary. Moreover, “detected:” indicates the proportion of malicious that could be
expelled among the reworked questions. This result shows that the malicious answers
were expelled and only the normal answers finally remained.

Fig. 3. The result of simulation

The Fig. 4 illustrate how many problems nodes solved. “ID” is a node-specific
identifier, and “num” represents the number of answers included in the final result.

Fig. 4. The number of problems nodes solved

In addition, the following Fig. 5 is a breakdown of nodes at the end. Malicious

nodes have made IP part display “***** malicious *****”. “Account” represents an
account name, and “rely” represents Reliability. The figure*** indicates that the nodes
with high Reliability become Calculate Node and the Reliability of malicious nodes is
low.
892 J. Shota et al.

Fig. 5. The states of nodes at the end

The same simulation was repeated nine times. The results are shown in the Table 2.
“Result” represents the number of malicious answers included in the final result.
Moreover, “Detected” represents malicious answer among the discarded questions. In
the Voting method, consider the case where three malicious nodes intruded into fifteen
nodes. The number of calculation nodes of the proposed method is four, and it is
necessary to take a majority vote at 3.75 to calculate with equivalent efficiency. Since it
is necessary for the majority vote to be an odd number of integers, think about three
and five. When taking a majority vote by five units, the probability of adopting an
incorrect answer is about 2.2%. Also, when taking a majority vote by three units, errors
occur with a probability of about 10.1%. From this, it can be seen that 0.556% of this
method is superior than existing method.

Table 2. Simulation result

1st 2nd 3rd 4th 5th 6th 7th 8th 9th Total
Result 1/60 0/60 0/60 1/60 1/60 0/60 0/60 0/60 0/60 0.556%
Detected 7/15 3/6 5/10 7/15 5/11 12/24 3/6 5/9 6/11 49.5%
 Simulation of Secure Volunteer Computing 893

Moreover, we think about Sybil Attack. If malicious nodes occupy half of the total
nodes in Voting, if attackers give a same wrong answer, the correct answer adopted rate
is less than 50%. Therefore, we suppose that the ratio of (Ready, Check, Calculate) is
(4, 8, 4) totaling sixteen units, and eight malicious nodes are included therein. And
malicious nodes share one machine power. In order to obtain a more reliable result, we
set the number of questions to 100. The results of nine times simulations are shows in
the following Table 3. The results show that this method is superior than existing
method of majority voting of VC.

Table 3. Simulation result for Sybil Attack

1st 2nd 3rd 4th 5th 6th 7th 8th 9th Total
Result 1/100 6/100 1/100 1/100 10/100 1/100 1/100 0/100 2/100 2.56%
Detected 17/30 17/33 24/44 30/57 12/23 29/53 30/57 35/67 39/77 52.8%

5 Conclusion

VC is a method that can solve large-scale calculations with low cost. However, there
are problems that VC can’t output a reliable result in a case that there are several
malicious participants. Also, VC has problem that it’s difficult to keep stable compu-
tational resources because of shortage of participants. In this paper, in order to solve
these problem, we adopted PoW as resistance to Sybil Attack by determining solution
independent of the number of nodes. And we propose the new system named
“Hierarchy-System”. In addition, we suggested motivation for participation by using
cryptocurrency in order to measure the stability of calculation by the increase of
participants. And, we show the effectiveness of our proposed method by simulation.

Acknowledgment. This research was supported by the National Institute of Science and
Technology Corporation (JST) Strategic International Collaborative Research Program
(SICORP) and the grant of JSPS Grant-in-Aid for Scientific Research JP18H03240, we thank
you here.

References
1. Foster, I., Kesselman, C.: The Grid: Blueprint for a New Computing Infrastructure. Morgan
Kaufmann Publishers, Burlington (1999)
2. Kondo, D., et al.: Cost-benefit analysis of cloud computing versus desktop grids. In: IPDPS,
vol. 9 (2009)
3. Anderson, D.P., Fedak, G.: The computational and storage potential of volunteer computing.
In: Sixth IEEE International Symposium on Cluster Computing and the Grid, CCGRID
2006, vol. 1. IEEE (2006)
4. Anderson, D.P.: Boinc: a system for public-resource computing and storage. In: Proceedings
of the Fifth IEEE/ACM International Workshop on Grid Computing. IEEE (2004)
5. Woltman, G.: GIMPS, The Great Internet Mersenne Prime Search (2007)
894 J. Shota et al.

6. Shirts, M., Pande, V.S.: Screen savers of the world unite! Science 290(5498), 1903–1904
(2000)
7. Anderson, D.P., et al.: SETI@ home: an experiment in public-resource computing.
Commun. ACM 45(11), 56–61 (2002)
8. FightAIDS@Home. http://fightaidsathome.scripps.edu/
9. Folding@home: Lessons from eight years of volunteer distributed computing. IEEEIPDPS
(2009)
10. Cochran, E.S., et al.: The quake-catcher network: citizen science expanding seismic
horizons. Seismol. Res. Lett. 80(1), 26–30 (2009)
11. Sarmenta, L.F.G.: Volunteer computing. Ph.D. thesis. Department of Electrical Engineering
and Computer Science, MIT, Cambridge, MA, December 2000
12. Douceur, J.R. The sybil attack. In: International workshop on peer-to-peer systems. Springer,
Heidelberg (2002)
13. Sarmenta, L.F.G.: Sabotage-tolerance mechanisms for volunteer computing systems. In:
Proceedings of the First IEEE/ACM International Symposium on Cluster Computing and the
Grid, pp. 337–346. IEEE (2001)
14. Watanabe, K., et al.: The effectiveness for spot-checking about Secure VC. Inst. Electroni.
Inf. Commun. Eng. 111(408), 103–108 (2012)
15. Nakamoto, S.: Bitcoin: a peer-to-peer electronic cash system (2008)
16. Lamport, L., Shostak, R., Pease, M.: The Byzantine generals problem. ACM Trans.
Program. Lang. Syst. (TOPLAS) 4(3), 382–401 (1982)
17. Bentov, I., Gabizon, A., Mizrahi, A.: Cryptocurrencies without proof of work. In:
International Conference on Financial Cryptography and Data Security. Springer, Heidelberg
(2016)
18. Castro, L.M., Liskov, B.: Practical Byzantine fault tolerance and proactive recovery. ACM
Trans. Comput. Syst. 20(4), 398–461 (2002)
19. Bastiaan, M.: Preventing the 51%-attack: a stochastic analysis of two phase proof of work in
Bitcoin (2015). http://referaat.cs.utwente.nl/conference/22/paper/7473/preventingthe-51-
attack-astochastic-analysis-of-two-phase-proof-of-work-in-bitcoin.pdf
 A Novel Movie Recommendation System
Based on Collaborative Filtering
and Neural Networks

Chu-Hsing Lin(&) and Hsuan Chi

Tunghai University, Taichung 40704, Taiwan

{chlin,g06350027}@thu.edu.tw

Abstract. Movie recommendation is one of the most common recommendation

systems, and the related technologies for recommendation are constantly
improving. It has evolved from traditional statistical analysis to collaborative
filtering and machine learning today. This paper aims to find a set of criteria that
are practical, reasonable, and accurate in using existing recommendation sys-
tems. We compare some techniques and tools and conduct experiments for
popular tools currently used, especially Scikit-learn and TensorFlow. The
experiments focus on the comparison of the advantages, error measure, and
process time of these tools. From the experimental results, we further propose a
novel recommendation system, based on collaborative filtering and neural net-
work, which maintains lower error measure.

1 Introduction

With the rapid development of the Internet, the value of recommendation systems have
been gradually discovered by major companies in a variety of fields. The related
theories and applications of recommendation systems have also developed rapidly.
Researchers and analysts are trying to find the most accurate recommendation method
to facilitate the products marketing. Up to now, among the currently publicized rec-
ommendation algorithms, the most commonly used one is the collaborative filtering
algorithm [2, 3]. However, with the trend of machine learning in recent years,
researches also switch their direction to use machine learning [4] for recommendation.
The concept of machine learning had emerged as early as the late 19th century, but
it was limited by the lack of hardware performance and data volume at that time. Until
recently, with the dramatic increase in hardware performance and the emergence of big
data applications, machine learning has gradually gained attention from many
researchers, and then many developers and companies have invested in it and devel-
oped a number of tools. Scikit-learn [5] and TensorFlow [6] are two of the more widely
known tools.
After referring to the article “Integration of Content-based approach and Hybrid
Collaborative Filtering for Movie Recommendation” [1], we conceived a new movie
recommendation system and conducted two experiments. At the same time, we will
compare the differences and advantages of both Scikit-learn and TensorFlow from
these experiments. As many developers know, Scikit-learn has a highly optimized

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 895–903, 2020.
https://doi.org/10.1007/978-3-030-15032-7_75
896 C.-H. Lin and H. Chi

library, which is good at processing data that has already extracted features, and is
extremely efficient in mathematical operations. On the other hand, TensorFlow per-
forms well on a complex and large system like neural networks and deep learning. In
this article, we developed both algorithms with Scikit-learn and TensorFlow, respec-
tively, and conducted experiments for movies recommendation. We described the pros
and cons of both methods and made valuable suggestions for users when adopting a
recommendation system. We further propose a novel recommendation system, based
on collaborative filtering and neural network, which maintains lower error measure.
The organization of this paper is as follow. In Sect. 2, we describe the background
knowledge of the paper. The design methodology appears in Sect. 3. In Sect. 4, we
present the experimental results. Finally, we have the proposed system and give a
conclusion.

2 Background
2.1 Collaborative Filtering Algorithms
Collaborative filtering algorithms [2] is an algorithm that recommends products
according to user preferences. According to the data used for analysis, there are three
kind of it:
I. User-based Collaborative Filtering
It analyzes the similarity between users according to the user’s basic information
(gender, birthday, place of residence, etc.). By using the k-Nearest Neighbor (K-
NN), the system filters the most similar K users, picks out items according to
user’s rating, and then recommends the best-matched items to users whom to
recommend.
II. Item-based Collaborative Filtering
Analyze the similarity between items according to the product information
(category, brand, price, etc.) of the items. According to the similarity between
the items, K items, the user may like, are chosen and recommended to the user.
III. Model-based Collaborative Filtering
Train a recommendation model according to the known rating, and the model is
used to predict the rating of the user who has not yet rated the item, and the one
with the higher score is selected to recommend to the user.

2.2 Neural Network

Neural network is a mathematical model that mimics the structure of a biological neural
network [4]. It can enhance the ability to predict results through training and learning.
The general neural network has three layers, namely an input layer, a hidden layer, and
an output layer.
Input layer receives external messages, and these values are computed with weights
and offsets within the neurons. The higher the weight of the neurons, the greater the
impact on the results.
 A Novel Movie Recommendation System 897

Hidden layer is the most important part of a neural network and can have one or
more layers. It can improve the complexity of the neural network and enhance its
learning ability. However, too many hidden layers will increase the complexity of the
operation and make the improvement effect limited.
Output layer is responsible for the overall and output operation results.
Another important feature of neural networks is the activation function. The acti-
vation function is a nonlinear function, and its introduction into the neural network can
make it better to deal with nonlinear problems that are easy to meet in real applications.
We used two activation functions in the experiments of this paper.
I. ReLU: It stands for Rectified Linear Unit. ReLU is the most commonly used
activation function in neural networks. Mathematically, it is defined as

f ðxÞ ¼ maxð0; xÞ ð1Þ

II. TanH: It stands for Hyperbolic Tangent Function. The function produces outputs in
scale of [−1, +1]. Mathematically, it is defined as

ðex  ex Þ
f ðxÞ ¼ tanhð xÞ ¼ ð2Þ
ðex þ ex Þ

2.3 Scikit-Learn and TensorFlow

Scikit-learn is an early machine learning tool with many ready-made libraries and
functions to call, and has powerful and fast mathematics capabilities. It’s easier for
developers who are just getting started, and you can get good results with a short code.
TensorFlow is a tool that has emerged in recent years with the development of deep
learning. Although it does not have a variety of libraries that can be used, it gives
developers a very high degree of freedom. Developer can define the model they need
according to the requirements, and then throw the data to be processed into the model.
With the concept of TensorFlow parallelization operation, multiple sets of data cal-
culated simultaneously, which greatly speeds up the machine learning.

3 Experimental Design

In 2013, Weng and Lee proposed an integration of content-based approach and hybrid
collaborative filtering for movie recommendation [1]. Based on the concept, we pro-
posed a new method and improved it in this paper. Firstly, let us review their method.
In their research, the authors designed a movie recommendation system, which pre-
dicted the rate of the movie that the user has not evaluated, and then choose a higher-
rated movie to recommend. The process is shown in Fig. 1. If a new movie comes in, it
will use the collaborative filtering based on movie information to predict the user’s
rating of the movie. If a new user comes in, the system will use user information to find
898 C.-H. Lin and H. Chi

similar users and use collaborative filtering to predict the rating. On the other hand, if it
is not a new user, the system will use historical rate record to find similar users and use
collaborative filtering to predict the rating. Finally, it uses matrix factorization to
calculate the final score prediction table and choose the Top-N of movies to recom-
mend. The final experiment results of [1] have a mean absolute error of about 1.3.

Fig. 1. Flow chart of Weng and Lee’s method

The main purpose of [1] is to solve the sparsity problem and the cold start problem
of the collaborative filtering. In this paper, we retains some basic idea, but dividing the
problem into two cases, sufficient score data and insufficient score data, to deal with. If
we have enough historical rating, we can use model-based collaborative filtering to
make prediction. Otherwise, if we have not enough historical rating, we have to rely on
users and movie information to make predictions.
Therefore, in this paper we design two sets of experiments. In the first experiment
set, we use model-based collaborative filtering for prediction. We first extract the user’s
historical rating of the movies to calculate the similarity between the users, choose the
K neighbors with the highest similarity for the recommender, and use the ratings of
these neighbors to predict the rating of the movie that the recommender still haven’t
watch.
In Experiment 1 suppose we have enough scores data and made predictions with
sufficient rating data. In Experiment 2, we use the basic information of users and
movies to make score predictions when we do not have enough rating data. We believe
that neural networks should be able to produce better results in this aspect than using
 A Novel Movie Recommendation System 899

similarity calculations, so in Experiment 2 we train a neural network to predict the

ratings.
The data used in this paper comes from MovieLens’s ml-100k database [7], which
contains 943 users’ comments on 1,682 movies and basic information of all users and
movies.
For the experimental environment, the Operating System is Windows 10, CPU is
i7-7700HQ, and the memory is 12G. Since Scikit-learn does not support GPU oper-
ations, neither Scikit-learn nor TensorFlow uses GPU operations.

4 Experiments and Results

4.1 Collaborative Filtering Recommendation

In Experiment 1, we use collaborative filtering recommendation. This experiment is
consists of the following four steps. Note that we make preprocessing to the data, 20%
of the rating data is extracted as testing data and the rest as training data.
Step 1: Use the training data to calculate the similarity between each user.
Step 2: Pick the most similar K neighbors for each user. This experiment sets K to 35.
Step 3: Average the rating of each movie by the 35 selected neighbors as the predicted
score.
Step 4: Calculate the Mean Absolute Error (MAE) between the predicted score and the
test data.
The data flow diagram of this experiment is shown in Fig. 2.

Fig. 2. Data flow diagram of experiment 1

900 C.-H. Lin and H. Chi

Although whether the collaborative filtering is a machine learning or not still has
some controversy, its concept is similar to the KNN algorithm, but the final processing
method and application is different. Considering that our purpose is to predict rating, it
is more appropriate to use collaborative filtering.
In the experiment, we also compute the MAEs and measure the process times with
different K values. We found that the best performance occurs when K was about 35, as
shown in Fig. 3 (this is the result of running with Scikit-learn).

Fig. 3. MAEs and process times with different K values

Comparison of Scikit-learn and TensorFlow:

The biggest difference between Scikit-learn and TensorFlow in this experiment is the
part of similarity calculation. Scikit-learn uses the pairwise_distances function in the
sklearn.metrics.pairwise library for similarity calculations, and TensorFlow uses the
basic mathematical functions in TensorFlow to build a model for calculating similar-
ities and then throws data into it.
The error measure used in this experiment is the Mean Absolute Error (MAE).
Using MAE, it subtract the predicted score from the rating in the test data and average
the absolute difference values. The result is shown in Table 1.

Table 1. The result of experiment 1.

Methods MAE values Process times
Scikit-learn 0.88 31.49 s
TensorFlow 0.88 104.33 s

As shown in Table 1, the MAE values are about the same, 0.88. However, in the
process time, Scikit-learn is obviously much faster than using TensorFlow. We can see
 A Novel Movie Recommendation System 901

that Scikit-learn is the better choice for the collaborative filtering algorithm. It is about
three times faster than using TensorFlow.

4.2 Neural Network Recommendation Model

In Experiment 2, we use neural network recommendation model. The neural network
model used in this experiment is shown in Fig. 4. The input values include user ID,
user gender, user age, user occupation, movie ID, and movie category.
The hidden layer has two layers, the first layer has 192 neurons, and its activation
function is ReLu, the second layer has 300 neurons, and its activation function is TanH.
Finally, the output layer outputs the predicted rating.

Fig. 4. Neural network recommendation model

Comparison of Scikit-learn and TensorFlow:

In this experiment, Scikit-learn uses the Regressor function of the sknn.mlp library to
build the neural network, and TensorFlow refers to [7] to design the neural network
model.
The experimental results are shown in Table 2. TensorFlow’s MAE is 0.76 and the
process time is 136 s. On the other hand, Scikit-learn’s MAE is around 1.4, and the
process time is about 1.74 h.
902 C.-H. Lin and H. Chi

Table 2. The result of experiment 2

Methods MAE values Process times
Scikit-learn 1.43 6264.01 s
TensorFlow 0.76 136.28 s

We can see that TensorFlow is much better than Scikit-learn in the neural network.
TensorFlow’s parallel computing ability can greatly shorten the training time of the
neural network. Scikit-learn does not have parallel computing capability. Even if the
underlying function has undergone multiple optimizations, it still takes a lot of time to
process. Also, since Scikit-learn’s freely adjustable parameters are not as numerous as
TensorFlow, it is not as good as TensorFlow on the result of process.

5 Novel Systems and Conclusions

From the experiments in this research, the results are indeed as stated at the outset,
Scikit-learn does have a good effect in the traditional machine learning field, and
TensorFlow is good at the neural network.
As the experimental results, we can see that the rating prediction by the user
information using the neural network is more accurate than using the collaborative
filtering through the historical rating data. In addition, the neural network is unlike the
collaborative filtering algorithm that needs enough rating data to make predictions. It
seems that the neural network can directly replace the collaborative filtering algorithm.
However, it is a good idea to use the user’s personal information to make predictions.
But in reality, not all users will provide correct personal information. It is more reliable
to use the user’s historical rating to make predictions than to use their personal
information to make predictions. Furthermore, in this research, we also found that
although the results of Experiment 2 are better than Experiment 1, the actual recom-
mended movies are not much different.
From the experiments and observations previously, we finally proposed a novel
recommendation system as shown in Fig. 5. If we can have enough scoring information
for collaborative filtering prediction, and can find 35 users with similarity higher than
0.35 (completely consistent is 1), the system will use collaborative filtering as the main
prediction basis, and choose N movies with the highest rating prediction to recommend.
If the above conditions not met, the system will use neural network as the main
prediction basis, and choose N movie with the highest rating prediction to recommend.
If the collaborative filtering system fails to provide a sufficient number of recom-
mendations, it has complement by a neural network system. Based on the collaborative
filtering and neural network, the proposed novel system can maintain the MAE to
below 1.0. This result has considerable progress compared to [1].
 A Novel Movie Recommendation System 903

Fig. 5. Proposed novel recommendation system

Acknowledgement. This research was partly supported by the Ministry of Science and Tech-
nology, Taiwan, under grant number MOST 107 - 2221 - E - 029 - 005 - MY3.

References
1. Weng, S.-S., Lee, C.-H.: Integration of content-based approach and hybrid collaborative
filtering for movie recommendation. In: BAI 2013 International Conference on Business and
Information (2013)
2. Mustafa, N., Ibrahim, A.O.: Collaborative filtering: techniques and applications. In: 2017
International Conference on Communication, Control, Computing and Electronics Engineer-
ing (ICCCCEE) (2017)
3. Luo, T., Chen, S., Xu, G., Zhou, J.: Collaborative filtering. In: Trust-based Collective View
Prediction, pp. 25–51 (2013)
4. Hansen, L.K., Salamon, P.: Neural network ensembles. IEEE Trans. Pattern Anal. Mach.
Intell. 12(10), 993–1001 (1990)
5. Pedregosa, F., Varoquaux, G., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn.
Res. 12, 2825–2830 (2011)
6. Abadi, M., Barham, P., et al.: TensorFlow: a system for large-scale machine learning. In: 12th
USENIX Symposium on Operating Systems Design and Implementation, pp. 264–283 (2016)
7. MovieLens Recommender System. https://github.com/chengstone/movie_recommender
 Response to Co-resident Threats in Cloud
Computing Using Machine Learning

Chu-Hsing Lin(&) and Hsiao-Wen Lu

Department of Computer Science, Tunghai University, Taichung 40704, Taiwan

{chlin,g06350005}@thu.edu.tw

Abstract. Virtualization technologies in cloud computing brings merits for

resource utilization and on-demand sharing. However, users can face new
security risks when they use the virtualized platforms. The co-resident attack
means that the malicious users build side channels and threaten the virtual
machines co-located on the same server. In 2017, Abazari et al. proposed a
multi-objective, under the constraints of minimum cost and threat, response to
co-resident attacks in cloud environment. In this paper, we aimed to propose a
novel method for countermeasures decision to the attacks. We used machine
learning to train the intrusion response model and conducted a set of experi-
ments to demonstrate the effect of the proposed model. It showed that the near
optimal solutions with good accuracy is obtainable and the response efficiency is
improved.

1 Introduction

Cloud computing provides on-demand services for users to share data, software, or
infrastructure with low cost. Users can obtain services conveniently without need to
know the details of the cloud environment and to maintain the devices. However, the
shared environment also exposes a variety of security threats to users. Ali et al. [2]
described the security issues associated with the outsourcing of data, software, service,
virtualization, and hypervisors in cloud computing. In addition to the well-known
security problems, attackers can maliciously use cloud as a source of attacks for various
purposes. Besides, virtualization technology brings merits for resource utilization and
on-demand sharing. Due to the virtualization property of the cloud, several virtual
machines (VMs) can run concurrently in a physical machine. However, if the VMs are
not isolated properly, a malicious VM is able to access the data from another VM on
the same physical host.
From recent investigations, researchers propose to develop the intrusion response
system (IRS) to against the cloud attacks. It is crucial to choose the instant and best
countermeasures on detecting a threat. In 2017, Abazari et al. proposed a multi-
objective response to co-resident attacks in cloud environment. [1] The authors in the
paper specifically discussed the security issues caused by shared infrastructure of cloud,
and sorted out the types of VMs co-resident attacks and the response countermeasures.
In [1], it used collaboration graphs, called co-resident time graph, to capture
information about the VMs to against the co-resident attack in the cloud. The node of a
co-resident time graph indicates a VM, and the edge between two nodes means that the
© Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 904–913, 2020.
https://doi.org/10.1007/978-3-030-15032-7_76
 Response to Co-resident Threats in Cloud Computing Using Machine Learning 905

two VMs co-resident in the same physical machine. The weight of an edge indicates the
length of co-resident time. There are several types of threats and each type can be
defending with specific countermeasures. Both of the effect of the countermeasure on
the threat and its cost are calculated. Now the problem is how to estimate threat level
based on the co-resident graph and suggests proper countermeasures based on threat
type with minimum cost in a dynamic cloud environment. In addition, we found that
the efficiency of the IRS is also a crucial issue. It will take considerable time to
calculate the required response countermeasures in a cloud environment with large
number of VMs. In this paper, inspired by the above research, we proposed a novel
method to solve the problems. We used machine learning and conducted a set of
experiments to demonstrate the effect of countermeasures. From the experimental
results, it showed that near optimal solutions with good accuracy is obtainable and the
response efficiency is improved.
The rest of this article organized as follows. We describe the co-resident attacks,
collaboration graph, and related background in Sect. 2. In Sect. 3, the proposed method
is introduced. In Sect. 4 we conduct a set of experiments to evaluate our method and
discuss the results, and finally, Sect. 5 concludes the paper.

2 Related Work

We focus on the VMs running on a physical machine in the cloud environment. Since
in the cloud it contains large amount of valuable information such as user data and
applications. Once a suspicious activity, such as port scans, excessive resource con-
sumption, and malicious traffic generation, detected in the system, it should be deter-
mined instantly whether the activity is a malicious one. Besides, the coexistence time
among VMs also affects the probability of being attack. Therefore, the coexistence time
has to add to the model of estimated threat level.

2.1 Co-resident Attack

Due to the shared infrastructure property of the cloud, a malicious VM can retrieve data
from other VMs in the same physical machine [2]. If one of the VMs attacked, other
VMs on the same machine may be threaten by the attack [3]. We can divide co-resident
attacks into three categories. The first category includes message stealing and other
side-channel attacks. The attacker collects timing information, cache, and power
consumption information through the physical machine, and obtains the privacy
information of the victim VMs. The second category is the spread of malicious soft-
ware between VMs [4]. In the third type of attacks, the attacker’s goal is to obtain
unfair sharing of resources [5], causing the victim VMs on the physical machine to
subject to DoS attacks [6].
906 C.-H. Lin and H.-W. Lu

2.2 Co-resident Graph

The probability of co-resident attack positively correlated with the coexistence time [7,
8]. The VMs co-resident graph is used to calculate the threat level of all VMs in the
cloud environment [9].
As shown in Fig. 1, the nodes in the co-resident graph indicate the VMs. The
connection between two nodes, an edge, means that the two VMs co-resident in the
same physical machine [10]. The weight on the edge indicates the length of co-resident
time. Note that we focus on two type of VMs: a set M of a malicious VMs and a set S
of victim VMs.

Fig. 1. A co-resident graph.

2.3 Estimated Threat Level

In order to estimate the threat level in the cloud, first we have the co-resident graph and
a set M of detected malicious VMs. When a malicious VM wants to obtain a victim’s
message through a cross-channel attack, its VM behavior exists an exception such as
resource usage, system call, and cache miss, and it can be found through continuous
monitoring [11].

To calculate the threat level for VMi, we define a triplet ti ¼ tiSC ; tiMP ; tiDOS as
Eq. 1 [1]. Where tiSC indicates the probability of side-channel attack (SC for short) on
VMi, tiMP is the probability of malicious software propagation (MP for short), and tiDOS
represents the probability of denial-of-service attack (DoS for short). Let wij indicate
the duration in which VMi and VMj coexist. The threat level ti means the probability
 Response to Co-resident Threats in Cloud Computing Using Machine Learning 907

for VMi to be attacked. In addition, a victim VM will increase the probability of been
attacked if it coexisted with a malicious VM for a longer time duration or co-resided
with multiple malicious VMs. Therefore, the threat level has to include the probability
of any threat occurrence and the probability of multiple threats occurrence simulta-
neously. We made some modification to the threat level equation used in [1], as showed
in Eq. 2. The total threat calculation is shown in Eq. 3, where n represents the number
of VMs.


ti ¼ tiSC ; tiMP ; tiDOS ð1Þ
8  
< ti ¼ ½111  ð½111  ti Þ  ½111  tj  wi;j
subj: to ð2Þ
:
VMj 2 M; 1  j  n
Xn
T¼ t
i¼1 i
ð3Þ

2.4 Countermeasure Table

Each type of threat can be prevented by using specific countermeasures [1]. We listed
the six countermeasures against co-resident attacks in Table 1, where the 6th coun-
termeasure means that there is no recommended response strategy. The 2nd Counter-
measure has different mitigation methods and the descriptions depending on the VM
state, but has the same defense capabilities and response costs. Let the array C ¼ ½c1 ;


c2 ;    ; cq T indicate the response strategy, where ci ¼ csc i ; ci ; ci
MP DoS
. Let
T
RC ¼ ½rc1 ;rc2 ;    ; rcq  be the response cost, rci is the cost of ci , 0  rci  1, and q
means the number of countermeasures. Each countermeasure has different execution
time and computational cost, and it can also protect the VM from different types of co-
resident attacks. The countermeasure array C and the cost vector RC are represented in
Eqs. 4 and 5, respectively.
2 3 2 sc DoS 3
c1 c1 cMP
1 c1
6 c2 7 6 csc cMP DoS 7
2 c2
6 7 6 2 7
C ¼ 6 .. 7 ¼ 6 .. .. .. 7 ð4Þ
4 . 5 4 . . . 5
sc MP DoS
cq cq cq cq
2 3
rc1
6 rc2 7
6 7
RC ¼ 6 .. 7 ð5Þ
4 . 5
rcq
908 C.-H. Lin and H.-W. Lu

Table 1. Countermeasure table.

Num VM Mitigation Description SC MP DoS Cost
state methods
1 M/S VM migration Frequently migrating Yes Yes Yes Increase power
VMs consumption and
probability of SLA
violation,
performance
overhead => High
2 M Alter hypervisor Adding latency to Yes No No Performance
[12] potentially malicious overhead
operations ¼> Medium
S Change Change access Yes No No Performance
application and pattern to shared overhead => Medium
OS component resources like
access patterns memory, CPU and
cache to a consistent
pattern. This
countermeasure
prevent attacker to
gain information
from behavioral
attributes
3 M/S Traffic isolation Utilizing traffic No Yes No Network
[13] engineering reconfiguration
capabilities of virtual overhead => Low
switches to
reconfigure the
virtual network
4 M/S Limit resource Change hypervisor No No Yes Increase power
allocation [14] configurations to consumption
lower the allowed => Medium
maximum
computation load or
cache capacity
5 M/S Limit web traffic Reduce the allowed No Yes Yes Network
rate [14] maximum traffic rate reconfiguration
of the associated VM overhead => Low
6 M/S No suggested Don’t do anything to No No No ¼> Zero
countermeasures the VM

3 The Proposed Methods

First, we use the fgoalattain toolbox in the MATLAB, to generate the dataset of the
aforementioned multi-objective optimization problem [1]. Then we use machine
learning tools and the dataset to train to build a model for that. Finally, we test the
 Response to Co-resident Threats in Cloud Computing Using Machine Learning 909

model and calculate the accuracy of the model. The multi-objective optimization means
minimizing total threats and minimizing total costs simultaneously. As shown  in Eq.  6,
F1 ð X Þ is the total threat function and F2 ð X Þ is the total cost function. Let F1 ; F2 be
the target solution of the problem. And the goal value of the problem is ð0; 0Þ. The
weight vector, W ¼ ðW1 ; W2 Þ; W1 þ W2 ¼ 1, represents the importance of each item,
trade-off between lower threat or lower cost. When W1 is close to 0, it means that we
will have a solution with very low threat by using very high cost. Using different vector
W it produces different solution X, shown in Eq. 7, where X is an n  q array,
xi;j ¼ f0; 1g. Finally, the multi-objective optimization problem is shown in Eq. 8.

minc;x c
subj:to
ð6Þ
F1 ð X Þ  W1 c  F1
F2 ð X Þ  W2 c  F2
2 3
x1;1 x1;2  x1;q
6 x2;1 x2;2  x2;q 7
6 7
X¼6 .. 7 ð7Þ
4 . 5
xn;1 xn;2  xn;q
( P  P  
Min Threat ¼ ni¼1 ½111  qj¼1 xi;j  Cj  ti
ð8Þ
Min Cost ¼ X  RC

For machine learning, we use Ridge Regression from scikit-learn, indicated as

sklearn for short. We need to use import sklearn when using Python. In addition to
many algorithms for machine learning, scikit-learn also provides many well-known
datasets (such as Iris and handwritten digits). In this paper, we generate the training
dataset by using MATLAB. The steps are as follows.
1. Generate a random VM co-resident time graph.
2. Randomly pick about 30% of VMs as malicious VMs.
3. Randomly generate malicious VM threat values.
4. Calculate the threat value for each VM.
5. Use the fgoalattain function to generate the solution.
Then we randomly selected 20% of the data as the test data, and the rest as training
data. For a simple example. We generate the response countermeasure array C from the
SC, MP and DoS column of Table 1, as shown in Eq. 9. And the response cost vector
RC is generated from the Cost column of Table 1, as shown in Eq. 10. In this case, if C
(i, j) = 1 it indicates that countermeasure i can response to threat j. For example, C(3,
2) = 1 means that traffic isolation can mitigate malware propagation threat. On the
other hand, RC(3) = 0.2 means that traffic isolation has a low cost.
910 C.-H. Lin and H.-W. Lu

2 3
1 1 1
61 0 07
6 7
60 1 07
C¼6
60
7 ð9Þ
6 0 177
40 1 15
0 0 0
2 3
1
6 0:5 7
6 7
6 0:2 7
6
RC ¼ 6 7 ð10Þ
7
6 0:4 7
4 0:3 5
0

4 Experimental Results

The experiment environment is described as follows. We use Intel i7-6700 CPU,

Windows10 operating system, the Math Works MATLAB version R2017b, and
Python3.6 for machine learning.
In the environment of 12 VMs, the execution time of MATLAB is 3.12 s on
average, which is acceptable. In the case of 12 VMs and 200 pieces of data, the average
response time required for machine learning is 0.017 s with an average accuracy of
71%. Under the setting of weight W = (0.5, 0.5), we randomly selected 2 victim VMs
to list their threat value T, and the recommended countermeasures x, as well as the total
threat, total cost, and execution time in the cloud, as shown in Tables 2 and 3,
respectively.
VM1’s MP threat value is high, and the rest is very low. It is correct to recommend
countermeasures 3 by MATLAB, and the same countermeasure is recommended by
machine learning. VM3 has a higher MP threat value and a lower DOS threat value.
The MATLAB recommends the use of countermeasures 5, the machine learning rec-
ommended countermeasures 3, both of which can correctly respond to MP threats, but
the countermeasure by the machine learning is lower in cost and higher in threats.

Table 2. 12 VMs
VMi Ti MATLAB x(i) Machine learning x(i)
VM1 T1(0.04, 0.56, 0.00) x1 = (0, 0, 1, 0, 0, 0) x1 = (0, 0, 1, 0, 0, 0)
VM3 T3(0.00, 0.42, 0.14) x3 = (0, 0, 0, 0, 1, 0) x3 = (0, 0, 1, 0, 0, 0)

Table 3. Total threat, total cost and execution time for 12 VMs.
W = (0.5, 0.5) Total threat Total cost Execution time (s)
MATLAB 3.10 2.20 3.12
Machine learning 3.21 1.92 0.017
 Response to Co-resident Threats in Cloud Computing Using Machine Learning 911

In the environment of 50 VMs, the execution time of the MATLAB averaged

48.23 s, which is no longer an immediate response. In the case of 50 VM and 300 data,
the machine learning response requires an average of 0.021 s and an accuracy of 83%.
From this, it is obvious that the time required for the response is largely improved. We
also randomly selected 2 victim VMs to list their information, as shown in Table 4.
Table 5 lists the total threat and the total cost of using the two methods by setting
W = (0.6, 0.4).

Table 4. 50 VMs
VMi Ti MATLAB x(i) Machine learning x(i)
VM12 T12(0.24, 0.56, 0.80) x12 = (0, 0, 0, 0, 1, 0) x12 = (0, 0, 0, 0, 1, 0)
VM36 T36(0.50, 0.83, 0.61) x36 = (1, 0, 0, 0, 0, 0) x36 = (0, 0, 0, 0, 1, 0)

Table 5. Total threat, total cost and execution time for 50 VMs.
W = (0.6, 0.4) Total threat Total cost Execution time (s)
MATLAB 25.7 8.0 48.23
Machine learning 31.2 7.8 0.021

Fig. 2. Threat vs cost with different weight settings

To compare MATLAB with machine learning, we conducted another experiment.

In the environment of 50 VMs, we use a set of threat values and different weights, from
W = (0.1, 0.9) to W = (0.9, 0.1) with increment (decrement) 0.1, and use MATLAB
and machine learning to generate response countermeasures. Then calculate the total
threat and total cost of the cloud after using the countermeasures. Figure 2
912 C.-H. Lin and H.-W. Lu

demonstrated the comparison of the total threat and total cost using the two methods by
setting different weights. As shown in Fig. 2, machine learning strategies have higher
threats but lower costs at the same weight setting.
From the above, we found that using both the MATLAB and the machine learning
methods we can obtain the appropriate response to the attack with acceptable total
threat and total cost. The machine learning model outperformed the MATLAB model
in the response time.

5 Conclusion

Once a potential attack detected in the cloud environment, immediate response is a key
to reduce the damage. Based on the co-resident graph, we proposed a novel method for
the multi-objective optimization problem by using machine learning. We found that it
is not enough to use traditional method to solve the problem for a cloud environment
with large number of VMs. From the experimental results, we can see that machine
learning can solve its problem very efficient. When the number of VMs and the amount
of dataset increased, the machine learning outperformed the MATLAB model and the
accuracy of machine learning model can also be improved. In the near future, we plan
to have experiments for large number of VMs. Besides, it is not limited to the VMs co-
resident attacks. For other security issues in cloud, we can also try to use the proposed
model to select the real-time response countermeasures.

Acknowledgement. This research was partly supported by the Ministry of Science and Tech-
nology, Taiwan, under grant number MOST 107 - 2221 - E - 029 - 005 - MY3.

References
1. Abazari, F., Analoui, M., Takab, H.: Multi-objective response to co-resident attacks in cloud
environment. Int. J. Inf. Commun. Technol. Res. 9(3), 25–36 (2018)
2. Ali, M., Khan, S.U., Vasilakos, A.V.: Security in cloud computing: opportunities and
challenges. Inf. Sci. 305, 357–383 (2015)
3. Han, Y., Chan, J., Alpcan, T., Leckie, C.: Using virtual machine allocation policies to defend
against co-resident attacks in cloud computing. IEEE Trans. Dependable Secure Comput. 14,
95–108 (2017)
4. Abazari, F., Analoui, M., Takab, H.: Effect of anti-malware software on infectious nodes in
cloud environment. Comput. Secur. 58(C), 139–148 (2016)
5. Varadarajan, V., Kooburat, T., Farley, B., Ristenpart, T., Swift, M.M.: Resource-freeing
attacks: improve your cloud performance (at your neighbor’s expense). In: Computer and
Communications Security, CCS 2012, pp. 281–292. ACM, New York (2012)
6. Zhou, F., Goel, M., Desnoyers, P., Sundaram, R.: Scheduler vulnerabilities and coordinated
attacks in cloud computing. In: IEEE 10th International Symposium on Network Computing
and Applications, Cambridge, MA, pp. 123–130 (2011)
7. Atya, A.O.F., Qian, Z., Krishnamurthy, S.V., Porta, T.L., McDaniel, P., Marvel, L.:
Malicious co-residency on the cloud: attacks and defense. In: IEEE INFOCOM 2017 - IEEE
Conference on Computer Communications, Atlanta, GA, pp. 1–9 (2017)
 Response to Co-resident Threats in Cloud Computing Using Machine Learning 913

8. Zhang, W., Jia, X., Wang, C., Zhang, S., Huang, Q., Wang, M., Liu, P.: A comprehensive
study of co-residence threat in multi-tenant public PaaS clouds. In: Lam, K.Y., Chi, C.H.,
Qing, S. (eds.) Information and Communications Security, ICICS, Lecture Notes in
Computer Science, vol. 9977. Springer, Cham (2016)
9. Altunay, M., Leyffer, S., Linderoth, J.T., Xie, Z.: Optimal response to attacks on the open
science grid. Comput. Netw. 55(1), 61–73 (2011)
10. Abazari, F., Analoui, M.: Exploring the effects of virtual machine placement on the
transmission of infections in cloud. In: 7th International Symposium on Telecommunica-
tions, Tehran, pp. 278–282 (2014)
11. Sun, Q., Shen, Q., Li, C., Wu, Z.: SeLance: secure load balancing of virtual machines in
cloud. In: IEEE Trustcom/BigDataSE/ISPA, Tianjin, pp. 662–669 (2016)
12. Wu, J., Ding, L., Lin, Y., Min-Allah, N., Wang, Y.: XenPump: a new method to mitigate
timing channel in cloud computing. In: IEEE Fifth International Conference on Cloud
Computing, Honolulu, HI, pp. 678–685 (2012)
13. Chung, C.-J., Khatkar, P., Xing, T., Lee, J., Huang, D.: NICE: network intrusion detection
and countermeasure selection in virtual network systems. IEEE Trans. Dependable Secure
Comput. 10(4), 198–211 (2013)
14. Balasubramanian, S., Lobbes, M.M., O’connell, B.M., Snitzer, B.J.: Automated Response to
Detection of Threat to Cloud Virtual Machine. US Patent 20,160,094,568, March 2016
 Automatic Graph-Based Clustering
for Security Logs

Hudan Studiawan(B) , Christian Payne, and Ferdous Sohel

Discipline of Information Technology, Mathematics, and Statistics,

Murdoch University, Perth, Australia
{hudan.studiawan,c.payne,f.sohel}@murdoch.edu.au

Abstract. Computer security events are recorded in several log files.

It is necessary to cluster these logs to discover security threats, detect
anomalies, or identify a particular error. A problem arises when large
quantities of security log data need to be checked as existing tools do not
provide sufficiently sophisticated grouping results. In addition, existing
methods need user input parameters and it is not trivial to find optimal
values for these. Therefore, we propose a method for the automatic clus-
tering of security logs. First, we present a new graph-theoretic approach
for security log clustering based on maximal clique percolation. Second,
we add an intensity threshold to the obtained maximal clique to consider
the edge weight before proceeds to the percolations. Third, we use the
simulated annealing algorithm to optimize the number of percolations
and intensity threshold for maximal clique percolation. The entire pro-
cess is automatic and does not need any user input. Experimental results
on various real-world datasets show that the proposed method achieves
superior clustering results compared to other methods.

1 Introduction
A server records a certain activity to system or kernel logs. Another log file
keeps authentication events for the operating system, such as auth.log. Both
successful and failed login attempts are recorded in this authentication log file
[11]. Other security applications also track their operations into specific log files,
such as Snort Intrusion Detection System (IDS) logs.
System administrators may need to investigate these log files to check for
incidents in relation to a certain application server. The use of command line
tools such as grep or awk are time-consuming because we need to develop the
correct search parameters [2]. These tools are unsuitable for convenient identifi-
cation and reviewing of security-relevant events. In addition, the command line
tools are non-automated and require considerable user expertise. Therefore, it
is difficult to determine what has occurred in the last couple hours in terms of
security attacks to a server. A sophisticated tool is required for the investigation
of log files.
The use of event log clustering is important for security threat discovery,
specifically by using clustering as a basis for anomaly detection [16,19]. Vaarandi
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 914–926, 2020.
https://doi.org/10.1007/978-3-030-15032-7_77
 Automatic Graph-Based Clustering for Security Logs 915

and Pihelgas [32] state that a cluster which has less log pattern frequency than
other clusters becomes an outlier. More recent work has used a log clustering
method to identify a problem or a particular error in large-scale online service
systems [20].
The existing methods use many user input parameters to cluster event logs.
Therefore, the users will get a difficulty to determine the appropriate parameter
values for various log files. For example, Simple Log Clustering Tool (SLCT)
needs one mandatory parameter and other 13 options that can be set by the
users [31]. The improved version of SLCT, known as LogCluster, requires one
mandatory parameter and other 26 options, excluding optional formatting fea-
tures such as color, debugging mode, and the version of the tool [32]. To find
any good parameter combination is a time-consuming task. The main challenge
in the log clustering case is that we need an automated technique to get good
clusters without any user intervention.
Contributions. We propose graph-based clustering for security logs and name
the proposed method ASLoC (Automatic Security Log Clustering). We focus
on security-related logs as log examination is greatly required for security anal-
ysis. In summary, the contributions of the paper are as follows:

1. ASLoC preprocesses log entries without any regular expressions or predefined

rules;
2. Security logs are represented as a graph data structure, so that the logs will
be grouped naturally as the log entries are connected by their similarity;
3. We model the logs as a maximal clique graph and relax it to a weighted graph.
This is because the proposed graph model considers the string similarity
between two log entries as an edge weight;
4. The clusters are determined using maximal clique percolation which needs two
parameters, specifically the number of percolations and the intensity value.
We use simulated annealing (SA) [18] to optimize these two parameters; and
5. ASLoC will make the investigation of the security logs significantly easier for
the system administrators as no input parameters are needed.

2 Related Work
Clustering-based event log analysis is a commonly-used approach although the
existing methods do not always focus on security logs. One of the techniques
for analyzing event logs is to cluster the logs based on the similarity of the text
in the event messages [29,32]. Vaarandi [31] initiated this research domain by
proposing a technique namely Simple Log Clustering Tool (SLCT), which utilizes
density-based clustering.
IPLoM (Iterative Partitioning Log Mining) uses several phases in a hierar-
chical approach to obtain clustering results [21]. Other work by [29] proposed
Baler for log clustering in high-performance computing (HPC) infrastructures.
This technique first parses log messages then generates tokens and patterns.
The tokens and patterns are treated as “meta-clusters” and processed using the
916 H. Studiawan et al.

Levenshtein distance to measure their similarity to other groups. The process is

continued until they form “meta-patterns” and final results are produced. The
clustering of event logs can also be applied to log data streams in real time [17].
Moreover, LogCluster is the improved version of SLCT [32]. The preprocess-
ing step of LogCluster is similar to that of SLCT, but LogCluster considers the
position of the word in the log lines. LogCluster maintains the frequent words
found in the log message as cluster candidates. It then selects some of the can-
didates based on a user-specified threshold. LogCluster will join the candidates
to the larger group for easier observation [32].

3 Proposed Method: ASLoC

ASLoC has four phases as shown in Fig. 1. First, the security log is parsed
to generate a unique message. Second, a graph is constructed to represent the
preprocessed raw logs. Furthermore, we describe improved maximal clique per-
colation for log clustering in the proposed weighted graph. In the last phase, the
optimal parameters are found using the simulated annealing (SA). Each phase
is discussed as follows.

3.1 Log Preprocessing and the Graph Model

The preprocessing procedure first reads every line of a log file and converts it to
lower case. This conversion is important for calculating string similarity in order
to create an edge in the proposed graph model. For example, the terms “Invalid”
and “invalid” will be considered as different words if they are not converted to
lower case. Any unnecessary symbols, such as underscore, dash, and colon, are
also removed from the log message. The next step is to delete numbers, so the
procedure leaves only alphabetic characters in the log messages.

Security logs Preprocessing log entries Building graph model

Final clustering results Finding the clustering optimal Clustering based on

parameters with SA maximal clique percolation

Fig. 1. A block diagram of ASLoC as the proposed clustering method

The last step will delete stop words such as “from” and “for”. Stop words
are extremely common words and are removed from computation because they
 Automatic Graph-Based Clustering for Security Logs 917

are insignificant for the clustering. We use stop words from the NLTK (Natural
Language Toolkit) Python library [3]. In addition, we remove any word that
always appears at the beginning of each line such as month name Dec. We
denote all raw logs as L = {l1 , l2 , . . . , l|L| }, where |L| is the number of L. We then
define L as the preprocessed security logs. We filter L and take only the unique
messages for the graph model. This is denoted by M = {m1 , m2 , . . . , m|M | }
where |M | is the number of unique messages.
The graph model provides a natural structure of text data because an edge
in a graph represents string similarity. Furthermore, we choose the graph repre-
sentation because it produces a natural initial grouping. If the string similarity
between two security log entries is zero, then we will not create an edge between
them. After completing the first phase, we construct an undirected and edge-
weighted graph, defined as G = (V, E). Every unique message in M is now
represented as a vertex v in the set V . E is a set of weighted edges. The edge eij
from vi to vj is constructed when the cosine similarity between two messages in
vi and vj is greater than zero. Please note that i ≥ 0, j < |V |, and i = j.
We then define w(v) as the vertex weight and w(eij ) as the edge weight.
The vertex weight w(v) is the number of members in the list of each element
in a unique message m. We can determine this value by maintaining a list that
contains the index of L. The list represents the individual row of security logs
that refers to m. In addition, this list will make the ASLoC easier to traverse
back to the original raw logs when presenting the clustering results. The edge
weight w(eij ) is a value of the cosine similarity between the two vertices in a
range [0, 1].

3.2 Weighted Maximal Clique Percolation as the Clusters

A clique is a complete subgraph of a graph G. The maximal clique provides
a clique that cannot be extended with one more adjacent vertex. This means
that a maximal clique is not a subset of a larger clique found [13]. To find all
maximal cliques in a graph G, we use the Bron-Kerbosch algorithm [4] and it is
implemented using NetworkX graph library [12].
In our case, we need an algorithm that considers the edge weight in the
proposed graph G. After generating all maximal cliques in graph G, we filter
them by using an intensity value. A low intensity value indicates a good cluster
candidate for an individual maximal clique. This intensity threshold has been
introduced in [9].
We denote I(C) as the intensity of a maximal clique C, E(C) is a set of edge
weights in C, and |E(C)| is the total number of edges in C. It should be noted
that this threshold has been used in k-clique percolation [9] but not used for
maximal clique percolation. We modify the power of I(C) to (|E(C)|−1 ) instead
of (2 / k(k − 1)) in [9] to adapt with the maximal cliques. In other words, I(C)
is the geometric mean of all edge weights in C as seen in following equation:
918 H. Studiawan et al.

⎛ ⎞|E(C)|−1

I(C) = ⎝ w(eij )⎠ (1)
eij ∈E(C)

For the percolation step, we consider only the valid maximal cliques, V (C).
V (C) is a maximal clique C with an intensity I(C) more than the threshold,
where: 
C, if I(C) > threshold
V (C) = (2)
∅, if I(C) ≤ threshold
We use the SA algorithm [18] to estimate an optimal value of the intensity
threshold because it can provide global solutions. SA is better than other meth-
ods such as hill climbing because the latter can reach only a local optima in the
search space.
Furthermore, a percolation is defined as an intersection between two maximal
cliques. These cliques share at least k − 1 vertices [24]. Note that k is the number
of percolations. A percolation is constructed using a maximal clique graph G =
(V  , E  ), where G is an undirected, unweighted graph, and V  contains every
maximal clique C in G [24]. The edge E will be created if vi and vj in V  share
k − 1 vertices. The percolation result will be equivalent to a cluster in G. When
no percolation is found, the maximal clique itself is considered as the cluster.
In a special case, such as in a small size security logs, if no maximal cliques
are found, each vertex in the graph G will be set as a cluster. To obtain the
number of generated clusters, we detect the connected components in G and
then continue the process in the initial graph G to retrieve each of the cluster
members.

3.3 Finding the Optimal Parameters Using Simulated Annealing

Two main parameters are applied to control the weighted maximal clique perco-
lation process namely the number of percolations k and the intensity threshold
I. To obtain the best parameters, we use the SA method. This process is applied
to the percolation step to get the best cluster configuration.
SA is derived from a metallurgical process in material cooling introduced by
Metropolis et al. [22]. A material such as metal is processed starting at a high
temperature and then cooled slowly according to a particular cooling schedule.
The algorithm will accept the new energy state based on Boltzmann’s probability
exp(−ΔE/Tcur ) where ΔE = Enew − Ecur is the difference between the new
and the current energy. Therefore, the system will achieve a thermodynamic
equilibrium at a certain time. The best configuration at a specific energy level
is achieved at the equilibrium and becomes the global solution.
In this paper, the cooling schedule in the annealing process uses the static
decrement rule by a cooling factor α where Tnew = αTcur [33]. In terms of
parameter optimization, the energy will be replaced by an objective function
first proposed by Kirkpatrick et al. [18].
 Automatic Graph-Based Clustering for Security Logs 919

Furthermore, the initialization of proposed ASLoC algorithm runs as follows.

First, the procedure finds the weighted maximal clique in graph G. The current
parameters kcur and Icur are selected randomly in a defined range. Subsequently,
a cluster is generated by performing percolation using current parameters. In
this case, the energy E is an internal evaluation of clustering using the Calinski-
Harabasz index (CH).
The next step in the algorithm is to set the initial value of the best parameter
(Cbest , Ebest , kbest , Ibest ) with the current configuration. The procedure then sets
the new temperature Tnew using the cooling factor α and initializes the iteration
counter to zero. The default design of SA is a minimization of energy, while CH
is looking for a maximal value. Therefore, we multiply CH by -1 to obtain the
minimum value and to conform with SA.
The start of the main ASLoC iteration is similar to the initialization step
except for the new random selection of a pair parameter kcur and Icur . The new
calculated Enew and previous energy Ecur are then evaluated. If new energy
Enew is less than or equal to the current energy Ecur and current Ebest , then
the best parameter is set to the current configuration. On the other hand, if
exp(−ΔE/Tcur ) is more than a random value from uniform distribution in [0,1],
then Ecur is set to new energy Enew . Before continuing to the next iteration,
the procedure updates new temperature Tnew and increments the counter. The
main iteration will stop if Tnew is bigger than Tmin and the counter is less than
or equal to the maximum iteration N . The procedure then returns the final best
parameters (Cbest , Ebest , kbest , Ibest ).

4 Experimental Results and Analysis

This section comprises the evaluation metrics, the description of the datasets,
the configuration of SA, and the main experiments. Also, we compare ASLoC
with other five methods to see their performance using the same datasets.
We use internal evaluation to examine the clustering performance based on
the clustering results themselves. We do not use external evaluation metrics
such as precision, recall, and F-measure, because the internal one does not need
a label from the dataset to evaluate the clusters. In addition, cluster labeling is
not feasible in real-world security log files which are very large. The performance
measurement of ASLoC uses two standard internal clustering evaluation metrics,
specifically Calinski-Harbasz index (CH) [5] and Davies-Bouldin index (DB)
[8]. We choose these metrics because CH and DB use a centroid to represent a
cluster which significantly reduces the computation time. A cluster centroid is
defined as a concatenation of the preprocessed log entries in a particular cluster.

4.1 Description of the Security Log Datasets

For this research, we used five security log datasets that are publicly available.
To test the robustness of ASLoC, these datasets have different types of format
and event messages. First, we use the first bundle of the dataset from Hofstede
920 H. Studiawan et al.

et al. [15], which includes mainly SSH brute-force attacks recorded in authenti-
cation logs. This data is taken from November 1 to December 31, 2014. It con-
tains 187,591 lines and has been anonymized by replacing any usernames with
“XXXXX” and hostnames with the anonymized IP address of the observed hosts.
The second authentication logs dataset was provided by Sconzo [25] in SecRepo
(Security Repository) website. These logs are from an operational Linux-based
server and recorded from November 30 to December 31, 2014. No sanitization
has been performed on the logs. This dataset contains 86,839 lines of event logs.
The third dataset consists of 69,039 lines of Snort IDS logs provided by
Chuvakin from The Honeynet Project [6]. This log is part of the Scan of the
Month 34 Challenge and intended for the forensic analysis of a honeypot system.
The next Snort IDS log dataset was created by National CyberWatch Center in
Mid-Atlantic Collegiate Cyber Defense Competition (MACCDC) 2012 [23] and
contains 469,010 lines. The last dataset is a syslog from a real-world honeypot
installed in a RedHat-based Linux [7]. There are 25,567 lines in this dataset.
Similar to other datasets, no anonymization and no modification are applied to
this syslog.

4.2 Experimental Results

There are several parameters to run the SA part of ASLoC. There is no formal
definition to set the parameters, so we refer to the previous research on SA [28].
We define Tmin = 10−99 , Tmax = 10, and α = 0.9. The optimum energy E to
achieve is the Caliski-Harabasz index (CH). The trial of k is started from 2 to the
maximum number of vertices of the biggest maximal clique found, x, because
the number of percolations cannot exceed this value. The intensity calculates
the geometric mean of edge weight in an individual maximal clique. This edge
weight is cosine similarity in the range [0, 1]. Therefore, we set the intensity
threshold I = {0.1, 0.2, . . . , 0.9}. SA parameters are defined internally and no
user input is required. The maximum iteration for a brute force approach is
the total combination of a set of k and I. ASLoC runs only 30% of this total
maximum iteration to achieve the optimal solution, so we set the maximum
iteration to N = 0.3.
To implement and manipulate the graph model, we used the NetworkX
library [12]. An example of finding the ASLoC parameters is depicted in Fig. 2.
To achieve the best performance in terms of internal evaluation of clustering
results, we need to define the best parameters k and I in the maximal clique
percolation using SA.
In Fig. 2, the data plotted is from an authentication log on December 4, 2014
from Hofstede et al. dataset [15]. The optimal parameters obtained by ASLoC
have a red-triangle marker with k ∗ = 6, I ∗ = 0.4, CH ∗ = 2338.49, while green-
cross markers indicate the trial process of SA. The x-axis shows the number of
percolations k, y-axis as intensity threshold I, and z-axis as SA energy, which is
CH in this case.
We evaluate the CH and DB per day for all log files in each dataset. After
that, we calculate the mean of the evaluation metrics for each dataset. The
 Automatic Graph-Based Clustering for Security Logs 921

Fig. 2. The example of CH for an authentication log (red-triangle dot is the optimum
value and the green crosses are the trial values in ASLoC)

mean of the best SA energy will be compared with existing log clustering algo-
rithms as shown in Table 1. Any similar messages in the log entries, such as
Failed password ... or Accepted password ..., will be grouped together
and this will enable the security analyst or system administrator to more easily
analyze the security logs as no parameter settings need to be inputted. This
output can be used as part of a broader process for security log analysis, such as
outlier detection. An outlier is defined as any cluster that has a small number of
log entries and has possibly recorded potential threats. This particular cluster
will require further investigation.
The larger the logs, the bigger will be the generated graph. This is because
large logs have more unique event messages and these are represented as vertices
in the graph. Therefore, ASLoC will need to undergo more trials to obtain the
optimal parameters. The ASLoC behavior of CH is similar to that of DB as
depicted in Table 1. ASLoC generates a high value of CH and a low value of DB
for all datasets, which indicates a good clustering performance.
ASLoC can produce different optimal parameters, k ∗ and I ∗ , for each dataset
because it will adapt to the input security logs, while simultaneously providing
high CH and low DB. If multiple parameters produce the best CH, as shown in
Fig. 2, ASLoC will take that appears the last in the iteration of the trial process.
To objectively measure the performance and demonstrate the robustness
of the proposed method, we compare ASLoC with other methods such as
Improved MajorClust [26], IPLoM [21], LogSig [30], LKE [10], and LogCluster
[32]. Improved MajorClust [26], LKE [10], and LogCluster [32] employ clustering
as the part of anomaly detection in event logs. On the other hand, IPLoM and
LogSig use clustering as the preprocessing step to generate log abstraction [21]
922 H. Studiawan et al.

or event signatures [30]. Only Improved MajorClust uses the graph data struc-
ture as the basis for the clustering model, while other methods use non-graph
approaches.

Table 1. Comparison of CH (the higher, the better) and DB (the lower, the better)
for all datasets. Note that the bold values indicate the best results

Method Auth logs I Auth logs II Snort logs I Snort logs II Syslog
CH DB CH DB CH DB CH DB CH DB
ASLoC 9333.64 0.63 3395.24 0.98 9618.57 0.53 1387.57 0.51 940.63 0.74
(proposed)
Improved 780.34 1.09 1454.05 1.44 5910.23 1.08 157.41 1.76 83.83 1.01
MajorClust
[26]
IPLoM [21] 486.93 2.33 577.59 1.47 4221.02 1.20 75.31 1.22 103.22 2.07
LogSig [30] 2516.98 1.07 3294.07 1.99 5105.77 0.94 289.52 1.86 290.55 1.36
LKE [10] 332.49 1.27 1946.88 1.19 4661.18 0.77 110.49 1.28 156.14 0.95
LogCluster 613.13 4.05 2885.87 1.12 1402.81 1.58 252.85 3.08 391.64 5.48
[32]

The main advantage of ASLoC compared with other methods is that it does
not require users to input parameters and determines the parameters automat-
ically. Only Improved MajorClust does not user input, but its performance is
not as good as that of ASLoC as shown in Table 1. To make the comparison,
we define the parameters used by other methods as follows. The parameters
used for IPLoM are file support threshold = 0, partition support threshold = 0,
upper bound = 0.9, lower bound = 0.25, and cluster goodness threshold = 0.175.
These IPLoM parameters are recommended in the source code implementation
of IPLoM [14]. The parameter for LogSig is the number of clusters where we set
the value from 2 to 9 and choose it based on the best CH discovered. Moreover,
the parameter for LKE is a threshold value for the edit distance between two log
entries. We select the threshold = 5 as recommended in [14]. We use an option
rsupport = (10, 20, . . . , 90) for LogCluster and take only the best evaluation
metrics after trialing these rsupport values. This parameter defines the relative
support threshold of the total number of log lines in percentages. For instance,
rsupport = 10 means LogCluster will find clusters that match at least 10% of
the log entries in an input file.
Table 1 shows the comparison of CH and DB for ASLoC and the compared
methods. The values in bold are the best values of each dataset. We can see
that the proposed method has the best CH and DB for all datasets. Although
processing similar datasets, ASLoC obtains a higher value of CH for authentica-
tion logs I with 9333.64 and a lower CH with 3395.24 for authentication logs II
because the first dataset is anonymized and the second is not. This anonymiza-
tion reduces the number of unique strings for the username, so the grouping
results are better. These internal evaluation metrics indicate that our method
 Automatic Graph-Based Clustering for Security Logs 923

outperforms the others for all datasets and obtains the highest CH for authen-
tication logs I dataset with 9333.64 and the best DB for Snort logs II dataset
with 0.51.
Improved MajorClust has a low performance, especially in regard to the
Syslog dataset with CH = 83.83 and DB = 1.01, because it only considers the
edge weight of the neighboring vertices. Although the preprocessed logs in a
neighboring vertex are not so similar, it is still considered as the same cluster as
long as the edge weight is greater than those of the other neighbor vertices.
On the other hand, IPLoM also produces a low clustering performance for
almost all datasets, specifically in Snort logs II with CH = 75.31. This value
is very low compared to that obtained by ASLoC (CH = 1387.57) as shown in
Table 1. IPLoM is unable to process the log message well because the parameter
needs to be repeatedly tuned to find the best performance, while we use only the
static parameter values for all datasets based on previous research [14]. It is not
trivial to achieve the best configuration for five IPLoM parameters for various
security log datasets.
We need to configure the number of clusters to run LogSig. LogSig uses a
technique similar to K-means clustering to group the event log messages [30].
Since it is not feasible to test all possible number of clusters, we use only 2 to
9 clusters in our experiments. This is the major drawback of LogSig. However,
LogSig has a generally stable performance for all datasets compared to other
methods although ASLoC still has better clustering results. LogSig achieved the
slightly inferior values compared to ASLoC. For example, the CH of LogSig and
ASLoC for authentication logs II dataset are 3294.07 and 3395.24, respectively.
LKE joins two clusters if any two log entries between them have an edit
distance smaller than an input threshold. Therefore, it cannot generalize well
in many types of datasets. In addition, the testing of various thresholds is very
time-consuming, taking days or even weeks [14] because LKE uses hierarchical
clustering to group the log lines.
We run LogCluster with rsupport = (0.1, 0.2, . . . , 0.9) and choose the cluster-
ing results based on the best CH and DB. LogCluster has a fair performance for
Authentication logs II dataset compared to other methods with CH = 2885.87
and DB = 1.12. However, LogCluster shows a poor performance for four other
datasets indicated by high DB as depicted in Table 1. The reason for this output
is that LogCluster has many optional parameters, such as wsize for filtering out
infrequent words from the word frequency estimation process and csize for fil-
tering out cluster candidates which match less than rsupport log entries. There
are 15 parameters related to cluster generation in total. On the other hand,
we use only the mandatory parameter, specifically rsupport in this experiment.
Similar to other methods, tuning many parameters of LogCluster is impractical
and requires security expertise to set appropriate parameter combinations.
The mean performance of ASLoC for all datasets are 4935.13 and 0.68 for CH
and DB, respectively. This good performance is greatly affected by the graph
data structure to model the security logs. Similar log entries are first grouped by
their unique messages and a single unique message becomes a vertex in the graph.
924 H. Studiawan et al.

The edge indicates the cosine similarity between two messages. These vertices are
then clustered by their maximal clique percolations. Following these procedures,
the security logs will be well-grouped and show good clustering results.

5 Conclusion and Future Work

In this paper, we present a graph-theoretic method for security log clustering.
The proposed approach has achieved state of the art performance. ASLoC proved
to be accurate at finding clusters, indicated by Calinski-Harabasz (CH) and
Davies-Bouldin index (DB) with a flexible number of percolations and intensity
threshold for weighted maximal clique percolation. Importantly, ASLoC can run
automatically without user inputs by employing SA to optimize the internal
parameters.
In future, we will use this clustering method to provide more benefits for the
users. For example, event log clustering can be deployed as a preprocessing step
for anomaly detection [19]. It will also be helpful to use automatic log parser in
the preprocessing step [27]. Moreover, the SA method can be improved to include
multi-objective functions [1] as we use only CH as the objective function at the
moment. We can use several internal evaluation metrics, such as both CH and
DB, in a single run of SA.

Acknowledgements. This work is supported by the Indonesia Lecturer Scholarship

(BUDI) from Indonesia Endowment Fund or Education (LPDP), Ministry of Finance
of Republic of Indonesia.

References
1. Abubaker, A., Baharum, A., Alrefaei, M.: Automatic clustering using multi-
objective particle swarm and simulated annealing. PLoS One 10(7), e0130995
(2015)
2. Basin, D., Schaller, P., Schläpfer, M.: Logging and log analysis. In: Applied Infor-
mation Security, pp. 69–80. Springer, Heidelberg (2011)
3. Bird, S., Klein, E., Loper, E.: Natural Language Processing with Python. O’Reilly
Media, Inc., Newton (2009)
4. Bron, C., Kerbosch, J.: Algorithm 457: finding all cliques of an undirected graph.
Commun. ACM 16(9), 575–577 (1973)
5. Caliński, T., Harabasz, J.: A dendrite method for cluster analysis. Commun. Stat.
3(1), 1–27 (1974)
6. Chuvakin, A.: Scan 34 2005 from The Honeynet Project (2005). http://old.
honeynet.org/scans/scan34/
7. Chuvakin, A.: Free Honeynet Log Data for Research (2006). http://honeynet.org/
node/456/
8. Davies, D.L., Bouldin, D.W.: A cluster separation measure. IEEE Trans. Pattern
Anal. Mach. Intell. PAMI 1(2), 224–227 (1979)
9. Farkas, I.J., Ábel, D., Palla, G., Vicsek, T.: Weighted network modules. New J.
Phys. 9(6), 180 (2007)
 Automatic Graph-Based Clustering for Security Logs 925

10. Fu, Q., Lou, J.G., Wang, Y., Li, J.: Execution anomaly detection in distributed
systems through unstructured log analysis. In: Proceedings of the 9th IEEE Inter-
national Conference on Data Mining, pp. 149–158 (2009)
11. Geisshirt, K.: Pluggable Authentication Modules. Packt Publishing, Birmingham
(2007)
12. Hagberg, A., Schult, D., Swart, P.: Exploring network structure, dynamics, and
function using NetworkX. In: Proceedings of the 7th Python in Science Conference,
pp. 11–15 (2008)
13. Harary, F.: Graph Theory. Addison-Wesley, Reading (1994)
14. He, P., Zhu, J., He, S., Li, J., Lyu, M.R.: An evaluation study on log parsing and its
use in log mining. In: Proceedings of the 46th IEEE/IFIP International Conference
on Dependable Systems and Networks (2016)
15. Hofstede, R., Hendriks, L., Sperotto, A., Pras, A.: SSH Compromise Detection
using NetFlow/IPFIX. ACM SIGCOMM Comput. Commun. Rev. 44(5), 20–26
(2014)
16. Islam, H., Ahmed, T.: Anomaly clustering based on correspondence analysis. In:
Proceedings of the 32nd IEEE International Conference on Advanced Information
Networking and Applications, pp. 1019–1025 (2018)
17. Joshi, B., Bista, U., Ghimire, M.: Intelligent clustering scheme for log data streams.
In: Proceedings of the 15th International Conference on Computational Linguistics
and Intelligent Text Processing, pp. 454–465 (2014)
18. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing.
Science 220(4598), 671–680 (1983)
19. Landauer, M., Wurzenberger, M., Skopik, F., Settanni, G., Filzmoser, P.: Dynamic
log file analysis: an unsupervised cluster evolution approach for anomaly detection.
Comput. Secur. 79, 94–116 (2018)
20. Lin, Q., Zhang, H., Lou, J.G., Zhang, Y., Chen, X.: Log clustering based problem
identification for online service systems. In: Proceedings of the 38th International
Conference on Software Engineering Companion, pp. 102–111 (2016)
21. Makanju, A., Zincir-Heywood, A.N., Milios, E.E.: Clustering event logs using iter-
ative partitioning. In: Proceedings of the 15th ACM SIGKDD International Con-
ference on Knowledge Discovery and Data Mining, pp. 1255–1264 (2009)
22. Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equa-
tion of state calculations by fast computing machines. J. Chem. Phys. 21(6), 1087–
1092 (1953)
23. National CyberWatch Center: Snort fast alert logs from The U.S. National Cyber-
Watch (MACCDC) (2012). http://www.secrepo.com/maccdc2012/maccdc2012
fast alert.7z
24. Reid, F., McDaid, A., Hurley, N.: Percolation computation in complex networks.
In: Proceedings of the IEEE/ACM International Conference on Advances in Social
Networks Analysis and Mining, pp. 274–281 (2012)
25. Sconzo, M.: SecRepo.com: security data samples repository (2014). http://www.
secrepo.com/auth.log/auth.log.gz
26. Studiawan, H., Payne, C., Sohel, F.: Graph clustering and anomaly detection of
access control log for forensic purposes. Digit. Invest. 21(June), 76–87 (2017)
27. Studiawan, H., Sohel, F., Payne, C.: Automatic log parser to support forensic
analysis. In: Proceedings of the 16th Australian Digital Forensics Conference, pp.
1–10 (2018)
28. Sun, L.X., Danzer, K.: Fuzzy cluster analysis by simulated annealing. J.
Chemometr. 10, 325–342 (1996)
926 H. Studiawan et al.

29. Taerat, N., Brandt, J., Gentile, A., Wong, M., Leangsuksun, C.: Baler: determinis-
tic, lossless log message clustering tool. Comput. Sci. - Res. Dev. 26(3–4), 285–295
(2011)
30. Tang, L., Li, T., Perng, C.S.: LogSig: generating system events from raw textual
logs. In: Proceedings of the 20th ACM International Conference on Information
and Knowledge Management, pp. 785–794 (2011)
31. Vaarandi, R.: A data clustering algorithm for mining patterns from event logs.
In: Proceedings of the IEEE Workshop on IP Operations and Management, pp.
119–126 (2003)
32. Vaarandi, R., Pihelgas, M.: LogCluster - a data clustering and pattern mining
algorithm for event logs. In: Proceedings of the 11th International Conference on
Network and Service Management, pp. 1–7 (2015)
33. Yang, W., Rueda, L., Ngom, A.: A simulated annealing approach to find the
optimal parameters for fuzzy clustering microarray data. In: Proceedings of the
25th International Conference of the Chilean Computer Science Society, pp. 45–54
(2005)
 Retrieving Text-Based Surrounding
Objects in Spatial Databases

Bojie Shen1(B) , Md. Saiful Islam2 , David Taniar1 , and Junhu Wang2
1
Monash University, Melbourne, Australia
[email protected], [email protected]
2
Griffith University, Gold Coast, Australia
{saiful.islam,j.wang}@griffith.edu.au

Abstract. Retrieval of textually relevant non-dominated surrounding

data objects has many potential applications in spatial databases such as
textually relevant nearby point-of-interest retrieval surrounding a user.
This paper presents a novel query, called textually-relevant direction-
based spatial skyline (TDSS), for retrieving textually relevant non-
dominated surrounding data objects in spatial databases. The paper
also presents efficient algorithms for processing TDSS queries in spa-
tial databases by designing novel data pruning techniques using keyword
inverted index and R-Tree data indexing scheme. The effectiveness and
efficiency of the proposed algorithms are demonstrated by conducting
extensive experiments.

1 Introduction

The retrieval of textually relevant non-dominated surrounding data objects has

potential applications in spatial databases. For example, consider a user who
is looking for nearby restaurants surrounding her (green point as shown in
Fig. 1) that provide “Shushi”. A textually-relevant direction-based spatial skyline
(TDSS) query can return a number of restaurants surrounding her by trading off
the direction and distance {r2 , r3 , r4 , r5 , r6 } as shown in Fig. 1 as well as match-
ing her preferred food item. Though {r1 , r8 } and r7 also provide “Shushi”, they
are in the same directions as r2 and r6 , respectively and are also far from the
user in comparison to r2 and r6 , respectively. There exists plenty of works on
the retrieval of textually relevant objects in spatial databases ([2,3,6,20,22,25]
for survey). Unfortunately, none of these works incorporates surroundingness in
the retrieval of textually relevant data objects from spatial databases.
The first work on direction based spatial skyline query (DSQ) for retrieving
surrounding data objects in spatial databases is proposed by Guo et al. [7,8].
The DSQ query retrieves all data objects that are closest to the given query
point and that are not dominated by other data objects in their directions w.r.t.
query point. The authors propose that a data object pi should dominate another
data object pj w.r.t. the query object q if (i) both pi and pj are in the same
direction according to the user-given acceptance angle, i.e., pi qpj ≤ τ , where τ
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 927–939, 2020.
https://doi.org/10.1007/978-3-030-15032-7_78
928 B. Shen et al.

Fig. 1. An application of TDSS queries for retrieving surrounding objects (results

produced by our query model)

is the user-given acceptance angle and (ii) pi is closer to q than pj , i.e., d(q, pi ) <
d(q, pj ), where d(q, pi ) denotes the Euclidean distance between q and pi . This
query model does not consider to emphasize textual relevance on surrounding
objects retrieval. The DSQ query also has two major problems as given as follows:
(i) missing result and (ii) instability. The missing result problem is caused by the
fact that a non-resultant data object can dominate and filter other data objects.
The instability problem is caused by the settings of the user given acceptance
angle. A small change in the acceptance angle can provide a completely different
result set and causes the instability issue.
Another work on surrounding objects retrieval is the nearest surrounder
queries (NSQ) proposed by Lee et al. [13,14]. The authors propose an app-
roach to retrieve nearest surrounder objects of arbitrary shapes w.r.t. the given
query point and argue that surrounder objects should be visible from the query
point. Unfortunately, none of these works [7,8,13,14] considers textual relevance
in their approaches.
To fill the research gap and alleviate the missing result and instability prob-
lems of DSQ queries [7,8], we propose a novel query called, Textually-relevant
Direction-based Spatial Skyline (TDSS), for retrieving textually relevant non-
dominating surrounding data objects in spatial databases. We propose direc-
tional zone (DZ) to measure directional similarities among spatial data objects.
We also develop novel data pruning techniques based on keyword inverted list
and R-tree data indexing and propose two different algorithms to process TDSS
queries in spatial databases. To be specific, our main contributions are summa-
rized below:
 Retrieving Text-Based Surrounding Objects in Spatial Databases 929

Fig. 2. A toy dataset for exemplifying the definitions and algorithms in this paper

1. we propose directional zone to measure directional similarity between two

spatial data objects (Sect. 2);
2. we present a novel query called TDSS to retrieve textually relevant non-
dominating surrounding point data objects (Sect. 2);
3. we propose efficient algorithms to process TDSS queries in spatial databases
by designing novel pruning techniques based on keyword inverted list and
R-tree data indexing scheme (Sect. 3); and
4. we experimentally evaluate our proposed algorithms (Sect. 4).

2 Preliminaries

Data Model. We assume that P is a set of spatial data objects and an individual
data object p ∈ P is modeled as a point in xy plane. The x and y coordinates of
p ∈ P are denoted by px and py , respectively. The query object is also a point
in xy plane and is denoted by q. Each object p ∈ P is associated with a set
of keywords and is denoted by p.ψ. The query keywords set is denoted by q.ψ.
Data objects and points are used interchangeably in this paper. We use the toy
dataset given in Fig. 2 to exemplify the definitions and algorithms provided the
paper.

Definition 1. A data object p is said to be a keyword-match object iff p.ψ =

q.ψ.

Definition 2. A keyword match data object pi dominates another keyword-

match data object pj w.r.t. a given query object q, denoted by pi ≺ pj , iff the
following holds: (a) pj is directionally similar to pi w.r.t. q; and (b) d(q, pi ) <
d(q, pj ), where d(q, pi ) denotes the distance between q and pi .
930 B. Shen et al.

From Definition 2, it is obvious that we

need to establish (i) directional similarity
metric and (ii) distance metric to decide
on the dominance between two spatial data
objects. For the distance metric, we rely on
the Euclidean distance measure. For direc-
tional similarity metric, in this paper we
propose directional zone to model the direc-
tional similarity among the spatial data
objects. Firstly, the direction of an arbitrary
point object pi w.r.t. the query q is modeled
by − →. Assume that the intersection point
qp i
of the perpendicular line from point pj to
−→ is denoted by I(p , −
qp → Fig. 3. The directional zone of p2
i j qpi ). The distance
d(pj , I) can be used to measure how far pj
deviates from the direction of pi w.r.t. q. On the other hand, d(pi , I) can be used
to measure how far pj is away from pi according to the direction − →. We consider
qp i
pj has the same direction as pi w.r.t. q if d(pi , I) > d(pj , I). To model the above,
we rotate the ray − → by 45◦ (up to this limit we get d(p , I) > d(p , I)) both in
qp i i j
clockwise and anticlockwise considering pi as the center of origin. Assume that
these rays are −p→ −→
i c and pi a, respectively. Now, the directional zone of pi , denoted
by DZ(pi ), is formed by the area bounded by − p→ −→
i a and pi c as illustrated in Fig. 3
for data object p2 .

Definition 3. A data object pj is considered to be directionally similar to a data

object pi w.r.t. the given query object q in spatial data space if pj ∈ DZ(pi ).

Lemma 1. If pj ∈ DZ(pi ), then we get d(q, pi ) < d(q, pj ).

Lemma 2. If pj ∈ DZ(pi ), then we get pi ≺ pj .

Definition 4. Given a set of spatial data objects P and a query point q, a

textually-relevant direction-based spatial skyline (TDSS) query for q, denoted
by T DSS(q), retrieves all keyword match data object pi ∈ P if any of the
following holds:

1.  ∃pj ∈ P such that pj ≺ pi , where pj is a keyword match data object; and

2.  ∃pk ∈ T DSS(q) and pj ∈ P such that pk ≺ pj and pj ≺ pi but pk ≺ pi ,
where both pj and pk are keyword match data objects.

The above definition of TDSS queries for retrieving the surrounding data
objects closely matches the idea of global skyline [5] (i.e., union of the skylines
in each quadrant of the query point in 2D) which is rotationally invariant in
spatial context. Here, we also emphasize textual relevance through Definition 1,
i.e., a TDSS query retrieves only textually relevant non-dominated surrounding
data objects.
 Retrieving Text-Based Surrounding Objects in Spatial Databases 931

3 Our Approach
This section presents our approach of processing TDSS queries in spatial
databases.

3.1 Dominance Checking

Here, we explain our idea of checking the directional dominance between a pair of
spatial data points pi and pj . Firstly, we identify the perpendicular intersection
point I from the point pj to the ray − → and derive the following vector based
qp i
calculation to calculate the coordinates of the perpendicular intersection point I.

I x = q x + t(pi x − q x ) (1)
y y y y
I = q + t(pi − q ) (2)
0 = (Ix − pj x )(pi x − q x ) + (I y − pj y )(pi y − q y ) (3)

By substituting the first two equations into the third, we get the following:

(pj x − q x )(pi x − q x ) + (pj y − q y )(pi y − q y )

t= (4)
(pi x − q x )2 + (pi y − q y )2

Now, the directional dominance between pi and pj can be decided by compar-

ing the distance d(pi , I) and d(pj , I). However, we need to ensure that the inter-
section point I follows pi on the ray − →, so that the p can be directionally dom-
qp i j
inated by pi . For the above, we derive the following property: d(q, I) > d(q, pi )
and d(q, I) > d(pi , I) from observation. Finally, for pj to appear inside DZ(pi )
we check the following condition: d(pi , I) ≥ d(pj , I). Otherwise, the point pj
deviates from the direction − → and it must not be inside the DZ(p ). Based on
qp i i
the above formulation, we can decide that p2 ≺ p1 and p2 ≺ p3 as illustrated
in Fig. 3.

3.2 TDSS Query Processing

This section proposes two algorithms for processing the TDSS queries in spa-
tial databases. Our first algorithm is called Keyword Filtering Based Approach
(KFBA). KFBA algorithm utilizes the keyword inverted index data structure
to filter out the non-keyword match objects based on the query keywords. The
second approach is a Branch and Bound Keyword Matching (BBKM) approach
which progressively matches query keywords after indexing the database objects
into an R-tree. BBKM also exploits the directional dominance between the cur-
rent skyline points and the bounding boxes in R-Tree data indexing to expedite
the query processing.
932 B. Shen et al.

3.2.1 Keyword Filtering Based Approach

The main idea of keyword filtering based approach (KFBA) is to take advan-
tage of the pruning power of the keyword inverted index, which is a mapping
between the database objects and the keywords. They keyword inverted index
allows us to quickly search for the objects that contain a specific keyword and
can help us to filter out the objects that do not match the query keywords q.ψ.
Consider our toy dataset given in Fig. 2 as an example, which contains a set of
restaurants and the corresponding keywords which describe the foods provided
by each restaurant. The inverted index lists all the keywords appeared and the
corresponding objects in our toy dataset as shown in Table 1. Now, if the user
is only interested in the restaurants which provide Australian food, then the
objects {p1 , p2 , p3 , p4 , p6 , p8 , p9 , p10 } are returned as keyword match objects. For
multiple query keyword, we can simply calculate the intersection set among key-
word match objects of each query keyword. After filtering data objects based
on keyword inverted index, the next step is to perform the dominance check-
ing among them. However, the access order of the database objects cannot be
random as per the following lemma.

Table 1. The keyword inverted index of the toy dataset given in Fig. 2

Keyword p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11

Australian 1 1 1 1 0 1 0 1 1 1 0
Chinese 0 0 0 0 1 0 0 0 0 0 0
Italian 0 0 0 0 0 0 1 0 0 0 0
Japanese 0 0 0 0 0 0 0 0 0 0 1
Burger 1 0 0 1 0 1 0 1 0 1 0
Fries 1 0 0 1 0 1 0 1 0 1 0
Steak 0 1 0 0 0 0 0 0 0 0 0
Dessert 0 1 0 0 1 0 1 1 1 0 0
Pizza 0 0 1 0 0 0 1 0 1 0 0
Risotto 0 0 0 0 0 0 1 0 0 0 0
Sushi 0 0 0 0 0 0 0 0 0 0 1
Noodle 0 0 0 0 1 0 0 0 0 0 0

Lemma 3. The TDSS of a query point q will be correct if and only if we access
the database objects in order of their distances to q given that we compare their
dominances with the objects accessed so far.
Proof. Assume that there are three points {p1 , p2 , p3 } and the following relation-
ships hold: (a) d(q, p1 ) < d(q, p2 ) < d(q, p3 ); (b) p2 ∈ DZ(p1 ); (c) p3 ∈ DZ(p2 )
and (d) p3 ∈ DZ(p1 ). Now, assume again that we access these points from the
database in the following order: first p2 , then p1 and p3 and compare their direc-
tional dominances with the points accessed so far only. The TDSS result for the
 Retrieving Text-Based Surrounding Objects in Spatial Databases 933

Algorithm 1. Keyword Filtering Based Approach (KFBA)

Input : q: query point, P : dataset
Output : S: a list of textually relevant spatial skyline objects
Initialization: S ← ∅
1 invertedIndex← buildInvertedIndex(P ); // build inverted index
2 P  ← searchKeywordMatchObjects(invertedIndex, q.ψ); // find keyword match
objects
3 H ← insert(P  ); // insert P  into min heap H
4 while H = ∅ do
5 e ← H .pop() ; // pop the root element
6 if  ∃s ∈ S : s ≺ e then
7 S ← append(e); // e is a spatial skyline object
8 return S; // final spatial skyline set

above would be {p1 , p2 }, which is incorrect as per Definition 4. However, if we

access them in the following order: first p1 , then p2 and p3 , the TDSS result
would be {p1 , p3 }, which is correct as per Definition 4. Hence, the lemma.

Algorithm Steps. The KFBA algorithm firstly constructs the keyword inverted
index of the dataset P . Then, it performs keyword based filtering to find keyword
match objects, ∀p ∈ P : p.ψ = q.ψ. After that, a min heap H is created by
inserting the filtered database objects P
in order of their distances to the query
point q. Finally, we initialize the spatial skyline set S to ∅, then repeatedly
retrieve the root element e from H until H become ∅ and append e to S iff
 ∃s ∈ S : s ≺ e. The above steps are pseudo-coded in Algorithm 1.

3.2.2 Branch and Bound Keyword Matching

In this section, we present our branch and bound keyword matching approach
which is based on the R-tree indexing structure. The main idea of the BBKM
approach is to avoid the object-to-object dominance checking as much as possible
by pruning R-tree bounding boxes as per the following lemma.

Lemma 4. A bounding box Rk in R-Tree can be safely pruned if ∃s ∈ S such

that all of the vertices of Rk are inside DZ(s), where S is the current skyline
of the database objects accessed so far in order of their distances to the query
point q.

Proof. Every database object p ∈ Rk is bounded by the vertices of the bounding

box Rk in R-Tree. These vertices are the corner points of the minimum bounding
box Rk in R-Tree. Therefore, every p ∈ Rk is inside DZ(s) as all the vertices
of Rk are inside DZ(s), i.e., p ∈ DZ(si ), ∀p ∈ Rk . Now, every database object
p ∈ Rk has the same direction as s w.r.t. q as well as d(q, s) < d(q, p) as
p ∈ DZ(s) (as per Sect. 2 and Lemma 1). Therefore, we get s ≺ p, ∀p ∈ Rk and
the bounding box Rk can be pruned safely without further processing. Hence,
the lemma.
934 B. Shen et al.

Fig. 4. R-Tree (MAX #entries = 4) bounding boxes of the data in Fig. 2 and dominance
checking for BBKM

Consider the dataset given in Fig. 2 and the R-tree indexing given in Fig. 4.
Assume that a user wants to search for the surrounding restaurants around
her which provide Australian food. In this case, the query keyword is q.ψ =
{“Australian”}. Here, the bounding box R4 can be directly eliminated as it is
directionally dominated by the data objects p9 as per Lemma 4. Also, the data
objects p5 and p7 are ignored during the processing as these two data objects
do not contain the keyword “Australian”, i.e., are nor keyword match objects.
Algorithm Steps. Based on Lemmas 3 and 4, the BBKM approach can be
explained as given below. Firstly, the algorithm index the dataset P into an
R-tree, initialize the skyline result set S to ∅ and insert the root element e of
R-Tree into a min-heap H . The BBKM keep accessing the top element e from
H until H becomes ∅, the element e is examined further if  ∃s ∈ S : s ≺ e.
There are three case to consider as follows: (i) e is an intermediate node: we
insert each child ei of e, which is a R-tree node, into H iff  ∃s ∈ S : s ≺ ei ; (ii)
e is a leaf node, then we insert each child ei of e, which is a database object,
into H iff  ∃s ∈ S : s ≺ ei and ei is a keyword match object; and (iii) e is a
database object and is a keyword match for q, i.e., e.ψ = q.ψ, we insert e into
the skyline result set S. The above steps are pseudo-coded in Algorithm 2. It
should be noted that the min heap H stores only the R-tree nodes and keyword
match objects.
 Retrieving Text-Based Surrounding Objects in Spatial Databases 935

Algorithm 2. Branch and Bound Keyword Matching (BBKM)

Input : q: query point, R: R-Tree indexing of dataset P
Output : S: a list of textually relevant spatial skyline objects
Initialization: S ← ∅;
1 H ←insert(getRoot(R)) ; // insert root of R into min-heap H
2 while H = ∅ do
3 e ← H .pop() ; // pop the root element
4 if  ∃s ∈ S : s ≺ e then
5 if e =leaf then // intermediate node in R-Tree
6 foreach child ei of e do
7 if  ∃s ∈ S : s ≺ ei then
8 H ←insert(ei ) ; // insert box ei into heap H
9 else if e ==leaf then // leaf node in R-Tree
10 foreach child ei of e do
11 if  ∃s ∈ S : s ≺ ei and isKeywordMatch(ei ) then
12 H ←insert(ei ) ; // insert object ei into heap H
13 else // a database point
14 S ← append(e); // e is a spatial skyline point
15 return S; // final spatial skyline set

4 Experiments

Setup.The experiments are conducted on both real and synthetic datasets. The
real dataset is a POI dataset, which contains 104770 locations in California (CA)
[16]. We randomly select 25K, 50K, 75K and 100K points from this dataset as
candidate points. We also generate synthetic (SYN) datasets consisting of 125K,
250K, 375K and 500K uniformly distributed points. We randomly create 50
query points following the distribution of the datasets to report the performance
of the tested algorithms. Each point is assigned 10 random keywords from a
keyword pool of 63 keywords. We randomly create 50 query points by following
the distribution of the dataset and assign 1–4 keywords to conduct our experi-
ments. To index the datasets in R-tree, we set MAX #entries in a R-Tree node
to 20–50 and default setting is 30. The algorithms are implemented in Java and
experiments are conducted on a Mac Laptop with 2 GHz Intel Core i7 CPU and
8 GB 1600 MHz DDR3 main memory.

4.1 Effectiveness Study

Here, we demonstrate the usefulness of our TDSS query model through a case
study. The case study result for an arbitrary user (query) in CA dataset is visu-
alized in Fig. 5, where the user is shown as green point. The gray points represent
the non-keyword matched POI objects, whereas the candidate keyword match
POI objects and the resultant objects are shown as blue and red, respectively.
It is obvious that our proposed TDSS query model can retrieve non-dominated
936 B. Shen et al.

surrounding keyword match objects for an arbitrary user by trading-off the direc-
tion and distance.

Fig. 5. Non-dominated keyword match surrounding objects retrieval for an arbitrary

user via TDSS query model in CA dataset

4.2 Efficiency Study

This section presents the efficiency study of the proposed algorithms.
Effect of Query Keywords. Here, we examine of the effect of query keywords
on the efficiency of the proposed algorithms and the results are illustrated in
Fig. 6. It is evident from Fig. 6 that the proposed BBKM algorithm outperforms
our KFBA algorithm for lower number of query keywords (1–3). On the other
150
10 20 30 40 50

144.92

KFBA BBKM KFBA BBKM

ProcessingTime(ms)

ProcessingTime(ms)

41.18
39.13 114.32
50 100

10.41 29.32
24.1 25.02
19.54
5.52
4.1 3.98
2.83 5.36 8.02
1.35
0

1 2 3 4 1 2 3 4
Number of Keywords Number of Keywords

(a) CA dataset (100K) (b) SYN dataset (500K)

Fig. 6. Effect of keywords: avg. processing time (ms) in (a) CA and (b) SYN datasets
 Retrieving Text-Based Surrounding Objects in Spatial Databases 937

5 10 15 20 25 30
KFBA KFBA

6
BBKM BBKM
ProcessingTime(ms)

ProcessingTime(ms)
1 2 3 4 50

0
25k 50k 75k 100k 125k 250k 375k 500k
Cardinality Cardinality

(a) CA dataset (b) SYN dataset

Fig. 7. Effect of cardinality: avg. processing time (ms) in (a) CA and (b) SYN datasets
(|q.ψ| = 2)

hand, KFBA algorithm performs better for higher number of query keywords
(3–4 and more). This is because, the KFBA algorithm ends up having only a
few data objects after keyword filtering and thereby, reduces the size of the heap
significantly at run time.
Effect of Data Cardinality. Here, we examine the effect of cardinality on the
efficiency of our proposed algorithms and the results are shown in Fig. 7. From
Fig. 7(a), it is evident that the KFBA algorithm runs faster than the BBKM
algorithm for lower cardinality (25K–50K). On the other hand, the BBKM algo-
rithm performs much better for higher cardinality settings. From Fig. 7(b), it
is obvious that the BBKM algorithm outperforms our KFBA algorithm for all
cardinality settings in SYN dataset. This is because the KFBA algorithm is time-
consuming when there are many objects after keyword filtering. Also, multiple
keywords searching in inverted index structure for higher cardinality setting is
time consuming.
Effect of R-Tree Parameter. Here, we Table 2. Avg. running time (ms) of
examine the effect of R-tree parameter on BBKM algorithm in CA and SYN
the efficiency of BBKM algorithm and the datasets for different settings of MAX
results are shown in Table 2 for |q.ψ| = 2. #entries in a R-tree node
The proposed BBKM algorithm is tolerant Approach 20 30 40 50
in all different R-tree MAX #entries for CA dataset 2.877 2.830 2.324 2.758
both CA and SYN datasets. However, it is SYN dataset 5.92 8.02 9.18 7.4
hard to decide which R-tree parameter set-
ting is optimum. For the larger R-tree parameter setting, the BBKM algorithm
may need to examine more objects to find the TDSS result. However, the larger
R-tree MAX #entries may also have the advantage of pruning more objects that
are inside a bounding box.

5 Related Work
The skyline operator proposed by Borzsony et al. [1] has gained significant
attention among the researchers [4,5,9–12,18,19] including the spatial database
938 B. Shen et al.

community [15,17,21,23,24]. However, none of these approach considers sur-

roundingness in the non-dominating objects retrieval by trading off the distance
and direction.
The direction-based spatial skyline query (DSQ) was first discussed by Guo
et al. [7,8] and they propose to retrieve data objects that surround the query
point based on user given angle threshold. The DSQ queries have missing result
and instability problems as discussed in Sect. 1. The nearest surrounder queries
(NSQ) proposed by Lee et al. [13,14] retrieve nearest surrounder objects of
arbitrary shapes that are visible from the given query point. Both DSQ and
NSQ queries do not emphasize textual relevance of the retrieved objects.
Existing works on the retrieval of textually relevant objects in spatial
databases are quite established ([2,3,6,20,22,25] for survey). These works
emphasize on trading off the spatial proximity and the textual relevance in
spatio-textual data context, rather than distance and direction. As these works
do not consider surroundingness in their retrieval system, they may return mul-
tiple objects in the same direction.
Our work differs from the existing works in the sense that we emphasize
not only the textual relevance in the retrieved objects, but also we trade off
the direction and distance to retrieve non-dominating surrounding objects for a
given query point. Our TDSS queries are also fair and free from missing result
and instability problems.

6 Conclusion

In this paper, we have presented a novel query called textually-relevant direction-

based spatial skyline (TDSS) for retrieving textually relevant non-dominating
surrounding data objects in spatial databases. We have also presented effi-
cient algorithms to process TDSS queries in spatial databases. The proposed
algorithms have been evaluated by experimenting with both real and synthetic
datasets.

Acknowledgement. This work was partially supported by a Griffith University’s

2018 New Researcher Grant with Dr. Md Saiful Islam being the Chief Investigator.
The first and second authors contributed equally in this paper.

References
1. Borzsony, S., Kossmann, D., Stocker, K.: The skyline operator. In: ICDE, pp. 421–
430 (2001)
2. Cao, X., Chen, L., Cong, G., Jensen, C.S., Qu, Q., Skovsgaard, A., Wu, D., Yiu,
M.L.: Spatial keyword querying. In: ER, pp. 16–29 (2012)
3. Cary, A., Wolfson, O., Rishe, N.: Efficient and scalable method for processing top-k
spatial boolean queries. In: SSDBM, pp. 87–95 (2010)
4. Chan, C.Y., Jagadish, H.V., Tan, K., Tung, A.K.H., Zhang, Z.: Finding k-dominant
skylines in high dimensional space. In: SIGMOD, pp. 503–514 (2006)
 Retrieving Text-Based Surrounding Objects in Spatial Databases 939

5. Dellis, E., Seeger, B.: Efficient computation of reverse skyline queries. In: VLDB,
pp. 291–302 (2007)
6. Felipe, I.D., Hristidis, V., Rishe, N.: Keyword search on spatial databases. In:
ICDE, pp. 656–665 (2008)
7. Guo, X., Ishikawa, Y., Gao, Y.: Direction-based spatial skylines. In: MobiDE, pp.
73–80 (2010)
8. Guo, X., Zheng, B., Ishikawa, Y., Gao, Y.: Direction-based surrounder queries for
mobile recommendations. VLDB J. 20(5), 743–766 (2011)
9. Islam, M.S., Liu, C.: Know your customer: computing k-most promising products
for targeted marketing. VLDB J. 25(4), 545–570 (2016)
10. Islam, M.S., Liu, C., Rahayu, J.W., Anwar, T.: Q+Tree: an efficient quad tree
based data indexing for parallelizing dynamic and reverse skylines. In: CIKM, pp.
1291–1300 (2016)
11. Islam, M.S., Rahayu, J.W., Liu, C., Anwar, T., Stantic, B.: Computing influence
of a product through uncertain reverse skyline. In: SSDBM, pp. 4:1–4:12 (2017)
12. Islam, M.S., Zhou, R., Liu, C.: On answering why-not questions in reverse skyline
queries. In: ICDE, pp. 973–984 (2013)
13. Lee, K.C.K., Lee, W., Leong, H.V.: Nearest surrounder queries. IEEE Trans.
Knowl. Data Eng. 22(10), 1444–1458 (2010)
14. Lee, K.C.K., Schiffman, J., Zheng, B., Lee, W., Leong, H.V.: Tracking nearest
surrounders in moving object environments. In: ICPS, pp. 3–12 (2006)
15. Lee, M.W., Son, W., Ahn, H.K., Hwang, S.W.: Spatial skyline queries: exact and
approximation algorithms. GeoInformatica 15(4), 665–697 (2011)
16. Li, F., Cheng, D., Hadjieleftheriou, M., Kollios, G., Teng, S.: On trip planning
queries in spatial databases. In: SSTD, pp. 273–290 (2005)
17. Lin, Q., Zhang, Y., Zhang, W., Li, A.: General spatial skyline operator. In: DAS-
FAA, pp. 494–508 (2012)
18. Lin, X., Yuan, Y., Zhang, Q., Zhang, Y.: Selecting stars: the k most representative
skyline operator. In: ICDE, pp. 86–95 (2007)
19. Papadias, D., Tao, Y., Fu, G., Seeger, B.: An optimal and progressive algorithm
for skyline queries. In: SIGMOD, pp. 467–478 (2003)
20. Rocha-Junior, J.B., Gkorgkas, O., Jonassen, S., Nørvåg, K.: Efficient processing of
top-k spatial keyword queries. In: SSTD, pp. 205–222 (2011)
21. Sharifzadeh, M., Shahabi, C.: The spatial skyline queries. In: VLDB, pp. 751–762
(2006)
22. Shi, J., Wu, D., Mamoulis, N.: Textually relevant spatial skylines. IEEE Trans.
Knowl. Data Eng. 28(1), 224–237 (2016)
23. Sohail, A., Cheema, M.A., Taniar, D.: Social-aware spatial top-k and skyline
queries. Comput. J. 61(11), 1620–1638 (2018)
24. Son, W., Lee, M.W., Ahn, H.K., Hwang, S.W.: Spatial skyline queries: an efficient
geometric algorithm. In: SSTD, pp. 247–264. Springer (2009)
25. Wu, D., Yiu, M.L., Cong, G., Jensen, C.S.: Joint top-k spatial keyword query
processing. IEEE Trans. Knowl. Data Eng. 24(10), 1889–1903 (2012)
 Forecasting Crypto-Asset Price Using
Influencer Tweets

Hirofumi Yamamoto1(B) , Hiroki Sakaji1 , Hiroyasu Matsushima1 ,

Yuki Yamashita2 , Kyohei Osawa2 , Kiyoshi Izumi1 , and Takashi Shimada1
1
The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8654, Japan
[email protected]
2
Information Services International-Dentsu, Ltd.,
2-17-1 Minato Minami, Minato-ku, Tokyo 108-0075, Japan

Abstract. Nowadays, crypto-asset is gaining immense interest in the

field of finance. Bitcoin is a one such crypto-asset with a trading volume
of more than 5 billion a day. On social networking services, there are
people who have a great influence on social media users; these people
are called influencers. In this study, we focus on crypto-asset influencers.
We consider that influencer tweets may affect crypto-asset prices. In
this research, we propose a method to predict whether bitcoin price will
increase or decrease using influencer tweets. For this, we collect influencer
tweets to extract features using natural language processing techniques;
these features are used as input for machine learning methods, which
also use bitcoin price data. The results of our experiment show that the
influencer tweets affect crypto-asset prices.

Keywords: Text mining · Machine learning · Crypto-asset

1 Introduction

Crypto-asset1 is currently attracting interest of many researchers in the field of

finance. Bitcoin (BTC) [12] is a one such crypto-asset with a trading volume of
more than 5 billion a day. Furthermore, not only investors but also engineers are
paying attention to crypto-asset. Because of the existence of blockchain technol-
ogy, originally devised for crypto-asset, its application to other fields is expected.
Therefore, there have been many tweets on the topic of blockchain technology
on Twitter.
In addition, Twitter data is beginning to gain attention from institutional
investors. In 2011, Bollen et al. showed that moods states obtained from tweets
are useful for forecasting the Dow Jones Industrial Average [1]. Their research
1
Crypto-asset is often called “Cryptocurrency.” However, since the Financial Services
Agency of Japan discusses that cryptocurrency should be called “crypto-asset,” we
call it crypto-asset in this paper. https://www.fsa.go.jp/news/30/singi/20181214.
html.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 940–951, 2020.
https://doi.org/10.1007/978-3-030-15032-7_79
 Forecasting Crypto-Asset Price Using Influencer Tweets 941

had a significant impact on many researchers and investors. We aim to forecast

BTC price using tweets, and not stock prices, by referring to their research.
On social networking services (SNSs), people who have a great influence on
users are called influencers. We focus on crypto-asset influencers. We consider
that influencer tweets may affect crypto-asset prices, i.e., we hypothesize that
tweets of influencers will affect crypto-asset. We confirm this hypothesis by pre-
dicting BTC price using influencer tweets. If the predictability of a method using
Twitter data and BTC price is better than a method using only BTC price, we
believe that our hypothesis will be proved.
We propose a method to predict whether BTC price will increase or decrease
using influencer tweets. First, we gather influencer tweets and price data of BTC.
Next, we extract features from tweets using natural language processing tech-
niques to use as input for machine learning methods. Then, our method predicts
BTC price using the machine learning methods. We employ several machine
learning methods such as neural networks, support vector machine (SVM) [4],
and random forest [2]. Then, we conduct experiments to confirm our hypothesis.

2 Methodology

In this section, we describe our method and the features selected for the machine
learning methods. The novelty of our method is using both tweet data and
crypto-asset price for predicting future crypto-asset prices. We describe the fea-
ture engineering and our model in the following subsections.

2.1 Features

In this section, we introduce how we chose the features. Firstly, nouns, verbs, and
adjectives included in more than 0.5% of tweet words in English and Japanese are
selected, and their importance is evaluated by term frequency-inverse document
frequency (TFIDF). T F IDF (w, t) is calculated by the following formula:

N
T F IDF (w, t) = tf (w, t) log2 , (1)
df (w)

where tf (w, t) is the frequency of word w in tweet t. N is number of tweets in

the learning data set. df (w) is the tweet frequency of word w.
Two inputs are selected: one is input of only the TFIDF data of the tweet text;
and the other includes four more features, namely, number of likes and number
of retweets, number of tweets, number of followers, and the rate of change of
the price of BTC from the previous day. However, these inputs are not used for
biLSTM, instead word2vec [10]2 (200 dimensions) of tweet data is employed as
input.

2
https://radimrehurek.com/gensim/.
942 H. Yamamoto et al.

2.2 Machine Learning Methods

In this section, we introduce the machine learning methods used. To predict
price fluctuation of BTC from tweets, logistic regression (LR), random forest
(RF), and multilayer perceptron (MLP) are employed. Scikit-learn3 is adopted
for LR and RF, and MLP and biLSTM are implemented using Pytorch4 . MLP
is constructed with one input layer, one hidden layer, and one output layer.
Hyperbolic tangent (tanh) is employed as an activation function. Each hidden
layer is constructed with 100 units. biLSTM is also constructed with one input
layer, one hidden layer, and one output layer.

2.3 Our Method

To further investigate the relationship between the price of BTC and tweet data,
we propose the following model (Fig. 1).
First, if the change in price of BTC from previous day is 1% or more, the
coefficient and the intercept of the model are calculated by using LR. Then,
using the intercept and the coefficient, the probability that the result is one is
calculated using the following equation:
1
probability = (2)
1 + e−β−α·v
where α is the coefficient, β is the intercept, and v is the vector connecting
TFIDF and other tweet data used as input for LR. Formula 2 is called Vectorize
as shown in Fig. 1. RF input data is generated as follows.
Step 1: Compute TFIDF from the texts of tweets.
Step 2: Concatenate the number of favorites and retweets of the tweet and
the number of followers of that user to the calculated TFIDF. This is called
“tweet data.”
Step 3: Train LR with the tweet training data and calculate its coefficients and
intercepts. This trained LR is “Learned LR” in Fig. 1.
Step 4: For each tweet, the probability that the price of BTC will rise after one
day is calculated by the LR coefficient and intercept, tweet data (Formula 1).
This probability is called the “LR score.” This score is the respective number
of the “Vector B” part of the input vector in Fig. 1.
Step 5: For every minute, calculate the rate of change of the BTC price from
the previous day (this is called “Rate of change”). This is the “Vector A”
part of the input vector in Fig. 1.
Step 6: Concatenate the LR score of 12 tweets tweeted during one minute to the
rate of change of one hour from one point which the price of BTC changed
more than 1% over the next day. Substitute 0.5 for the lack of tweets. This
is the input data for RF and the input vector in Fig. 1.
For example, there are 7 tweets from October 9th between 21:58–21:59 as
shown in Table 1. The LR scores calculated using these tweets are as follows.
3
http://scikit-learn.org/stable/.
4
https://pytorch.org/.
 Forecasting Crypto-Asset Price Using Influencer Tweets 943

Table 1. Examples of tweets

0.473, 0.433, 0.508, 0.529, 0.530, 0.457, 0.606

Therefore, the input data of October 9th at 21: 59 is as follows.

[−0.041, . . . , −0.040, 0.473, 0.433, 0.508, 0.529, 0.530, 0.457, 0.606, 0.5, 0.5, 0.5, 0.5, 0.5]

Fig. 1. Our method

944 H. Yamamoto et al.

3 Experiment and Evaluation

In this section, we evaluate our method. To investigate the relationship between
BTC price change and influencer tweets, we conducted the following experiments:

– Experiments to ascertain the difference in accuracy of prediction by our

method and prediction without tweet data.
– Experiment using the biLSTM method with word2vec as input.
– Experiments in the same conditions as our method but using other
approaches.

Furthermore, we also assumed that not only the value but also the volume
and the tweet might be related; therefore, we also experimented to predict the
change of the transaction volume from the tweets.

3.1 Experimental Settings

Three experimental settings were set up. The first was to predict whether the
price for BTC rises or falls after a certain period, considering three durations:
one day, one week, and two weeks. Second was to predict whether the transac-
tion amount of BTC increases or decreases after a certain period. Again, three
durations were considered as in the case of price. Finally, we predicted whether
the price will increase or decrease only when the price of BTC increased by 1%
or more after one day or when it decreased.
The tweets were classified according to the following three criteria: language
(Japanese or English), tagged as BTC or GEN (general), and whether there was
BTC price change by 1% or more after the tweet. Here, if tweets included a topic
related to BTC, the tweets were tagged as BTC. Additionally, if tweets included
general topics related to crypto-asset, they were tagged as GEN. Based on to
the combination of these criteria, we obtained eight types of inputs.

3.2 Evaluation Data

In this section, we show how to generate data for evaluating our method. MeCab5
is employed as a Japanese morphological analyzer, and spaCy6 is utilized for
English tweets. Tweets starting with RT were removed as they are retweets. We
used tweet data from July 16 to October 24, 2018 for evaluation. We gathered
influencer tweets manually. First, we surveyed people who often post tweets
related to crypto-asset. Then, we traced their followers. Finally, we selected
influencers among these people, and collected their tweets. Tweet data from
July 16 to September 24 was considered as training data, and from October 4
to October 24 as test data. Table 2 shows our data set.
Among the features, TFIDF value was normalized by Scikit-learn. Additional
features, i.e., the number of favorites and retweets, rate of change of BTC price
5
http://taku910.github.io/mecab/.
6
https://spacy.io/.
 Forecasting Crypto-Asset Price Using Influencer Tweets 945

Table 2. Number of tweets of different types used as input

Train data Test data

English 25,986 14,542
Japanese 70,400 23,581
English (Tagged as BTC or GEN) 2,765 1,693
Japanese (Tagged as BTC or GEN) 7,915 2,431
English (rate of change > 0.01) 16,864 2,552
Japanese (rate of change > 0.01) 47,864 4,305
English (Tagged as BTC or GEN, rate of change > 0.01) 1,824 336
Japanese (Tagged as BTC or GEN, rate of change > 0.01) 5,508 413

from the previous day, and the number of followers of the that user were divided
by the maximum of their respective absolute values and the maximum value
was 1.

3.3 Results

Table 3 shows the prediction results of our proposed method and RF prediction
using only BTC price and not tweet data. In addition, we used the same data
as our method to predict the price fluctuation of BTC using LR, RF, and MLP
methods. Only those inputs with rate of change 1% or more on the next day
were used.

Table 3. Evaluation results

Our method Only value data LR RF MLP

ACC 0.666 0.646 0.435 0.608 0.499
PRE 0.665 0.639 0.344 0.517 0.403
REC 0.605 0.598 0.440 0.435 0.499
F1 0.629 0.613 0.386 0.472 0.446

“ACC” indicates accuracy, “PRE” denotes precision, “REC” indicates recall,

and “F1” is the f measure. Here, these values are defined as follows.

true condition
predicted TP FP
condition FN TN
946 H. Yamamoto et al.

When defined as above,

TP + TN
ACC =
TP + TN + FP + FN
TP
P RE =
TP + FP
TP
REC =
TP + FN
2 × P RE × REC
F1 = .
P RE + REC
Moreover, with word2vec as input, the result of learning with biLSTM is as
shown in the following table (Table 4).

Table 4. biLSTM results

English word2vec Japanese word2vec

1 day 1 week 2 weeks 1 day 1 week 2 weeks
PRE 0.515 0.120 0.424 0.521 0.157 0.411
REC 0.544 0.398 0.480 0.516 0.531 0.499
F1 0.529 0.184 0.450 0.518 0.242 0.451

In addition, the results of learning data with a change rate of 1% or more

with LR, RF, and MLP are as shown in Figs. 2 and 3. As input data, the number

Fig. 2. Value change, rate of change > 0.01

 Forecasting Crypto-Asset Price Using Influencer Tweets 947

Fig. 3. Value change, rate of change > 0.01, tagged as BTC or GEN

of favorites and retweets, the rate of change of BTC price from the previous day,
and the number of followers of that user were used in addition to the tweet text.
In English and Japanese tweets, two kinds of inputs, namely, only tweets tagged
as BTC or GEN, and all tweets, were used.
Finally, we examined whether transaction volume can be predicted from
tweet data. As input data, the TFIDF data of all tweets in English and Japanese,
and other four features (i.e., the number of favorites and retweets, the number of
followers of that user, and the rate of change of the BTC price from the previous
day), were used.

4 Discussion

Table 3 shows that our method outperforms other methods. Here, we focus on
the fact that the performance of our method is better than the performance
when using “only value data.” This result indicates that influencer tweets affect
crypto-asset price in this scenario, which is obtained using our method only.
We attempted to predict the price using other machine learning methods;
however, these methods did not achieve the results as shown in Fig. 3. Therefore,
our method can effectively utilize tweet data to forecast crypto-asset price.
From Figs. 2 and 3, we can see that using tagged tweets for forecasting gives
better results. Furthermore, in these results, LR with Japanese tweets outper-
forms other methods. Therefore, we decided to develop our method using LR.
948 H. Yamamoto et al.

Figures 4 and 5 confirm that volume forecasting does not depend on lan-
guages. From this result, we see that it is difficult to forecast volume using
tweets. Consequently, we developed a method for forecasting the price of BTC
using tweets.

Fig. 4. Volume change, input: English

Fig. 5. Volume change, input: Japanese

 Forecasting Crypto-Asset Price Using Influencer Tweets 949

5 Related Work
Bollen et al. showed that tweet moods are useful for forecasting the Dow Jones
Industrial Average [1]. In their research, they used self-organizing fuzzy neural
network for forecasting. As a result, they could predict rise and fall with an
accuracy of more than 80%. Schumaker et al. proposed a machine learning app-
roach for predicting stock prices using financial news article analysis [16]. Their
research predicts indicators and stock prices; it does not predict crypto-asset
prices.
Concerning financial text mining, Koppel et al. proposed a method for classi-
fying news stories of a company according to its apparent impact on the perfor-
mance of the company’s stock [8]. Low et al. proposed a semantic expectation-
based knowledge extraction methodology (SEKE) for extracting causal relation-
ships [9] that used WordNet [3] as a thesaurus for extracting terms representing
movement concepts. Ito et al. proposed a neural network model for visualizing
online financial textual data [5,6]. Additionally, their neural network model could
acquire word sentiment and its category. Milea et al. predicted the MSCI euro
index (upwards, downwards, or constant) based on fuzzy grammar fragments
extracted from the report published by the European Central Bank [11].
With regards to financial text mining for the Japanese language, Sakai et
al. proposed a method for extracting causal information from Japanese financial
articles on business performance [13]. Their method uses clues for extracting
causal information. Additionally, their method can gather clues using the boot-
strapping method automatically. Sakaji et al. proposed a method to automat-
ically extract basis expressions that indicate economic trends from newspaper
articles by using a statistical method [15]. Additionally, Sakaji et al. proposed an
unsupervised approach to discover rare causal knowledge from financial state-
ment summaries [14]. Their method extracts basis expressions and causal knowl-
edge using syntactic patterns. Kitamori et al. proposed a method for extracting
and classifying sentences indicating business performance forecasts and economic
forecasts from summaries of financial statements [7]. The classification method
is based on a neural network by using a semi-supervised approach.
These financial text mining researches target one language only. In contrast,
our method uses tweets of multiple languages.

6 Conclusions

In this paper, we proposed a method for forecasting BTC price. The method uses
RF and LR as machine learning methods and uses tweets and crypto-asset price
as features. In particular, the novelty of our method was to use learned coefficient
and intercept by LR as features of RF. Through this method, we showed that
influencer tweets affect crypto-asset. Moreover, we experimented using tweets in
two languages and also attempted to predict the volume of crypto-asset.
950 H. Yamamoto et al.

Our method considers a rate of increase of 1% or more; therefore, in future

research, we will attempt to find a method to predict crypto-asset price with-
out this limitation. Additionally, we will also simulate virtual trading using our
method.

References
1. Bollen, J., Mao, H., Zeng, X.: Twitter mood predicts the stock market. J. Comput.
Sci. 2(1), 1–8 (2011)
2. Breiman, L.: Random forest. Mach. Learn. 45(1), 5–32 (2001). https://doi.org/10.
1023/A:1010933404324
3. Fellbaum, C.: WordNet: An Electronic Lexical Database. The MIT Press, Cam-
bridge (1998)
4. Hearst, M., Dumais, S., Osuna, E., Platt, J., Scholkopf, B.: Support vector
machines. IEEE Intell. Syst. Appl. 13(4), 18–28 (1998). https://doi.org/10.1109/
5254.708428
5. Ito, T., Sakaji, H., Izumi, K., Tsubouchi, K., Yamashita, T.: GINN: gradient inter-
pretable neural networks for visualizing financial texts. Int. J. Data Sci. Anal.
(2018). https://doi.org/10.1007/s41060-018-0160-8
6. Ito, T., Sakaji, H., Tsubouchi, K., Izumi, K., Yamashita, T.: Text-visualizing neu-
ral network model: understanding online financial textual data. In: Pacific-Asia
Conference on Knowledge Discovery and Data Mining (PAKDD) (2018)
7. Kitamori, S., Sakai, H., Sakaji, H.: Extraction of sentences concerning business
performance forecast and economic forecast from summaries of financial state-
ments by deep learning. In: 2017 IEEE Symposium Series on Computational Intel-
ligence, SSCI, pp. 1–7 (2017). https://doi.org/10.1109/SSCI.2017.8285335, http://
ieeexplore.ieee.org/document/8285335/
8. Koppel, M., Shtrimberg, I.: Good news or bad news? Let the market decide, pp.
297–301. Springer, Dordrecht (2006)
9. Low, B.T., Chan, K., Choi, L.L., Chin, M.Y., Lay, S.L.: Semantic expectation-
based causation knowledge extraction: a study on Hong Kong stock movement
analysis. In: Pacific-Asia Conference on Knowledge Discovery and Data Mining,
PAKDD, pp. 114–123 (2001)
10. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed repre-
sentations of words and phrases and their compositionality. In: Advances in Neural
Information Processing Systems, pp. 3111–3119 (2013)
11. Milea, V., Sharef, N.M., Almeida, R.J., Kaymak, U., Frasincar, F.: Prediction of
the MSCI Euro index based on fuzzy grammar fragments extracted from European
central bank statements. In: 2010 International Conference of Soft Computing
and Pattern Recognition, pp. 231–236, December 2010. https://doi.org/10.1109/
SOCPAR.2010.5686083
12. Nakamoto, S.: Bitcoin: a peer-to-peer electronic cash system (2008)
13. Sakai, H., Masuyama, S.: Extraction of cause information from newspaper articles
concerning business performance. In: Proceedings of the 4th IFIP Conference on
Artificial Intelligence Applications & Innovations, AIAI 2007, pp. 205–212 (2007)
14. Sakaji, H., Murono, R., Sakai, H., Bennett, J., Izumi, K.: Discovery of rare causal
knowledge from financial statement summaries. In: The 2017 IEEE Symposium on
Computational Intelligence for Financial Engineering and Economics, CIFEr, pp.
602–608 (2017)
 Forecasting Crypto-Asset Price Using Influencer Tweets 951

15. Sakaji, H., Sakai, H., Masuyama, S.: Automatic extraction of basis expressions that
indicate economic trends. In: Pacific-Asia Conference on Knowledge Discovery and
Data Mining, PAKDD, pp. 977–984 (2008)
16. Schumaker, R.P., Chen, H.: Textual analysis of stock market prediction using
breaking financial news: the AZFin text system. ACM Trans. Inf. Syst. 27(2),
12:1–12:19 (2009). https://doi.org/10.1145/1462198.1462204
 Challenges and Strategies for Developing
Decentralized Applications Based on
Blockchain Technology

Thanh Chung Dao(B) , Binh Minh Nguyen, and Ba Lam Do

School of Information and Communication Technology,

Hanoi University of Science and Technology, Hanoi, Vietnam
{chungdt,minhnb,lamdb}@soict.hust.edu.vn

Abstract. This paper reveals challenges and strategies of data models,

deployment scenarios, business processes, and consensus models when
enterprises or individuals want to apply blockchain to their information
system in order to leverage advantages of the technology. A blockchain is
a digital ledger, which is encrypted and stored on multiple computers in
a public or private network and the data stored therein cannot be altered
or deleted by a single individual. The changes for existing applications
using blockchain bring difficulties for both managers and developers. The
decentralized manner is different from centralized ones in aspects of how
to store data in blocks, who has reading and writing permission, and
how to validate transactions among many parties. We illustrate our pro-
posal by developing an example blockchain application that enables to
authenticate Vietnamese certificates at schools and universities.

1 Introduction
After the advent of the Internet few decades ago, blockchain is judged to be the
next technology that could radically change the operation way of world econ-
omy during the fourth industrial revolution era. A blockchain is a digital ledger,
which is encrypted and stored on multiple computers in a public or private
network. The ledger consists of “blocks” that store electronic transaction data.
When these storage blocks are aggregated and connected in a sequence, the data
stored therein cannot be altered or deleted by a single individual. Instead, these
data will be verified and managed by different computers. As such, no central
entity can decide whether a data change is valid or not. All computers in the
network thus will use a unified protocol to check the transaction data and create
automatic consensus about accepting or rejecting the change. This verification
process, along with modern encryption methods, can effectively prevent unau-
thorized data changes on the ledger. Basically, blockchain brings three basic
functionalities, including:

• Anti-forgery information, in which the ledger is used to store, and encrypt

the original information that can not be forged.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 952–962, 2020.
https://doi.org/10.1007/978-3-030-15032-7_80
 Challenges and Strategies for Developing DApps 953

• Transaction verification, where the ledger is used to record transaction infor-

mation for traceability.
• Smart contract, in which the ledger is used to store the contract conditions.
When conditions are met, terms can be executed automatically.

Many organizations as well as enterprises today would like to apply

blockchain to their information systems in order to take all advantages of the
technology in increasing their operation performance. However, the changes for
applications bring several difficulties for developers. Firstly, the developers have
to consider about how to migrate existing data to “blocks”. Secondly, the orga-
nizations as well as their developer teams should think about the consensus
models, which suit the blockchain deployment scenario and the system busi-
ness flows. Thirdly, although the organizations can use directly permissionless
blockchain network like Ethereum. In the most cases, the organizations must use
permissioned blockchain, which is deployed by themselves in a certain architec-
ture that suit application requirements. This demands the careful designs for use
cases of systems. Finally, change of business processes of applications as com-
pared with existing one is also a big barrier for the developers when they rewrite
codes. Consequently, analyzing challenges and based on that to figure out the
right strategies for developing blockchain-based decentralized applications are
practical requirement today. In this paper, we concentrate on studying problems
when building application using blockchain. Then we propose strategies to over-
come the difficulties. We illustrate our proposal by developing a typical example
blockchain application that enables to authenticate Vietnamese certificates.
The paper has the following structure. Section 2 presents problems and res-
olution strategies in building blockchain-based applications in detail. Section 3
will discuss related work to highlight blockchain application domain nowadays.
We also would like to point out challenges when developing the applications here.
Section 4 introduces a typical example of decentralized application with degree
authentication function. While Sect. 5 describes the tests with the applications,
the last Section concludes our paper with some future works.

2 Problems in Developing Blockchain-Based DApps

2.1 Data Transformation and Mapping
One of main issues when developing a blockchain-based application is to figure
out which data should be stored on the network. This selection mainly depends
on features of the origin data source which can reside in many organizations and
can be stored by different methods. A huge number of input databases will create
a huge challenge for data integration task in the application. In this case, to col-
lect and represent data from heterogeneous sources, identifying a common data
schema is a useful approach. The role of this shared schema is to create a con-
solidated representation of names and coding for all different representations of
data and point out which data needs to be stored. Next, we need to perform data
transformation for input databases which can be heterogeneity of storage models
954 T. C. Dao et al.

such as relational databases (e.g., Oracle, MySQL, Microsoft SQL Server), key-
value databases (e.g., Redis), document oriented databases (e.g., MongoDB).
This data transformation process, therefore is a time consuming task that devel-
opers must to face with. They can either build different tools for each database
or make use of a mapping definition language for data transformation.
We summarize following criteria when considering deploying an application
on blockchain networks:
1. The number of input databases.
2. Model type of each input database.
3. Mechanism for data transformation.

2.2 Deployment of DApps on Blockchain Networks

The first issue is the scalability of a deployed application on a blockchain network
which means how many transactions (throughput) can be processed per second
and how low the latency in response to clients is. The number of transactions
increases when there are more participants using the DApp since transactions
play a role as a logbook recording all activities. More transactions can be pro-
cessed, more requests and responses can be received and sent.
The second issue is which type of blockchain networks we should choose to
deploy our DApps: permissioned (private) or permission-less (public) blockchain.
Bitcoin and Ethereum are examples of public blockchain networks since anyone
can join the network, create transactions, and read blocks’ information. More-
over, participants might be anonymous identification by using any private key
to sign transactions. By contrast, a private blockchain requires authenticated
identification by using registered keys. In permissioned networks, roles and per-
mission are defined, such as who can create, endorse, and validate transactions.
The third issue is how much resources including RAM, CPU, and network
bandwidth are necessary to deploy a DApp on a blockchain networks. Blockchain
has been evolved from supporting payment functionality into complex decen-
tralized applications, a so-called blockchain 3.0. For the purposes of payment
information and simple smart contracts, resource consumption is not significant
on full nodes. A node that stores the distributed ledger and executes requests is
called a full node.
We summarize following criteria when considering deploying an application
on blockchain networks.
1. Figuring out how many users the application will serve. The developer should
estimate the number of requests.
2. Choosing whether data in the application can be public or not.
3. Finding the good metric of using RAM, CPU, and network bandwidth.

2.3 Developing Blockchain-Based Business Processes

At the present, blockchain technology is expected to apply to many application
domains such as trading [13], data security [4], supply chain management [16],
 Challenges and Strategies for Developing DApps 955

certification authentication [7], voting [19], and so forth. The main reason is
that blockchain promises to improve efficiency in numerous business processes.
Even though the technology also create opportunities to build other operations.
But almost all blockchain-based applications are doing quite simply because of
fear of missing out, without a clear understanding of how it can be useful and
when it should be applied. In the terms of development and operation, current
business processes in information systems rely on data. A common procedure
today is moving the data that processes need between fragmented information
systems. For complex systems, the approach is expensive and sometime results
in outdated and inconsistent data. Even more concerning, lack of transparency
and trust is the consequence.
Meanwhile, current blockchain technology brings not only data integrity fea-
ture but also smart contract functionality, which could change operational pro-
cesses of applications because they can resolve the trust, and transparency issues
in the fragmented systems as described above. While the immutable data stored
in block with a certain cryptography mechanism that ensures the trustworthiness
for system business processes, smart contracts control the execution of transac-
tions between untrusted parties and ensure that contractual conditions are met,
and obligations are enforced.
Although the big barrier when applying blockchain technology to decentral-
ized applications is to replace traditional business processes or build new func-
tions using blockchain. This barrier thus appears in both cases of developing new
applications and restructuring existing systems. Based on the analytics about
disadvantage points of traditional application development manners as well as
blockchain technology advantages, we propose three criteria for business process
development as follows.

1. Designing which data will be stored in distributed ledgers. Instead of putting

all system data into blockchain network, some data can be stored in local
databases. Otherwise, only data of application that requires immutable will
be packaged inside blocks. This helps decrease storage, replication and lookup
costs for the developed decentralized applications.
2. Figuring out all business processes that read/write data stored in the dis-
tributed ledger to design and implement effectively.
3. Building smart-contract-based business processes for system functions, which
create transactions between untrusted parties.

2.4 Consensus Model

The consensus model depends on how to deploy a DApp on blockchain networks
and its business flows. If it is a public blockchain, the DApp can inherit the
underlying consensus model, such as proof of work and proof of stake. The DApp
developers only need to define smart contracts and how to write and retrieve data
in transactions. On the other hand, the consensus model is not often available
and the developers are free to define a relevant one. For example, practical
Byzantine fault tolerance (PBFT) can be configured during deployment in HF.
956 T. C. Dao et al.

We describe three criteria for choosing relevant consensus models for DApps as
follows:
1. Choosing whether participants can be anonymous or not.
2. Classifying who can validate and sign transactions.
3. Transforming current business processes into decentralized management.

3 Related Work
Regarding deployment of DApps, scalability, network type, and resource alloca-
tion are main research topics. While Bitcoin can only process approximately 35
transactions per second, Ethereum is changed to perform up to 15–30 ones each
second. In order to complement to Bitcoin, the lightning network is designed
to process thousands of transactions per second [15]. Almost all blockchain net-
works are public and there are few private ones, such as Hyperledger Fabric [2],
Quorum [12], and R3 Corda [5]. In the rise of inter-blockchain communication, if
a blockchain can accept tokens from other blockchains, it is classified as a cross
chain [20] but otherwise a single chain. A blockchain can be a side chain if it helps
a single chain improve scalability. DApps can choose to only pay transaction fees
based on the data size of transaction [21] or run the app on a decentralized plat-
form and pay fees for RAM, CPU, and network consumption [6].
A huge number of blockchain applications have been developed in recent
years [1,22] and [8,14]. We classify existing blockchain applications into four
main groups, namely finance (e.g. payment services [13] and [17], insurance
fraud prevention [3]), public services (e.g. voting [19], health care [18]), Iden-
tification and authentication management (e.g. supply chain management [16]),
and Transportation (e.g. car renting [9], ride sharing [11]). However, almost all
blockchain-based applications have their specific business processes that are dif-
ferent as compared with normal applications. For example, with a supply chain
management application, because the number of product data is very large, just
some data is chosen to stored inside blockchain network. Otherwise, the rest data
can be stored in local databases. Indeed, the transaction data which records a
product is moved from a distributor to another will be store in “blocks”. Mean-
while, names of distributors can be stored in traditional databases. This also
effects to operation processes that interact with both data types (i.e. in dis-
tributed ledger and in traditional database). For other applications with many
untrusted actors like voting or payment, smart-contract is a powerful function
that helps create trustworthy among them. The interaction with blockchain thus
will change the application operation processes.
Consensus model has been evolving fast in both industry and academia.
There are many white papers proposing new consensus models, such as dele-
gated proof of stake (DPoS) [6], practical byzantine fault tolerance (PBFT) [2],
and proof of space (PoSpace) [10]. EOS [6] uses voting system to choose rep-
resentatives called block producers and they have permission to create a new
block. There are 21 block producer having the most votes. Practical byzantine
fault tolerance can be deployed in HF to validate transactions.
 Challenges and Strategies for Developing DApps 957

4 Sample Blockchain-Based DApp Design

4.1 Motivation

In Vietnam, Ministry of Education and Training (MOET) takes responsibility

for all education and training at the national level. Structurally, the basic edu-
cation system in Vietnam has complete all levels including pre-school, primary,
secondary, high school, and higher educations. Besides full-time education, there
are in-service (part-time), and distance learning types in Vietnam. In Vietnam,
there are also many training education centers, which offer the huge number of
short courses like English, informatics, and other specialties. With large number
of these centers, the management, operation audit as well as training quality
evaluation of government agencies are quite difficult. As a result, certificates
issued by the centers usually do not have quality assurance and origins.
Technically, no one can ensure the trustworthiness of diploma or certificates
issued by the educational institutions except themselves. Although several of
them update the lists of graduates on their official websites, most institutions
do not publish the certificate and diploma data of learners. On the other hand,
in fact to authenticate precisely a learner certificate, not only the certificate
database but also the training process data of the learner also need to be pro-
vided by education institutions. Until now there are no efficient way to authen-
ticate certifications in Vietnam. Hence, the need of having a system that can be
authenticate these documents and training processes is very urgent in Vietnam
nowadays.

Fig. 1. System architecture

958 T. C. Dao et al.

4.2 ECefblock DApp’s Deployment and Consensus

We name blockchain-based DApp for authenticate grades and certificates in

schools and universities ECefblock. It has two main components described in
Fig. 1. Its interface is deployed on a centralized server and users access the
ECefblock application through this interface. Data of ECefblock is written to
a blockchain network by using smart contracts. There are two methods of data
writing: packaging those data directly as a transaction [21] and calling a smart
contract on the blockchain [2]. The interface can be considered as a View and
the chosen blockchain is a Model in MVC programming model.
Data in the application is private, so a permissioned blockchain network
is relevant, such as Hyperledger Fabric [2] and Quorum [12]. Users consists of
students, teachers, school managers, and MOET. Students have only reading
permission of their own grades and certificates. Teachers can create grade trans-
actions and school managers can issue certificates. MOET has full permission of
writing and reading. In the current design, the application itself still run on a
central server, but its data is recorded on a blockchain network through smart
contracts. The consensus model is PBFT in order to validate transactions by
several parties.

Fig. 2. Data structures

4.3 Data Transformation and Mapping

Figure 2 describes four data structures used for data transformation to

blockchain network. They contain one input relational database, a common
schema, and two transactions. To consolidate different representations of data,
we make use of a common schema which defines unified names for attributes and
 Challenges and Strategies for Developing DApps 959

points out which data used for authenticating transcripts and degrees of learn-
ers. The data transformation process in this example application is performed
through a tool that is tailored to the input database. The result of this process is
two types of transactions: (i) the first type stores transcripts and (ii) the second
type stores certificates of learners.

4.4 Blockchain-Based System Business Processes

Education institutions thus play the role of defining training smart contract
conditions, which point out the number of credit course must be collected by
student before he/she can receive certificate. In other word, this contract is the
education programme with a courses set. Learners must complete the contract
conditions through collecting credits of courses. Once all the smart contract con-
ditions are met, a transaction of certificate issue is automatically created and
stored on blockchain network. Using our data mapping structure, the certificate
issue information will be sent to the education institution. Based on the cer-
tificate issue information, institution will issue both certificate in form of paper
and electronic (certificate codes) for learners. For enterprises that would like to
look up a certificate, they query data stored in blockchain transactions through
certificate codes issued for learners before.

5 Experimental Results
Our ECefblock DApp is implemented to run on Hyperledger Fabric [2] and its
interface is written in NodeJS. Hyperledger Fabric is a private blockchain and
developers can configure permission. Organization and ordering service are two
main components in HF. For example, an university can play a role as an organi-
zation. Throughput of the ledger in HF and latency of committing transactions
are measured in our experiments.
We measure the performance of writing and reading transaction to/from the
ledger. We use two clusters: our local lab machines and m5.2xlarge Amazon
EC2 instance. The hardware configuration of our lab nodes is weaker than the
m5.2xlarge instances on Amazon EC2. While the Amazon EC2 instances has 8
vCPUs and 16 GB of RAM, our lab node is equipped with 2 vCPUs and 2 GB
of RAM. We want to show the scalability when deploying on cloud services.

5.1 Transaction Writing

This experiment aims to show our ECefblock DApp is working on HF and how
fast data writing to the blockchain network is. We change a transaction rate for
each configuration and the transaction rate is how many transactions are submit-
ted to the blockchain per second. Figure 3a and c show writing performance on
Amazon EC2 and our lab cluster, respectively. The peak throughput on Amazon
EC2 cluster is 8000, but it is only temporary for small transaction rates. The
average throughput is 330 when the transaction rate is more than 32. On the
960 T. C. Dao et al.

8000

8000
15

15
Throughput Throughput
Latency Latency

12

12
6000

6000
Latency (second)

Latency (second)
Throughput (tps)

Throughput (tps)
9

9
4000

4000
6

6
2000

2000
3

3
0

0
0

0
2 4 8 16 32 64 128 2 4 8 16 32 64 128

Transaction arrival rate each second Request arrival rate each second

(a) Writing performance on Amazon EC2 (b) Reading performance on Amazon EC2

15

15
Throughput Throughput
Latency Latency
400

400
12

12
300

300
Latency (second)

Latency (second)
Throughput (tps)

Throughput (tps)
9

9
200

200
6

6
100

100
3

3
0

0
0

2 4 8 16 32 64 128 2 4 8 16 32 64 128

Transaction arrival rate each second Request arrival rate each second

(c) Writing performance on lab cluster (d) Reading performance on lab cluster

Fig. 3. ECefblock DApp performance

other hand, the peak throughput on our lab cluster is just less than 100 and it is
low (3–5) for the transaction rate less than 32. The reason is the network delay
on our lab cluster is higher than Amazon EC2’s one. We measure throughput
and latency based on response times at clients. Moreover, the latency on our lab
cluster is increased linearly, but it is nearly unchanged on Amazon EC2 cluster.

5.2 Transaction Reading

The purpose of this experiment is to figure out how long it takes to retrieve
data from HF for our ECefblock DApp. Although ECefblock still keeps data in
a local database, but for the trusted data, we should read it from the ledger
of HF. Figure 3b and d show reading performance on Amazon EC2 and our
lab cluster, respectively. While the peak throughput on Amazon EC2 is 4000,
it is just only 120 on our lab cluster. The latency of reading is similar to the
writing experiment. The latency of searching on our lab cluster is high, and
it is increasing for higher transaction rates. The reason is transaction retrieval
requires more CPU time and our lab cluster’ hardware is limited.
 Challenges and Strategies for Developing DApps 961

6 Conclusion and Future Work

In recent years, blockchain technology has been received a huge interest of devel-
opers and researchers because of its promising advantages. However, developing
a blockchain-based application is not straightforward because we need to firmly
understand foundation of this technology. Our aim in this paper, therefore, is
to show main challenges and strategies when developing decentralized applica-
tions based on blockchain technology. We identify four main challenges including
(i) data transformation and mapping, (ii) deployment of DApps on blockchain
networks; (iii) developing blockchain-based business processes; and (iv) consen-
sus model. For each challenge, we also point out useful strategies to solve the
challenge. Next, we present an example application to explain identified chal-
lenges and strategies. Finally, we evaluate our application in terms of writing
and reading performance.
In the future, we will extend the present challenges and strategies to provide
a more detailed overview for developers and researchers. In will be necessary to
study other problems such as privacy, cryptography, 51% attack, etc. Finally, we
also plan to extend the existing application to provide more convenient mecha-
nisms for users in checking the validity of transcripts and certificates.

Acknowledgment. The authors acknowledge financial support from National Foun-

dation for Science and Technology (NAFOSTED) of Vietnam.

References
1. Abeyratne, S., Monfared, R.: Blockchain ready manufacturing supply chain using
distributed ledger. Int. J. Res. Eng. Technol. 05(09), 1–10 (2016)
2. Androulaki, E., Barger, A., Bortnikov, V., Cachin, C., Christidis, K., De Caro, A.,
Enyeart, D., Ferris, C., Laventman, G., Manevich, Y., et al.: Hyperledger fabric: a
distributed operating system for permissioned blockchains. In: Proceedings of the
Thirteenth EuroSys Conference (2018)
3. Asia Insurance Review Magazine: India: 15 insurers in financial sectors’s first
blockchain project
4. Barclays: Barclays files two digital currency and blockchain patents with U.S.
patent office
5. Brown, R.G.: Introducing R3 Corda: a distributed ledger designed for financial
services, 2016 (2017)
6. EOS: Eos.io technical white paper v2. EOS.IO Documentation, vol. 1 (2018)
7. Gräther, W., Kolvenbach, S., Ruland, R., Schütte, J., Torres, C., Wendland,
F.: Blockchain for education: lifelong learning passport. In: Proceedings of 1st
ERCIM Blockchain Workshop. European Society for Socially Embedded Technolo-
gies (EUSSET) (2018)
8. Hamida, E.B., Brousmiche, K.L., Levard, H., Thea, E.: Blockchain for enterprise:
overview, opportunities and challenges. In: International Conference on Wireless
and Mobile Communications (2017)
9. HireGo: Decentralised shared mobility platform (2018). https://www.hirego.io/
10. Hønsi, T.: SpaceMint-a cryptocurrency based on proofs of space. Master’s thesis.
NTNU (2017)
962 T. C. Dao et al.

11. LaZooz: A value system designed for sustainability (2018). http://lazooz.org/

12. Morgan, J.: Quorum whitepaper (2016). https://github.com/jpmorganchase/
quorum-docs/blob/master/Quorum%20Whitepaper%20v0.1.pdf
13. Narayanan, A., Bonneau, J., Felten, E., Miller, A., Goldfeder, S.: Bitcoin and
Cryptocurrency Technologies: A Comprehensive Introduction. Princeton Univer-
sity Press, Princeton (2016)
14. Nofer, M., Gomber, P., Hinz, O., Schiereck, D.: Blockchain. Bus. Inf. Syst. Eng.
59(3), 183–187 (2017)
15. Poon, J., Dryja, T.: The bitcoin lightning network: scalable off-chain instant pay-
ments (2016). https://lightning.network/lightning-network-paper.pdf
16. Project Provenance Ltd.: Blockchain: the solution for transparency in product
supply chains (2018). https://www.provenance.org/whitepaper
17. R3 Consortium: R3 consortium (2018). https://www.r3.com/
18. Ruubel, M.: Estonian e-health authority partners with guardtime to accelerate
transparency and auditability in health care (2016)
19. Follow My Vote: Follow my vote (2018). https://followmyvote.com/
20. Wang, H., Cen, Y., Li, X.: Blockchain router: a cross-chain communication proto-
col. In: Proceedings of the 6th International Conference on Informatics, Environ-
ment, Energy and Applications (2017)
21. Wood, G.: Ethereum: a secure decentralised generalised transaction ledger.
Ethereum project yellow paper, vol. 151, pp. 1–32 (2014)
22. Zheng, Z., Xie, S., Dai, H.N., Chen, X., Wang, H.: Blockchain challenges and
opportunities: a survey. Int. J. Web Grid Serv. 14, 352–375 (2018)
 Design of Remote Heart Monitoring System
for Cardiac Patients

Afef Benjemmaa1(&), Hela Ltifi1,2, and Mounir Ben Ayed1,3

1
Research Groups in Intelligent Machines,
National School of Engineers (ENIS),
University of Sfax, BP 1173, 3038 Sfax, Tunisia
[email protected],
{hela.ltifi,mounir.benayed}@ieee.org
2
Computer Sciences and Mathematics Department,
Faculty of Sciences and Techniques of Sidi Bouzid,
University of Kairouan, Kairouan, Tunisia
3
Computer Sciences and Communication Department,
Faculty of Sciences of Sfax, University of Sfax, Sfax, Tunisia

Abstract. At present, the people monitoring, in particular outside clinical sites,

has become a major research concern. With the rapid technological advances,
remote monitoring is considered as a vital component of future e-Health sys-
tems. The techniques of the Internet of Things (IoT) offer the opportunity to
change the nature of health services in an omnipresent way, and to create them
based on the patients’ physical condition rather than their feelings. Nevertheless,
these new technologies have also made this area more difficult to manage and
manipulate. In this paper, we proposed a monitoring system that can send and
analyze the collected real-time data for effective decision-making in the case of
Remote Cardiac Patients. We propose to design and develop a Remote Heart
Monitoring System (RHMS) that combines two complementary technologies
(1) machine learning for generating intelligent valuable information, and
(2) visual analytics for gaining insight the collected real-time data. The success
of these systems is based on the quality of their design and development. For
this reason, we propose to use the multi-agent architecture. The developed
prototype was evaluated by 30 participants to verify its feasibility. It has been
validated in terms of utility and usability, RHMS time of response and multi-
agent modeling quality. The multi-agent modeling of a RHMS based on auto-
matic and interactive machine learning (visual data mining) presented interesting
results. It is a distributed architecture of a real-time monitoring system for
cardiac patients helping remote monitors to improve the management of their
out-of-hospital data.

1 Introduction

Staying in a hospital after an operation and waiting for recovery or for medical
treatment is very embarrassing, monotonous and yet expensive. Follow-up visits for
patients are usually tiring, especially for older people who need a few minutes of
consultation but must wait a long time [1]. Their monitoring process is based on

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 963–976, 2020.
https://doi.org/10.1007/978-3-030-15032-7_81
964 A. Benjemmaa et al.

repetitive activities, by taking vital signs values. In fact, if the patient’s condition is not
critical, such monitoring can be performed remotely at patient’s home. In this case, a
remote monitoring system is needed. We aim in particular to propose a real-time
Remote Heart Monitoring System (RHMS) that offers cardiac patients the opportunity
to remotely obtain advanced healthcare services [2]. Developing a system to predict
critical heart conditions may be the best way to increase the chances of detecting heart
defects and help saving many heart patients.
Generally, an effective analysis of data to prevent medical events is of potential
benefits for decision-making [3–5]. In fact, based on the rapid progress of the body
sensors and connected objects, machine-learning methods have to be applied to make
automatic remote prediction for patients’ surveillance [6]. It consists, in particular, of
analyzing real-time patient data in order to detect risk situations and generate best
decisions [7]. It uses supervised and unsupervised learning algorithms to discover data
sets properties for making predictions. These algorithms can be applied for data mining
and therefore for RHMS: data mining based RHMS [8]. Real-time aspect characterizes
data in such systems.
Another approach for making predictions consists in interactively representing data
sets using visualization techniques for visual interpretation and analysis: visual data
mining. It is an interactive machine learning approach aiming at interacting with
humans for improving their cognitive ability of learning [9].
In our work, we focus on the intersection of all these fields. It is the visual data
mining based RHMS that combines artificial intelligence with human intelligence in
decision-making. In fact, the main objective of such RHMS is to design advanced
automatic and interactive machine learning solutions in order to improve confidentiality
and communications security [10]. However, it is characterized by its interdisciplinary
and the heterogeneity of its communication devices, which leads to the complexity of
its decision-making activity. As a promising solution to deal with this complexity, we
suggest using the multi-agent architecture. Our main contribution in this paper is the
use of Multi-Agent (MAS) technology for modeling the automatic and interactive
RHMS. The introduction of MAS in our RHMS do not only aim to facilitate the
awareness and knowledge sharing among patients and physicians but also to offer fast
responses using distributed computing. It allows decomposing the RHMS function into
specific, interactive and automatic modules. These modules work as autonomous
entities and cooperate, so that the whole RHMS is working.
The paper is structured as follows. Section 2 introduces the theoretical background
of our research by presenting RHMS, visual data mining and multi-agent systems.
Section 3 describes the proposed multi-agent-based modeling of RHMS. Sections 4
and 5 provide the implementation and experimental validation of the proposed mod-
eling. Finally, Sect. 6 concludes the paper and suggests future work.

2 Study Background

Remote Heart Monitoring System (RHMS) is a specific decision support system. It

aims to collect real-time health data from sensors to provide care for cardiac patients in
their homes. It allows a simulated doctor/patient consultation that enables to see the
 Design of Remote Heart Monitoring System for Cardiac Patients 965

patient’s vital data, track and analyze their trends immediately. It consists of equipping
cardiac patients with a remote monitoring system for reducing unnecessary hospital and
emergency department visits [11, 12].
Several research projects and systems dealing with various diseases have been
developed. Shuicai et al. [13] in their work proposed a new physiological multi-
parameter remote monitoring system based on the (Browser/Server) B/S model. Kwon
et al. [14] developed a model-based method combined with conventional reasoning
such as multiple regressions and the boosting strategy. Botia et al. [15] developed a
system called ‘Necessity’ which helps to monitor patient living alone. It offers the
possibility of adapting with patient behavior. In [16], the authors used an android-based
Internet of Things (IOT) application for home community healthcare, which applied
multi sensor data fusion technique for data collection and cloud computing capacity for
medical data diagnosis. Zhang et al. [17] develop a multi-agent platform to support the
decision-making between stakeholders in a diabetic healthcare scenario for children.
Software agents provide intelligent decision support to the patients in order to decrease
the communication and coordination burden of the healthcare management, and thus,
improving the healthcare quality. This includes monitoring of the patient, decision
support to diagnosis, as well as meeting arrangements. However, shared decision
making is also not provided. The work of [18] proposed a new approach for remote
healthcare monitoring risk detection. Based on the concept of a multi-agent system,
authors present an architecture, which takes advantage of the technological diversity of
several DSS used in this research field. They suggest a real-time scheduling mechanism
by using several priority levels. Laleci et al. [19] proposed the SAPHIRE project using
a multi-agent system for the monitoring of chronic diseases both at hospital and in-
home environments based on a semantic infrastructure. The system is capable of
deploying and executing clinical guidelines in a care environment including many
disparate care providers having heterogeneous information systems.
In summary, surveillance systems are an essential tool of today’s health services.
Those who monitor multiple parameters are more useful than those who only monitor
one. However, the accuracy of analysing and transmitting a large amount of data for
multiple parameters is a significant problem that can even affect the lives of patients.
Machine-learning methods have to be applied to make automatic remote prediction for
patients’ surveillance. It consists, in particular, of analyzing real-time patient data in
order to detect risk situations and generate best decisions. It uses supervised and
unsupervised learning algorithms to discover data sets properties for making
predictions.

3 Multi-agent Modeling of Visual Data Mining Based RHMS

This section is organized into two sub-sections. The first one introduces the proposed
RHMS architecture and the description of these different layers. The second sub-
section represents the modeling of our architecture.
966 A. Benjemmaa et al.

3.1 System Architecture

Figure 1 presents an overview of our proposed solution based on MAS. We have
developed this solution to monitor and help cardiac patients to get immediate care
during emergencies. Our system will be able to send information stored in the patient’s
Smartphone by Internet of Things (IoT) devices to our system’s database via WIFI
connection to trigger subsequent processes. The system provides interactivity for
visualizing collected and prepared data from sensors, analyzing them, identifying new
interesting knowledge and progressively learning from them to generate guidance to
the patient under supervision. To design our architecture using multi-agent technology,
we have to identify and assign intelligent and cooperative agents to each task of the
visual data mining based RHMS. These agents can be organized into three layers:
sensing monitoring layer, data processing layer and Decision-making layer.

Fig. 1. System architecture

3.2 RHMS Layers

3.2.1 Sensing Monitoring Layer
This layer is done by an agent named Data collected agent responsible for the inter-
action of patients with IoT devices (sensors) for the acquisition and storage of data
(heart rate, blood pressure, temperature, oxygen saturation, etc.) via Bluetooth con-
nection. Patients are able to see their own story and even shared them with their
parents, doctors and others. These data help in the early detection of diseases and
emergencies cases such as Bradycardia, hypotension, hypertension and Tachycardia
through alarming system based on upper and lower threshold values.

3.2.2 Data Processing Layer

The performance of the RHMS depends on the efficiency of the data processing
module. Indeed, the doctors’ decision (diagnosis and treatment) is based on the data
analysis activities collected from the sensors. The data processing layer is the most
important layer that is responsible for data mining and recommendation to patient. The
data are stored in the database. Then, the data mining agent prepare the data (raw data
 Design of Remote Heart Monitoring System for Cardiac Patients 967

formatting, missing data values treatment, and attributes selection appropriate) to the
data mining algorithm. This algorithm will be then executed. An appropriate technique
for exploring medical data is the decision tree [20]. It can be selected because it is an:
(1) Automatic technique: it applies an algorithm to predict the value of a result
variable (represented by leafs) based on many input variables (represented by
inner nodes). The path from the root to the leaf through the inner nodes identifies
and describes the diagnosis of the patient. It can detect any possible deterioration
of the patient’s condition.
(2) Interactive technique: it as a simple representation. It graphically presents the
applied classification method allowing the decision-maker (i.e. remote monitor) to
interactively interpret the diagnosis, and to cognitively acquire knowledge.
The knowledge base agent evaluates the performance and predictive ability of the
knowledge extracted by the decision tree. This algorithm allows the remote-monitor to
visually interact with the decision tree by changing values, filtering paths, etc.
This knowledge is then translated by this agent into recommendation(s) to assist in
decision-making by a simple comparison between the knowledge:
For each new extracted knowledge do
Look for knowledge base of the patient
If find ok
Else
Look for knowledge base of others patient
If find ok
Else
New recommendation

Such interaction helps remote monitor to estimate the importance of the extracted
knowledge. The decision makers of our system, thanks to this base, build new
knowledge based on the synergy between the stored explicit knowledge and their tacit
knowledge (related to their experience).

3.2.3 Decision-Making Layer

The prediction agent in this layer is responsible to predict the state of the patient to
produce suitable decision for decision-makers using the validated recommendations
received from the data processing layer.
The three layers of our RHMS involve a common agent, which is the coordinator
agent (CA). It coordinates, negotiates and collaborates with other agents for resolving
conflicts, time synchronization and distributing resources. This agent communicates
with the other agents to demand services or delegate actions.
The three layers of our RHMS involve a common agent, which is the coordinator
agent (CA). It coordinates, negotiates and collaborates with other agents for resolving
conflicts, time synchronization and distributing resources.
968 A. Benjemmaa et al.

3.3 System Modeling

The IoT Devices agent transmits each reading directly to the smartphone, which in turn
stores and sends the reading directly to the database of our system. This direct trans-
mission helps the doctor focusing on patient care rather than storing data one by one.
This can reduce and eliminate the possibility of human errors when compiling medical
information. Figure 2 depicts the transmission of information extracted by the patient’s
heart rate sensor to the doctors. Indeed, the patient must be able to transmit this
information using the smartphone every 2 s at the frequency of 1 Hz.

Fig. 2. Flow chart of the RHMS

Figure 3 shows detailed system sequence diagram for both Wireless Sensors and
Android Smartphone data management.

Fig. 3. Sequence diagram for wireless sensors scenario

 Design of Remote Heart Monitoring System for Cardiac Patients 969

4 Prototype Implementation

In this section, we provide a description of the design and development of our multi-
agent architecture presented in the previous section. Forty heart patients (Above 65
year-old) were participated to the implementation and evaluation process of our system.
Each patient has been equipped with various portable sensors to collect medical
information. Each subject has been tested with respective conventional machines for
evaluation (the test duration is 10 min). All patients’ sensors transfer their data
continuously.
The calibration of the sensors was done with appropriate devices used in hospital.
For the collection of heart rate information for patients requiring continuous
monitoring, the Zephyr BT handheld sensor was selected. This choice was based on the
criteria of accuracy, reliability, cost, availability and comfort. In addition, low-power
Bluetooth technology allows this sensor to be used longer, increase system durability
and help to minimize the risk of false alarms.
The second measured parameter is the Blood Pressure (BP). Sometimes, an acute
BP trend can be seen as an early indication that helps medical experts to perform
diagnostics. We choose Omron Wireless Blood Pressure Monitor to collect and
transmit blood pressure information wirelessly. The selection of this sensor is based on
its handling criteria, its lightweight and its long-term operation.
For the measurement of the body temperature, we choose the G plus Bluetooth
based temperature sensor. Then the collected information of temperature is transmitted
wirelessly.
Figure 4 represents the transmission process from the sensors to the Smartphone
via Bluetooth. STX is the beginning of the text in ASCII character by delimiting the
message. The message ID is in the binary format, and used to identify the type of
message. For example, for the heart rate sensor, the standard message ID is 0  26.
The number of bytes within the data payload has been specified by the data length code
(DLC). Data payload contains the actual data transferred to the Smartphone’s listening
port with data ranges from 0 to 128 bytes. CRC is an 8-bit CRC polynomial of 0  8C
(CRC-8). The accumulator is initialized to zero before starting the CRC calculation.
ETX is a standard ASCII control character that terminates the message. The data stored
within the smartphone is then transferred to the database via Wi-Fi connection. For data
security, the smartphone keeps record history in the memory.

Fig. 4. Transmission process

970 A. Benjemmaa et al.

Bluetooth technology is a short wavelength radio transmission in ISM band from

2400 to 2480 MHz used for standard data transmission in short ranges. In this current
study, this technology is used as a serial port transmission having sufficient baud rate of
115200 bps.
The aim of our work is to develop a real-time RHMS for cardiac peoples. It
measures heart rate, blood pressure, body temperature and oxygen saturation using
wearable sensors. An android interface has been developed to collect this parameter.
This interface is composed of a set of modules: Patient data (contains personal infor-
mation of the patient: name, age, gender, and contact details), Users management
(connectivity status between the sensor and smartphone), and Real-time monitoring (cf.
Fig. 5).

Fig. 5. An android interface

Figure 6 represents the machine-learning agent implemented in our RHMS pro-

totype. The decision tree is an appropriate predictive technique [20]. It is simple, fast
and efficient, particular for small datasets. It presents the advantages of an easy
interpretation and the possibility for decision makers to compare the results with their
domain knowledge for decision validation and justification. Motivated by this, we have
developed this technique. The main objective is to build a predictive model that meets
the needs of users (remote monitors). It is based on patients’ real-time data and aims to
predict particular attributes for diagnosis. The prediction model of each new user inputs
is provided by comparing his/her data with historical data and based on similarity
between them.
Among the main objectives of our model is to translate the collected data from their
numerical format to an interactive visual form to help decision-makers (i.e. remote
monitors/doctors) to visually monitor the patient’s health status and to interact with its
user interface in real time by displaying the results. The vital signs of the patient can be
viewed at any time through the visualization agent on a single screen without having to
check several devices or wires interfering with the procedure (cf. Fig. 7).
 Design of Remote Heart Monitoring System for Cardiac Patients 971

Fig. 6. Decision tree visualization

Fig. 7. Alarming interface of remote patient monitoring system (high value of blood pressure
and heart rate)

The data processing layer allows providing validated patterns. These patterns have
to be interactively visualized, interpreted and finally integrated in the RHMS for
generating appropriate recommendations by the decision agent. The taken decision can
be interactively presented to the remote monitor by sending a request to the visual-
ization agent. The remote monitor interface agent is responsible of displaying these
visualizations to the remote monitor and then receiving him/her instructions. These
visual representations give doctors the option to directly interpret the patient’s condi-
tion by presenting the received parameters with specific colors: the normal values in
green and those of alert cases in red (cf. Fig. 8).
The developed RHMS has been accepted by the users (Patient, doctors and Staff).
Therefore, it was possible to deduce a first validation of the proposed architecture.
Different evaluations of our system are explained in the next section.
972 A. Benjemmaa et al.

Received parameters

Diagnostic + treatment

Fig. 8. Remote monitor interface

5 Evaluation and Discussion

5.1 Utility and Usability Evaluation

To measure the utility and usability of our system, an evaluation questionnaire was
used [16]. This evaluation is based on the criteria of ease of use, an overall utility and
satisfaction with the visualization tools. We have invited 30 participants to conduct this
evaluation. We provided each participant with a 5-min tutorial about the visualization
tools. The participants’ profiles are as follows (Table 1):

Table 1. User profile

Types of participants Number Age Skill
Knowledge- 10 Health Informatics 35–40 Experience in using computer:
intermittent professionals 15–17 years
Expert 10 physicians 35–45 Experience in using computer:
10–13 years
Patient 10 Patients 25–30 Experience in using computer:
8–10 years

Table 2 shows usability and utility questions related to our application and the
answers provided by the participants.
 Design of Remote Heart Monitoring System for Cardiac Patients 973

Table 2. Usability and utility evaluation

I I I partially disagree
I
partially disagree
agree
agree
1. The system helps me to collect data (BG, 20 6 2 0
insulin, diet....) faster and more efficiently
2. The system helps me to understand and 26 2 0 0
analyze my monitoring data faster and
effectively
Usability

3. It was easy to learn how to use the system 25 2 1 0

and become familiar with it
4. The system is convenient to use 25 3 1 1
5. The use of the system has made my daily 0 4 4 20
life more difficult
6. I like the system 20 6 2 0
1. The system helps me to communicate 26 2 0 0
with the physician quickly and effectively
2. The system helps me to make decisions 26 2 0 0
regarding my state
3. The system increases the flexibility of my 27 1 0 0
Utility

therapy
4. The system has helped me to reduce the 19 4 1 4
number of acute problems
5. The use of the system makes me feel 25 4 0 0
more secure in handling my disease
6. I would recommend the use of the system 26 2 0 0
to other patients

The overall results of utility and usability evaluation proved that our system is able
to provide a better accuracy while applying visualization technique. Most participants
had a positive opinion of the functionalities and utility of the system (28 out of 30
participants answered all the questions).

5.2 Running Time and Alarming Response Time

It is necessary to check if an agent can perform its tasks in a short time without losing a
lot of memory space. For this reason, we have measured the transmission time of an
alarm message. The role of the alarm monitor agent is to generate an alarm message if
the measured value exceeds the upper or lower limits of the threshold. The transmission
time of an alarm message to the doctor is 20 s.
Table 3 represents a comparison of the average transmission time of an alarming
message using Wi-Fi (the value for each type of alarm is the average value of 30
alarms) between our system and the system proposed by [21]. According to AHA the
ideal time to save life of a heart patient between a sudden fall or increase in cardiac
vital signs and sending an alarm message to the cardiologist is between 4 to 6 min,
which is called the golden period. In our system the time required to send an alarm
974 A. Benjemmaa et al.

message is less than this period. This proves the applicability of remote care monitoring
and its ability in continuous screening of patients residing at a considerable distance
from hospitals.

Table 3. Average data transmission time using Wi-Fi

Alarm for Our system average time Average time of [21]
Tachycardia 00:00:19 00:00:29
Bradycardia 00:00:20 00:00:30
Hypertension 00:00:21 00:00:31
Hypotension 00:00:22 00:00:33

The gain of 10 s presented by the Table 3 can be explained by the distributed

execution of the different stages of our RHMS, which is based on the parallel execution
of more than one task among the previously defined tasks. This advantage is offered by
our MAS architecture.

5.3 Capacity of Prediction

To assess the prediction capacity of our proposed system based on multi-agent system,
we compared it with its previous version [22]. Table 4 presents their confusion
matrixes based on 50 patients.

Table 4. Confusion matrix of the current version of the RHMS and its old version

Predicted results of the current version

Yes No Total
Yes True Positives= 12 False Positives= 3 15
No False Negatives = 2 True Negatives= 33 35
Total 14 36 50
The classification rate 90% = (TP+TN)/(P+N)
The positive prediction capacity 80%=TP/(TP+FP)
Observed results

The negative prediction 94,43%= TN/(TN+FN)

capacity
Predicted results of the old version [22]
Yes True Positives= 10 False Positives= 5 15
No False Negatives= 5 True Negatives= 30 35
Total 15 35 50
The classification rate 80% = (TP+TN)/(P+N)
The positive prediction capacity 66%=TP/(TP+FP)
The negative prediction 85%= TN/(TN+FN)
capacity
 Design of Remote Heart Monitoring System for Cardiac Patients 975

The confusion matrix of our prototype shows that the classification rate of the
current rate of the RHMS is equal to 90%. While the confusion matrix of its old version
shows its classification rate is equal to 80%. From this, we can conclude that the multi-
agent modeling of a machine-learning based RHMS improves its predictive capacity.

6 Conclusion

In this paper we have studied the importance of two approaches of machine learning:
data mining (as automatic machine learning approach) and visualization (as interactive
machine learning approach) for developing such RHMS, what we have called visual
data mining based RHMS. It is a predictive system that aims at reducing unnecessary
hospitalization and monitoring costs, improving treatment, providing reliable diagnosis
and accelerating decision-making.
In this paper, we have proposed a Multi-Agent design (architecture and agent
interaction modeling) of visual data mining based RHMS. The proposed architecture is
composed of three related layers: (1) sensing monitoring layer for collecting and
visualizing real-time date, (2) data processing layer based on the decision tree tech-
nique for analyzing, predicting and learning from data and extracted patterns, and
(2) decision-making layer for taken and executing decision. In each layer, a set of
specific intelligent agents is proposed. Each agent performs a set of parallel tasks
relating to the remote monitoring and data mining processes. They ensure multi-agent
tasks like coordination, communication, and sharing of resources in real-time, which
improves performance of our RHMS.
The designed system was developed and evaluated. The system has been validated
in terms of time of response, utility and usability. We plan to extend our work by
implementing another data mining techniques such as association rules or Dynamic
Bayesian Networks and compare their prediction abilities.

References
1. Ludwig, W., Wolf, K.H., Duwenkamp, C., Gusew, N., Hellrung, N., Marschollek, M.,
Wagner, M., Haux, R.: Health-enabling technologies for the elderly – an overview of
services based on a literature review. Comput. Methods Programs Biomed. 106(2), 70–78
(2012)
2. Rashid, M.S.A., Mustaffa, N., Keikhosrokiani, P.: Mobile healthcare locator system for heart
patients (MHLSHP). In: 2016 3rd International Conference On Computer And Information
Sciences (ICCOINS), 978-1-5090-2549-7/16/$31.00 ©2016 IEEE (2016)
3. Ltifi, H., Kolski, C., Ayed, M.B.: Combination of cognitive and HCI modeling for the design
of KDD-based DSS used in dynamic situations. Decis. Support Syst. 78, 51–64 (2015)
4. Ltifi, H., Ayed, M.B., Kolski, C., Alimi, A.M.: HCI-enriched approach for DSS
development: the UP/U approach. In: IEEE Symposium on Computers and Communications
(ISCC 2009), pp. 895–900 (2009)
5. Annicchiarico, R., Cortés, U., Urdiales, C. (eds.): Agent Technology and e-Health.
Whitestein Series in Software Agent Technologies and Autonomic Computing.
BirkhäuserVerlag, Basel (2008)
976 A. Benjemmaa et al.

6. Fox, J., Beveridge, M., Glasspool, D.: Understanding intelligent agents: analysis and
synthesis. AI Commun. 16(3), 139–152 (2003)
7. Revuelta-Zamorano, P., Sánchez, A., Luis Rojo-Álvarez, J., Álvarez-Rodríguez, J., Ramos-
López, J., Soguero-Ruiz, C.: Prediction of healthcare associated infections in an intensive
care unit using machine learning and big data tools. Springer, Switzerland (2016)
8. Rani, N.S., Vimala, K., Kalaivani, V.: A remote healthcare monitoring system for faster
identification of cardiac abnormalities from compressed ECG using advanced data mining
approach. In: Mohan, S., Kumar, S. (eds.) Proceedings of the Fourth International
Conference on Signal and Image Processing (ICSIP 2012). Lecture Notes in Electrical
Engineering, vol. 222. Springer, India (2012)
9. Holzinger, A.: Human-computer interaction and knowledge discovery (HCI-KDD): what is
the benefit of bringing those two fields to work together? In: International Conference on
Availability, Reliability, and Security, pp. 319–328 (2013)
10. Piette, F., Caval, C.: A multi-agent system for resource privacy: deployment of ambient
applications in smart environments. In: Proceedings of the 15th International Conference on
Autonomous Agents and Multiagent Systems (AAMAS 2016) (2016)
11. Lan, M., Alshurafa, N., Ghasemzadeh, H., Sarrafzadeh, M.: WANDA: an end-to-end remote
health monitoring and analytics system for heart failure patients. In: Wireless Health
Conference (2012)
12. Lee, S.I., Ghasemzadeh, H., Mortazavi, B.J., Lan, M., Alshurafa, N., Ong, M., Sarrafzadeh,
M.: Remote health monitoring systems: what impact can data analytics have on cost? In:
Wireless Health (2013)
13. Shuicai, W., Peijie, J., Chunlan, Y., Haomin, L., Yanping, B.: The development of a tele-
monitoring system for physiological parameters based on the B/Smodel. Comput. Biol. Med.
40(11), 883–888 (2010)
14. Kwon, O., Shim, J., Lim, G.: Single activity sensor based ensemble analysis for health
monitoring of solitary elderly people. Expert Syst. Appl. 39(5), 5774–5783 (2012)
15. Botia, J.A., Villa, A., Palma, J.: Ambient assisted living system for in-home monitoring of
healthy independent elders. Expert Syst. Appl. 39(9), 8136–8148 (2012)
16. Sung, W.T., Chang, K.Y.: Evidence-based multi-sensor information fusion for remote health
care systems. Sens. Actuat. A: Phys. 204, 1–19 (2013)
17. Zhang, P., Carlsson, B., Johansson, S.J.: Enhance collaboration in diabetic healthcare for
children using multi-agent systems. Int. J. Agent Oriented Softw. Eng. 4(2), 155–173 (2010)
18. Dhouib, M.A., Bougueroua, L., Węgrzyn-Wolska, K.: Combination of decision support
system (DSS) for remote healthcare monitoring using a multi-agent approach. In: Chaki, N.,
Cortesi, A. (eds.) CISIM 2011. Communications in Computer and Information Science, vol.
245, pp. 74–84. Springer, Heidelberg (2011)
19. Laleci, G.B., Dogac, A., Olduz, M.: SAPHIRE: a multi-agent system for remote healthcare
monitoring through computerized clinical guidelines. In: Whitestein Series in Software
Agent Technologies, pp. 25–44. Birkh¨auserVerlag, Basel (2007)
20. El Haddioui, A., El Mamoune, S., Loubna, C.: New system forecast e-health by using hybrid
of association rules and decision tree. In: 2014 5th Workshop on Codes, Cryptography and
Communication Systems (WCCCS), University, Faculty of Sciences, Tetuan, Morocco
21. Kakria, P., Tripathi, N.K., Kitipawang, P.: A real-time health monitoring system for remote
cardiac patients using smartphone and wearable sensors. Int. J. Telemed. Appl. 11 (2015)
22. Benjemmaa, A., Ltifi, H., Ayed, M.B.: Multi-agent architecture for visual intelligent remote
healthcare monitoring system. In: The 15th International Conference on Hybrid Intelligent
Systems (HIS 2015) in Seoul, South Korea, pp. 223–235, November 2015
 An Efficient Virtual Machine Placement
via Bin Packing in Cloud Data Centers

Aisha Fatima1 , Nadeem Javaid1(B) , Tanzeela Sultana1 ,

Mohammed Y. Aalsalem2 , Shaista Shabbir3 , and Durr-e-Adan4
1
COMSATS University Islamabad, Islamabad 44000, Pakistan
[email protected]
2
Farasan Networking Research Laboratory, Department of Computer Science
and Information System, Jazan University, Jazan 82822-6694, Saudi Arabia
3
Virtual University of Pakistan, Kotli Campus, Azad Kashmir 11100, Pakistan
4
Mohi-ud-Din Islamic University Nerian Sharif, Azad Jammu and Kashmir, Pakistan
http://www.njavaid.com/

Abstract. Virtual machine (VM) consolidation is an intelligent and effi-

cient strategy to balance the load of cloud data centers. VM consolidation
includes a most important subproblem, i.e., VM placement problem. The
basic objective of VM placement is to minimize the use of running phys-
ical machines (PMs). An enhanced levy based particle swarm optimiza-
tion algorithm with variable sized bin packing (PSOLBP) is proposed for
solving VM placement problem. Moreover, the best fit strategy is also
used with the variable sized bin packing problem (VSBPP). Simulations
are performed to check the performance of the proposed algorithm. The
proposed algorithm is compared with simple particle swarm optimiza-
tion (PSO) and the hybrid of levy flight and particle swarm optimization
(LFPSO). The proposed algorithm efficiently minimized the number of
running PMs. Matlab is used for simulations.

Keywords: Cloud computing · Particle swarm optimization ·

Levy flight algorithm · Virtual machine placement ·
Variable sized bin packing

1 Introduction

In the cloud environment, many tasks may require resources from multiple
resource providers [1]. There are a vast number of physical servers in the cloud
data center [2]. Virtualization technology provides a facility of virtual machines
(VMs). The users can run their applications on VMs. The facilities a virtual-
ization technology provides are sharing of memory, CPU and storage resources
[2]. Multiple applications on VMs perform a task submitted by a user [3]. A
bi-directional communication between a user and application needs to use the
network resources. In task processing, the capacity of network resources plays
an important role.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 977–987, 2020.
https://doi.org/10.1007/978-3-030-15032-7_82
978 A. Fatima et al.

There are two main parts of cloud computing: front end and back end. A user
can see only front end consists of a browser, connections etc. Back end comprises
of servers, components, data storage, hosts etc. [4]. Mathematically, the process
of managing and designing cloud computing is formulated as an optimization
problem [4]. The main objective is to balance the load of a cloud data center. Two
approaches are generally used to balance the load of a network, i.e., intelligent
task assignment and VM integration. VM integration includes VM migration
and VM placement. Efficient placement of VMs according to the capacity of each
host is the main concept of VM placement. VM migration includes the process
of moving tasks from over-loaded or underloaded server to normal server. VM
migration is triggered when a critical value is reached. Another technique to
balance the load of a network is intelligent task assignment. It is the process
of selecting the appropriate VM according to the size and capacity of a VM.
However, migration of a task increases the total cost. Intelligent task assignment
also increases cost. VM placement is an efficient way to balance the load of a
cloud data center. Most of the cloud computing problems including scheduling
and load balancing can be done through bin packing problem (BPP) variants.
This paper presents an enhanced levy based PSO with variable sized bin pack-
ing (PSOLBP) algorithm based on best fit strategy. Variable sized bin packing
is used to balance the load of a cloud data center. The initial steps are similar
to simple PSO. The process of making swarm in search space, finding local best
solution and global best solution is the same as PSO. If the probability is greater
than 0.5, the position and the velocity of each particle are updated using simple
PSO method otherwise levy flight is used to update the velocity and position of
the particle.

1.1 Contributions
In this paper, an efficient technique is proposed to solve the VM consolidation
problem in the cloud data centers. The main contributions of the work are listed
below:
• An improved levy based PSO algorithm is proposed to solve the VM place-
ment problem
• Variable sized bin packing is used to minimize the utilization rate of the
running PMs
• A best fit strategy is used to achieve an optimal solution without wasting
space of a running PM.
The remaining paper is organized as: Sect. 2 covered related work, proposed
algorithm is discussed in Sect. 3 while bin packing problem is presented in Sect. 4.
The results are described in Sect. 5. Conclusion and future work is written in
Sect. 6.

2 Related Work
In [1], the authors proposed a decentralized belief propagation-based method
(PD-LBP) to improve the performance of task allocation in the cloud environ-
 Efficient Resource Allocation 979

ment. In PD-LBP, the set of tasks and agents both change constantly. In this
paper, two steps (pruning and decomposition) are considered. The achievements
of the proposed method involve the best allocation of big scale tasks and effi-
cient performance in the dynamic cloud environment. However, PD-LBP is not
feasible, i.e., an agent can execute only one sub-task.
Guo et al. proposed a shadow routing based approach to solve the problem
of VM auto-scaling and VM-to-PM packing in [2]. The main objective of the
proposed approach is to minimize the usage of PMs by intelligently packing
a number of VMs in it. It is done to save energies and minimize operational
cost. The proposed algorithm runs continuously without the need to resolve the
problem “from scratch”. It adapts automatically according to the demand of
requests. However, when there are a large number of VMs in a single PM, there
must be a chance of congestion in it.
In [5], Jensi et al. enhanced PSO by introducing a levy flight to update the
velocity of a particle. The proposed algorithm is verified on 21 test functions.
Proposed approach enhanced the capability of global search and increased con-
vergence efficiency. However, this algorithm is used to solve linear problems.
Mirjalili et al. proposed a multi objective gray wolf optimization algorithm
in [6]. It mimics the hunting behavior of grey wolves. A fixed size archive is also
integrated to save and retrieve Pareto optimal solutions. However, the proposed
algorithm cannot handle uncertainties.
In [7], authors formulated cost as a function of consumed and overhead energy
by the server. This paper provides an efficient VM placement approach. However,
there is no guarantee of renewable energy availability.
Chekired et al. proposed a priority assignment algorithm in [8]. Four prior-
ity levels are defined for charging and discharging vehicles to stable grid load
especially during on-peak hours. Electric vehicles communicate with power sup-
ply station. Station take electricity from the grid. Grid then send data to the
cloud to store it permanently. However, the disposal of batteries results in global
warming.
The authors proposed a pricing scheme in [9]. Two types of instances are
considered. One is on-demand instance and other is reserved instance. The tariff
policy for on-demand instance is like pay as you go. Reserved instance is for the
users having long-term demands. However, the user has to give upfront payment
for reserved instance.
Wang et al. considered energy aware VM consolidation and VM migration
for green cloud computing in [10]. The authors proposed a space-aware-best-fit-
decreasing algorithm (SABFD). VM migration is also done on the basis of high
CPU utilization rate. The paper saves energy and assures service level agreement
(SLA). However, the migration cost is great when VMs migrate too many times.
A hybrid genetic-wind-driven (GWD) algorithm is proposed by Javaid et
al. in [11]. In smart grid, an energy management controller is designed for the
residential area based on the heuristic algorithm to flatten the load in grid area
network. The proposed work scheduled the load of a single and multiple homes.
However, there is a chance of delay whenever the request rate is high.
980 A. Fatima et al.

In [12], authors proposed network-topology-aware redundant VM placement

optimization algorithm to minimize the consumption of network resources. The
sole objective of a proposed technique is to increase the reliability of cloud
resources. However, this approach does not work for the complex cloud.
The authors in [13] proposed a novel orchestration method, i.e., a placement
algorithm to deploy the VMs automatically. It also manages the high availability
zone. The main objective of the paper is the global load balancing of multi-zone
clouds. The proposed work efficiently schedule the tasks of a cloud. However,
the cost is increased.
Vakilinia et al. proposed integer-quadratic-program (IQP) with linear and
quadratic constraints in [14]. The main objective of the proposed work is an
optimization of power in data center. The proposed work presented optimized
the power consumption of servers, migration cost, and network communication
via VM placement. However, the given approach does not work for the large-scale
system.
The authors in [15] proposed a hybrid artificial bee ant colony optimization
(HABACO) for resource management in the smart grid. The proposed work
enhanced the performance of cloud computing. The main focus of the presented
work is the efficient utilization of smart devices. However, the proposed algorithm
does not work well for multiple load balancing applications.

3 Proposed Levy Flight Particle Swarm Optimization

with Variable Sized Bin Packing

The PSO is a swarm-intelligence-based meta-heuristic technique for the solution

of the continual optimization problem [5]. It is a simple, intelligent and powerful
algorithm. In PSO, a particle is a candidate solution for an optimization problem.
A number of particles integrate together to make a swarm and search for a good
solution of a given optimization problem. A search space is a set of all possible
positions. Each particle has some position and it shows a possible solution of the
fitness function in d-dimension search space. Each particle i has two vectors in
search space [5]. The position of the particle consists of a value obtained through
the objective function. Each particle memorizes the position of the best solution
while moving in search space. Following equation is used to find the position of
a particle [5].
(t+1) (t) (t+1)
Xi = Xi + V i (1)
(t+1) (t+1)
where Xi represents the position of ith particle at iteration t + 1, Vi
is the velocity of ith particle at iteration t+1 and Vi is the velocity vector. A
particle moves with a velocity at each iteration of algorithm. Following equation
is used to find the velocity of a particle [5]:
(t+1) (t)
Vi = ω × Vi + c1 × rand() ⊕ (pbesti − Xit )
(2)
+c2 × rand() ⊕ (gbest − Xit )
 Efficient Resource Allocation 981

where ω is inertia weight, c1 is cognitive weighting factor, c2 is a social weighting

factor, rand () is a stochastic component of the algorithm which is 0.5 and ⊕
shows the element by element multiplication.
Inertia weight (ω) has a controlling effect of the previous velocity. Following
equation is used to calculate ω [5]:
ωmax − ωmin
ω(t + 1) = ωmax − (3)
tmax
where t is the current iteration, tmax and tmin shows the maximum and mini-
mum inertia weights and tmax is the maximum number of iteration. The fitness
value of each particle is calculated, pbest and gbest updates are performed. This
procedure is continued until the stopping criteria are satisfied. Levy flight is a
class of random walk whose step length is drawn from levy distribution. Levy
flight starts from one best-known location and generates a whole new genera-
tion at a distance which is randomly distributed according to levy distribution.
The new generation will be evaluated to select the most encouraging one. In
PSO, there is a problem of trapping and premature convergence in local optima.
Many researchers used different techniques to improve PSO. This work focuses
on updating the velocity using the levy flight method. The initial steps are simi-
lar to PSO like random distribution of particles in search space, evaluation of all
particles fitness value and the finding of pbest and gbest. When the probability
is greater than or equal to 0.5, the velocity and position of each particle are
updated using Eqs. 1 and 2. Otherwise, use Eq. 5 and the velocity of particle
becomes its position [5].
 
it
ω = 0.1 + 0.8 × 1 − (4)
M axit
(t)
Vit+1 = ω × Levywalk (Xi ) + c1 × rand() ⊕ (pbesti − Xit )
(5)
+c2 × rand() ⊕ (gbest − Xit )
The particles take long jump toward its best positions. In the distribution of
levy flight, β parameter performs a main role. In this paper, the value of β is
constant i.e., 1.5.
The proposed phenomenon avoided loss of divergence while updating the
velocity. This work improves the particle’s position after each iteration. The
next position of a particle is calculated using Eq. 6 [5].
Xit+1 = Vit+1 (6)
and
(t) (t)
Levywalk (Xi ) = (Xi ) + step ⊕ random(size(Xi )) (7)
where
(t)
step = stepsize ⊕ Xi (8)
The fitness value of each particle is calculated after updating its position and
velocity. Update pbest, if the particle has better value. Otherwise, there is no
need to update the pbest value. Then find the gbest value of the whole swarm.
Repeat the procedure to achieve global optimum value.
982 A. Fatima et al.

4 Bin Packing Problem

Bin packing is a process of fitting tasks neatly and efficiently inside the finite
number of the bins. The sole purpose of bin packing is to minimize the number
of PMs (bins) [4]. There are three main steps involved in bin packing problem.
• Lower bound for the problem
• First-fit algorithm
• Best-fit algorithm.
The best fit strategy is used in the proposed work.

4.1 Best Fit Algorithm

Arrange a group of tasks in descending order and then place them into a mini-
mum number of bins as shown in Fig. 1.

Fig. 1. Best fit algorithm

5 Simulation Results
The simulation results are discussed in this section. For comparison purpose,
the two algorithms are also implemented: PSO and LFPSO [5]. The proposed
algorithm is a hybrid of PSO algorithm and levy flight for VM placement based
on VSBPP. The best fit strategy is used in VSBPP. The goal of this paper is
to minimize the number of running PMs. The proposed algorithm is tested on
 Efficient Resource Allocation 983

Table 1. Benchmark functions.

S. No. Function Formula Dimension Search range

d
1 Sphere∗ f1 (x) = 2
i=1 xi 30 [−100, 100]
n
2 N oise∗ f4 (x) = 4
i=1 ixi + random()(0, 1) 30 [−100, 100]
 
3 Schwef el2.26+ f5 (x) = 418.9828 ∗ d − di=1 [xi sin( |xi |)] 30 [−100, 100]
d
4 Rastrigin+ f6 (x) = 10d + i=1 [x2i − 10 cos(2πxi )] 30 [−100, 100]
 
5 Ackley + f7 (x) = −20exp[−0.2 1/d d 2
i=1 xi ] 30 [−100, 100]

known unimodal and multimodal benchmark functions [5]. All the algorithms
are implemented on Matlab with Windows operating system using Intel Core
i7, 2.60 GHz, 3.88 GB RAM. The population size is based on the dimension,
i.e., 30.

5.1 Benchmark Functions

Five known benchmark functions are given in Table 1 in [5]. These test functions
are used to verify the adaptiveness of the proposed algorithm. Two types of
benchmark functions are used: unimodal and multimodal. The functions having
a single optimal solution are unimodal, i.e., sphere and noise in this paper. They
can be taken as a multimodal function in high dimensional search space. Multi-
modal functions have two or more optimal solutions. Three complex multimodal
functions are schwefel 2.26, rastrigin and ackley.

Fig. 2. Sphere
984 A. Fatima et al.

5.2 Comparison of Algorithms

Three algorithms PSO, LFPSO, and PSOLBP are tested for each function inde-
pendently for about 10 times on 30 dimension search space. PSOLBP outper-
formed other algorithms for three test functions, i.e., noise, rastrigin and ack-
ley. LFPSO performs better for schwefel 2.26. While simple PSO gets optimal
results for the sphere. Simple PSO works better for unimodal functions and being
trapped for multimodal functions. Proposed algorithm PSOLBP with bin pack-
ing performs much better than the simple PSO and LFPSO algorithms (Figs. 2,
3, 4, 5 and 6).

Fig. 3. Noise

Fig. 4. Schwefel 2.26

 Efficient Resource Allocation 985

Fig. 5. Rastrigin

Fig. 6. Ackley

5.3 PSOLBP

PSOLF outperforms other two algorithms when tested on benchmark functions

and then it is used with VSBPP to minimize the running PMs. The utilization
rate of PMs is minimized to save energies and cost. Maximum number of VMs
are packed into one PM using best fit strategy. New PM is allocated when an
old one is fully packed (Fig. 7).
986 A. Fatima et al.

Fig. 7. PSOLBP

6 Conclusion and Future Work

Cloud computing is turning into an emergent field. Cloud data centers are very
complex. The technology evolves causing the influx of a large amount of infor-
mation. The efficient management of cloud computing is needed. VM integra-
tion and intelligent task assignment are used to optimally manage the load of a
cloud data center. This paper proposed an efficient VM placement algorithm. An
enhanced PSOLBP is applied to the variable sized bin packing problem. This
is the hybrid of two algorithms: levy flight and PSO. The best fit strategy is
used in BPP. Extensive simulations are performed to verify the efficiency of a
proposed algorithm. The performance of cloud data center is enhanced. Five test
functions are used to verify the adaptiveness of the proposed algorithm.
In the future, the proposed algorithm will be tested on more benchmark
functions to test the adaptivity of the proposed work.

References
1. Kong, Y., Zhang, M., Ye, D.: A belief propagation-based method for task allocation
in open and dynamic cloud environments. Knowl.-Based Syst. 115, 123–132 (2017)
2. Guo, Y., Stolyar, A., Walid, A.: Online VM auto-scaling algorithms for application
hosting in a cloud. IEEE Trans. Cloud Comput. (2018, accepted)
3. Fu, X., Chen, J., Deng, S., Wang, J., Zhang, L.: Layered virtual machine migration
algorithm for network resource balancing in cloud computing. Front. Comput. Sci.
12(1), 75–85 (2018)
4. Abdel-Basset, M., Abdle-Fatah, L., Sangaiah, A.K.: An improved Lévy based whale
optimization algorithm for bandwidth-efficient virtual machine placement in cloud
computing environment. Cluster Comput. 1–16 (2018)
5. Jensi, R., Jiji, G.W.: An enhanced particle swarm optimization with levy flight for
global optimization. Appl. Soft Comput. 43, 248–261 (2016)
 Efficient Resource Allocation 987

6. Mirjalili, S., Saremi, S., Mirjalili, S.M., dos S. Coelho, L.: Multi-objective grey wolf
optimizer: a novel algorithm for multi-criterion optimization. Expert Syst. Appl.
47, 106–119 (2016)
7. Khosravi, A., Andrew, L.L.H., Buyya, R.: Dynamic VM placement method for
minimizing energy and carbon cost in geographically distributed cloud data centers.
IEEE Trans. Sustain. Comput. 2(2), 183–196 (2017)
8. Chekired, D.A., Khoukhi, L.: Smart grid solution for charging and discharging
services based on cloud computing scheduling. IEEE Trans. Ind. Inform. 13(6),
3312–3321 (2017)
9. Cao, Z., Lin, J., Wan, C., Song, Y., Zhang, Y., Wang, X.: Optimal cloud computing
resource allocation for demand side management in smart grid. IEEE Trans. Smart
Grid 8(4), 1943–1955 (2017)
10. Wang, H., Tianfield, H.: Energy-aware dynamic virtual machine consolidation for
cloud datacenters. IEEE Access 6, 15259–15273 (2018)
11. Javaid, N., Javaid, S., Abdul, W., Ahmed, I., Almogren, A., Alamri, A., Niaz,
I.A.: A hybrid genetic wind driven heuristic optimization algorithm for demand
side management in smart grid. Energies 10(3), 319 (2017)
12. Zhou, A., Wang, S., Cheng, B., Zheng, Z., Yang, F., Chang, R.N., Lyu, M.R.,
Buyya, R.: Cloud service reliability enhancement via virtual machine placement
optimization. IEEE Trans. Serv. Comput. 10(6), 902–913 (2017)
13. Moreno-Vozmediano, R., Montero, R.S., Huedo, E., Llorente, I.M.: Orchestrating
the deployment of high availability services on multi-zone and multi-cloud scenar-
ios. J. Grid Comput. 16(1), 39–53 (2018)
14. Vakilinia, S.: Energy efficient temporal load aware resource allocation in cloud
computing datacenters. J. Cloud Comput. 7(1), 2 (2018)
15. Zahoor, S., Javaid, S., Javaid, N., Ashraf, M., Ishmanov, F., Afzal, M.: Cloud fog
based smart grid model for efficient resource management. Sustainability 10(6),
2079 (2018)
 Commodore: Fail Safe Container
Scheduling in Kubernetes

Christos Christodoulopoulos and Euripides G. M. Petrakis(B)

School of Electrical and Computer Engineering, Technical University of Crete (TUC),

Chania, Greece
[email protected], [email protected]

Abstract. Kubernetes is a tool to facilitate deployment of multiple vir-

tualized applications using container technology. Kubernetes scheduling
mechanism orchestrates computing resources per application at runtime.
However, resource allocation is static, as the maximum amount of com-
puting resources that each application can use, must be reserved in
advance. If the application requests more resources than the maximum,
a fail scheduling event is generated. Although solutions to the problem of
automatic scaling in Kubernetes are known to exist and automatic scal-
ing is supported by cloud providers such as Amazon and Google, these
solutions are fully proprietary and not generic (e.g. do not apply to all
Kubernetes distributions). Our solution, referred to as “Commodore”,
is capable of allocating (or de-allocating) resources based on the actual
demands of running applications. Taking advantage of the virtualiza-
tion features of cloud computing, applications are deployed on worker
machines (nodes) as Virtual Machines (VMs). This not only results in
better utilization of computing resources (i.e. CPU, memory and network
are defined virtually) but also, in enhanced software security by isolat-
ing services or applications from each other. The experimental results
demonstrated that Commodore responds to the increasing (or decreas-
ing) resource demands of each application leading to significantly faster
response times compared to a non-auto scaled implementation.

1 Introduction
Scalability [5] refers to the ability of a system to continue to function well when
its size (i.e., in terms of computing resources) has to change (i.e. increase or
decrease) in order to respond to increasing or decreasing workloads. A system
is scalable if its performance changes uniformly (e.g. linearly) with the size of
the workload. Scalability can appear as a problem of infrastructure, where the
lack of computing resources can decrease the performance of the system or, as
a problem of architecture when the design of the system itself does not allow it
to scale.
This work focuses on the infrastructure aspect of the problem. If an applica-
tion is experiencing an increased workload, the addition of computing resources
(e.g. more CPU or memory) and a reconfiguration of the run-time environment
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 988–999, 2020.
https://doi.org/10.1007/978-3-030-15032-7_83
 Commodore 989

may solve the problem (vertical scaling). If this is not good enough to guar-
antee uniform change of performance, more instances of the application can be
spawned in the same or different physical or virtual machines (horizontal scaling).
Cloud computing along with virtualization technology facilitate such operations
by providing an abstraction layer that decouples the hardware from the envi-
ronment that runs on top of it and by supporting features like process isolation
and resource customization [2,5]. Containers1 provide the benefits of cloud vir-
tualization in a lightweight manner. Solutions to the problem of autoscaling of
resources in virtualized environments are known to exist [1,4].
Kubernetes2 Container orchestration and management system that auto-
mates deployment, scaling and management of containerized applications Each
application can run in many copies in clusters of nodes (each node is typically
deployed as a VM). Each node can run more than one instance of the applica-
tion. The maximum number of nodes is specified in advance. Inside a node, an
application is deployed in clusters of containers referred to as “pods”; each pod
is deployed as a separate Docker environment running one or more containers.
Kubernetes resumes responsibility for scheduling and managing each application
and its replicas inside each cluster. Kubernetes can also be configured to bal-
ance the traffic across clusters. If the system experiences a heavy workload and
a derating of performance is observed, the solution is to increase the number of
pods (horizontal scaling). An alternative would be to support vertical scaling
(in the example of [1] or live container migration between machines (e.g. VMs)
when there is no enough resources on a machine [1]. However, this solution is
rather static as it is constraint by the allocated resource capacity of the running
deployment and, it does not show how new VMs can be allocated or deallocated
according to workload.
Kubernetes can attempt to schedule an additional instance of the application
in an existing node. However, if the maximum number of nodes is reached,
Kubernetes fails to allocate an extra node and emits a fail event. Commodore
[3] is a solution to the problem of infrastructure scalability on a virtualized cloud
environment managed by Kubernetes. It adds infrastructure scalability features
to Kubernetes, by supporting addition or removal of computing resources (as
VMs) depending on cluster state. In Kubernetes, a pod is defined by specifying
the maximum amount of resources (i.e. CPU and RAM) it is allowed to consume.
Typically, Kubernetes scheduler decides where to put the new pod based on this
amount of resources and places the pod in a node only if the capacity of the node
is less that the resource requests of the pod (even though the actual resource
usage within the node can be low). Although simple and fast, this strategy is
not always optimal (i.e. may result in underutilization of resources).
As an example, consider a two node Kubernetes cluster with an applica-
tion deployed in it as illustrated in Fig. 1. The nodes are identical in terms of
computational resources (i.e. each can run up to two pods). The application is
deployed in 3 replicas (pods) running in these two nodes (i.e., one copy of the

1
https://www.docker.com/resources/what-container.
2
https://kubernetes.io.
990 C. Christodoulopoulos and E. G. M. Petrakis

application in Node 1 and two copies of the same application in Node 2). Each
pod consumes half of the computational resources of each node. Now consider
the following scenario:

• The application experiences heavy load slowing down response times (i.e. the
pods can not cope with the increased workload).
• The system observes the derating and decides to increase the number of pods
from 3 to 5 in order to reduce the load per running application (pod).
• The system instructs Kubernetes to schedule two pods in the existing nodes.
• Kubernetes tries to schedule two new pods.
• Only one additional pod can be scheduled in Node 1 (Fig. 1). No more
resources are available and the fifth pod can’t be scheduled.
• Kubernetes emits a fail event and can not take any further action.

Fig. 1. A Kubernetes deployment with two nodes and three instances of an application
(pods).

Kubernetes is lacking an optimal mechanism for providing the additional

resources (see Sect. 2 for technical detail). This is the problem Commodore is
dealing with that is, the problem of infrastructure scalability in a virtualized
environment managed by Kubernetes. Commodore service is capable of auto-
matic provisioning or de-provisioning of computational resources automatically
in a Kubernetes cluster: New work nodes can be added to deal with increasing
workloads or, un-used work nodes can be freed. Continuing our previous scenario
but with Commodore service in place:

• Commodore received the schedule fail event emitted by Kubernetes.

• Commodore assumes that a smaller node (i.e. a VM with half the resources
of the existing VMs) can be used to accommodate the fifth pod.
• Commodore contacts Kubernetes about the needed resources.
 Commodore 991

• Commodore inquires the infrastructure provider (i.e. the Nova service3 of

Openstack in our implementation) for a VM with the computational resources
that Kubernetes needs.
• The cloud provider creates a new VM.
• The new VM is initialized and bind to the Kubernetes cluster as Node 3 (as
illustrated in Fig. 2).
• Kubernetes schedules the fifth pod of the application to run in Node 3.

Fig. 2. A Kubernetes deployment scaled-up to three nodes and five pods.

2 Background and Contributions of the Work

In the following, existing solutions to the problem of scaling in Kubernetes envi-
ronments are discussed highlighting the contributions of Commodore.
Azure Kubernetes Service (AKS)4 (formerly known as Azure Container Ser-
vice) is the container management service by Microsoft Azure. It allows quick
deployment of a production ready Kubernetes (DC/OS2 or Docker Swarm) clus-
ter. It runs on a standard VM infrastructure using a container optimized Linux
image (a stripped-down version of Linux kernel) as host operating system. In
order to deploy a Kubernetes cluster on AKS, a user must use Azure CLI (Com-
mand Line Interface), which is a shell specific for Azure operations. A cluster can
be created by specifying the number of VMs needed. In terms of scaling, worker
nodes can be added or removed from the cluster using Azure CLI. Autoscaling
is not supported.
3
https://docs.openstack.org/nova/latest/.
4
https://azure.microsoft.com/en-us/services/kubernetes-service/.
992 C. Christodoulopoulos and E. G. M. Petrakis

Amazon Elastic Container Service (ECS)5 by Amazon Web Services (AWS)

allows management of Docker containers on a cluster of Amazon Elastic Com-
pute Cloud (Amazon EC2) instances (VMs). It can be used to launch and stop
containerized applications by making API calls, allows monitoring the state of
the cluster from a centralized service and integrates with many familiar AWS fea-
tures like Elastic Load Balancers, CloudTrail, CloudWatch etc. The cluster setup
is a single step process in which the number and flavor of EC2 instances needed
for the cluster is specified. The rest of the setup process, as well as the manage-
ment of those instances is handled by the ECS service. In terms of autoscaling,
ECS provides a mechanism which lets a user configure policies on how scaling
operations take place. A policy consists of a set of rules and a set of actions.
Rules typically refer to thresholds defined upon utilization metrics and actions
refer to scaling operations. An example policy would be the following: If CPU
utilization in all cluster nodes is above x%, create a new node with y amount of
computing resources and add it to the cluster. ECS, autoscaling operations are
driven my measurements of the actual computing resources consumed. In addi-
tion to scaling, ECS can move applications (tasks) across the cluster of nodes in
order to achieve better utilization, meaning that ECS’s scheduler operates based
on the actual run-time utilization of each application.
Google Kubernetes Engine (GKE)6 uses a proprietary Kubernetes distribu-
tion and runs on top of Google Compute Engine (GCE). The cluster setup pro-
cess, is carried out in steps during which, a developer can specify (a) the size of
the cluster in terms of VMs, (b) the flavor and the deployment location of these
VMs, (c) the operating system image for the VMs, (d) the Kubernetes version
to be deployed. GKE provides logging and monitoring tools (Stackdriver) which
can be enabled during the setup process. Autoscaling decisions are based on
CPU utilization, HTTP utilization (i.e. rate of requests), load balancing policies
or Stackdriver Monitoring metrics7 but does not apply an elaborate rule-based
auto-scaling policy similar to Amazon ECS.
Cluster Autoscaler (CA)8 is a Kubernetes plug-in (it is part of the official
Kubernetes project) that enables infrastructure scalability. Autoscaling decisions
are based on the amount of resources that pods request, not the amount of
resources they are actually using (as Amazon ECS and Google GKE do). CA
automatically resizes a Kubernetes cluster by adding new nodes to the cluster
if there are not enough computing resources available; conversely, if a node is
under-utilized and its pods can run on other nodes, CA moves the pods and
deletes the node. However, CA’s autoscaling is rather static: The minimum and
maximum size for each node pool in a cluster must be specified in advance and
CA makes rescaling decisions within these boundaries. On the one hand, it is
reasonable not to allow an application to scale without limits; on the other hand,

5
https://aws.amazon.com/ecs/.
6
https://cloud.google.com/kubernetes-engine/.
7
https://cloud.google.com/compute/docs/autoscaler/.
8
https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler.
 Commodore 993

this policy is not optimal and results (often) in under-utilization of resources

since CA reserves the maximum amount of resources in advance.
In regards to autoscaling, Amazon ECS and Google GKE are optimal or
almost optimal respectively. Both CA and Commodore rely on non-optimal
strategies: Autoscaling decisions and requests for new pods are based on the
amount of resources that pods request, rather than on actual utilization met-
rics (e.g. CPU, RAM). As explained in Sect. 1, this policy, although simple and
fast, may result in under-utilization of resources. CA, Google GKE and Ama-
zon ECS, may move pods from one node to another in order to achieve better
resource utilization. Commodore does not support this functionality (i.e. moving
pods among nodes is left as future work). Notice that, Amazon ECS and Google
GKE are fully proprietary solutions running on Amazon’s and Google’s platform
respectively. Commodore and CA can operate on any Kubernetes infrastructure.
Finally, unlike CA which makes scaling decisions within limits (i.e. the maxi-
mum size of the node pool must be specified in advance so that the maximum
amount of computational resources are reserved), Commodore is fully dynamic
(i.e. scales-up or down within limits or, if desired, as long as it is needed) but
without reserving any resources in advance.

3 Architecture

Commodore is implemented in Openstack. Application services are deployed

in worker machines (nodes) which are realized as Virtual Machines (VMs). This
allows for definition of virtualized resources including network, CPU and memory
leading to better utilization of physical resources while ensuring software security
by isolating applications from each other. Commodore runs in cooperation with
the Kubernetes master node (alternatively it can be implemented within the
master node). It makes periodic checks to the cluster state, by requesting the
latest events emitted by Kubernetes scheduler. Commodore acts when there
are scheduling events that fail (i.e. events emitted by the scheduler when it
fails to deploy one or more pods in the cluster). Then, a scale-up operation is
initiated following the steps described in Sect. 1. If there are no failed scheduling
events, Commodore checks if a scale-down operation can take place: Inquires
Kubernetes API server regarding what pods are deployed on each node. If there
are nodes with no pods deployed (empty nodes) Commodore removes them from
the cluster.
Commodore comprises of many interconnected components (or services)
shown in Fig. 3, the most important of them being: The Database, the Col-
lector Service, the Infrastructure manager, the Kubernetes manager, the Rule
Engine and the User Interface. For implementation details of all Commodore
services please refer to [3].
Commodore produces large amounts of usage data (logs) that account for
utilization scores from the cluster, events from the Kubernetes API server, as
well as history data regarding scaling operations. All data is stored in an Apache
994 C. Christodoulopoulos and E. G. M. Petrakis

Fig. 3. Commodore architecture.

Cassandra (NoSQL) database9 which suits best the nature of semi-structured

log data and for operations allowed on such data (ACID transactions are not
mandatory).
Collector (Fig. 4) collects utilization scores, stores them in the database and
exposes an API for accessing the database. It consists of two components, a
daemon process and a Web server. The daemon is a script that runs on each
cluster node. It collects utilization scores from the node at regular time intervals
(i.e. every second) including CPU, disk and memory usage and reports them to
the server. It is implemented as a Web application that exposes a set of REST
services that can be invoked in order to submit and retrieve utilization metrics.
It also performs a moving average aggregation on the incoming data, which
compute the average utilization of each node per minute. This aggregated score
is a more accurate representation of the actual resource utilization within the
node since instant resource usage metrics change sharply over time. These scores
are used only for visualization. Utilization scores may potentially power a more
advanced autoscaling mechanism based on actual resource utilization metrics
(similar to Amazon ECS). This is left as future work (see Sect. 5).
The User Interface (Web application) allows cluster administrators to moni-
tor the actual cluster load without the need of extra tools. It is responsible for
monitoring (a) the health status of all services, (b) the number, flavor and state
of the Kubernetes nodes, (c) the actual utilization of resources within each node
and (d) the pods that deployed within each node. It provides also an API inter-
face and it is invoked to get the statistics and metrics needed for monitoring and
visualization purpose.
Infrastructure Manager manages the cluster and allows CRUD operations to
be executed at infrastructure level like addition, resize or deletion of cluster nodes
(VMs). It exposes a set of RESTful Web services which can be invoked from other
services. and implements the following operations: (a) listing available flavors,

9
http://cassandra.apache.org.
 Commodore 995

Fig. 4. Commodore Collector.

Algorithm 1. Rule Engine - Timeout Loop.

1: procedure decide
2: decision, f lavor ← scaleU p()
3: if decision = ’up’ then
4: inf rastructureM anager.addN ode(f lavor)
5: saveDecision(decision, f lavor)
6: sleep(longT ime)
7: else
8: decision, nodeList ← scaleDown()
9: if decision = ’down’ then
10: for node in nodeList do
11: inf rastructureM anager.deleteN ode(node)
12: sleep(shortT ime)

(b) listing nodes and pods, (c) getting details of a specific node, (d) resizing
and deleting nodes from the cluster, (e) adding nodes, (f) listing events. The
Infrastructure Manager communicates with the cloud provider (i.e. OpenStack)
and Kubernetes Scheduler.
The Rule Engine implements infrastructure scalability functionality. It is
responsible for managing the number and the flavor of the worker nodes (VMs)
in use. It receives information about the cluster state from Infrastructure Man-
ager and decides whether or not an infrastructure change (i.e. scale-up or down)
is necessary. If a change is mandatory, Rule Engine initiates a sequence of opera-
tions for the purpose of adjusting the cluster (addition or deletion of machines).
The operation is triggered periodically by a timeout value which is set at run-
time. Algorithm 1 illustrates the basic logic that implements the service. Initially,
the method checks if a scale up operation is needed by calling the “scaleUp()”
method (Algorithm 2). This method returns a decision and a VM flavor. If the
decision is “up”, it means that Kubernetes scheduler has failed to deploy a num-
ber of pods on the existing cluster nodes (that information is retrieved from
Kubernetes Manager). scaleUp() computes also the resources that are needed in
order to successfully deploy the needed pods (i.e. by summating the requested
996 C. Christodoulopoulos and E. G. M. Petrakis

Algorithm 2. Rule Engine - Scale Up.

1: procedure scaleup
2: decision ← ’neutral’
3: events ← kubernetesM anager.getF ailedSchedulingEvents()
4: if events is empty then return decision, null
5: pods ← kubernetesM anager.getP ods(events)
6: if pods is empty then return decision, null
7: def aultCpu ← 0.1
8: def ualtM emory ← 128
9: reqCpu ← 0.02
10: reqM emory ← 0.0
11: for pod in pods do
12: containers ← pod.getContainers()
13: for container in containers do
14: if container requires cpu then
15: reqCpu ← reqCpu + container.getCpu()
16: else
17: reqCpu ← reqCpu + def aultCpu
18: if container requires memory then
19: reqM emory ← reqM emory + container.getM emory()
20: else
21: reqM emory ← reqM emory + def ualtM emory
22: f lavors ← inf rastructureM anager.getF lavors()
23: f lavor ← f indBestF it(f lavors, reqCpu, reqM emory)
24: if f lavor is empty then return decision, null
25: decision ← ’up’ return decision, f lavor

computing resources from all pods). Based on this score, the VM flavor that
best matches this score is returned. Then, a new node with the specific flavor is
created by invoking the “addNode()” function of Infrastructure Manager; finally
the thread sleeps. The time that the thread sleeps depends on the time the cloud
(i.e. OpenStack in our case) needs to start a new VM and initialize it.
If scale-up is not necessary the method checks if scale-down can be applied
by invoking “scaleDown()” method (Algorithm 3). The service returns a decision
and a “nodeList”. If the decision is “down”, it means that there are nodes in the
cluster with no pods deployed on them. Those nodes are added in “nodeList”. By
iterating through the “nodeList” the “deleteNode()” function of Infrastructure
Manager in invoked to remove these nodes from the cluster. Finally, regardless
of whether a scale-down operation can be applied or not, the thread sleeps. The
time that the thread sleeps is user defined (i.e. one minute in our case), since
node deletion is an instant process and there is no need for the whole process to
run more frequently.
 Commodore 997

Algorithm 3. Rule Engine - Scale Down.

1: procedure scaledown
2: decision ← ’neutral’
3: nodes ← inf rastructureM anager.getN odes()
4: for node in nodes do
5: if node.canbeDeleted() then
6: if kubernetesM anager.isN odeRegistered(node) then
7: pods ← kubernetesM anager.getP ods(node)
8: if pods is empty then
9: nodesT oDelete.add(node)
10: if nodesT oDelete is empty then return decision, null
11: decision ← ’down’ return decision, nodesT oDelete

4 Experiments

The experimental infrastructure comprises of three VMs deployed in OpesSrtack,

one running Commodore, one running Kubernetes master node and one running
a worker node with 1 CPU and 2 GB RAM (small flavor). Therefore, the cluster
starts with one worker node. Each node can run up to two pods or two copies of
an application (i.e, each pod is scheduled to run on 1/2CPU and 1 GB RAM).
We consider an application that computes all prime numbers from 1 through
106 and runs on 1 pod. To run an instance of the application we issue an HTTP
request. We measure the response time for an increasing number of requests
up to N = 500, from which 4 run concurrently. The average response time (over
N requests) for an application running on 1 pod is 5,000 ms. Response times are
obtained using Apache Bench.10
Next, we instructed Kubernetes to schedule the application on 2 pods. The
response time dropped to 2,600 ms. Kubernetes balanced the load between the
two pods. Commodore, did not perform any scaling operation (i.e. the scheduler
deployed the new pod on the same node successfully, since there is still 1 pod
available). Next, Kubernetes scaled-up to 4 pods. Kubernetes had no available
resources and emitted 2 fail events. Commodore received the fail events and
computed the needed resources. In order to deploy 2 more pods, Commodore
started a new small flavor VM. The response time dropped to 1,700 ms.
Next, the application scaled-up to 8 pods. Kubernetes could not deploy the 4
new pods and emitted 4 fail events. Commodore received the events, computed
the requested resources and started a new medium flavor VM (i.e. 2 CPU cores
and 4 GB of RAM) for the new pods. The response time dropped to 1,500 ms.
It is worth noticing that, response time did not improve as much as before (e.g.
from 2 to 4 pods). Lastly, we scaled the application up to 14 pods. Kubernetes
could not schedule the pods on the available nodes and emitted fail events.
Commodore started two new VMs (one medium and one small flavor VM). The
response time is 1,400 ms.

10
https://httpd.apache.org/docs/2.4/programs/ab.html.
998 C. Christodoulopoulos and E. G. M. Petrakis

The response time scales almost linearly until 4 pods. From this point onward,
scaling-up the infrastructure did not yield the same benefits as before (i.e.
resources are idle). Figure 5 shows the response time as a function of the number
of running instances of the application. Notice that creation of VMs may take
several seconds, however, latest versions of OpenStack reduced the VM creation
process to 2–3 s. An alternative would be to allow spawning VMs in advance.

Fig. 5. Response time.

5 Conclusions and Future Work

Commodore offers infrastructure scalability features to Kubernetes by allocating

or de-allocating resources based on cluster load. Compared to Cluster Autoscaler
(CA), Commodore is fully dynamic (i.e. scales-up or down without restrictions).
Autoscaling solutions by Google and Amazon, are fully proprietary (Amazon)
and are bound to specific platform (Google, Amazon).
Incorporating an elaborate decision-making mechanism for triggering the
scaling of resources is an important direction for future research. Commodore
collects utilization scores from the cluster nodes. Although the current imple-
mentation does not use these scores for autoscaling, implementation of such an
autoscaling policy is rather straightforward. We envision a mechanism based on
machine learning considering actual computation usage statistics received from
running pods and implementing autoscaling strategies such as [4]. This feature
would potentially power an additional service for applying pod-level operations
such as altering the requested resources of the deployed pods, changing the num-
ber of pods based on actual utilization and also, taking decisions on the optimal
placement of workload (by moving pods) across nodes.
 Commodore 999

To better support our claims of efficiency of the solution it is necessary to

test Commodore in a more realistic experimental set-up (e.g. a database holding
large amounts of real or synthetic data) and stress the application with high user
concurrency and high incoming data streams.

References
1. Al-Dhuraibi, Y., Paraiso, F., Djarallah, N., Merle, P.: Autonomic vertical elastic-
ity of docker containers with ELASTICDOCKER. In: International Conference on
Cloud Computing (CLOUD 2017), pp. 472–479, Honolulu, USA, June 2017
2. Bondi, A.B.: Characteristics of scalability and their impact on performance. In:
2nd International Workshop on Software and Performance, pp. 195–203, NY, USA
(2000)
3. Christodoulopoulos, C.: Commodore: fail safe container scheduling in kubernetes.
Technical report TR-TUC-ISL-07-2017, School of Electrical and Computer Engi-
neering, TUC, Chania, Greece, December 2017
4. Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A review of auto-scaling tech-
niques for elastic applications in cloud environments. J. Grid Comput. 12(4), 559–
592 (2014)
5. Sharma, P., Chaufournier, L., Shenoy, P., Tay, Y.C.: Containers and virtual
machines at scale: a comparative study. In: 17th International Middleware Con-
ference, pp. 1:1–1:13, NY, USA (2016)
 Dynamic Ranking System of Cloud SaaS
Based on Consumer Preferences - Find
SaaS M2NFCP

Mohammed A. Ikram(B) , Nabin Sharma(B) , Muhammad Raza(B) ,

and Farookh Khadeer Hussain(B)

Centre for Artificial Intelligence, Faculty of Engineering and IT, School of Software,
University of Technology Sydney, Ultimo, NSW, Australia
{Mohommed.Ikram,Nabin.Sharma,Muhammad.Raza,Farookh.Hussain}@uts.edu.au

Abstract. Software as a Service (SaaS) is a type of software application

that runs and operates over a cloud computing infrastructure. SaaS has
grown more dramatically compared to other cloud services delivery mod-
els (i.e. PaaS and IaaS) in terms of the number of available services. This
rapid growth in SaaS brings a lot of challenges for consumers in selecting
the optimum services. The aim of this article is to propose a ranking sys-
tem for SaaS based on consumer’s preferences called Find SaaS M2NFCP.
The proposed ranking system is based on measuring the shortest distance
to the minimum and maximum of the selected consumer’s non-functional
preferences. In addition, linguistic terms are taken into account to weight
the most important non-functional preferences. The proposed system is
evaluated against traditional SaaS ranking systems using data collected
from online CRM SaaS and achieved improved results.

Keywords: Cloud services · Service selection ·

Consumer’s preferences · Ranking system · Software as a Service

1 Introduction
Software as a Service (SaaS) is a type of software application that runs and
operates over a cloud computing infrastructure [16]. SaaS is a relatively new
software delivery model for software applications, also known as software on-
demand. SaaS is accessed and utilized via the global network infrastructure (i.e.
the Internet) rather than installing software on computer hardware on-premises
[15]. In recent years, there has been a significant growth in the adoption of SaaS
as it can reduce the cost of building software applications and services, espe-
cially for small and medium enterprises (SMEs) [14]. SaaS has therefore become
the leading and fastest growing software development paradigm [2,12]. There
are many types of SaaS available to consumers, such as customer relationship
management (CRM) and enterprise resource planning (ERP) to name a few. In
addition to this, large vendors such as Oracle, SAP, IBM etc. have started to
migrate their services onto the cloud as SaaS.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1000–1010, 2020.
https://doi.org/10.1007/978-3-030-15032-7_84
 Find SaaS M2NFCP 1001

In recent years, research on SaaS selection has gained much attention [3–
5,8,11]. However, none of the previous studies considered measuring the selected
non-functional/QoS attributes and their values as thresholds to rank and select
a service. In addition, previous works have proposed selection methods that deal
with a small number of services.
The aim of this paper is to propose a ranking system for SaaS, taking into
account the minimum and maximum values of the selected non-functional con-
sumers preferences as threshold values as well as weights of the most important
non-functional parameters. In addition, the proposed approach measures the
shortest distance to their non-functional preferences to rank the services based
on the consumer’s request.
The rest of the paper is organized as follows. Section 2 discusses the related
work on SaaS selection and ranking; Sect. 3 describes the proposed architectural
framework. The experimental setup and results are presented in Sect. 4. Finally,
Sect. 5 concludes this work with a summary.

2 Related Works
Selecting SaaS can be a tedious and challenging task for a consumer. There are
many different services available for consumers which can be accessed directly via
a web browser or an application program interface (API). Some large IT vendors,
such as SAP, IBM and Oracle, have launched their services to the public as SaaS
to reduce the cost for consumers when building their application infrastructures.
Therefore, consumers may be confused when trying to select the best available
services. To deal with this issue, researchers have proposed different frameworks
based on decision-makers [3–5,8,11], or by addressing the most important QoS
when selecting a SaaS [1,6,7,10,13].
When looking at SaaS selection from the decision-maker’s perspective, the
analytical hierarchy process (AHP) method has been widely used to weight the
most important criteria as well as to make service selection [5,8]. Badidi pro-
posed a framework for SaaS selection using an aggregate utility function [4].
Arman et al. proposed a selection method for cloud services based on a rating
vector to weight the important criteria and rank the services using the TOPSIS
MCDM technique [3]. Limam and Boutaba (2010) proposed a framework for a
reputation-aware system when making a SaaS selection [11]. To evaluate and
select each service, their approach aggregates the quality, cost and reputation
parameters into a single metric.
Several studies have been conducted to determine the importance of QoS
when selecting a SaaS. Burkon compares the SaaS characteristics with other
types of software applications to obtain the QoS for managing SaaS applications.
The author concludes that the main QoS features that are used for managing
a SaaS are availability, performance, reliability, scalability, security, support,
interoperability, modifiability, usability and testability [6]. Repschlaeger et al.
followed a design science research methodology to obtain the most important
criteria for selecting a SaaS. They conducted a systematic literature review,
1002 M. A. Ikram et al.

carrying out an extensive market analysis of 651 SaaS providers, and found that
there are six criteria for selecting a SaaS provider, namely maintenance and
service cycle, functional coverage, service category, user scaling, probability of
data and web browser compatibility [14].
However, the previous studies do not take into account the minimum and
maximum non-functional preferences as threshold values for ranking and select-
ing SaaS. In addition, the previous studies only considered a small number of
services. In this work, service selection is based on multi-criteria decision making,
whereas the previous research only considers the weight of the most important
criteria.

3 The Proposed Find SaaS M2NFCP Ranking System

The main contribution of this work is to rank and sort SaaS by measuring the
shortest distance to the consumer’s non-functional preferences. Our proposed
ranking system is called the Find SaaS Minimum and Maximum Threshold-
based Non-Functional Consumer’s Preferences (Find SaaS M2NFCP). In our
previous work [9], we considered single values of non-functional customer prefer-
ences. SaaS consumers should be able to represent their preferences as threshold
values. For instance, our proposed method can represent the price of a service by
providing a price range (for instance 20to40/month) for the consumers. More-
over, consumers can select the criteria preferences during the ranking process.
For example, a consumer can select the Availability and Service rating criteria
to rank the services, without any concerns about other selection criteria such as
Price, or Service reviewers etc. Some SaaS consumers prefer services with high
availability and a lower response time, irrespective of the price. This is an impor-
tant advantage of our proposed approach since the existing methods consider all
the criteria during the ranking process. In addition to this, our proposed method
uses linguistic terms to weight these criteria to rank and sort the services. The
main elements of the Find SaaS M2NFCP are as follows: (1) consumer’s request
handler (CRH), (2) non-functional/QoS attributes and (3) ranking system engine
(RSE). Each of these elements are discussed in the following sub-sections.

3.1 Consumer’s Request Handler (CRH)

This component of the Find SaaS M2NFCP interprets consumer preferences.

The ranking approach begins when a consumer submits a new service request
that includes functional and non-functional preferences of SaaS. After matching
with a SaaS functionality such as CRM SaaS, the consumer must present their
preferences for non-functional attributes. Following this, the consumer selects
the criteria that must be considered during the ranking process. For example,
a consumer (x) selects criteria such as service ratings, service price and service
founded to rank and sort the services. After this, the consumer must provide
their preferences for these criteria in two categories: (1) minimum and maximum
values for each criterion as threshold values; (2) the weights for each criterion
 Find SaaS M2NFCP 1003

as linguistic terms such as extremely important (EI) or important (I). After

gathering this information from the consumer, the consumer request handler
(CRH) generates the best-ideal service (BIS) and worse-ideal service (WIS) for
the ranking system engine which will rank and sort all the available services for
the consumer.
To deal with the linguistic terms to weight the importance of the criteria,
we apply the widely used fuzzy logic method in this work. There are different
techniques to weight the importance of criteria, such as ranking the criteria
starting from the most important to the least important and using the ratio
weighting technique. However, the linguistic terms for weighting the criteria
using the fuzzy logic concept are implemented in the different ranking systems.
The fuzzy logic method shows superior results when describing the consumer
preferences of weighting non-functional attributes.

Table 1. Linguistic terms with their fuzzy numbers

Linguistics term Triangular fuzzy number

Extremely important (EI) (0.9, 1.0, 1.0)
Very important (VI) (0.7, 0.9, 1.0)
Important (I) (0.5, 0.7, 0.9)
Somewhat important (SI) (0.3, 0.5, 0.7)
Not important (NI) (0.1, 0.3, 0.5)
Not very important (NVI) (0.0, 0.1, 0.3)
Not important at all (NIA) (0.0, 0.0, 0.1)

As shown in Table 1, seven linguistic terms have been allocated to describe

consumer preferences regarding the non- functional criteria. In addition, the
triangular fuzzy set has been introduced to deal with these linguistic terms.
During the defuzzification process for the linguistic terms, the center of gravity
method is selected to deal with the triangular fuzzy set as shown in Eq. (1).
d(A) = 1/3(a + b + c) (1)
Equation (2) calculates the weights for each criterion.
d(Aj )
wj = n (2)
j=1 d(Aj )

At this stage, a set of weights W = w1 , w2 , ...., wn is calculated which is based

on consumer preferences.

3.2 Non-functional/QoS Attributes

This section discusses the main attributes considered during the ranking process.
The importance of QoS in SaaS selection is one of the main foci in recent studies.
1004 M. A. Ikram et al.

In addition, QoS can be calculated in terms of quantitative or qualitative values.

However, this study ranks the services in numeric form representing each service.
We classify the QoS into three categories: (1) the QoS from the service provider
(2) the QoS from old consumer feedback (3) the QoS from third party monitoring
tools.

1. Service provider
A service provider launches their services along with multiple related QoS
attributes. One of the most important parameters is the price or the cost of
using the service. Most of the pricing models for the SaaS today include the
pay-as-you-go option. SaaS providers also offer other options, such as monthly
payments per user or paying upfront. In addition, most service providers offer
a one-month free trial for customers before they require a full contract for
their services. Mostly, the services offered during the one-month free trial
come with limited features. A second criterion that we can obtain from a
service provider is the history of a service or meta- data since the service
was launched. This provides a useful indicator in relation to the quality of
the service. Therefore, a SaaS with a longer history has a good impact and
instills more confidence in consumers when they are selecting relevant services.
Usually, services with a longer lifespan results in more user feedback and more
training resources which are available to the public. It does not reflect that
relatively new services are less credible, rather it provides more information in
the form of user feedback and ratings. Also, other useful criteria are considered
frequently, such as service features, offered training support, and customer
support. As all these criteria are considered qualitative, it is much easier to
define them in terms of good value or otherwise.
2. Third Party Monitoring Tool
This includes qualities that can be collected using publicly available web mon-
itoring tools. A number of web monitoring tools are available on the Inter-
net such as up-down, statusOK and site24x7. All these services are available
through subscription and can be used to monitor different resources on the
Internet. The majority of monitoring tools measure the performance of SaaS,
web services or a specific network by calculating availability, response time
and throughput. This research considers three attributes collected from a
third-party monitoring tool: (1) availability: which refers to the number of
successful invocations/total invocations and is calculated as a percent- age;
(2) response time: which refers to the time taken to send a request and receive
a response (in milliseconds); and (3) throughput: which refers to the number
of invocations for a given period of time.
3. Previous consumer feedback
There are many parameters that can be obtained from the feedback of previ-
ous consumers, such as performance and support. Due to consumer privacy,
many useful parameters cannot be obtained for assessment. However, two
important attributes are very useful and can help to gather important infor-
mation, such as service ratings and service reviewers. Typically, service ratings
are in the range of 1 to 5, where 5 means the service has a high rating and
 Find SaaS M2NFCP 1005

a lower value means the service received a lower rating. The second crite-
rion, service reviewers, is the number of previous consumers who assisted in
the rating of the service. Normally, services with high popularity indicates a
higher number of service reviewers.

3.3 Ranking System Engine (RSE)

The ranking system is based on measuring the shortest distance to the con-
sumer’s service preferences. It is calculated by gathering the upper and lower
threshold value of the consumer’s preference for each service criteria along with
the linguistic terms for weighting the preferences of the consumer request. The
RSE receives the best-ideal service (BIS) vector and worse-ideal service (WIS)
vectors as well as the weights of the criteria to rank and sort the services for the
consumer.
Let’s assume that the BIS = {bis1 , bis2 , ...., bisn } is the vector for the best-
ideal service and W IS = {wis1 , wis2 , ...., wisn } is the vector for the worse-
ideal service. In addition, we assume that W = {w1 , w2 , ...., wn } is the vector of
weights for the importance of criteria. The RSE performs a number of steps to
rank the services as follows.
Step 1: To normalize all the data, the BIS and W IS are added to the pool
of similarity functional services.
Step 2: The normalization process is necessary to keep the data in the range
[0,1]. Equation (3) normalizes the positive criteria which means that it prefers
the biggest value, especially for criteria such as availability and throughput. In
addition, Eq. (4) normalizes the data with negative criteria which means that
it prefers smaller values, especially for criteria such as the price and response
time.
N Fi j
ri j = m (3)
2
i=1 N Fi j
1/N Fi j
ri j = m (4)
2
i=1 1/N Fi j

Step 3: This step calculates the weighted normalized ratings using Eq. (5).

vi j = wj .ri j (5)

where wj is the weighting value of criterion j, and ri j refers to the normalized

value for service i and criteria j.
Step 4: Measuring the distance to the BIS and W IS using Euclidean dis-
tance. Equation (6) is used for measuring the distance to the best-ideal service.


 n
DBj =  (vi j − bis∗j )2 (6)
j=1
1006 M. A. Ikram et al.

where vi j is the weighted and normalized value for service i and criteria j.
Also, bis∗j denotes to the best-ideal service vector after the weighted normalized
process.
Equation (7) is used for measuring the distance to the worse-ideal service.


 n
DWj =  (vi j − wis∗j )2 (7)
j=1

where vi j is the weighted and normalized value for service i and criteria j.
Also, wis∗j denotes the worse-ideal service vector after the weighted normalized
process.
Step 5: Measures the similarity to the best-ideal service (CSB) using Eq. (8)
to obtain the most similar service to the consumer’s preferences. The service
that has the highest number is the service that has the most similarity to the
consumer’s preferences.
DWj
CSBj = (8)
DBj + DWj
Step 6: Ranking all the services using the CSB starting from the highest
number to the lowest number in order to rank and sort all the services according
to the consumer’s preference.

4 Validation and Verification

This section describes the experiment results from the proposed Find SaaS
M2NFCP method. We compared the efficiency of our method with two exist-
ing ranking methods ([3,4]) using the NDCG validation ranking mechanism. We
also introduced a new dataset which includes the top 20 cloud services for CRM
services from the Capterra web site.

4.1 Data Set

The dataset used in the experiments is based on the top 20 CRM SaaS from
the Capterra website. As shown in Table 2, the service price is based on monthly
usage per user, and the service founded criterion indicates the lifetime of the ser-
vice. Service ratings and service reviewers are based on the users of the Capterra
website. In addition, the availability, response time and throughput is gathered
from the Site24x7. The data collected from the monitoring tool is for the month
of September 2018. Table 2 shows the service dataset with all the criteria used
for the experiments.

4.2 Experiment Setup

Three algorithms were implemented for ranking and sorting the available CRM
SaaS using RStudio version 1.1.383. The experiments were run on a MAC oper-
ating system. In the first part, we run Find SaaS M2NFCP with 2, 4 and 6 crite-
ria, respectively. After this, we apply other methods with the same importance
 Find SaaS M2NFCP 1007

Table 2. Top 20 CRM SaaS services based on Capterra website

Number Service name Service Service Service Year Availability Response Throughput
price rating reviewers founded time
1 Salesforce 25 4 5869 1999 100 583 681
2 Zoho 12 4 881 1996 100 420 104
3 Odoo 20 4 110 2002 99.86 1499 35
4 Microsoft 65 4 1545 1975 100 335 596
dynamics
5 OroCRM 295 5 7 2012 99.92 508 118
6 HubSpot 50 4.5 865 2006 100 360 1194
CRM
7 Act! 10 4 182 1987 100 183 199
8 Pipedrive 15 4.5 1317 2010 99.81 673 126
9 Maximizer 60 4 49 1987 100 1452 52
10 SAP 60 4 58 1972 100 362 1345
11 Insightly 29 4 245 2010 100 1597 37
12 Oracle sales 600 4 43 1977 100 174 345
cloud
13 Apptivo 10 4.5 167 2010 100 255 555
CRM
14 Infusionsoft 99 4 533 2001 100 815 91
15 SugarCRM 40 4 183 2004 100 654 43
16 SuiteCRM 0 4 10 2006 100 3230 30
17 Sage CRM 45 3 25 1981 99.39 1839 79
18 Base 45 4.5 93 2008 100 229 517
19 Salesboom 14 4.5 22 2003 100 352 262
20 Bpm’online 25 5 67 2002 100 1009 214

of criteria which means that the weights used are the same as in our method.
Since our method uniquely considers the threshold values for the non-functional
criteria, we tested other methods without considering these values. Therefore,
our method shows the importance of considering the value of consumer’s non-
functional preferences when ranking the services. In the end, we measured and
compared the performance of Find SaaS M2TNFP against the other algorithms
using the NDCG metric which is widely used to measure the performance of
ranking systems.

4.3 Experimental Results and Discussion

In this section, the algorithms that are used to evaluate Find SaaS M2NFCP
are described briefly.
The framework for SaaS selection and provisioning (FSSP) [4] aggregates the
utility function, then determines the service that has the highest value and ranks
it first. This process is carried out throughout the available set of services. The
results from the FSSP do not consider the threshold values for the consumer’s
non-functional preferences which is then compared with the results from our
method.
Another important work using a consensus-based approach for selecting cloud
plans (CASCP) [3] used the TOPSIS method to rank cloud services. The TOPSIS
1008 M. A. Ikram et al.

method is widely used in multi-criteria decision making in several domains. How-

ever, it does not consider non-functional customer preferences as a range of
threshold values.
To evaluate the Find SaaS M2NFCP against the other algorithms, we employ
the normalized discounted cumulative gain (NDCG) metric, which is widely used
to evaluate ranking systems. NDCG works by providing the ideal descending
services’ ranking and predicting descending services’ ranking. The NDCG is
then calculated for the top-k using Eq. (9).
DCGk
N DCGk = (9)
iDCGk
where DCGk is the discounted cumulative values of the top-k for the predicted
service ranking system, while iDCG is the ideal value of the top-k for ideal
service ranking. The value of DCGk can be calculate using Eq. (10).
k
 relj
DCGk = (10)
j=1
log(1 + Rj )

Table 3. NDCG comparison of different ranking system

Approach NDCG (10) - 2 criteria NDCG (10) - 4 criteria NDCG (10) - 6 criteria
Find SaaS M2NFCP 0.846153057 0.831564531 0.868169158
FSSP 0.4487994 0.608195292 0.772991349
CASCP 0.487136044 0.59006853 0.75410915

Table 3 shows the evaluation results of our proposed method against the
selected algorithms. The NDCG value of our proposed method achieves better
results compared to the other methods. The best results (i.e. 0.86) are achieved
when the number of criteria is increased to 6. The same improvement can be
seen with the other algorithms as well. The FSSP method and CASCP gave
similar results since the non-functional values are not taken into account using
the same dataset.
Overall, consumer non-functional values is an important factor that should be
considered when ranking the cloud SaaS to represent the consumer’s preferences.

5 Conclusion
This paper proposed a ranking system for similar SaaS taking into account
the consumer’s non-functional preferences as threshold values (called Find SaaS
M2NFCP). This system is based on measuring the shortest distance from the
threshold values to the non-functional preferences. In addition, this work takes
into account the new quantitative values the consumer should consider when
 Find SaaS M2NFCP 1009

ranking SaaS. The performance of Find SaaS M2NFCP shows superior results
compared to the other algorithms by measuring the NDCG metric.
We intend to extend our method to cover qualitative non-function values
combined with quantitative values in order to rank SaaS.

Acknowledgment. This research is supported by Umm Al-Qura University, Saudi

Arabia.

References
1. Al-Shammari, S., Al-Yasiri, A.: Defining a metric for measuring QoE of SaaS cloud
computing. In: Proceedings of PGNET, pp. 251–256 (2014)
2. Alkalbani, A.M., Ghamry, A.M., Hussain, F.K., Hussain, O.K.: Blue pages: soft-
ware as a service data set. In: 2015 10th International Conference on Broadband
and Wireless Computing, Communication and Applications (BWCCA), pp. 269–
274, November 2015. https://doi.org/10.1109/BWCCA.2015.83
3. Arman, A., Foresti, S., Livraga, G., Samarati, P.: A consensus-based approach for
selecting cloud plans. In: 2016 IEEE 2nd International Forum on Research and
Technologies for Society and Industry Leveraging a Better Tomorrow (RTSI), pp.
1–6. IEEE (2016)
4. Badidi, E.: A framework for Software-as-a-Service selection and provisioning. arXiv
preprint arXiv:1306.1888 (2013)
5. Boussoualim, N., Aklouf, Y.: Evaluation and selection of SaaS product based
on user preferences. In: 2015 Third International Conference on Technological
Advances in Electrical, Electronics and Computer Engineering (TAEECE), pp.
299–308. IEEE (2015)
6. Burkon, L.: Quality of service attributes for Software as a Service. J. Syst. Integr.
4(3), 38 (2013)
7. Ghani, A.A.A., Sultan, A.B.M.: SaaS quality of service attributes. J. Appl. Sci.
14(24), 3613–3619 (2014)
8. Godse, M., Mulik, S.: An approach for selecting Software-as-a-Service (SaaS) prod-
uct. In: 2009 IEEE International Conference on Cloud Computing, CLOUD 2009,
pp. 155–158. IEEE (2009)
9. Ikram, M.A., Hussain, F.K.: Software as a Service (SaaS) service selection based
on measuring the shortest distance to the consumer’s preferences. In: Advances in
Internet, Data & Web Technologies, The 6th International Conference on Emerging
Internet, Data & Web Technologies, EIDWT-2018, Tirana, Albania, 15–17 March
2018, pp. 403–415 (2018). https://doi.org/10.1007/978-3-319-75928-9 36
10. Jagli, D., Purohit, S., Chandra, N.S.: Evaluating service customizability of SaaS
on the cloud computing environment (2016)
11. Limam, N., Boutaba, R.: Assessing software service quality and trustworthiness at
selection time. IEEE Trans. Softw. Eng. 36(4), 559–574 (2010)
12. McGrath, B.: Worldwide SaaS and cloud software 2015–2019 forecast and 2014
vendor shares. In: IDC, August 2015
13. Monteiro, L., Vasconcelos, A.: Survey on important cloud service provider
attributes using the SMI framework. Procedia Technol. 9, 253–259 (2013)
14. Repschlaeger, J., Wind, S., Zarnekow, R., Turowski, K.: Selection criteria for Soft-
ware as a Service: an explorative analysis of provider requirements (2012)
1010 M. A. Ikram et al.

15. Seethamraju, R.: Adoption of Software as a Service (SaaS) enterprise resource

planning (ERP) systems in small and medium sized enterprises (SMEs). Inf. Syst.
Front. 17(3), 475–492 (2015)
16. Tsai, W., Bai, X., Huang, Y.: Software-as-a-Service (SaaS): perspectives and chal-
lenges. Sci. China Inf. Sci. 57(5), 1–15 (2014)
 A Methodology for Automating the Cloud
Data Center Availability Assessment

Guto Leoni Santos1(B) , Daniel Rosendo1 , Demis Gomes1 , Leylane Ferreira1 ,

Andre Moreira1 , Djamel Sadok1 , Judith Kelner1 , Glauco Goncalves2 ,
Mattias Wilderman3 , and Patricia Takako Endo4
1
Universidade Federal de Pernambuco, Recife, Brazil
{guto.leoni,daniel.rosendo,demis.gomes,
leylane.ferreira,andre,jamel,jk}@gprt.ufpe.br
2
Universidade Federal Rural de Pernambuco, Recife, Brazil
[email protected]
3
Ericsson Research, Stockholm, Sweden
[email protected]
4
Universidade de Pernambuco, Recife, Brazil
[email protected]

Abstract. Organizations worldwide are rapidly migrating their IT ser-

vices and infrastructure to cloud data centers in order to reduce costs
and guarantee high availability, scalability, and security. Considering
that service downtime translates into major financial losses, new mecha-
nisms need to be developed to assess the availability of cloud data center
dynamically. However, data center availability analysis remains a com-
plex task and one that is prone to human error due the large number
of components and their interconnections. In this work we propose a
methodology for acquiring information about a data center infrastruc-
ture and, automatically, generating computational models to assess its
availability. We make use of the Redfish standard to acquire information
about the data center infrastructure, the main standard for data center
management. To demonstrate the applicability of our proposal, we con-
duct a study to analyze availability and failure costs of an application
hosted in a cloud and we compare different scenarios with redundant
servers according to the TIA-942 data center standard. Results show
that a lower tier level with redundant servers, in some cases, is more
suitable (more available and less costly) than higher tier levels without
redundant servers hosting a cloud application.

1 Introduction
Over recent years cloud computing consolidated itself as a powerful paradigm for
delivering Information Technology (IT) resources, such as server, storage, and
network, through the Internet. This technology is very attractive to companies,
since they considerably reduce their acquisition costs and can take advantage of
resources with high availability, scalability, security, and elasticity [3].
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1011–1023, 2020.
https://doi.org/10.1007/978-3-030-15032-7_85
1012 G. L. Santos et al.

Despite these advantages, even the shortest downtime of a cloud infrastruc-

ture can result in significant financial and reputation damage to cloud providers.
For instance, according to the International Working Group on Cloud Computing
Resiliency (ICWGR), an outage of one hour can result in a financial cost between
$89,000 and $225,000 for highly critical applications [16]. Another example to
illustrate the criticality of cloud availability is related to credit-card authoriza-
tion services, whose costs due to downtime reach about $2.6 million per hour in
transactions that cannot be completed [3].
Several factors can cause interruptions, including human errors, software fail-
ures, and cyberattacks [3,9]. Hence the cloud data center must rely on resource
redundancy to overcome possible failures. However, adding more redundant
devices and systems into a data center implies in a more complex infrastructure
to manage, once the additional components can fail and introduce additional
vulnerability. Therefore, flexible mechanisms to manage and assess the availabil-
ity of data center infrastructure are necessary. It is important to highlight that
this availability assessment needs to be performed automatically whenever new
devices are added or some devices fail.
Authors in [12] presented a tool able to transform an input of data center
model in Unified Modeling Language (UML) into a Petri Net model and then
estimate the data center availability. Despite having similarities between [12] and
our proposal, we mainly differ in terms of collecting data center information.
In our case, this is performed in an automatic way thanks to the use of the
Redfish standard. In addition, as we are using RBD to model the servers
present in a data center, our model is able to obtain a solution faster than a
SPN model, as explained in [14].
In [13] authors presented a tool called Redfish Conformance Test Tool
(RCTT) to allow customers to test their services in an automatic way, under-
standing specifications and identifying the compatibility between these specifica-
tions and the client system requirements. However, the RCTT does not provide
models to estimate data center availability. In [11] compares the Redish standard
against the IPMI (Intelligent Platform Management Interface) tool in terms of
reliability, security, latency and scalability using a benchmarking environment
for it. However, in contrast to our work, [11] does not use Redfish to collect
information and measure the data center availability.
Some of our previous works have addressed the availability of data centers,
designed with different availability levels and infrastructures [6,17,18]. However,
we considered fixed infrastructures and disregarded the fact that data center
resources can dynamically change, for example, when adding or removing new
servers, disk units, network devices, etc.
In this context, this work addresses the following main question: How to
automatically generate computational models to evaluate data center
infrastructure availability? To answer this question, we propose a method-
ology that acquires information directly from the data center hardware com-
ponents (using the Redfish industry standard API), and automatically gener-
ates computational models to assess the data center availability. We implement
 A Methodology for Automating the Cloud Data Center 1013

this methodology, and present some availability results considering different IT

subsystem configurations of a data center (based on the TIA-942 data center
standard).
The main contributions of this work are:

• a methodology to automatically generate computational models to estimate

data center availability; and
• a use case that demonstrates the methodology applicability in different
data center configurations according to the TIA-942 data center standard.

This paper is organized as follows: Sect. 2 describes the proposed method-

ology, detailing each step taken to assess the data center availability. Section 3
presents evaluations performed using our methodology to measure the availabil-
ity of a data center. Finally, we conclude with Sect. 4 and present future works.

2 Automating the Data Center Availability Assessment

In this section, we present the methodology that defines our approach to acquire
information from a data center infrastructure and automatically generate models
to estimate its availability. Figure 1 summarizes the proposed methodology that
is composed of two main steps. Firstly, we need to gather information about
the data center infrastructure, and then we can automatically generate the
computational models to evaluate the availability according to such collected
information.

Fig. 1. Methodology implementation: information acquisition and model generation in

the DCAV tool
1014 G. L. Santos et al.

2.1 Acquiring Data Center Infrastructure Information

The acquisition of information about the data center (IT subsystem composed
of many servers) is the first step to be accomplished. This step is relevant for
the methodology because commonly data centers are very complex systems com-
posed of hundreds (or more) equipment (e.g. compute, storage, and network).
Therefore, their one-by-one manual description and that of their interconnec-
tions, and their physical characteristics is a time-consuming task and it is prone
to human errors.
In this step, industry standards and protocols for data center software and
hardware management should be combined to deliver information about the data
center architecture. These include standards defined by the Internet Engineering
Task Force (IETF)1 , the Distributed Management Task Force (DMTF)2 , or the
Institute of Electrical and Electronics Engineers (IEEE)3 .
In our methodology, we use the Redfish industry standard and specification
defined by the DMTF, once it provides a rich set of highly detailed information
about server resources present in data centers. Through the Redfish REST API,
we can obtain information about hardware components of servers present at
the data center. Those resources are composed of processors, memories, network
interfaces, and disks. This data is provided in JSON based format, making it
easily interpretable by scripts and administrator programs.
Figure 2 illustrates an example of a JSON file gathered from Redfish. The
example shows three processors (CPU0, CPU1, CPU2) present at a server (called
System-4). Each processor has its own URL which can be used to retrieve pro-
cessor’s specific information, such as its manufacturer, processor architecture,
number of cores, etc. Data about other server components, such as memories,
disks, and network interfaces, are accessible in a similar way.

Fig. 2. Example of JSON gathered from Redfish API

1
https://www.ietf.org/.
2
https://www.dmtf.org/.
3
https://www.ieee.org/.
 A Methodology for Automating the Cloud Data Center 1015

2.2 Modeling the Data Center Infrastructure

The next step of our methodology consists in automatically generating availabil-

ity models based on the information obtained in the previous stage. The idea is
to translate the data acquired from Redfish into stochastic models to calculate
the data center availability. To achieve this, mathematical modeling languages,
such as Reliability Block Diagram (RBD), Markov models, Product-Form Queu-
ing Networks, Stochastic Petri Net (SPN) models, among others, could be used
to model complex systems. The model should cover components interactions and
their relationships, including the composition, interconnection, and behaviour of
the data center architecture, its subsystems (power, cooling, and IT), and low
level components (compute, storage, and network).
In this work, we chose RBD to model the data center infrastructure because
it is a simple technique that allows to evaluate dependability metrics based
on a simple logic system, whose components are represented as blocks [21]. In
addition, it is possible to derive closed-form equations from RBD models, thus
the results are usually obtained faster than those from other methods [14].
Figure 3 illustrates an example of a data center infrastructure (obtained from
Redfish) and its respective RBD. For the sake of simplicity, we are considering an
example with only two servers, A and B, present in a data center infrastructure.

Fig. 3. Representation of a computer system in an RBD model

The Algorithm 1 shows the translation process adopted to automatically cre-

ate RBD models from Redfish information. Basically, in the translation process,
each hardware and software resource of a server (processor, memory, network
interface, and disk) is represented by an RBD block and arranged in series,
meaning that the failure of any hardware or software resource is reflected as a
failure of the entire server. In case the redundant resources, they are arranged
in parallel, and then the server failure will happen only when all the redundant
resources of the same type fail.
1016 G. L. Santos et al.

In line 2 of the Algorithm 1, we have a method to gather the information from

Redfish API. The loop in line 5 is used to, for each server present in data center,
generate the parallel RBD correspondent to the redundant hardware resource
(e.g. processors, memories, etc), as show in for loop in line 8. This function is
illustrated in line 10. Afterwards, all parallel blocks are arranged in series, as
shown in line 14, composing the RBD model for each server present in the data
center.
After generating the RBD model of each server, the loop in line 19 is used to
solve the models in order to obtain its respective mean time to failure (MTTF)
and mean time to repair (MTTR) values, as shown in line 21. These values are
used to generate a secondary RBD model (SA and SB blocks, at bottom of the
Fig. 3), with each block representing one server of the data center infrastructure
(line 25). This step is necessary to avoid high time-consumption to solve the RBD
model, once a data center may typically have hundreds to thousands servers.
Finally, in line 26, the algorithm solves the final model, calculating the avail-
ability of the data center, and then returns the result (line 27).

3 Availability Analysis

We conduct a case study that evaluates the availability of an application hosted

at different data center configurations (tiers I to IV with redundant servers, con-
sidering the TIA-942 standard). Figure 4 shows the configuration of a server in
our scenario. It is composed of three CPUs, three disks, four memories, and two
network interfaces (hardware components). Besides, once we have an applica-
tion hosted at each server, we consider a software stack composed of the host
operational system, hypervisor, and application.

Fig. 4. Evaluation scenario

 A Methodology for Automating the Cloud Data Center 1017

Algorithm 1. Methodology for generating availability models based on Redfish

data
Require: information from DC
Ensure: DC RBD model, DC availability
1: # Read data center information from RedFish API
2: DCinf rastructure ← redf ishAP I()
3: # Create an empty list of RBD models of servers
4: serverM odelsList ← list
5: for all serverInf o ∈ DCinf rastructure do
6: # Create an empty list of RBD model of redundant components (in parallel)
7: RedundantComponentsList ← list
8: for all redundantComponents ∈ serverInf o do
9: # Generate RBD parallel model of server redundant components
10: blocks ← generateP arallelRBD(redundantComponents)
11: RedundantComponentsList.add(blocks)
12: end for
13: # Arrange the parallel RBD blocks in series
14: serverRBD ← generateSeriesRBD(RedundantComponentsList)
15: serverM odelsList.add(serverRBD)
16: end for
17: # Create an empty list of blocks regarding to each server in DC
18: dcRBD ← list
19: for all serverM odel ∈ serverM odelsList do
20: # Calculate the MTTF and MTTR values of each RBD model
21: mttf, mttr ← solve(serverM odel)
22: dcRBD.add(block(mttf, mttr))
23: end for
24: # Arrange all servers models in parallel
25: model ← generateP arallelRBD(dcRBD)
26: av ← solve(model)
27: return (model, av)

For this use case, we add the network and storage components to assess
the availability of the whole IT subsystem of a data center. These additional
components are modeled after the generation of the RBD models, and we are
using SPN based on the models presented previously in [18].
We vary the IT subsystem tier from I to IV and the number of redundant
servers is defined as 2k , where k varies from 0 to 3 for each tier. Here, we are
considering that Tier I has no redundant network components, but has redundant
servers in order to better compare the tiers. The MTTF and MTTR values of
server components (hardware and software) are shown in Table 1.
Assuming a scenario with two redundant servers, if one server fails, the other
one will keep the application running. The servers follow the N-away active
redundancy model, standardized by the Service Availability Forum (SAF) [10].
This model defines that a workload may be assigned to one or multiple active
service units, and all service units are active for any workload. We chose this
redundancy model because it provides higher availability than the others [7].
1018 G. L. Santos et al.

Table 1. MTTF and MTTR values of the hardware and software components [1, 19]

Component MTTF (in hours) MTTR (in hours)

CPU 2,500,000.00 2.5
Memory 480,000.00 2.5
HD 200,000.00 2.5
NIC 120,000.00 2.5
OS 1,440 1
VM 2,880 0.17
APP 6,865.3 0.25

Table 2. Availability results (in %)

Number of servers Tier I Tier II Tier III/IV

1 99.779649 99.904804 99.921066
2 99.858363 99.983653 99.999937
4 99.858426 99.983716 99.999999
8 99.858426 99.983717 99.999999

To integrate both models SPN and RBD (IT subsystem and redundant
servers, respectively), we calculated the MTTF and MTTR of the RBD with
redundant servers, then we fed these values to the SPN model to estimate the
availability of the whole data center (IT subsystem with redundant servers).
This integration follows the strategy presented in [18].

3.1 Results

Table 2 and Fig. 5 show the availability results estimated by our methodology.
Tier I presented the lowest availability compared to tiers II and IV. For a single
server (not redundant), the availability is about 99.7796%. With the addition
of one more redundant server the availability increases to 99.8583%, a notice-
able difference of 0.07871%. However, when changing from two to four and eight
redundant servers, the availability demonstrates a slight increase, only. The avail-
ability for two servers was 99.8583% and for eight servers was 99.8584%, a small
difference of 0.00006278%.
The availability behavior resulting from increasing the number of servers for
tiers II to IV is similar to tier I. Using two servers instead of a single server
results in a larger increase in availability when compared to increasing from
two to eight servers. Thus, increasing the number of servers above two does not
impacts considerably the data center availability. The lesson we take here is that
investing in redundant resources can be helpful to a given point. Managers must
therefore always weight the gain in availability against the cost augment.
 A Methodology for Automating the Cloud Data Center 1019

Fig. 5. Availability results regarding tier I, II, and III/IV when increasing the number
of redundant servers

Table 3. Downtime results (in hours/year)

Number of servers Tier I Tier II Tier III/IV

1 19.302790079 8.3391833 6.91463
2 12.407367778 1.431955575 0.005535876
4 12.401885641 1.426453354 0.000077981
8 12.401867524 1.426427495 0.000077967

The availability of tier IV is the highest compared to the other tiers. The
availability for one and eight servers is 99.921065% and 99.999999%, respectively.
However, the availability of tier II with two servers is bigger than the availability
of tier IV with only one server. So, depending on the availability level require-
ments and budget, hosting an application in redundant servers in a tier
II data center infrastructure is more worthwhile than hosting in a tier
IV without redundant servers. This is indeed an interesting result, one that
shows that data center managers must carefully choose where to invest their
resources and that following blindly the tier based configurations is not always
the best way to increase resource availability. This demonstrates the need for an
availability evaluation methodology such as our proposal.
Table 3 shows the downtime results based on the availability of our scenarios.
From these results, tier I presented the highest values of downtime, while tier
IV presented the lowest ones. In tier I, with a single server, the downtime was
as long as 19.30 h per year.
1020 G. L. Santos et al.

Table 4. Cost of downtime (in $)

Number of servers Tier I Tier II Tier III/IV

1 3,860,558.02 1,667,836.66 1,382,926.00
2 2,481,473.56 286,391.11 1,107.18
4 2,480,377.13 285,290.67 15.60
8 2,480,373.50 285,285.50 15.59

Considering cost per hour of downtime is $200,000.00 [5], the loss in tier I with
one server reaches $3,860,558.02 per year. By increasing the number of servers
in tier I to eight (redundant servers), the downtime decreases from 19.3 h to
12.4 h, an annual downtime cost reduction from $3,860,558.02 to $2,480,373.50,
corresponding to a difference of $1,380,184.52 per year. Table 4 summarizes the
cost of downtime for each data center configuration.

3.2 Work Limitations

Despite the efficiency and simplicity of RBD to model and evaluate system
dependability aspects, the methodology showed in this paper presents some limi-
tations. RBD models cannot represent the dynamical behaviour of a system, such
as mechanisms of active redundancy (warm-standby) [2], neither repair depen-
dencies between different system components [4]. Usually, other state-based tech-
niques can be employed to address RBD limitations, such as Petri nets [15] and
Markov models [2]. However, those modeling techniques may suffer with the
state space problem [20]. This problem happens when the number of states in a
discrete model become too large to be verified, even when clever algorithms and
very powerful computers are used. Therefore, such models cannot be solved and
evaluated [8].
Combining these techniques (RBD and state-based methods) and using a
heterogeneous model approach is essential to represent large and complex sys-
tems [2,4]. This way, RBD can be used to represent the simple relationships
between components, while more complex behaviors can be modeled by other
state-base methods. However, generate these heterogeneous models automati-
cally is still a hard and time-consuming task due the complexity of the models,
the components’ relationship, and the integration process to put different mod-
eling techniques together.

4 Conclusions and Future Works

In this paper, we presented a methodology to acquire information of computa-
tional resources of a data center and to generate automatically stochastic models
to assess its availability. This methodology is essential when evaluating a large
data center infrastructure, commonly composed of hundreds to thousands of
hardware.
 A Methodology for Automating the Cloud Data Center 1021

We showed the applicability of our methodology by evaluating some data

center scenarios focusing on the improvement of the availability when replicat-
ing an application in redundant servers. From the availability results, simply
increasing the number of redundant servers does not impact considerably data
center availability. So, for applications that require a higher availability, it is
recommended to increase the tier level of the data center following the TIA-942
standard, instead of only adding redundant servers. However, we noted that a
tier II data center with two or more redundant servers presented a higher avail-
ability when compared to a tier IV data center without redundant servers. So,
depending on the application, it may be more advantageous to add more servers
in a tier II data center than to invest in increasing the tier level of the data
center as a whole.
We highlight that the tier IV presented a greater difference compared to
other tiers. The annual costs due to downtime decreased from $1,382,926.00
with a single server to $15.60 with four redundant servers, reducing $1,382,910,4.
But, as explained previously, the impact from four to eight redundant servers was
negligible. Therefore, we can consider that increasing from one to four redundant
servers reduced considerably the costs related to downtime, but adding more
redundant servers does not produces noticeable impact and it may not be worth
the investment of acquiring and setting up new servers to make the application
even more redundant.
As future works, we plan to improve the stochastic models generated by
our methodology to consider other aspects of a data center, such as software
management components, chassis, power, and cooler equipment. We also plan
to add optimization algorithms to suggest different configurations and resource
allocations in order to find the optimized data center availability considering
constraints such as hardware costs and number of hardware components.

References
1. Araujo, J., Maciel, P., Torquato, M., Callou, G., Andrade, E.: Availability eval-
uation of digital library cloud services. In: 2014 44th Annual IEEE/IFIP Inter-
national Conference on Dependable Systems and Networks (DSN), pp. 666–671.
IEEE (2014)
2. Dantas, J., Matos, R., Araujo, J., Maciel, P.: Eucalyptus-based private clouds:
availability modeling and comparison to the cost of a public cloud. Computing
97(11), 1121–1140 (2015)
3. Endo, P.T., Santos, G.L., Rosendo, D., Gomes, D.M., Moreira, A., Kelner, J.,
Sadok, D., Gonçalves, G.E., Mahloo, M.: Minimizing and managing cloud failures.
Computer 50(11), 86–90 (2017)
4. Ferreira, J., Dantas, J., Araujo, J., Mendonca, D., Maciel, P., Callou, G.: An algo-
rithm to optimize electrical flows of private cloud infrastructures. In: 2015 IEEE
International Conference on Systems, Man, and Cybernetics (SMC), pp. 771–776.
IEEE (2015)
1022 G. L. Santos et al.

5. Gagnaire, M., Diaz, F., Coti, C., Cerin, C., Shiozaki, K., Xu, Y., Delort, P., Smets,
J.P., Le Lous, J., Lubiarz, S., et al.: Downtime statistics of current cloud solutions.
Technical report, International Working Group on Cloud Computing Resiliency
(2014)
6. Gomes, D., Endo, P., Gonçalves, G., Rosendo, D., Santos, G., Kelner, J., Sadok,
D., Mahloo, M.: Evaluating the cooling subsystem availability on a cloud data
center. In: IEEE Symposium on Computers and Communications. IEEE (2017)
7. Gonçalves, G., Endo, P.T., Rodrigues, M., Kelner, J., Sadok, D., Curescu, C.: Risk-
based model for availability estimation of SAF redundancy models. In: 2016 IEEE
Symposium on Computers and Communication (ISCC), pp. 886–891. IEEE (2016)
8. Groote, J.F., Kouters, T.W., Osaiweran, A.: Specification guidelines to avoid the
state space explosion problem. Softw. Test. Verif. Reliab. 25(1), 4–33 (2015)
9. Guo, C., Yuan, L., Xiang, D., Dang, Y., Huang, R., Maltz, D., Liu, Z., Wang, V.,
Pang, B., Chen, H., et al.: Pingmesh: a large-scale system for data center network
latency measurement and analysis. In: ACM SIGCOMM Computer Communica-
tion Review, vol. 45, pp. 139–152. ACM (2015)
10. Heidari, P., Hormati, M., Toeroe, M., Al Ahmad, Y., Khendek, F.: Integrating open
SAF high availability solution with open stack. In: 2015 IEEE World Congress on
Services (SERVICES), pp. 229–236. IEEE (2015)
11. Hojati, E., Chen, Y., Sill, A.: Benchmarking automated hardware management
technologies for modern data centers and cloud environments. In: Proceedings of
the 10th International Conference on Utility and Cloud Computing, pp. 195–196.
ACM (2017)
12. Jammal, M., Kanso, A., Heidari, P., Shami, A.: Evaluating high availability-aware
deployments using stochastic Petri net model and cloud scoring selection tool.
IEEE Trans. Serv. Comput. (2017)
13. Kumari, P., Saleem, F., Sill, A., Chen, Y.: Validation of redfish: the scalable plat-
form management standard. In: Companion Proceedings of the 10th International
Conference on Utility and Cloud Computing, pp. 113–117. ACM (2017)
14. Maciel, P., Matos, R., Silva, B., Figueiredo, J., Oliveira, D., Fé, I., Maciel, R.,
Dantas, J.: Mercury: performance and dependability evaluation of systems with
exponential, expolynomial, and general distributions. In: 2017 IEEE 22nd Pacific
Rim International Symposium on Dependable Computing (PRDC), pp. 50–57.
IEEE (2017)
15. Melo, C., Matos, R., Dantas, J., Maciel, P.: Capacity-oriented availability model
for resources estimation on private cloud infrastructure. In: 2017 IEEE 22nd Pacific
Rim International Symposium on Dependable Computing (PRDC), pp. 255–260.
IEEE (2017)
16. Rocha, É., Endo, P.T., Leoni, G., Braga, J., Lynn, T.: Analyzing the impact of
power infrastructure failures on cloud application availability. In: 2017 IEEE Inter-
national Conference on Systems, Man, and Cybernetics (SMC), pp. 1746–1751.
IEEE (2017)
17. Rosendo, D., Santos, G., Gomes, D., Moreira, A., Gonçalves, G., Endo, P., Kelner,
J., Sadok, D., Mahloo, M.: How to improve cloud services availability? Investigating
the impact of power and it subsystems failures. In: HICSS Hawaii International
Conference on System Sciences. HICSS (2017)
18. Santos, G., Endo, P., Gonçalves, G., Rosendo, D., Gomes, D., Kelner, J., Sadok,
D., Mahloo, M.: Analyzing the it subsystem failure impact on availability of cloud
services. In: IEEE Symposium on Computers and Communications. IEEE (2017)
19. Smith, W.E., Trivedi, K.S., Tomek, L.A., Ackaret, J.: Availability analysis of blade
server systems. IBM Syst. J. 47(4), 621–640 (2008)
 A Methodology for Automating the Cloud Data Center 1023

20. Sousa, E., Lins, F., Tavares, E., Cunha, P., Maciel, P.: A modeling approach for
cloud infrastructure planning considering dependability and cost requirements.
IEEE Trans. Syst. Man Cybern.: Syst. 45(4), 549–558 (2015)
21. Verma, A.K., Ajit, S., Karanki, D.R.: Reliability and Safety Engineering, vol. 43.
Springer, London (2010)
 Job Scheduling Simulator for Assisting
the Mapping Configuration Between
Queue and Computing Nodes

Yuki Matsui1(B) , Yasuhiro Watashiba2 , Susumu Date2 , Takashi Yoshikawa2 ,

and Shinji Shimojo2
1
Graduate School of Information Science and Technology, Osaka University,
Osaka, Japan
[email protected]
2
Cybermedia Center, Osaka University, Osaka, Japan

Abstract. In computer centers responsible for providing users with

high-performance computing resources such as computer clusters, job
throughput is an important criterion in satisfying users’ computing
needs. The setting and configuration parameters in a job scheduler affect
the criterion. In particular, the mapping between queues deployed in the
job scheduler and computing nodes is highly related to job throughput.
In most cases, however, the mapping configuration is conducted based
on the administrators’ experience and knowhow partly because the tools
that facilitate the determination of the mapping are not available. In this
paper, we propose a job scheduling simulator that allows the administra-
tors to investigate how the mapping affects job throughput. In the eval-
uation, the behavior of the proposed job scheduling simulator is assessed
through a comparison with an actual computer cluster. In addition, real
cases using the proposed job scheduling simulator are discussed.

1 Introduction
Recently, the amount of computational power demanded by researchers has
been increasing [14]. Some of the reasons for this increase in demand can be
explained by increasing interest in machine learning, AI, and big data analy-
sis. Concomitantly, computer centers must now accommodate the computing
needs and requirements from scientists and engineers efficiently and effectively.
The number of jobs completed as well as the rising number of job requests are
important criteria in measuring the satisfaction of the users’ computing needs.
Currently, administrators struggle to configure job schedulers to be deployed on
computing resources, such as by a computer cluster, to gain higher job through-
put.
The mapping which defines how the queues on a job scheduler are linked to
computing nodes of a computer cluster becomes important in gaining higher job
throughput. In reality, however, administrators currently configure the mapping
based on the expected job requests and their knowhow because of a lack of
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1024–1033, 2020.
https://doi.org/10.1007/978-3-030-15032-7_86
 Job Scheduling Simulator for Assisting the Mapping Configuration 1025

tools that facilitate the determination of the mapping. In this paper, a novel job
scheduling simulator that allows administrators to investigate how the mapping
affects job throughput is proposed. Most of the existing job scheduling simulators
do not allow the mapping to be configured in simulations.
The remainder of the paper is organized as follows. In Sect. 2, technical
issues are clarified through an explanation of computer clusters and the map-
ping configuration. Section 3 describes the related work. In Sect. 4, the design
and implementation of the proposed job scheduling simulator is explained. In
Sect. 5, experimental evaluations are conducted. In Sect. 6, the conclusion and
future works are discussed.

2 Computer Cluster and Mapping Configuration

A computer cluster is composed of multiple computing nodes. Computing nodes
are connected on a high-bandwidth and low-latency network called an inter-
connect. Usually, a job scheduler is deployed onto the computer cluster for effi-
cient load-balancing and higher job throughput purposes. The job scheduler
receives users’ computing requests as jobs and then allocates an appropriate set
of resources to each job. Examples of job schedulers include Sun Grid Engine [5],
DQS [6], Portable Batch System [7] and Slurm [15].
Generally, a job scheduler has multiple queues to classify jobs based on the
priority and degree of parallelism. The mapping between queues and computing
nodes is an important parameter to control the behavior of the job scheduler.
Available computing nodes to perform a job are restricted by the mapping con-
figuration for a queue where they are stored.
Figure 1 illustrates how the mapping is configured on a computer cluster.
Assume that the job scheduler has two queues and manages six computing nodes.
Queue A is assigned to nodes A to D, and the queue B is assigned to nodes C to
F. Under this configuration, Job-α can only be executed on nodes A to D and
Job-β can only be executed on nodes C to F.
Importantly, the mapping configuration highly affects job throughput. As
illustrated in Fig. 1, a job stored in a queue can be allocated only to the com-
puting nodes linked to the queue. In other words, the computing nodes to be
linked only to the other queue are not provided for the job even if there are
enough computing nodes to satisfy the job’s requirements. For this reason, if the
appropriate mapping configuration for a specific computing environment is not
set, lower job throughput may easily occur.
In reality, however, the mapping configuration, in the most cases, tends to
be decided based on the administrator’s experience and knowhow in addition to
the past statistics and the history of jobs. Furthermore, recent high performance
computing demands make the mapping configuration problem more difficult. To
achieve higher computational performance, current computer clusters have been
increasingly scaled up [12] and the more computing nodes and queues, the more
complex the combination of the mapping.
To assist administrators in deciding the mapping, the authors focus on a
tool that can investigate the relationship between the mapping and the behavior
1026 Y. Matsui et al.

Fig. 1. An example of the mapping.

of computer clusters. The tool enables administrators to use the outputs as

indexes to determine the mapping. For such a tool, a job scheduling simulator is
considered suitable because job scheduling simulators can adapt the setting and
configuration of various parameters in the job scheduler [1,13].

3 Existing Job Scheduling Simulator

To date, many job scheduling simulators have been developed. Obaida’s job
scheduling simulator [9] can determine the allocation of computing nodes to
jobs according to various algorithms. MERPSYS [4] and gem5-gpu [10] can han-
dle heterogeneous computer clusters with multiple different types of processors.
GridSim [2] and Alea [8] are capable of dealing with not only computer clusters
but also a grid system that collaborates with multiple cluster systems. Slurm
Simulator [11] has been developed to evaluate the actual computer cluster man-
aged by Slurm [15].
Since these job scheduling simulators were developed mainly for the evalua-
tion of job scheduling algorithms and are not needed to manage the condition or
the behavior of computing nodes individually, the existing job scheduling simu-
lators do not have any mechanism to simulate behavior under a specific mapping
configuration.

4 Design and Implementation of the Job Scheduling

Simulator

In this section, we propose a novel job scheduling simulator that can assign jobs
to computing nodes by considering the relationship between computing nodes
and queues to investigate how the mapping affects job throughput on a computer
cluster. From the viewpoint of the multi-functionality and the extensibility of the
 Job Scheduling Simulator for Assisting the Mapping Configuration 1027

job scheduling simulator, Alea is adopted as the platform of research and devel-
opment. In Sect. 4.1, the functionalities required to adopt the mapping to Alea
are analyzed. In Sect. 4.2, the architecture of the novel job scheduling simulator
based on the analysis in Sect. 4.1 is mentioned.

4.1 Requirements for the Job Scheduling Simulator

Figure 2 shows the simulation flow of job execution in Alea. The simulation flow
of Alea is composed of three steps. First, the job scheduler selects a job according
to a job scheduling algorithm. Next, the job scheduler judges whether the job can
run on the computer cluster. Finally, the jobs which are judged to be executable
are allocated to some computing nodes and start running.

Fig. 2. Alea’s simulation behavior.

To build mapping configuration functionality into this simulation flow, the

following three requirements must be satisfied. First, the mapping configuration
should be allowed in this flow. The original Alea has a queue configuration file,
with which such queue parameters as the priorities and the time limits of running
the jobs can be retained. However, the information on the mapping cannot be
set up in the current queue configuration file.
Second, the status of each computing node should be managed indepen-
dently. In the aimed for job scheduling simulator, the job scheduler is required
to have the functionality of counting the number of available computing nodes
per queues. However, Alea does not have a means of knowing whether each com-
puting node is available or not. Instead, the job scheduler makes decisions based
on the number of total available computing nodes in the computer clusters, not
per queues.
Third, the selection of a computing node to be allocated to a job should be
controllable. In the aimed for job scheduling simulator, the job scheduler must be
able to distinctively select computing nodes to allocate to the job. However, in
Alea, the computing nodes are selected automatically. These three requirements
have to be solved to realize the job scheduling simulator.
1028 Y. Matsui et al.

4.2 Node Management Module

To satisfy the three requirements described in Sect. 4.1, we have built an exten-
sional module named the node management module. Figure 3 shows how the
node management module is integrated into the original Alea. The underlined
portions are the functionalities performed by the node management module.

Fig. 3. Simulation behavior with the node management module.

The node management module consists of basically three functions: the node
status management function, the mapping setting function, and the job exe-
cution control function. The node status management function allows Alea to
understand and control each computing node status independently. To under-
stand the status, the node status management function enhances the function
to access each computing node status, which is as an internal value in Alea.
The mapping setting function enables the mapping configuration as a queue’s
parameter. To set the mapping configuration, the mapping setting function
extends the queue configuration file. The mapping configuration can be described
per computing node basis to the extended queue configuration file.
The job execution control function enables the job scheduler in Alea to judge
whether the job can run or not, and this function decides the computing nodes
to allocate to the job by considering the mapping configuration. To be specific,
the job execution control function refers to the mapping configuration and the
status of each computing node. Moreover, the job execution control function also
enables the job scheduler to select the computing nodes to allocate to the job.
By linking the node management module to Alea, the proposed job scheduling
simulator is realized.

5 Evaluation
In Sect. 5.1, to assess the effect of the node management module on the proposed
job scheduling simulator, the job scheduling simulator and an actual environment
 Job Scheduling Simulator for Assisting the Mapping Configuration 1029

are compared. In Sect. 5.2, as a case study to utilize the proposed job scheduling
simulator as a tool to assist administrators, several job sets are submitted to the
proposed job scheduling simulator and the results are discussed.

5.1 Assessment of the Behavior of the Job Scheduling Simulator

In this subsection, the behavior of both environments was compared by submit-
ting the same job set. The job set is composed of 10,000 jobs. Each job is set to
have three parameters: arrival time, the number of parallelisms, and execution
time. Arrival time is the time when a user submits the job and it is randomly set
within 60 s from the arrival of the previous job. The number of the parallelism is
assumed to be the number of computing nodes that a job requires for running.
In this evaluation, we assumed that jobs required one or four computing nodes.
Execution time is the time required for the job to finish running and is randomly
set within 120 s.
In the evaluation, the structure of the computer cluster on both environments
consists of 16 computing nodes and the job scheduler has two queues. Queue H
stores four parallel jobs and has a high priority. Queue L stores one parallel job
and has a low priority. To execute a job requiring four nodes, the mapping for
Queue H has to consist of more than or equal four computing nodes. For the
job scheduling algorithm to determine the running order of jobs, First-Come
First-Served (FCFS) is adopted. The computing nodes can be linked to both
queues in the mapping configuration. In the actual computer cluster, the queues
and scheduling algorithm are set by Slurm [15]. The specification of a computing
node in the actual cluster is shown in Table 1.

Table 1. Details of the actual environment.

Component Detail
CPUs Intel
R
Xeon
R
E5-2630L 2.00 GHz 6core * 2/1 node
Memories 64 GB(DDR3 1600 8 GB DIMM * 8/1 node
Network Gigabit Ethernet
Topology Two level Fat-Tree
OS CentOS 6.9

Assuming that the performance of the computing nodes is identical, the job
execution time does not change regardless of which computing node is allocated
to a job. Each computing node accommodates only one job at the same time.
Moreover, computing nodes which are only assigned to one queue in the mapping
configuration are preferentially used for job execution. In the example of Fig. 1
in Sect. 2, nodes A and B are preferentially used instead of nodes C and D for
the execution of Job-α. The number of the mapping patterns, each of which is an
assignment of computing nodes to queues defined in the mapping configuration,
was 142 patterns in total.
1030 Y. Matsui et al.

Figure 4 shows the job throughput of the proposed job scheduling simulator
and the actual environment. The results indicate that the tendency of the job
throughput on the proposed job scheduling simulator is similar to the result
of the actual computer cluster. In all points, the difference of job through-
put between simulator and the actual environment is less than 0.01%. The job
throughput of the actual environment was slightly changed. The reason for this
is that the proposed job scheduling simulator does not consider the overheads
which are caused by the action of the job scheduler and the communication
between computing nodes. However, since these overheads are extremely small,
both job throughputs show a similar tendency. There is no effect on searching
for the appropriate mapping configuration using the proposed job scheduling
simulator.

Fig. 4. Job throughput comparison: The axis of Queue H and Queue L show the number
of computing nodes assigned to the each queue. The axis of job throughput shows the
number of executed jobs per hour.

5.2 Case Study

This subsection shows a case study that assists the mapping configuration. The
case study is conducted by submitting three job sets to the proposed job schedul-
ing simulator. Each job set consists of 10,000 jobs. Each job in the job sets has
three parameters as described in Sect. 5.1. In this evaluation, the arrival time is
randomly set within 3,600 s from the arrival of the previous job. The execution
time is randomly set within 43,200 s. The differences between job sets are the
proportion of the number of parallelisms as shown in Table 2. Each job randomly
requested 1 to 32 computing nodes to run. A job requiring less than or equal 16
nodes is regarded as a small job and greater than or equal 17 computing nodes,
a large job.
The computer cluster consists of 64 computing nodes and the job scheduler
has two queues. The system structure of the computer cluster is assumed to be
the VCC located in the Cybermedia Center [3]. Queue H only accommodates
large jobs and has a high priority. Queue L only accommodates small jobs and
has a low priority. To execute all types of jobs with the assumed number of
 Job Scheduling Simulator for Assisting the Mapping Configuration 1031

Table 2. Details of each job set.

Name The ratio of small job The ratio of large job

Job set A 70% 30%
Job set B 50% 50%
Job set C 30% 70%

parallelisms, the number of computing nodes assigned to Queue H is set more

than or equal 32 computing nodes, and the number linked to Queue L is set over
16 computing nodes. The setting of the job scheduling algorithm and the features
of computing nodes are the same as the condition in Sect. 5.1. The number of
mapping patterns was 1,481 patterns in total.
Figure 5 shows the job throughputs of each job set. From the simulation
results, it is possible to acquire useful indexes to determine the mapping configu-
ration. First, the results allow us to find the pattern of the mapping configuration
which contributes the best performance of job throughput. For example, when
the job throughput achieves the maximum in job set A, 63 computing nodes were
assigned to Queue H and 46 computing nodes were assigned to Queue L. Even
in the two other patterns, the mapping configuration to achieve the highest job
throughput can be confirmed. Such identification of the pattern of the mapping
configuration from the simulation results is useful in determining the mapping
configuration when the jobs to be submitted are known.

Fig. 5. All simulation results: The axis of queue H and queue L show the number of
computing nodes assigned to the each queue. The axis of job throughput shows the
number of executed jobs per hour.

Furthermore, the results show the tendency of job throughput depending on

the mapping configuration. For example, in job set C, it was confirmed that the
job throughput was improved with increasing the number of computing nodes
were assigned to Queue H. On the other hand, in job sets A and B, even if all
1032 Y. Matsui et al.

computing nodes were assigned to Queue H, the job throughput decreased when
the computing nodes assigned to Queue L were too few. The tendency of job
throughput derived from the simulation results clarifies the relationship between
the job sets and the mapping configuration. Such a relationship is helpful when
administrators determine the mapping configuration based on the past trend of
jobs.

6 Conclusion
In this paper, we proposed a novel job scheduling simulator that allows admin-
istrators to investigate how the mapping affects job throughput. To realize the
job scheduling simulator, we designed and implemented the node management
module which provides the functionalities of mapping for Alea. From the evalu-
ation, it was confirmed that the proposed job scheduling simulator behaves just
like an actual computer cluster. In addition, the simulation results from the job
scheduling simulator can be used as an index to analyze the job throughput.
In the future, to facilitate simulation targeting on larger-scale computer clus-
ters, we plan a decrease in the processing time of the job scheduling simulator.
Furthermore, we will explore the relationship between the mapping and the job
throughput by analyzing the simulated results of the job sets that are used in
actual computer clusters and by considering more diverse parameters.

Acknowledgement. This work was partially supported by JSPS KAKENHI Grant

Numbers JP16H02802, JP17K00101.

References
1. Buchert, T., Ruiz, C., Nussbaum, L., Richard, O.: A survey of general-purpose
experiment management tools for distributed systems. Futur. Gener. Comput.
Syst. 45, 1–12 (2015)
2. Buyya, R., Murshed, M.: Gridsim: a toolkit for the modeling and simulation of dis-
tributed resource management and scheduling for grid computing. Concurr. Com-
put.: Pract. Exp. 14, 1175–1220 (2002)
3. Cybermedia Center, Osaka University: Large-Scale Computer System (2018).
http://www.hpc.cmc.osaka-u.ac.jp/en
4. Czarnul, P., Kuchta, J., Matuszek, M., Proficz, J., Rościszewski, P., Wójcik, M.,
Szymański, J.: MERPSYS: an environment for simulation of parallel application
execution on large scale HPC systems. Simul. Model. Pract. Theory 77, 124–140
(2017)
5. Gentzsch, W.: Sun grid engine: towards creating a compute power grid. In: Pro-
ceedings of the 1st IEEE/ACM International Symposium on Cluster Computing
and the Grid, pp. 35–36. IEEE (2001)
6. Green, T.P., Snyder, J.: DQS, a distributed queuing system. Florida State Univer-
sity (1993)
7. Henderson, R.L.: Job scheduling under the portable batch system. In: Workshop
on Job Scheduling Strategies for Parallel Processing, pp. 279–294. Springer (1995)
 Job Scheduling Simulator for Assisting the Mapping Configuration 1033

8. Klusáček, D., Rudová, H.: Alea 2: job scheduling simulator. In: Proceedings of the
3rd International ICST Conference on Simulation Tools and Techniques, pp. 1–10.
ICST (2010)
9. Obaida, M.A., Liu, J.: Simulation of HPC job scheduling and large-scale parallel
workloads. In: Proceedings of the 50th Winter Simulation Conference (WSC), pp.
920–931. IEEE (2017)
10. Power, J., Hestness, J., Orr, M.S., Hill, M.D., Wood, D.A.: gem5-gpu: a heteroge-
neous CPU-GPU simulator. IEEE Comput. Arch. Lett. 14(1), 34–36 (2015)
11. Simakov, N.A., Innus, M.D., Jones, M.D., DeLeon, R.L., White, J.P., Gallo, S.M.,
Patra, A.K., Furlani, T.R.: A slurm simulator: implementation and parametric
analysis. In: Proceedings of the 8th International Workshop on Performance Mod-
eling, Benchmarking and Simulation of High Performance Computer Systems, pp.
197–217. Springer (2017)
12. TOP500.org: TOP500 The List (2018). http://www.top500.org
13. Tourancheau, B.: Experimental methodologies for large-scale systems: a survey.
Parallel Process. Lett. 19(3), 399–418 (2009)
14. Tulasi, B., Wagh, R.S., Balaji, S.: High performance computing and big data ana-
lytics - paradigms and challenges. Int. J. Comput. Appl. 116(2), 28–33 (2015)
15. Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple linux utility for resource
management. In: Proceedings of the 9th Workshop on Job Scheduling Strategies
for Parallel Processing, pp. 44–60. Springer (2003)
 MVMM: Data Center Scheduler Algorithm
for Virtual Machine Migration

Nawel Kortas(&) and Habib Youssef

Prince Research Laboratory, ISITCom of Hammam Sousse,

University of Sousse, Sousse, Tunisia
[email protected], [email protected]

Abstract. Migration algorithms and conception of data centers have significant

impact on cost and service qualities. Researchers are looking to find effective
solutions for Virtual Machine (VM) migration in data centers to reduce power
consumption under the given quality of service constraints. This paper presents
a model for the consistency of VM migration in a data center by Dynamic
Bayesian Networks (DBN). The novelty of our work is to construct a DBN
according to the probabilistic dependencies between the parameters in order to
make decisions about the corresponding placement of VM in the distributed data
centers. In addition, to evaluate the proposed model, we add a new scheduler
algorithm inspiring from the DBN model into GreenCloud simulator, called
MVMM (Minimization of Virtual Machine Migration), which provides a VM
score to make decisions about the matching placement of VMs in order to allow
a minimization of future VMs migrations. The results reveal that the use of the
proposed approach can reduce energy consumption until 8% compared to other
scheduler algorithms.

Keywords: Virtual machine
Data center
Cloud computing

Dynamic Bayesian Networks
SLA
VM migration algorithm

1 Introduction

Applying the SLA (Service Level Agreement) in a dynamic environment such as cloud
information is a complex challenge. It requires effective management of available
resources to avoid breaches of contractual commitments. Effective resource manage-
ment is problematic due to uncertainties in the operation of equipment and uncertainty
about customer behavior. A supplier that does not have the capacity to manage its
resources properly in order to meet its commitments, risks having a bad reputation and
a low level of trust with its customers. Noting that the principal problems are mainly
due to the complexity and the high cost of data centers resource management in cloud
environment. In this context, we propose to model a resource consistency service in a
Data center using the DBN, which accords probabilistic dependencies between the
parameters in order to make decisions about the corresponding placement of VM in
data centers so as to reduce the energy consumption by minimizing the number of VMs
migrations.

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1034–1046, 2020.
https://doi.org/10.1007/978-3-030-15032-7_87
 MVMM: Data Center Scheduler Algorithm 1035

This paper is organized as follow. The second part presents the basic support
available for the DBN, the cloud environment and VM migration. In the third part, we
present the created DBN model for the consistency of VM services in a datacenter.
Then we display the simulation parameters and results. Finally, we present a conclusion
to synthesize our work as a set of perspectives for future work.

2 Bayesian Networks and Cloud

2.1 Bayesian Networks
Bayesian Networks (Bayesian Networks) [1–3] (also known as Bayes model or
probabilistic directed acyclic graphical model, etc.) are probabilistic graphical models
that encode probabilistic Acyclic graphs directed DAG (Directed Acyclic Graphs).
Probabilistic relationships are defined as a set of conditional affirmations of CI (Con-
ditional Independence) independence.
Each node in a Bayesian network carries a label that is one of the problem attributes
(Fig. 1). Example:
• A and C are conditionally independent so we can
B say the probability P(A|B,C) = P(A|B). This means
that the probability of A depends only on that of B.
• The joint probability of all variables is:
A C
PðA; B; CÞ ¼ PðAjBÞ : PðBÞ : PðCjBÞ
Fig. 1. Bayesian network nodes

2.2 Dynamic Bayesian Networks

Dynamic Bayesian Networks (DBN) [4–6] range the representation of BN to dynamic
processes. A DBN encrypts the JPD (Joint Probability Distribution) of a time evolution
set X[t] = {X1[t]… Xn[t]} of variables. In other words, it represents the belief about
the possible trajectory of the dynamic process X[t]. After a similar notation with the
static representation of BN, the JPD for a finite interval of time [1, T] is factored as:

Y
T Y
n
P ðX½1; . . .; X½TÞ ¼ P ðXi ½t; jPi ½tÞ
T¼1 i¼1

Here Pi[t] indicates the parents of Xi[t] in the structure graph of the DBN. The
graphical structure of a DBN can be viewed as concatenation of several static BNs
linked with temporal arcs. We call each of these static networks a time slice (a time
slice is defined as a collection of the set of variables X[t] in a single instantaneous time t
and their associated parents P[t] in the graph structure) of the DBN. In general, DBNs
are defined for stationary processes where the structure and network parameters are
time invariable.
1036 N. Kortas and H. Youssef

2.3 Cloud Computing and Virtualisation

Cloud Computing [8], is a model that allows remote access to the demand of a set of
configurable computational resources. The virtual machines hosting applications can
be migrated based on Peak loads, Energy consumption and/or satisfaction of customer
service quality contracts (SLA).
Virtualization is the foundation of cloud computing by enabling deployment and
migration of virtual machines on demand and based on availability and workload of
physical machines, hosted in a local or remote datacenter, where virtualization is a set
of techniques for operating multiple systems, isolated from one another, on a single
physical system. With virtualization, a cloud can adapt its IT environment to the
evolution of its activities.
Virtual machines have the advantage of allowing a logical separation of the
resources of the machine. For example [9], two virtual machines can be allocated on the
same host, allocating 70% of the CPU to the first virtual machine and the remaining
30% to the other. As a result, task management is facilitated by virtualization tech-
nologies, including the provisioning of tasks.
VMs also have a significant advantage of migration. Indeed, the current virtual-
ization technologies make it possible to migrate a VM from one physical machine to
another. This can be done in two ways, the first method is to pause the machine,
migrate it and resume it once the migration is complete and the second way allows a
“live” migration [15]. A task is a set of virtual machines dedicated to running a parallel
application. A user to Entropy (Autonomous dynamic placement manager) submits
this. The life cycle of a task corresponds to the different states traversed by the virtual
machines during their take-up by Entropy. We consider from a theoretical point of view
that the change of state of a group of virtual machines belonging to the same task is an
atomic operation.

2.4 Related Work

Sonnek and Chandra [14] classify VMs like most suitable for being associated on a
single host. They propose to multiplex VMs based on their CPU to co-locate VMs with
advanced potential of memory sharing.
Govindan et al. [11] use statistical multiplexing of applications to identify appli-
cations that appropriate into given power budgets. VM multiplexing consolidates VMs
more densely on hosts. Nevertheless, VM multiplexing is not degraded by VM
multiplexing.
Authors in [12], suggest Virtual Router on the Move (VROOM), a primitive for
virtual network management. It offers a free transfer of virtual resources (routers and
links) from one physical machine to another to simplify physical network-management
tasks.
Nathuji et al. [13] have suggested the organization of a data center’s resource
managing system where resource is separated into local and global policies. However,
authors have not suggested a precise policy for automatic resource organization at the
global level.
 MVMM: Data Center Scheduler Algorithm 1037

Beloglazov et al. [7] suggested an experimental for determining the time to migrate
VMs from a host built on exploitation thresholds. It is established on the idea of setting
higher and lesser employment thresholds for hosts and protection the total utilization of
the CPU by the VMs.
Red Hat’s System Scheduler [16] afford placement policies with the objective of
load balancing and the saving energy consumption. They continuously display the
utilization of available hosts and, if needed, achieve a remapping using live migration,
which promises the satisfaction of policies during runtime.
Sharkh et al. [17] introduce a comprehensive solution to hold the problem of
resource allocation in a cloud. They demonstrate that the problem was formulated as a
mixed integer linear model. This formulation was solved using an optimization library
for a small data set. However, results prove that the optimal solution for bigger
practical scenarios is not possible by using the optimal formulation.
As we know, there is no reference present a model for the consistency of VM
migration in a data center by dynamic Bayesian networks. In this study, we will
construct a DBN according to the expert knowledge extracted and the probabilistic
dependencies between the parameters in order to make decisions about the placement
of VM in the distributed data centers to reduce the cost of the energy consumption.

3 DBN Model for VM Migration

3.1 Dynamic Allocation of Resources

The idea is to dynamically allocating additional resources to a VM to prevent damages
while minimizing the cost of allocated resources. Overall, this proposal makes it
possible to define the terms of the SLA, based on:
• The parameters of QoS required by the client;
• Describe a matching strategy by interpreting the requirements of the client’s
requests in terms of resource requirements;
• Reduce the complexity and the cost of data centers resource managements by
minimizing the number of operations into cloud environments.
• And implement a scheduling mechanism to minimize the risk of breach by seeking
VMs with characteristics similar to those being used by the client to minimize the
cost of allocations and VM migration.

3.2 Coherence Model of the VMs in the RBD

Initially, the model is defined as a graph of six nodes and six transitions, of which It and
It + 1 are the values of virtual machines at time t and t + 1. Vt and Vt + 1 are the values
of the quantity of services of VMs at times t and t + 1. St and St + 1 are the data to be
transferred at times t and t + 1, but in order to simplify the notation, an observation
variable Yt = (It + 1, It) which subsumes a pair of successive sequences (Fig. 2).
1038 N. Kortas and H. Youssef

It I t +1 Yt Y t +1

Y t = ( I t , I t +1 )
V t Vt V t +1
V t +1

S t
S t +1
S t S t +1
Fig. 2. Dynamic Bayesian Network for the estimation of VM migration

Since all the movements of virtual machines from one physical machine to another
are observed, the probability P(It) in terms of P(It + 1 | Vt, It) = P(It + 1, It | Vt) is a
constant which can be noticed in the model.

3.3 The Probability of Observation

The probability of observation P(Yt | Vt) is defined by assuming that the probability of
a momentum Vtx is related to a placement of a virtual machine Itxþ 1 located at the turn
of x at the instant T + 1, and which at time t was located to x  Vtx Dt. Assume that
Dt = 1 toPsimplify the calculations.
P Let S (x, l, R, v) be the t-distribution of student and
N ðx; l; Þ ¼ lim Sðx; l; ; vÞ is the normal distribution of the variable x with l is
v!1
the average, R is the covariance value and v is the degree of liberty. We then define:
 
  r2I
P Y t
jvtx ¼S Ixt þ 1 ; Ixv
t
t; ; mI ð1Þ
x Nðx; x0 ; @I Þ

With the following considerations:

• VMs are considered as vectors,
• The covariance r21 =Nðx; x0 ; @I Þ is a diagonal that depends on x′ relative to the
datacenter,
• N (x, x′, ∂I) implements a Gaussian weighting of the locality centered around x for
Itxþ 1 and around x  Vtx for ItxVt and ∂I defines the spatial scope of the VM and ∂I
x
is the value of the variance of the demand for displacement.
 MVMM: Data Center Scheduler Algorithm 1039

Output: Observation of datacenter parameters The difference

between the VMs
v=1 demand values is sig-
foreach time t do nificant, if we calculate
S = S + S[v] (assembling Datacenter
the Euclidean distance
parameters)
if S != P[v] or S != P[v+1] then between two physical
v=v+1 machines and we find a
else large value, then it will
Update the parameters in P[v] using S[v] be classified as aber-
End if rant, hence the high
End for robustness of the Stu-
dent distribution.
Algorithm 1: The observation algorithm

3.4 The Probability of Movement of VMs

The definition of the transition probability P (Vt + 1
, St | Vt) includes two factors.
We suppose that:
• The data flow field (Data) changes value relative to itself,
• The set of physical machines imposes an additional factor when calculating the
probability of movement similar to Vt + 1,
• It is assumed that the origin of a data flow vector carrying from one physical
machine to another Vtx at position x has been a flow vector vtx0 at a position x′.
• rv is the transition failure between Vtx and Vtxþ 1 .
vxt þ 1 : Sðvxt þ 1 ; vtx0 ; rV ; V v Þ ð2Þ

Which leads to a spatiotemporal robustness if one wants to estimate the value of x′

in the presiding physical machine, which can be inferred from the field of movement
flow itself via:
x0 : N(x0 ; x  vxt þ 1 ; @v Þ

Vtxþ 1 is used to extract the previous point.

If we combine the two factors and integrate x′ we get:
 X
vtxþ 1 V t : N(x0 ; x  vtxþ 1 ; @v ÞSðvxt þ 1 ; vtx0 ; rV ; V v Þ ð3Þ
x0

• Where rv is the transition failure between Vtx and vtxþ 1

• Where ∂v is the uncertainty in the spatial identification between two PM,
• Vv controls the robustness against the outliers with large and small values, which
increases and decreases the influence of the incoherence of the services in motion
with ∂v the field of spatial values obtained.
1040 N. Kortas and H. Youssef

3.5 Consistency: VMs Location and Data Flow Transfer

Each data set of a VM corresponds to flow pattern qs(Vt) which corresponds to an a
priori knowledge about the data stream, so we get
Y
qs ð V t Þ ¼ Nðvx ; As x þ ts ; rs Þ ð4Þ
x

Moreover, ts are affine parameters, which describe the motion, such that:
• As is a 2 * 2 matrix that describes any bearing.
• ts is an As location vector of size 2 * 1.
Assuming that the distribution of the parameters in the Datacenter cluster structure
is uniform, we can specify the minimum number of VMs for each variable noted
(MVM). This next algorithm does not make a superfluous search for structure learning,
but it maximizes the search in the cluster substructures of a Datacenter by choosing the
optimal parent variables for each current variable.
Input: qs (V): The data segmentation domain of each VM
Output: MVM score
For t = 1 to n do
qs (V t) = 0;
S = score (V t, qs (V t))
MVM= score (V t+1)
While S and qs (V t) < MVM do
Search Z preceding V t
which maximizes score (V t, qs (V t) {Z})
New= score (V t, qs (V t) {Z})
if New > S then
S=New
qs (V t) = qs (V t) {Z})
Endif
End while
MVM=S
End for
Algorithm 2: VMs location and data flow transfer
The data segmentation domain in a Datacenter qs which, for any data stream,
associates a label Vt
{1… k} specifies the correspondence of each VM data segment
to the corresponding flow pattern.
The data flow transition probability is added to this data segmentation in a Data-
center as an additional factor, whereby the transition probability is obtained as:
 X  0   
Pðvxt þ 1 jSt þ 1 ; V t Þ ¼ qstxþ 1 vtxþ 1 N x ; x  vtxþ 1 ; @V S vtxþ 1 ; vtx0 ; rV ; mV ð5Þ
x0

• t assume the time of the location circuit of Vm changes according to the data stream
vector and depends on location at the previous time.
• When a data segment belongs to segment qs, it is possible to change its locations
domain according to the change of data stream field.
 MVMM: Data Center Scheduler Algorithm 1041

• If we proceed as before for vtxþ 1 | Vt, we obtain:

X    
Pðsxt þ 1 jSt Þ ¼ N x0 ; x  qs;x
tþ1
; @s Q sxt þ 1 ; stx0 ; ms ð6Þ
x0

tþ1
• Where the average of qs, t+1
x is depend on x and Sx is the number of segments lost
during uniform displacement with discrete random variable such that:

0 1ms si s' = s
Qðs; s ; m s Þ ¼
ms sinon

3.6 The Global Algorithm (MVMM Algorithm)

The global algorithm (MVMM algorithm) is a combination of the two previous
algorithms; the observation, VMs location and data flow transfer algorithms, which
combine the parameters and the components lists of one datacenter applying in the
GreenCloud [10] simulator in order to record each virtual machines tasks to a match
physical machine setting. We try to follow the DBN model as well as can by using the
available parameters into the simulator environment.

Input: NLT: The list of tasks , VML: the list of available VMs, L(I): available PMs
Output: Map each VM tasks to a match PM
1) NLT ListTask.size();
2) VML VMList();
3) Weighted Coeff α =0.5, β=0.2 ;
4) Weighted Coeff t ;
5) J ServelLoad ;
6) q queue size ;
7) For n 0 to NLT do
8) TS Tasklist.get(n);
9) I VML (n);
10) V VMList.get(I);
11) stepIn TransferTime(TS, V, in);
12) stepOut TransferTime(TS, V, out);
13) exec ExecuteTime(TS, V);
14) L(I) PM load ;
15) Lr (I) Rack load ;
16) Lm(I) Module load ;
17) Qr(q) Connection rack ;
18) Qm(q) Connection Module ;
19) Fs(I,q) Computer metric factors for PM ;
20) Fr(I,q) Metric factors for Rack ;
21) Fm(I) Metric factors for module ;
22) if (TS.Qm(I) + stepIn + exec + stepOut + V.Qr(I) ≤ J.L(I)+ J.Lr (I)+ J.Lm(I)) then
23) sendTask (TS, V);
24) Sum = (α· Fs(I,q) + β· Fr(I,q) + Fm(I) · t);
25) MVMM weighted of Sum;
26) update (V);
27) else
28) Drope (TS),
29) Endif
30) EndFor
Algorithm 3: The global Algorithm (MVMM algorithm)
1042 N. Kortas and H. Youssef

The index of the selected VM for the current Task is computed using: I = VML(n),
where I is the index to the selected VM and VML is the list of available VMs. We
define Sum as a weighted combination of server-level Fs, rack-level Fr, and module-
level Fm functions: Sum = a
Fs(I,q) + b
Fr(I,q) + Fm(I)
t, where a value favor
the selection of highly loaded PM in light racks. b values will rank computationally
loaded racks with low network traffic. In addition, t values favor selection of time
loaded. The t parameter is an imperative design variable for task link in data centers. In
the end MVMM get, the weighted of Sum (best match score).

4 Simulation Results

We use the GreenCloud simulation [10], which proposes a complete modeling of the
energy consumed by the total components of the data center, such as physical
machines, virtual machines, links and switches. Additionally, GreenCloud offers a
specific analysis of workload distributions. In addition, a specific attention is devoted
on the packet level simulations of communications in the data center infrastructure.
Moreover, the simulator supports various Datacenter scheduler such us Green, Round
Robin, Radom, and Best DENS. GreenCloud is a good open source device that analyze
the performance of a Datacenter by varying parameters of the cloud. It can be used to
develop innovative solutions like resource allocation, scheduling as well as optimiza-
tion of communication protocol and network schedules (Fig. 3).

Fig. 3. Architecture of the GreenCloud simulator

 MVMM: Data Center Scheduler Algorithm 1043

• The energy consumption results within various scheduling

We provide an example that studies simulations of an energy-aware data center.
The simulation parameters specifies the task that can be scheduled under VMs or
physical hosts, based on the deadlines of the task. We fixed the number of Cloud user,
access switch, aggregation node, nominal disk, task size and memory.
We choose the three-tier Debug architecture while we varied the number of
physical machines and the datacenter schedulers (Green scheduler, Round Robin
scheduler, Heros scheduler and the new integrated scheduler algorithm MVMM
(Minimization of Virtual Machine Migration).
Before we present the results, we will make a short presentation of each scheduler
implemented in GreenCloud simulator:
HEROS: (Heterogeneous Energy-efficient Resource allocation Optimizing Scheduler)
[18]: Is a load-balancing algorithm for the allocation of energy-efficient resources in
heterogeneous systems. HEROS which proceed into account the heterogeneity of a
system through the decision method and uses a full representation of the system.
As a result, servers that comprise multiple types of resources (computing, memory,
storage, and networking) that have changing internal structures of their mechanisms
can be used more competently.
Round Robin: Round Robin is also a very simple scheduling policy, but it focuses
primarily on distributing the load equally among all nodes. Using this algorithm, the
Scheduler allocates a VM to a node cyclically. The planner traverses the nodes, one
after the other, assigning VM. An advantage of this algorithm is that it uses resources in
a uniform way by helping to balance the system load.
Green: Green Information Technology [19], the Green scheduling algorithm combines
with a neural network predictor to reduce power consumption in cloud computing. In
this case, the server predicts the load from time t to the time required to restart and
calculate the peak load.

Fig. 4. Results of energy consumption with various scheduling

1044 N. Kortas and H. Youssef

Figure 4 presents the variance of energy consumption of Datacenter with various

scheduling. One cloud user and 100 PMs were set to perform the simulations. As it is
shown, with MVMM and Green algorithms we got less energy consumption. The
average results of MVMM is (312.33 W*H), and the average results of Green
scheduler is (339.68 W*H), so we mark a proportion of 8.16% between the results,
thus due to the VM score, which make decisions about the matching placement of VM
in order to allow a minimization of future migrations. Green algorithm is less successful
in the energy consumption. HEROS algorithm can be used in both homogeneous and
heterogeneous data centers to provide energy-efficient load balancing. Round robin
scheduling is simple, easy to implement, however beside MVMM and Green algo-
rithms, it does not present great performance within datacenter environment.

Fig. 5. Energy summary of MVMM vs Round Robin

We present in Fig. 5 the distribution of energy consumption between all equipment

in one data center with one cloud user and 20 physical machines, using Round robin
and MVMM scheduling algorithms. This awareness is obviously reflected in the IT
field, and encourages cloud service providers to make their equipment less and less
energy intensive. In addition, they also have to integrate their systems management
techniques in order to use as few resources as possible and thus reduce the energy
consumption of all their devices.

5 Conclusion and Future Work

The application of the Service Level Agreement (SLA) in the cloud environment is a
complex problem given the dynamic nature of the info-cloud. The uncertainty sur-
rounding the infrastructure of the cloud system (Datacenter, servers, VM, data, links,
 MVMM: Data Center Scheduler Algorithm 1045

etc.) and the uncertainty about the client’s behavior in terms of requests which makes it
increasingly difficult to comply with the clauses of SLA by the supplier.
This work exploited the consistency of VM services by dynamic Bayesian network
as a method to make decisions about the corresponding placement of VM in the
distributed data centers in order to reduce the energy consumption by decreasing the
number of VM migration in a datacenter. The results reveal that the use of the proposed
approach within MVMM algorithm can reduce energy consumption until 8% compared
to other scheduler algorithms such as Green and Round Robin.
Dynamic Bayesian networks are ubiquitous, and as in conceive, many fields use
this concept to solve problems such as telecom network design, scheduling problem,
medical industrial diagnostics, CBR case reasoning, etc.

References
1. Cheng, J., Greiner, R., Kelly, J., Bell, D.A., Liu, W.: Learning Bayesian networks from data:
an information-theory based approach. Artif. Intell. 137, 43 (2002)
2. Gonzales, C., Jouve, N.: Learning Bayesian networks structure using Markov networks. In:
Proceedings of PGM, pp. 147–154 (2006)
3. Deviren, M., Daoudi, K.: Structural learning of dynamic Bayesian networks in speech
recognition. In: Proceedings of EUROSPEECH (2001)
4. Ghahramani, Z.: Learning dynamic Bayesian networks. In: Adaptive Processing of
Sequences and Data Structures, pp. 168–197. Springer, Heidelberg (1998)
5. Lähdesmäki, H., Shmule, I.: Learning the structure of dynamic Bayesian networks from time
series and steady state measurements. Mach. Learn. 71, 185–217 (2008)
6. Robinson, J.W., Hartemink, A.J.: Learning non-stationary dynamic Bayesian networks.
J. Mach. Learn. Res. 11, 3647–3680 (2010)
7. Beloglazov, A., Abawajy, J., Buyya, R.: Energy-aware resource allocation heuristics for
efficient management of data centers for cloud computing. Future Gener. Comput. Syst. 28
(5), 755–768 (2012)
8. Borgetto, D.: Allocation et Réallocation de services pour économies d’énergie dans les
clusters et les clouds’, IRIT Institut de recherche en info de Toulouse (2013)
9. Emir, I., Dobrisa, D.: Grid infrastructure monitoring system based on nagios. In: Workshop
on Grid monitoring, pp. 23–28. ACM (2009)
10. Kliazovich, D., Bouvry, P., Khan, S.U.: GreenCloud: a packet-level simulator of energy-
aware cloud computing data centers (2010)
11. Choi, J., Govindan, S., Urgaonkar, B., Sivasubramaniam, A.: Profiling, prediction and
capping of power consumption in consolidated environments. In: IEEE International
Symposium on Modeling, Analysis and Simulation of Computers and Telecommunication
Systems (MASCOTS) (2008). Print ISSN: 1526-7539
12. Wang, Y., Keller, E., Biskeborn, B., Merwe, J., Rexford, J.: Virtual routers on the move: live
router migration as a network-management primitive. IEEE SIGCOMM (2008)
13. Nathuji, R., Schwan, K., Somani, A., Joshi, Y.: VPM tokens: virtual machine-aware power
budgeting in datacenters. Cluster Comput. 12(2), 189–203 (2009)
14. Sonneck, J., Chandra, A.: Virtual putty: reshaping the physical footprint of virtual machines.
In: HotCloud Workshop in Conjunction with USENIX Annual Technical Conference (2009)
1046 N. Kortas and H. Youssef

15. Clark, C., et al.: Live migration of Virtual machines. In: Proceedings of the 2nd Symposium
on Networked Systems Design & Implementation (NSDIS). USENIX Association, CA,
USA (2005)
16. RedHat Enterprise Virtualization: System Scheduler. Data sheet. http://www.redhat.com/f/
pdf/rhev/doc060-
17. Sharkh, M.A., Shali, A., Ouda, A.: Optimal and suboptimal resource allocation techniques in
cloud computing data centers. J. Cloud Comput.: Adv. Syst. Appl. 6, 6 (2017)
18. Guzek, M., Kliazovich, D., Bouvry, P.: HEROS: energy-efficient load balancing for
heterogeneous data centers. In: IEEE 8th International Conference on Cloud Computing
(2015). Print ISSN: 2159-6182
19. Saha, S., Deogun, J., Xu, L.: Energy models driven green routing for data centers. In: 2012
IEEE Global Communications Conference, GLOBECOM, pp. 2529–2534 (2012)
 A Large-Scale Wired Network Energy
Model for Flow-Level Simulations

Loic Guegan, Betsegaw Lemma Amersho, Anne-Cécile Orgerie(B) ,

and Martin Quinson

Univ. Rennes, Inria, CNRS, IRISA, Rennes, France

{loic.guegan,martin.quinson}@ens-rennes.fr, [email protected]

Abstract. The use of simulators to predict network energy consump-

tion is a good way for scientists to improve and develop new algorithms
and also to assess them. However, the average size of a network plat-
forms is continuously increasing with the emergence of new technologies
like the Internet Of Things and Fog Computing. Packet-level simula-
tors start to reach their limits in terms of performance and this calls for
newer solutions in the domain of large-scale platform energy models. In
this paper, we propose two energy models for wired networks adapted
to flow level simulators in order to estimate the energy consumption
of large platforms. An evaluation of these models is proposed and it
demonstrates their applicability in practice and their accuracy. Indeed,
we obtain simulation results with a relative error lower than 4% com-
pared to an ns-3-based solution, and our flow-based simulation is 120
times faster.

1 Introduction
Since the beginning of Internet, distributed systems are widely used and
researchers put large efforts in this field in order to improve performance mod-
els and provide new solutions. Since their usage tends to grow with the arrival
of new technologies like Internet of Things (IoT) and Fog computing [1], their
energy consumption is an increasingly important metric to optimize. Informa-
tion and communication technology (ICT) devices, that are part of distributed
systems, can be categorize in three classes [12]: end-user devices, communica-
tion networks, and data centers. Researchers estimate that the overall energy
consumption growth rate of each of these classes is respectively 5%, 10% and
4%. With the highest expected growth, communication networks constitute a
key player in the ICT’s global electricity consumption. Energy-efficient solutions
are required for curbing this trend.
Several methods are available to study energy saving techniques. First, test-
beds can be used to perform experiments in order to be as close to the reality as
possible. However, test-beds face three main drawbacks: (1) Deploying equip-
ment and building a real network is expensive especially at large scale. (2) Such a
deployment requires a lot of setting-up time. (3) Experiments are hardly repro-
ducible due to numerous uncontrollable parameters. Another option to study
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1047–1058, 2020.
https://doi.org/10.1007/978-3-030-15032-7_88
1048 L. Guegan et al.

energy saving techniques consists in using simulators. They allow to conduct

diverse experiments on any types of platforms in a convenient way. Moreover,
they permit to save time compared to real deployments, and experiments can be
reproduced easily. On the other hand, network simulators need to be carefully
calibrated and assessed to produce valid results.
Several types of simulators can estimate the energy consumption of networks.
However, current network platforms and traffic volumes push their scalability to
their limits. Indeed, with the arrival of new technologies like the Internet Of
Things and fog computing, the number of connected devices increases. Besides,
generated and exchanged data volumes are augmented consequently in these
increasingly distributed systems, leading to an unprecedented rise of the global
energy consumption attributable to ICT.
This calls for scalable yet accurate models and simulators to estimate network
energy consumption. Existing simulators that model the energy consumption
work at fine grain – simulating each packet exchange – and thus are not scalable
for modern research purposes. Their over-approximation is not compatible with
efficient simulations. Nevertheless, scalable simulators exist but to the best of
our knowledge, none of them is modeling network energy consumption or they
are limited to study specific environments or protocols. Flow-based simulators
operate at a flow level without simulating each packet, thus saving memory and
computations. In practice, they constitute a nice compromise between network
abstraction and scalability, in terms of memory usage and computation time.
In this work, we propose to extend an existing flow-based simulator with
network energy models. We show that modeling the network energy consump-
tion into a flow-level simulator leads to predictions that are in line with the
predictions of packet-level simulators. We also show that simple linear models
are sufficient on realistic settings to obtain accurate values, while more complex
models were proposed in literature. Indeed, this model exhibits a relative error
lower than 4% compared to an ns-3-based solution and provides the simulation
results 120 times faster with using more than 17 times less memory.
The rest of this paper is structured as follows. Section 2 details the different
simulators and the classical network energy models. Section 3 presents energy
modeling in flow-level simulators. Section 4 presents the proposed energy models.
Section 5 details our evaluation methodology. An evaluation of the energy models
in terms of validity, scalability and applicability is proposed in Sect. 6. Finally,
we conclude in Sect. 7.

2 Related Work
2.1 Network Simulators
In the literature several packet-level simulators were extended in order to simu-
late the energy consumption of network devices.
The first well-known simulator that enters in this category is ns-2. It has
been extended into GreenCloud [6] which is designed for cloud simulations using
different architectures: two-tiers, three-tiers and three-tiers high-speed. However,
 A Large-Scale Wired Network Energy Model for Flow-Level Simulations 1049

Greencloud can be used only in cloud context and thus, it is very limited. Fur-
thermore, as it relies on ns-2, its scalability is limited in terms of number of
simulated devices and exchanges volumes of data.
As a replacement for ns-2, another packet-level simulator called ns-3 was pro-
posed. It provides several modules including ECOFEN [8]. This module is able
to simulate the energy consumption of network communications based on three
energy models namely basic, linear and complete. Even if they provide accurate
results [9], the packet-level nature of ns-3 prevents the usage of ECOFEN for
large-scale networks.
While these simulators offer accurate predictions, they fail to scale. In fact,
event rate of packet-level simulators increases very quickly on large-scale plat-
forms with real workloads. Flow-level simulators work at a higher level of abstrac-
tion in order to have very efficient performance models. Hence, this feature makes
them suitable for large-scale network simulations. Moreover, they show good
prediction abilities when their are properly instantiated. This makes them good
candidates for network energy consumption simulations.
CloudSim belongs flow-level simulators and is built on top of GridSim. It is
a well-known event-based simulator capable of simulating cloud environments
and estimating their energy consumption. It implements green strategies like
Dynamic Voltage Frequency Scaling and energy-conscious VM provisioning tech-
niques. On the other side, CloudSim is designed for cloud environment which
makes it difficult to use outside of this context. Another point is that, CloudSim
models are very similar to the GridSim ones which are known to contain band-
width sharing model aberrations [13].
SimGrid is another flow-level simulator, it is very versatile and can be used
to simulate High Performance Computing, Grid, Peer-to-peer etc. It is also scal-
able [10], but unfortunately it does not provide network energy model. Con-
sequently, we propose to extend SimGrid by adding network energy models.
However, because of the coarse granularity nature of flow-level simulators, we
should carefully instantiate and validate these energy models.

2.2 Classical Network Energy Models

Simulators are able to abstract the complexity of network devices’ energy con-
sumption. Even if a network equipment is not subject to any network traffic, it
consumes energy. Indeed, the different components involved in network devices
should be maintained powered on to ensure a fully functional network communi-
cation in case of packet arrival. This energy consumption part is called the idle
or static part. In some cases, the idle energy cost of a network device can repre-
sent 85% of its peak power [4]. Classically, the energy consumption is estimated
t
using the energy definition E(t) = 0 P (t)dt. Considering a network equipment,
its energy consumption varies over time according to the network traffic. Since
its power is composed of two parts: one static and one dynamic, we can define
its power in Eq. 1 with S the set of ports on the given network device, N pi and
1050 L. Guegan et al.

N bi respectively the number of packets and the number of bytes going through
the port i, and P pi and P bi the power costs per packet and per byte for the
port i.

Pnetdev = Pidle + N p i × P p i + N b i × P bi (1)
  
i∈S
Static   
Dynamic

Existing energy models in literature are based on this equation and consequently,
they are not directly compatible with flow-level simulators. Indeed, flow-level
simulators do not provide any notion of port in their models nor packets. These
energy models should be adapted to this network representation.

3 Background on Simulation
Simulation models can be built by following different approaches. One of them is
to model every details of the reality. In this case, lots of parameters increase the
model instantiation complexity. For the same reasons, the observations generated
by these models are more difficult to interpret. On another hand, well detailed
models can lead to accurate predictions, but they still should be validated in spite
of their detailed nature. Another approach is to use abstract models. Thereby,
no assumptions can be made on their level of accuracy. The last approach –
this is the one that we chose – is to use simpler models that are still able to
do accurate predictions. A good reference related to these types of models is
the LogP model proposed by Coller et al. [3]. They propose a simple model
for parallel computing composed of only four parameters. Still, LogP is taking
into account communication delay and CPU overhead, which makes it suitable
for accurate performance predictions and algorithm optimizations. Thereby, our
energy model is part of this latter approach.
As aforementioned, the energy models proposed in this paper are imple-
mented on a flow-level simulator called SimGrid. This simulator is based on a
modeling philosophy where the different models are simplified but nevertheless
assessed with the aims of providing valid results. It goes against two classical
believes. First, a model with more parameters is necessarily more accurate and
thus gives results closer to the reality. Indeed, we showed in previous work that
simple models are sometimes enough to capture complex effects [7,13]. Second,
a model that implements all the parameters present in the reality are somehow
correct by design (these parameters can be routing protocols, link delays, bit
error rate, etc.). Yet, model validations are mandatory in all cases. It can be
justified by the fact that some published simulators suffer from trivial imple-
mentation bugs [13] that have never been fixed. Hence, in this work we provide
validation results that can be reproduced by everyone (our code is open source
and available on the SimGrid web page).
SimGrid does not include Network Interface Controllers (NICs) and routers.
Instead, the paths between two hosts are represented by routes. Each route is
composed of several entities, called links that represent both, the wire and the
 A Large-Scale Wired Network Energy Model for Flow-Level Simulations 1051

NIC at each end. A route between two hosts can be composed of multiple links
but not necessarily of multiple hosts. To this end, there is no notion of port in
flow-level simulators. In spite of this, we can attach the energy consumption of
ports on the links. Then, it is possible to
define the power of the overall network
platform using the equation Ptotal = i∈L LinkPower i × LinkUsage i with L
representing the set of links on the simulated network platform.
However, in every networks, each link is connected to two different ports
with potentially different energy consumption schemes. Thus, flow-level simula-
tors abstraction introduces difficulties to transpose packet-level energy models
into them. To solve this issue, we propose to define two energy models, one for
homogeneous energy platform (same energy consumption for ports on a given
link) and one for heterogeneous energy platform. We will see later that this
distinction can in fact be simplified.

4 Contribution
4.1 Homogeneous Model for Flow-Based Simulators
We first introduce an energy model for network topologies that uses homogeneous
energy consumptions. As said previously, links will hold the energy consumption
of each of their ports and additionally, will ensure the bandwidth sharing. To
achieve this goal, we propose to use one split-duplex link between each end-node.
Then, to overcome the lack of port representation into SimGrid, we double the
dynamic energy consumption of each link to account for the two ports (one at
each end of the link). Since split-duplex links cause the creation of two links, the
idle power will be already doubled and we only need to double the dynamic part
of the power. More generally, the energy consumed by ports follows a linear model
with a minimum value equals to the idle energy consumption and a maximum
value define as:

max = idle + (BW × ByteCons + P ktCons/M T U ) × 2 (2)

In this equation, BW represents the port rate in Bps, ByteCons its energy
consumption per byte in Joules, P ktCons represents the network device energy
consumption in Joules needed for handling a packet. Finally, M T U , the Max-
imum Transmission Unit is used as an over-approximation of the packet size.
Considering a one-way communication, the link consumes energy for the trans-
mitting (Tx) and receiving (Rx) ports at the same time and vice-versa. If there
are multiple flows, the bandwidth sharing occurs on the up or down links accord-
ing to the flows’ directions. Figure 1 depicts this homogeneous energy model for
both kinds of simulators.
This energy model has multiple advantages. First, it is very easy to instantiate
because there is only one energy cost per link. Consequently, this makes it easier
to implement on flow-level simulators since these simulators use at least links
for communications (but not necessarily ports). Finally, this model provides
low computational overhead. However, many real topologies use heterogeneous
networking devices. Thus, the energy consumption from one NIC to another
1052 L. Guegan et al.

Fig. 1. Homogeneous model on packet/flow level simulators

varies according to its specifications. Consequently, this heterogeneity cannot

be explicitly represented using the homogeneous model. Hence, we decide to
introduce another model to overcome this limitation.

4.2 Heterogeneous Model

In SimGrid, routes from one host to another can be composed of multiple links.
Therefore, we can take advantage of this feature to build the heterogeneous
model. This model is represented on Fig. 2.

Fig. 2. Heterogeneous model on packet/flow level simulators

In order to model two ports with different energy consumption on a single

link, we introduce routes made of 3 links. The first link models the energy con-
sumption of the first port, the second link handle the bandwidth sharing and
finally, the third link models the energy consumption of the second port. How-
ever, we are using split-duplex links, thus two links will be created by SimGrid
and consequently the idle power will be multiplied by two. To overcome this
issue, we simply have to divide by two the idle power of each energy link. Thus,
this energy model follows a linear behavior with a minimum value equals to idle/2
and a maximum value defined as:

max = idle/2 + (BW × ByteCons + P ktCons/M T U ) (3)

 A Large-Scale Wired Network Energy Model for Flow-Level Simulations 1053

5 Methodology and Experimental Setup

5.1 Methodology

The evaluation of the two energy models is divided in two steps. The first step
is dedicated to assessing the models by means of microbenchmark simulations.
Hence, the experiments are running on small platforms in order to do simulations
on a simple and controlled environment. The second step of the evaluation is
focused on testing the energy models on a real large-scale platform. In this
way, we can demonstrate that the models are suitable for real research purposes
and thus, are useful for the scientific community. In addition, it demonstrates
that the energy models are scalable regardless to the simulated platform and
the workload size. It is important to note that our evaluation focuses on the
network energy consumption estimation and not on the network performance
model. Yet, it is mandatory to have accurate time estimations to get accurate
energy consumptions.
To conduct the validation experiments, we decided by lack of real large-
scale platforms to do simulations. In fact, validating these energy models using
test-beds requires to measure the energy consumption per byte and per packet
on each network devices. However, these measurements require high-precision
instruments. Nevertheless, these energy measurements results are proposed in
the literature [2,11]. Therefore, we based our experiments on these works.
To validate the two energy models, the experiments are done in parallel on
a packet-level simulator that acts as a trusted party. We choose to use ns-3 and
ECOFEN for this purpose. It is a logical choice since ECOFEN has been assessed
as accurate in the literature [9]. Moreover, ECOFEN is a packet-level simula-
tor, thus it will be useful for the scalability experiments in order to compare its
simulation performances to SimGrid. It is important to mention that all the sim-
ulations ran on Grid’5000, a large-scale test-bed for experiment-driven research
in computer science. In fact, microbenchmark scenarios require 96 unique sim-
ulations for only one experiment point (4 data transfer scenarios, 3 topologies
with 2 energy instantiation, 3 models are tested including one which is an opti-
mization). Moreover, the real use case experiment was also very computationally
intensive for ns-3 and ECOFEN.
All the experiments presented in this paper are available online [5]. We did our
best effort to provide reproducible experiments that can be run with different
parameters and require the least possible amount of user interaction for the
reader desirous of reproducing our experiments.

5.2 Experimental Setup

To conduct microbenchmark simulations, we designed different experimental sce-

narios. These scenarios are built in order to account for the effects on the energy
consumption of the following parameters: (1) The number of nodes and their
connections in the network (2) Different flow configurations, and thus different
bandwidth sharing patterns (3) The heterogeneity of the devices in terms of
1054 L. Guegan et al.

energy profile. For this, we define three platforms. The first one is composed of
two hosts connected by one link. The second platform is an extension of the first
one where we add two hosts and two links. Finally, the last platform, called Dog-
bone, is composed of six hosts connected together in order to have a bottleneck
on the two central hosts. Furthermore, the platforms can be set up with homoge-
neous or heterogeneous energy consuming devices. Thanks to these two variants,
the ability of each model to predict the overall energy consumption of platforms
with homogeneous and heterogeneous devices is highlighted. Then, each plat-
form is running with different TCP flow configurations. Other classical network
parameters are also defined. These parameters are equals for all the microbench-
mark scenarios: the latency is set to 10 ms and the bandwidth to 1 Gbps for each
link since the energy value for such a NIC is available in the literature [2,11].
Thus according to these papers, the energy values used in our microbenchmarks
for 1 Gbps links are 1.12 W for idle power consumption, 3.4 nJ for energy con-
sumed per byte and 197.2 nJ for the energy consumption per packet. However,
for the heterogeneous platform instantiation, we used energy values of 10 Gbps
links namely 0.53 W for idle power consumption, 14 nJ and 1504 nJ respectively
for energy consumption per byte and per packet. Finally, the data flow size is
varying randomly between 10 MB to 100 MB (x-axis) in order to build a power
profile of the network platforms.

6 Evaluation: Validity, Scalability and Real Use Case

6.1 Validity of the Proposed Models

As a first step, we simulate the microbenchmark scenarios on homogeneous

energy consumption platforms. The amount of data sent by the hosts are varied
according to the parameters defined in Sect. 5.2. Due to space limitation, we only
provide the results for two scenarios on Fig. 3. The flow configuration used for
these scenarios is the following: 3 flows overall (2 up and 1 down) between the
extreme nodes of each platform.
These first results show the ability of each model to predict the energy con-
sumption on two platforms namely Dogbone and 4 hosts and 3 links with homo-
geneous network devices. They clearly show a linear relation between the amount
of data sent between hosts and the overall energy consumption. One interest-
ing phenomenon to note is that both SimGrid energy models predict the exact
same amount of energy. Their energy relative error are close to 0, which means
they have the same energy prediction ability. In fact, it is not surprising since
the first energy model is a subset of the second one. Thus, they show similar
predicting abilities on homogeneous platforms. It is also interesting to note that
both homogeneous and heterogeneous energy models show a very small relative
error (less than 1%) compared to ns3 with ECOFEN.
Next, to demonstrate the ability of each model to predict the energy con-
sumption on heterogeneous platforms, we did similar microbenchmark exper-
iments using the scenarios with heterogeneous devices in terms of energy
 A Large-Scale Wired Network Energy Model for Flow-Level Simulations 1055

Fig. 3. Microbenchmarks energy consumption on platforms with homogeneous devices.

consumption. Thereby, we expect the homogeneous model to have a lower pre-

diction ability since it is not designed to handle heterogeneous energy platforms.
The simulation results are shown on Fig. 4. As expected, the homogeneous
model is less accurate than the heterogeneous model. In fact, we can clearly see
that the heterogeneous model energy estimation corresponds to the energy pre-
dicted by ns-3 and ECOFEN. However, the homogeneous model is doing wrong
approximation. This phenomenon is explain by the fact that the granularity of
the first energy model is not fine enough to fully capture the dynamic energy
consumption of the network devices.
Nonetheless, we decided to modify the instantiation of the homogeneous
model in order to improve its prediction capability. Hence, for each link we
attached an energy consumption equal to the average energy consumption of
their ports. The results of these simulations are also visible on Fig. 4 in pink. We
can see that by using this simple optimization we surprisingly observe that the
homogeneous model produces very accurate results almost equal to the hetero-
geneous model. This improvement can be explain by the fact that on real plat-
forms, the difference between the energy consumption of the ports is small and
thus, taking the mean has a limited impact on the overall energy consumption.
Obviously, the relative error between ECOFEN and the homogeneous model
will increase if this difference becomes larger. Hopefully, in real platforms this
difference remains reasonable. Moreover, the heterogeneous energy model mul-
tiply by three the number of links used in the SimGrid simulation, which is not
negligible on large-scale platforms. This is why, by following the Occam’s razor
Principle, we decide to use the homogeneous energy model with the arithmetic
mean instantiation in the remaining experiments.
1056 L. Guegan et al.

Fig. 4. Microbenchmarks energy results on heterogeneous energy platforms.

6.2 Realistic Use Case

To evaluate the scalability of our approach, we propose to simulate a datacen-
ter’ network which is a classical large-scale platform widely used in research.
Therefore, the final experiment has two main goals. Firstly, it demonstrates that
our modified homogeneous energy model scales up in terms of execution time
and memory usage. Secondly, it shows that this energy model is usable for real
experiments and thus is interesting for the scientific community. The data center
platform is based on a classical three-tier architecture used by Greencloud [6].
It is composed of 8 core nodes that provide access to the data center. These two
core nodes are linked to 16 aggregation nodes by 10 Gbps links. These aggrega-
tion nodes are linked to 512 access switches by 1 Gbps links that provide access
to 1,536 servers by mean of 1 Gbps links. Each link latency is fixed to 0.2 ms
which is the average latency measured between two Grid’5000 internal nodes.
The platform is set with the energy consumption of 1 Gbps/10 Gbps links refer-
enced in the literature [2,11].
The simulation scenario is defined as follows. First, 10 to 400 clients requests,
modeled by TCP flows, are generated and arrive simultaneously to the data cen-
ter by the 4 core nodes. Next, the flows are spread among the different aggre-
gation switches randomly and then reach the servers. Finally, the requests are
handled by servers to answer the clients’ requests. The experiment runs on both
simulators, SimGrid and ns-3 in order to compare their energy consumption
estimation and also their performance in terms of execution time and memory
usage. For each point of the simulation, we run several experiments using differ-
ent random seeds in order to better estimate the accuracy of our proposition.
The energy and scalability results are shown in Fig. 5. The energy consump-
tion results show that we have similar values on both simulators. The homoge-
neous model provide similar results to ns-3 with ECOFEN. Even in the worst
cases, it still predicts energy values in the ECOFEN confidence interval. The
 A Large-Scale Wired Network Energy Model for Flow-Level Simulations 1057

Fig. 5. Energy and scalability simulations results

homogeneous model has a relative error to ECOFEN lower than 4%, which is a
reasonable accuracy regarding to its level of granularity. Performance and scala-
bility results are also shown on Figs. 5(a) and (b). The execution time is clearly
higher on ns-3 than on SimGrid. In fact, for 258 requests, ns-3 takes more than
12 h against only 6 min for 897 requests on SimGrid. This is why we stopped
ns-3 experiments earlier than SimGrid ones. On this example, SimGrid is more
than 120 times faster than ns-3 with ECOFEN. Similarly, the memory usage is
also reduced: for 258 requests ns-3 requires 3 GB of memory whereas SimGrid
uses at most 169.08 MB. It is logical since flow level simulators do not model
every packets that are transferred from one host to another. Hence, their mem-
ory footprint and their computational overhead are drastically reduced. In this
case, SimGrid uses 17 times less memory than ns-3.
1058 L. Guegan et al.

7 Conclusion
In this paper, we proposed a network energy model for flow based simulators.
We showed that classical models have to be adapted for flow-based simulators
due to their coarse grain nature. To evaluate these energy models, we performed
microbenchmark experiments. They showed that with simple energy models,
we can produce energy consumption estimations close to the ones provided by
packet level simulators. We also demonstrated that a fine grain modeling of plat-
forms energy heterogeneity is not required to have suitable energy estimations.
Moreover, on simulations of a data center network comprising more than 1,500
servers, we obtained a precision close to packet-level simulators with less than
4% relative error, and with simulation runtimes 120 times faster on flow-level
simulators. This realistic use case highlighted how it can be used by the sci-
entific community on large-scale platforms. The implementation is open-source
and already available on the website [5].

References
1. Bonomi, F., Milito, R., Zhu, J., Addepalli, S.: Fog computing and its role in the
internet of things. In: MCC Workshop on Mobile Cloud Computing (2012)
2. Cornea, B.F., Orgerie, A.C., Lefèvre, L.: Studying the energy consumption of data
transfers in Clouds: the Ecofen approach. In: IEEE CloudNet, pp. 143–148 (2014)
3. Culler, D., et al.: LogP: towards a realistic model of parallel computation. ACM
SIGPLAN Not. 28, 1–12 (1993)
4. Fiandrino, C., Kliazovich, D., Bouvry, P., Zomaya, A.Y.: Performance metrics for
data center communication systems. In: IEEE CLOUD (2015)
5. Guegan, L., Orgerie, A.C., Quinson, M.: Reproducible experiments repository
(2018). https://gitlab.inria.fr/lguegan/flowlvlwiredenergy
6. Kliazovich, D., Bouvry, P., Audzevich, Y., Khan, S.U.: GreenCloud: a packet-level
simulator of energy-aware cloud computing data centers. In: IEEE GLOBECOM
(2010)
7. Legrand, A., Marchal, L., Casanova, H.: Scheduling distributed applications: the
SimGrid simulation framework. In: IEEE/ACM CCGrid (2003)
8. Orgerie, A.C., Lefèvre, L., Guérin-Lassous, I., Pacheco, D.M.L.: ECOFEN: an end-
to-end energy cost model and simulator for evaluating power consumption in large-
scale networks. In: Sustainet (WoWMoM Workshop) (2011)
9. Orgerie, A.C., et al.: Simulation toolbox for studying energy consumption in wired
networks. In: International Conference on Network and Service Management (2017)
10. Quinson, M., Rosa, C., Thiéry, C.: Parallel simulation of peer-to-peer systems.
In: IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing
(CCGrid) (2012)
11. Sivaraman, V., Vishwanath, A., Zhao, Z., Russell, C.: Profiling per-packet and per-
byte energy consumption in the NetFPGA gigabit router. In: INFOCOM Work-
shops (2011)
12. Van Heddeghem, W., Lambert, S., Lannoo, B., Colle, D., Pickavet, M., Demeester,
P.: Trends in worldwide ICT electricity consumption from 2007 to 2012. Comput.
Commun. 50, 64–76 (2014)
13. Velho, P., Schnorr, L.M., Casanova, H., Legrand, A.: On the validity of flow-level
TCP network models for grid and cloud simulations. ACM Trans. Model. Comput.
Simul. 23(4) (2013)
 An Inter-slice Management Solution for Future
Virtualization-Based 5G Systems

Borja Bordel(&), Ramón Alcarria, Diego Sánchez-de-Rivera,

and Álvaro Sánchez

Universidad Politécnica de Madrid, Madrid, Spain

{bbordel,diegosanchez,asanchez}@dit.upm.es,
[email protected]

Abstract. Future 5G systems are envisioned to provide communication ser-

vices to a large catalogue of heterogeneous applications. Then, in order to
support services with so different requirements, the concept of network slicing
was proposed. In network slicing, network resources are packed and assigned to
users according to their needs in an isolated manner. Although different tech-
nological solutions may be employed to create slices, virtualization techniques
are probably the most promising ones. With this approach, functional data
components are configured as virtual network functions and most intra-slice and
inter-slice management functionalities are covered by native instruments from
virtualization technologies. In the intra-slice domain, algorithms to divide virtual
functions into efficient groups are needed. In the inter-slice domain, solutions to
address and coordinate global events affecting the entire mobile network are
required. Therefore, this paper proposes an inter-slice management mechanism
to control events in 5G systems. The solution is based on queue and graph
theory, to avoid the network congestion and manage all events in the fastest
manner. In order to evaluate the performance of the proposed solution, an
experimental validation based on simulation scenarios was carried out.

Keywords: 5G networks
Network slicing
Inter-slice domain

Management
Virtualization
Kubernetes

1 Introduction

Future 5G systems [1] are the next-generation of mobile networks, characterized by a

high throughput and a minimum delay. 5G solutions are envisioned to provide com-
munication services to a very large and heterogeneous catalogue of applications,
including Internet of Things [2], Cyber-Physical Systems [3] and traditional voice
services [4]. These services present extremely different requirements which should be
satisfied using different network architectures. However, an efficient network deploy-
ment must be homogenous, in order to simplify the network management and opera-
tion. To join both ideas, network slicing arises [5].
In network slicing solutions, network resources are packed and assigned to users in
an isolated manner according to their requirements. In this way, each slice may present
different security, availability, pricing, etc., properties using a common hardware

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1059–1070, 2020.
https://doi.org/10.1007/978-3-030-15032-7_89
1060 B. Bordel et al.

infrastructure. Besides, different applications and communication and Over-the-Top

(OTT) services [6] could be deployed (such as process execution systems [7], man-
agement solutions for critical infrastructures, etc.) in isolation from each other. Thus, a
global failure in a certain slice does not affect services within other slices.
Network slicing defines two different operation domains: the intra-slice domain,
and the inter-slice domain [8]. Intra-slice domain includes all functions focused on
managing services within the slice, while inter-slice domain coordinates the operation
of all slices to make them compatible. Besides, in the inter-slice domain handovers
between slices and other global changes in the network are also addressed.
To implement this environment, different technologies have been studied, such as
cloud computing deployments (Infrastructure as a Service, or IaaS) [9], Virtual Net-
work Functions (VNF) [10] or specific negotiation middleware. Nevertheless, the most
interesting technology to provide isolated slices over the same infrastructure is
virtualization.
In fact, different works have evaluated the performance of virtualization tech-
nologies in network slicing, proving that certain standard and existing solutions (such
as Docker or Kubernetes) may be successfully employed in this context [11]. However,
some functionalities cannot be covered by native instruments within virtualization
technologies, and additional algorithms are required. In particular, algorithms to pack
virtual network functions to be deployed in the same physical machine are needed
(intra-slice domain); and solutions to address global events in the network, orches-
trating the operation of different slices competing for the same common network
resources (inter-slice domain) are required.
Therefore, the objective of this paper is to design an inter-slice management
solution to coordinate the events generated by all slices in a future 5G network. The
algorithm is based on a queue system model and graph theory. The objective is to
address all events in the fastest and most efficient way, avoiding congestion in all cases.
The rest of the paper is organized as follows: Sect. 2 describes the state of the art on
virtualization solutions and 5G networks; Sect. 3 describes the inter-slice management
solution; Sect. 4 presents an experimental validation using simulation scenarios; and
Sect. 5 concludes the paper.

2 State of the Art on 5G Systems and Virtualization Solutions

In this section, main technologies in relation to future 5G networks, and virtualization

technologies to support network slicing in that context are described and presented.

2.1 Virtualization Technologies: Containers, Docker and Kubernetes

In this section, two main and commercial virtualization solutions are analyzed: Docker
[12] and Kubernetes (K8s) [13].
In general, works about virtualization and future 5G networks are focused on the
definition of new mechanisms to create resource pools, including antenna, frequency
spectrum and other similar specific resources. However, these solutions cannot entirely
address the creation and management of slices, and additional control, orchestration,
 An Inter-slice Management Solution 1061

and (even) virtualization technologies would be required. Moreover, these solutions are
totally linked to certain hardware equipment models, what prevents the referred
equipment from being freely chosen and makes costs more expensive.
These issues, nevertheless, can be solved using existing commercial technologies,
which are focused on lightweight virtualization (containers) and container manage-
ment, orchestration and control solutions (Kubernetes).
Hardware-level virtualization generates different virtual machines (VM) which
employ isolated slices of hardware capabilities and resources. These solutions require
the use of virtualization software named as “hypervisor’’. Two different types of
hypervisor may be found: “Type 1” runs directly on the bare hardware, and “Type 2”
runs as an additional layer over a guest Operating System (OS). Although in Sect. 3,
when employing VM, we are referring this taxonomy, in this section and discussion we
are not doing any distinction between both hypervisor types.
VM provides the highest level of isolation and security among all the virtualization
technologies, as to cause a general failure in all the VM running in the same server, the
physical machine must fail [14]. However, as hypervisor-based virtualization only
provides access to hardware resources, it is also needed to install an OS into each VM.
Thus, as a fixed resource quota is assigned to each VM and guest OS (regardless the
VM is currently employing them or not), hardware resources (RAM memory, CPU
time, bandwidth, etc.) are quickly consumed.
To address these problems, virtualization based on containers was developed.
Containers, basically, divide the OS resources (not the hardware resources) into por-
tions that are dynamically consumed by packed application. In this case, it is said the
Operating System is virtualized. Packed applications in containers are isolated, and
they have no notice about they are sharing resources. Containers are abstracted from all
OS resources such as memory or network interfaces. Containers are, besides, isolated
from the host OS. Two main benefits might be cited. First, the use of resources is more
efficient. If a container is not executing anything, it does not consume resources which
may be taken by other container. Second, creation and destroy of containers are faster
as booting an OS is not required
On the contrary, containers are less safe than VM. A container includes the packed
application and all its dependencies, so it can be run in any system independently from
other applications; however, all containers in the server are executing on the same
kernel. Moreover, to create containers, the host OS must implement a container dae-
mon, which poses a critical attack vector. Although this scheme is quite strong, it is
weaker than hypervisor-based virtualization as if the kernel or the daemon fail, all
containers fail [15]. This, in general, is an easier approach for a cyberattack than
targeting a physical server. To understand these differences, Fig. 1 shows a comparison
among the different virtualization technologies.
A second factor which makes VM more competitive is their maturity. Contrary to
most modern container solutions (such as Docker), hypervisor-based virtualization is
nowadays implemented in a large catalogue or production scenarios; which make VM
more trustworthy. Nevertheless, in some applications, benefits of containers surpass
these problems.
1062 B. Bordel et al.

Fig. 1. Comparison: VM or containers

Many different instruments to create containers have been reported in the last years.
However, due to its incredible performance, its successful application in cloud services
and the fact that it is an open source project, Docker is currently the preferred solution.
Although Docker is a very efficient technology, large and complex systems such as
future 5G networks, require deploying many different and interconnected components
whose management is an extremely challenging situation. To address this problem
Kubernetes technology was proposed. Kubernetes is a container orchestration and
coordination solution, which distributes virtual instances in the most efficient manner
among existing physical nodes (and respecting also the user preferences) and guar-
antees a high availability and reliability rate.
Kubernetes includes different components to replicate containers when they fail,
install virtual services from images stored within remote repositories, and interconnect
components using virtual network resources. These functionalities may be successfully
employed to manage network slices, which are virtual elements operating together over
the same infrastructure.

2.2 5G Networks: Relevant Proposals

In future 5G networks, radio access networks (RAN) and physical resources are also
virtualized and shared among all the slices deployed in the system. Depending on the
slice’s service requirements, a slice may be standalone (if it uses the spectrum and the
antenna as proprietary resources), may have its own spectrum (the antenna is shared),
etc. Up to four different slice types may be distinguished (e.g. resource unaware slices
share the entire RAN).
The use of virtualization technologies at this level, makes RAN to have a software
defined architecture including two separated planes (data plane and control plane) and
two basic layers (Medium Access Control and Packet Data Convergence Protocol). In
this context, seven main characteristics and innovative functionalities are envisioned
for RAN in future 5G networks.
 An Inter-slice Management Solution 1063

• LTE/5G tight integration and internetworking. Future 5G networks are defined as an

evolution of current LTE networks [16], and inter-RAN handovers between both
technologies must be as lightweight as inter-RAT handovers in 5G solutions.
• Double Radio Resource Management (RRM). In future 5G networks all function-
alities are distributed. Thus, RRM will include two management levels: a central-
ized level (cRRM) and a distributed group of functions (dRRM).
• Initial access using low frequency assistance. Cells in 5G networks will be smaller
than in current deployments, as the employed frequency bands are higher. That
makes the initial synchronization and alignment process very complex and poten-
tially slow. In this context, the use of macrocells using lower frequency bands
during the initial access will significantly improve the performance.
• Dynamic traffic steering. Although it has not been defined yet, most authors agree
that the use of hard and heavy handovers to support the traffic steering is not a valid
solution for future 5G networks. More dynamic and lightweight mechanisms are
required.
• Energy-efficient RAN. Future radio technologies could operate for a long time
without power supply. Thus, future radio equipment should be energy-efficient to be
able to operate using batteries and other similar solutions.
• Mobility management using millimeter-wave clustering. As said, cells in future
mobile networks will be smaller, due to the use of higher frequencies. However, for
some purposes, it is advisable to create macro-cells composed of a group of
common cells which are organized around a cluster head. In that way, User
Equipment (UE) will perform two handover types: among cells and among macro-
cells; which will improve the mobility management.
• Self-backhauling. In scenarios with a high device density, coverage problems may
easily arise. Then, in 5G networks, infrastructure will have capabilities to extend
coverage and modify its topology automatically. No solution to support these
functions has been reported yet.
With respect to the physical infrastructure, only some first ideas have been pro-
posed until now. Basically, fronthaul and backhaul will be integrated in a unique
network, which will be governed by real-time measures obtained through enhanced
monitoring systems.
On the other hand, service management and orchestration in future 5G networks is
a very challenging situation. Traditional static service architectures, typical in previous
mobile networks, will be replaced by dynamic deployments where several different
applications and scenarios coexist.
In this situation, four techniques are considered as enabling technologies: multi-
tenancy support, virtualization, cloud platforms and network programmability. In
network slicing, service MANO (MANagement and Orchestration) implies both, the
service management itself and the management of all underlying virtual network
functions. Services will be described by service providers using any of the existing
technologies such as TOSCA [17], a language and model to describe services, com-
ponents, etc., (see Fig. 2).
1064 B. Bordel et al.

Considering these provider descriptions (or templates), services are composed on-
demand. Before deployment, these services are verified and permissions for them
granted in the slice and global network.

Fig. 2. Service template

In a context where several operators are interconnected and must operate over a
common infrastructure, machine learning technologies can improve the performance of
orchestration policies. Services are, then, deployed over machine learning services (see
Fig. 3), which manage and control some planning services (actions to be taken for
managing a service). These machine learning services include, mainly, quality assur-
ance services, network demand prediction services and location-based services.

Fig. 3. Service layers

Basically, then, service MANO operations in future 5G networks consist of feed-

back control loops where five steps are identified: (i) monitor the service situation
(using the data collection services), (ii) analysis of the acquired information,
(iii) planning the policies and recommended actions to be taken, (iv) execute the
defined policies, and (v) manage network resources and functions to adequate them to
the new situation.
Virtualization mechanisms include solutions to perform most of these actions, but
specific algorithms are required to manage global network resources considering the
particular requirements of each communication service. These algorithms are deployed
in the inter-slice domain and are investigated in this paper.
 An Inter-slice Management Solution 1065

3 An Inter-slice Management Solution

Regarding the inter-slice domain, slices in a 5G network are seen as a queue system
where different sources generate events to be managed by intra-slice components, inter-
slice components or both. Figure 4 shows this view.

Fig. 4. Reference model

As can be seen, every slice is associated to a queue with an intra-slice server. This
server processes all generated events and decides if they should be redirected to the
inter-slice management engine or may be managed at intra-slice domain. Then, if inter-
slice components must participate, events are stored in a second queue. All secondary
queues from all slices are processed by the same inter-slice engine. In the proposed
model, however, the particular behavior of mobile networks must be considered. In
future 5G systems, as in current 4G solutions, processing one event may (potentially)
generate a set of new and additional events. To consider this situation, in our model,
servers, when processing events, may create new events to be stored in the corre-
sponding queues (see Fig. 4).
Thus, the inter-slice management engine must extract from secondary queues the
events, and process them. This procedure has to be developed in most efficient and
fastest manner, employing to measure these abstract objectives some relevant indica-
tors. Namely:
1066 B. Bordel et al.

• Availability ðAvÞ: The availability of all slices must be guaranteed, avoiding net-
work (queue) congestion. If congestion cannot be avoided for all slices, those
services with a higher guaranteed Quality-of-Service (QoS) will be priority.
• Constant and minimum event volume ðAeÞ: Events must be extracted from sec-
ondary queues in an order such that traffic volume in the queue system is constant
and minimum. In that way, probabilistic event losses in secondary queues (which
have a finite capacity) are reduced as much as possible.
• Constant and maximum QoS: Events must be extracted from secondary queues in
such a way that the QoS within the slices remains constant and as high as possible.
To ensure the fulfillment of these requirements, not only the number and type of
events in the secondary queues must be considered, but also the new future events that
may be generated when processing each already existing event. In that way, two
different event types may be distinguished:
• Atomic events: Events which are processed as a unit. They do not cause the gen-
eration of new events as they encapsulate all required actions to address the original
change, problem or situation.
• Cascade events: Events which causes the generation of new additional events. In
this case, events (when processed) cannot trigger all required actions to address the
original network change or problem; so different new events are generated to
represent new additional actions to be taken. These events are represented by
directed weighted graphs indicating the different events which may be generated
and their appearance probability (see Fig. 5). Addressing these situations implies to
go through the graph from the initial node (see Fig. 5, green node) to the final node
(see Fig. 5, red node). Every possible path is valid.

Fig. 5. Graph associated to cascade events

The problem to be solved by the inter-slice management engine, then, is ordering

all the events in the secondary queues to be processed, fulfilling three optimization
 An Inter-slice Management Solution 1067

conditions (1–3). In these conditions, Top is the total operation time, Tint the initial time,
q the network congestion rate, i indicates the i-th queue (including the m secondary
queues).
i
Tcong
Avi ¼ ¼ 1 for Top ! 1
Top ð1Þ


being qi Tint  t  Tint þ Tcong
i
1

Besides, ki is the event generation rate in the i-th queue, and li the event processing
rate in the same queue. k is a constant parameter, as low as possible.

ki ð t Þ
Aei ¼ ! k\1 for Top ! 1 ð2Þ
li ðtÞ

Finally, Ri represents the guaranteed throughput in the i-th slice for the different
offered communication services, and ri represents the real throughput in the same slice.

1X
QoSglobal ¼ QoSi ! 1 for Top ! 1
m i
ð3Þ
ri ðtÞ
QoSi ¼ ! 1 for Top ! 1
Ri

For each one of these conditions, and depending on the order in which events are
processed, a different temporal function will be generated. The area enclosed by each of
these functions must be calculated, selecting event processing order closest to the
optimization objectives (4).

1 TZop
Avobj
i ¼ Avi ðtÞ ¼ 1
Top 0
1 TZop
Aeobj
i ¼ Aei ðtÞ ¼ k\1 ð4Þ
Top 0
1 TZop
QoSobj
global ¼ QoSglobal ðtÞ ¼ 1
Top 0

As there are three different conditions, the global stop condition for the opti-
mization algorithm is the weighted addition of these three partial conditions (5).

Obj ¼ a1 Avobj obj obj

i þ a2 Aei þ a3 QoSglobal ð5Þ

As said, moreover, processing cascade events causes the generation of new addi-
tional events which must be considered. To do that, the adjacency matrix A of the
corresponding graph is employed. Using this matrix, all paths between the initial and
the final nodes are calculated, including the accumulated probability of each path and
1068 B. Bordel et al.

the number of new generated events for each path. For a cascade event, the number of
future events to be generated when processing corresponds to the mean value among all
possible paths (6). In this expression, n is the number of nodes in the graph.

X
1
j
E ½num event  ¼ j
A1;n ð6Þ
j¼1

Using a permutation algorithm [18], a well-known technological solution, it is

possible to evaluate all possible paths in a very fast manner; selecting the one for which
the optimization objective has a maximum value.

4 Experimental Validation: Simulations

In order to evaluate the performance of the proposed solution, a simulation scenario

was built using the NS3 network simulator. The NS3 network simulator is a tool to
design virtual scenarios based on C++ programs, which may be easily employed over
Linux systems.
The proposed scenario included twenty (20) different independent slices, inter-
connected to the same inter-slice domain. Each slice was composed of a set of ten base
stations and a random number of user equipment between fifty (50) and one hundred
and fifty (150). The inter-slice domain was composed by a unique management engine
implementing the described solution.
As the proposed solution is a complex algorithm, it must be implemented in a
different process or threat, separated from the NS3 process. To do that, TAP bridges
and ghost nodes are employed. Ghost nodes are integrated as a standard component in
the simulation scenario, being configured as a traffic sink. Then, from this node, it is
deployed a TAP bridge which connects the NS3 simulation with an external virtual
machine. The referred virtual machine was an LXC container. In this virtual instance,
the proposed algorithm was deployed using C programing.
The simulation and virtual machine were based on Linux Ubuntu operating system.
The simulation was configured to represent a total operation time of twenty four
(24) hours. Slices were configured to operate normally, except at two random time
instants when a large amount of inter-slice events will be generated. In order to obtain
statistically valid results, not affected by exogenous and uncontrolled variables, the
simulation was repeated twelve times.
Simulation logs were configured to collect information about the Quality of Ser-
vice, the availability and the traffic volume in the network. The temporal evolution of
these parameters, when employing the proposed solution, can be seen in Fig. 6. This
figure also includes a comparison with results obtained from a standard FIFO (First-In
First-Out) management policy.
As can be seen, in general, the proposed algorithm enables a faster network reaction.
QoS and availability reduction, when a large number of events are generated, is deeper if
a traditional FIFO strategy is considered, and more time is required to return to the
original stable situation. The greater improvement is obtained in terms of QoS, being
 An Inter-slice Management Solution 1069

1.05 1.6

Event volume (erlang)

1 1.4
1.2
Availability

0.95 1
0.9 0.8
0.6
0.85 FIFO
FIFO 0.4
0.8 0.2
Proposed Proposed
0.75 0
0 10 20 30 0 10 20 30
Time (hours) Time (hours)

1.05
1
0.95
0.9
0.85
QoS

0.8
0.75
FIFO
0.7
0.65 Proposed
0.6
0 5 10 15 20 25 30
Time (hours)

Fig. 6. Temporal evolution of proposed relevant global indicators

this parameter up to 18% more stable if controlled by the proposed solution. In terms of
availability and event volume, we have reduced up to 15% the network congestion.
Required time to return to the initial situation is similar for the QoS parameter and
both algorithms (our proposal and the traditional FIFO strategy), but it is around 50%
lower if availability or event volume is considered together with the proposed solution.

5 Conclusions and Future Works

In this paper we present an inter-slice management mechanism to control events in 5G

systems. The solution is based on queue and graph theory, to avoid the network
congestion and manage all events in the fastest manner. Three relevant indicators are
employed to represent the optimization objectives to be considered. Besides, the
adjacency matrix is employed to estimate the mean value of new events generated
when processing one of the currently existing event.
The proposed solution is evaluated using a simulation scenario. Results show the
proposed solution enables a faster network reaction and a more efficient event
management.
Future works should consider enhanced optimization algorithms to improve the
performance of existing generic solutions with specific proposals for future 5G net-
works. Moreover, predictive solutions to approximate and evaluate the future evolution
of network congestion and queues capacity could be considered in order to improve the
global QoS and availability in future 5G networks.
1070 B. Bordel et al.

Acknowledgments. The research leading to these results has received funding from the
Ministry of Economy and Competitiveness through SEMOLA project (TEC2015-68284-R) and
from the Autonomous Region of Madrid through MOSI-AGIL-CM project (grant P2013/ICE-
3019, co-funded by EU Structural Funds FSE and FEDER).

References
1. Andrews, J.G., Buzzi, S., Choi, W., Hanly, S.V., Lozano, A., Soong, A.C., Zhang, J.C.:
What will 5G be? IEEE J. Sel. Areas Commun. 32(6), 1065–1082 (2014)
2. Xia, F., Yang, L.T., Wang, L., Vinel, A.: Internet of Things. Int. J. Commun. Syst. 25(9),
1101–1102 (2012)
3. Bordel, B., Alcarria, R., Robles, T., Martín, D.: Cyber–physical systems: extending
pervasive sensing from control theory to the Internet of Things. Pervasive Mobile Comput.
40, 156–184 (2017)
4. Chandrasekhar, V., Andrews, J.G., Gatherer, A.: Femtocell networks: a survey. IEEE
Commun. Mag. 46(9), 59–67 (2008). https://doi.org/10.1109/MCOM.2008.4623708
5. Mareca, P., Bordel, B.: An intra-slice chaotic-based security solution for privacy
preservation in future 5G systems. In: World Conference on Information Systems and
Technologies, pp. 144–154. Springer, Cham (2018)
6. Sujata, J., Sohag, S., Tanu, D., Chintan, D., Shubham, P., Sumit, G.: Impact of over the top
(OTT) services on telecom service providers. Indian J. Sci. Technol. 8(S4), 145–160 (2015)
7. Bordel, B., Alcarria, R., Jara, A.: Process execution in humanized cyber-physical systems:
soft processes. In: 2017 12th Iberian Conference on Information Systems and Technologies
(CISTI), pp. 1–7. IEEE (2017)
8. Bordel, B., Orúe, A.B., Alcarria, R., Sánchez-de-Rivera, D.: An intra-slice security solution
for emerging 5G networks based on pseudo-random number generators. IEEE Access 6,
16149–16164 (2018)
9. Bhardwaj, S., Jain, L., Jain, S.: Cloud computing: a study of infrastructure as a service
(IAAS). Int. J. Eng. Inf. Technol. 2(1), 60–63 (2010)
10. Han, B., Gopalakrishnan, V., Ji, L., Lee, S.: Network function virtualization: challenges and
opportunities for innovations. IEEE Commun. Mag. 53(2), 90–97 (2015)
11. Bordel, B., de Rivera, D.S., Alcarria, R.: Virtualization-based techniques for the design,
management and implementation of future 5G systems with network slicing. In: World
Conference on Information Systems and Technologies, pp. 133–143. Springer, Cham (2018)
12. Merkel, D.: Docker: lightweight linux containers for consistent development and deploy-
ment. Linux J. 2014(239), 2 (2014)
13. Hightower, K., Burns, B., Beda, J.: Kubernetes: Up and Running: Dive Into the Future of
Infrastructure. O’Reilly Media Inc., Sebastopol (2017)
14. Smith, J.E., Nair, R.: The architecture of virtual machines. Computer 38(5), 32–38 (2005)
15. Dua, R., Raja, A.R., Kakadia, D.: Virtualization vs containerization to support PaaS. In: 2014
IEEE International Conference on Cloud Engineering (IC2E), pp. 610–614. IEEE (2014)
16. Sesia, S., Baker, M., Toufik, I.: LTE-the UMTS Long Term Evolution: From Theory to
Practice. Wiley, Hoboken (2011)
17. Brogi, A., Soldani, J., Wang, P.: TOSCA in a nutshell: promises and perspectives. In:
European Conference on Service-Oriented and Cloud Computing, pp. 171–186. Springer,
Heidelberg (2014)
18. Ramanan, S.V., Jordan, H.F., Sauer, J.R.: A new time domain, multistage permutation
algorithm (switching systems). IEEE Trans. Inf. Theory 36(1), 171–173 (1990)
 IoT Security Viewer System Using Machine
Learning

Yuya Kunugi1, Hiroyuki Suzuki2, and Akio Koyama2(&)

1
Department of Informatics, Faculty of Engineering, Yamagata University,
4-3-16 Jonan, Yonezawa, Yamagata 992-8510, Japan
[email protected]
2
Department of Informatics, Graduate School of Science and Engineering,
Yamagata University, 4-3-16 Jonan, Yonezawa, Yamagata 992-8510, Japan
{shiroyuki,akoyama}@yz.yamagata-u.ac.jp

Abstract. Recently, Internet of Things (IoT) are spreading and various things
are connected to the Internet. As the results, it is possible to get various data and
operate the devices remotely. With spreading of IoT, attacks of malwares like
Mirai and Hajime which target the IoT devices are increasing. For instance, a
large-scale DDoS attack by infected devices as a springboard has occurred.
Network monitoring tools for IoT devices have also been developed as attack
measures against IoT devices. However, the developed tools have many types of
software to be introduced and visualization of the network topology are not
performed, so there is a problem that visually instantaneous abnormalities
cannot be recognized. In this research, we develop a system that detects
abnormality by machine learning, visualize a network topology, and notifies
abnormality by alert visualization on the network topology. We measure the
accuracy of abnormality detection and the real time property of visualization of
alerts by actual machine experiments and show the effectiveness of the proposed
system.

1 Introduction

In recent years, the Internet of Things (IoT) that everything are connected to the
Internet are spreading and attracting attention. In IoT, everything are connected to the
Internet not only IT related equipment such as personal computers, servers and printers
but also other things like TV, HD recorder, car, wearable device, meter, sensor, etc.
Furthermore, they communicate each other and perform remote control and measure-
ment etc. By IoT, we can know the state of the far things, the environment (temper-
ature, humidity, atmospheric pressure, illuminance, noise, etc.), movement (impact,
vibration, tilt, turn over, drop, etc.). In addition, remote control of objects (control of air
conditioner and lighting etc.) can be performed using IoT technology [1, 2]. According
to white paper of Ministry of Internal Affairs and Communications Japan, the number
of IoT devices was 15.4 billion in 2015, and it is expected to be about 30 billion in
2020, about twice as many in 5 years [3].
Even though IoT is convenient, various dangerous incidents are occurring. As a
springboard of an attacker, a hard disk recorder was used in 2004, a smart TV and a

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1071–1081, 2020.
https://doi.org/10.1007/978-3-030-15032-7_90
1072 Y. Kunugi et al.

refrigerator was used in 2014. So, IoT equipment used as a springboard was tem-
porarily unavailable due to load [4]. Also, a 1.2 Tbps attack was carried out for the
DNS provider service “Dyn”. It is estimated that about 100 thousand IoT devices were
used as bots by malware Mirai specialized in IoT devices for this attack [5]. To cope
these security incidents, many information security engineers are needed. According to
the Ministry of Economy Trade and Industry Japan, although information security
engineer is lack currently, it is expected that it will become more and more deficient in
the future [6]. Network monitoring tools targeted to IoT devices are developing to deal
with lacking engineer and complicated attacks [7]. However, there are problems that it
is difficult to grasp at real time what kind of attacks occurring, and detailed information
in the attack cannot be obtained. Furthermore, there is also a problem that there are
many software to be introduced.
In this research, targeting IoT equipment, we develop a system that detects
abnormality by machine learning, visualize a network topology, and notifies abnor-
mality by alert visualization on the network topology.
The rest of paper is organized as follows. In Sect. 2, we introduce Mirai which is
malware of IoT. In Sect. 3, we describe related works. In Sect. 4, we explain the
proposed system. In Sect. 5, we discuss performance evaluation. In Sect. 6, we
describe conclusions and future works.

2 Mirai

Mirai is a malware that attacks to Linux server and IoT related equipment confirmed in
early August 2016 [8].
Malware is a generic name for malicious software or malicious code created with
the intention of acting harmfully. Malware includes viruses, worms, Trojan horses,
spyware, keyloggers, backdoors, bots and so on [9].
Mirai is classified as a bot among them. A bot is a program that executes commands
from an attacker via a network and attacks other computers and networks. Mirai
searches the infect terminals by using randomly selected IP addresses, and login to the
terminals and downloads the Bot. In this case, the ID and password set by default are
used. A terminal infected with Mirai will connect via telnet and attempt login when
there are terminals that can infect surrounding areas. When it can logged in, that
terminal also infects Mirai. In this way Mirai builds a botnet [10]. As shown in Fig. 1,
an attack command is sent from a C&C (Command and Control) server (an attacking
server controlling the bots through the Internet [11]) to IoT devices infected with Mirai
and attacks the specified attack target [12].
The source code of Mirai was released in October 2016, and since then many
variants have been confirmed, it is a malware that needs vigilance and countermeasures
in the future.
 IoT Security Viewer System Using Machine Learning 1073

4. The Scan Receiver sends the Victim node’s IP

address and the authentication information to Loader.
7. Attacker uses
Attacker Mirai to attack.

Scan Receiver Loader

3. Bot registers Victim node’s IP HTTP Server

C&C Server address and the authentication
information to the Scan Receiver.

1. BOT is registered 5. The Loader logins the Victim

to C&C Server. node.

8. Attack
Target IoT(BOT) IoT(Victim)
2. Try to login by telnet to 6. The Victim node
a node of randomly selected downloads Bot from
IP address. HTTP Server.

Fig. 1. System structure of MIRAI botnet.

3 Related Works

As related works, we introduce Trend Micro’s security solution for IoT [7], network
alert and topology visualization tool VizAlert [13].

3.1 Trend Micro’s Security Solution for IoT

As shown in Fig. 2, Trend Micro considers the IoT architecture as the four layers such
as device, network, control center which controls devices, and data analyzer which
makes data analysis. Data generated by the IoT device is accumulated in the control
center via the network. The accumulated data is analyzed by the data analyzer and
utilized. As described above, in the IoT environment, since data to be protected are
scattered, it is necessary to protect not only the device itself but also the network.
Products provided by Trend Micro’s security solution for IoT is different for type of
business. This time we introduce solutions for smart home as shown in Fig. 3. A device
layer adopts Virus Buster Cloud and ITrend Micro IoT Security. A network layer
adopts Trend Micro Smart Home Network and Virus Buster for Home Network.
A control center adopts Trend Micro IoT Security and a data analyzer adopts Trend
Micro SMART Protection Network. Although it is possible to protect various devices
and networks in this way, there are drawback as follows.
1074 Y. Kunugi et al.

• They cannot grasp outside attacks in real time.

• They cannot see the detailed information of attacks.
• They need to use multiple solutions.

Data Analyzer Analysis Utilization

Control Center Accumulation Control

IoT
Data
Network Gathering Instruction

Device Data generation Action

Fig. 2. Information for protecting in IoT environment.

Trend Micro
SMART Protection
Data Analyzer Network

Control Center Trend Micro IoT Security

Network Trend Micro Virus Buster for

Smart Home Network Home Network

Device
Virus Buster Cloud Trend Micro IoT Security

Fig. 3. Smart home [7].

3.2 VizAlert
VizAlert is a tool to visualize alerts and topology by using IDS (Intrusion Detection
System) logs. As shown in Fig. 4, information (when, where, what) of three attribute of
alert is displayed by using polar coordinate system, and topology is displayed at the
center. The radius of the polar coordinate system indicates the elapsed time of the alert,
and the angle indicates the type of alerts. The size of the icon on the topology becomes
large as the terminal receives multiple attacks. Namely, the icon size and the number of
attacks are proportional. However, the drawback is that since alert logs are used, the
alerts cannot be visualized in real time.
 IoT Security Viewer System Using Machine Learning 1075

Network Topology

Fig. 4. VizAlert.

4 Proposed System

We propose a system which visualizes alert by detecting abnormality using network

topology and machine learning.

4.1 Outline
In the proposed system, it is assumed that the system is implemented in all nodes in the
network. Also, the proposed system covers only the same segment (class C network).
As an operation after the proposed system is activated, a classifier for abnormality
determination is generated at each node using the learning data. Packet capture is
performed at each node after generation of the classifier, and abnormality judgment is
made by using a classifier for each captured packet. When it is judged to be abnormal,
the packet data are recorded into a log file and transmitted to the monitoring node. In
the monitoring node, the nodes in the same segment are confirmed using the ARP table,
and the topology is visualized. When there is a notification of abnormality detection
from another node, change the display color of the corresponding node, visualize the
alert, and visually notify.

4.2 Visualization of Network Topology

In the proposed system, nodes on the same segment are grasped by using the ARP
table. The monitoring node transmits ping packets using threads to each IP address on
the same segment and waits for a response from each node. When there were
responses, the MAC address and IP address of the nodes on the same segment are
added to the ARP table of the monitoring node. Extracts the IP address from it and
holds it as data of the node list. As shown in Fig. 5, the topology is visualized by using
the data of the node list. As shown in Fig. 6, by moved the mouse cursor over the node
drawn on the GUI, displayed the IP address of the node.
1076 Y. Kunugi et al.

Target
Attacker

Fig. 5. Visualization screen.

Fig. 6. Display of IP address.

4.3 Abnormal Detection Using Machine Learning

In the proposed system, each node performs an abnormal detection using machine
learning and notifies to the monitoring node when there are abnormalities.

4.3.1 Machine Learning Algorithm

As a machine learning algorithm, this time we used a Random Forest. The Random
Forest is a group learning algorithm with multiple decision trees. In the decision tree,
each node judges to move to which branch, and by repeat it, the final output result is
corresponded to the leaf of a tree [14]. In the Random Forest, a plurality of random
decision trees are generated from the same learning data, and the majority decision of
the output result of each decision tree is taken as the final output result as shown in
Fig. 7.

4.3.2 Abnormal Detection

At each node, a machine learning classifier is generated from learning data at system
startup. This time we use as learning data only the information that can be acquired by
packet capture based on the data of KDD Cup 1999 [15]. As shown in Table 1, the type
of protocol (TCP = 0, UDP = 1, ICMP = 2), packet size, and the number of sending
 IoT Security Viewer System Using Machine Learning 1077

The extracted packet data ???

Decision trees
generated by
supervised data

Classification results
of each decision tree
Normal DoS Normal

Final classification result Normal

Fig. 7. Flow of classification by Random Forest.

packets to the same host in the last two seconds are feature amounts. 5000 learning data
with ‘Normal’ or ‘DoS’ as supervised data are prepared. By changing learning data, it
can be used for detecting abnormality of various networks. Packet capture is performed
at each node after generating the classifier, and abnormality judgment is performed for
each captured packet by using the classifier generated at startup. When it is judged to be
abnormal, packet log data up to one minute before the corresponding packet reception
time is extracted and a log file is generated. By sending the generated log file to the
monitoring node, it notifies that there is an abnormality to itself.

Table 1. A part of learning data.

Protocol Packet size Number of sending packets in the last 2 s Judgment (Dos/Normal)
1 28 179 DoS
0 15 1 Normal
2 183 1 Normal

4.4 Alert Visualization

As show in Fig. 8, on the monitoring node, check the contents of the log file when
receiving the log file, change the color of the corresponding node part in the visualized
topology to blue in the case of target, change it to red in the case of an attacker, and
make a visualize of alert. As shown in Fig. 9, by clicking the node whose color is
changing on the GUI, we can check the contents of the sent log file in another window.
1078 Y. Kunugi et al.

Fig. 8. Visualization of alert.

Fig. 9. Log file.

5 Performance Evaluation

As a performance evaluation, we compare the precision, learning time and classifica-

tion time of Random Forest and other machine learning algorithms, and show the
superiority of Random Forest in this research. In addition, we construct the proposed
system by using a wireless LAN and multiple PCs, and measure the time taken to
visualize alerts by pseudo attack, and show the real time property.
 IoT Security Viewer System Using Machine Learning 1079

5.1 Evaluation Way

5.1.1 Comparison Between Machine Learning Algorithms
We compared the Random Forest used in the proposed system with other machine
learning algorithms like Support Vector Machine (SVM) and KD trees. In order to
make same conditions, as the number of learning data we used 5000 and as the number
of evaluation data we used 292000. Both the learning data and the evaluation data are
those of KDD Cup 1999. The learning and classification time were an average time for
five times. The number of decision trees of Random Forest set to 5.

5.1.2 Evaluation of Real-Time Performance

In order to evaluate the real-time nature of alert visualization, we construct the proposed
system using a wireless LAN and multiple PCs and conduct experiments. Activate the
proposed system on the PC, and perform pseudo attack using the packet generator, and
measure the time it took from the start of attack to visualization on the monitoring PC.
We performed a SYN flood attack which is a type of DoS attack that generates a large
amount of traffic (access) as an attack. The SYN flood attack is an attack to continuously
transmit the first stage SYN packet in the TCP 3 way handshake [16].

5.2 Results and Discussion

5.2.1 Results

• Evaluation of Machine Learning Algorithms

Table 2 shows the evaluation results of the machine learning algorithms. The
accuracy of the Random Forest (RF) was about 95%, which was higher than other
algorithms. However, the learning time was 0.570 s, about 0.27 s slower than the KD
tree, and the classification time was about 0.32 s slower than the SVM.

Table 2. Comparison of machine learning algorithms.

Algorithm Learning time [s] Classification time [s] Accuracy [%]
RF 0.57 0.691 95.01
SVM 0.583 0.372 83.99
KD tree 0.302 5.776 90.99

• Evaluation of Real-Time Performance

The experimental results for real-time performance are shown in Tables 3 and 4.
The attack judgment time is from attack starting time to judged to be attack. The
processing time is from judged to be attack to generating the log file, transmitting the
log file, and visualizing at the monitoring PC. Total time is sum of attack judgment
time and processing time. As we can be seen from the table, in many cases visualizing
time is within 4 s from the start of the attack. However, in some cases, processing time
has increased significantly.
1080 Y. Kunugi et al.

Table 3. Experimental results for real-time performance (3 PCs)

Attack judgment time [s] Processing time [s] Total time [s]
0.200 0.488 0.688
0.221 0.517 0.738
0.433 0.601 1.034
0.202 3.579 3.781
0.656 3.139 3.795
0.527 4.990 5.517

Table 4. Experimental results for real-time performance (4 PCs)

Attack judgment time [s] Processing time [s] Total time [s]
0.399 0.406 0.805
0.401 0.420 0.821
0.402 2.017 2.419
0.418 2.956 3.374
1.110 2.534 3.644
0.775 12.517 13.292

5.2.2 Discussion
From the experimental results, the proposed system can detect many DoS attacks
within 4 s and display alerts. From Table 2, it can be said that the Random Forest is
more useful algorithm than other machine learning algorithms in the proposed system.
From the experimental results for the real-time performance, it turned out that it
takes about 1 s for attack judgment and about 3 s for processing such as log file
generation, and it can be visualized in about 4 s. However, regarding some alerts, the
processing time has increased to about 13 s. It is thought that because the node is being
attacked as a cause of significant increase in processing time, resources to be used for
processing such as log file generation were lacked.

6 Conclusions

In this paper, we described topology visualization for IoT devices and proposal of
abnormal detection system using machine learning.
Since the scope of visualization is on the same segment, ping is sent to all nodes
with IP addresses that can exist on the same segment, the ARP table is updated, and the
topology is visualized using the information of the ARP table.
We used a Random Forest of machine learning algorithm to detect abnormality, and
send the packet log file to the monitoring node when abnormality are detected, and
visualized the alerts by receiving a log file by the monitoring node.
We evaluated the proposed system by real machine experiments and comparison of
machine learning algorithms. As the results, we could see the following things.
 IoT Security Viewer System Using Machine Learning 1081

• The classification accuracy of the Random Forest is as high as about 95%, which is
useful in the proposed system.
• Many of the attacks can be visualized within 4 s from the start of the attack, so it
can be said that there is real-time nature.
• If resources for creation of log files are insufficient due to attacks, transmission of
the log file is delayed and cannot be notified to the monitoring PC in real time.
As a future work, we will plan to improve the system which can detect various
attacks.

References
1. Atzori, L., Iera, A., Morabito, G.: The Internet of Things: a survey. Comput. Netw. 54(15),
2787–2805 (2010)
2. Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.: Internet of Things (IoT): a vision,
architectural elements, and future directions. Future Gener. Comput. Syst. 29(7), 1645–1660
(2013)
3. Ministry of Internal Affairs and Communications JAPAN: 2017 Information and Commu-
nications White Paper, Development of IoT in the Information and Communications
Industry. http://www.soumu.go.jp/johotsusintokei/whitepaper/ja/h29/pdf/n3300000.pdf
4. IoT Security Headlines: IoT also needs security? https://www.trendmicro.com/jp/iot-
security/special/26
5. Trend Micro Security Blog: Look back on the main cyber security case in 2016. http://blog.
trendmicro.co.jp/archives/14203
6. Ministry of Economy: Trade and Industry JAPAN, Survey Results on the Recent Trends and
Future Estimates of IT Human Resources. http://www.meti.go.jp/policy/it_policy/jinzai/
27FY/ITjinzai_report_summary.pdf
7. Trend Micro: Security Solutions for the IoT Age. https://www.trendmicro.com/ja_jp/
business/capabilities/solutions-for/iot.html
8. Trend Micro Security Blog: The Latest Trend of Threats Aimed at Linux. http://blog.
trendmicro.co.jp/archives/13870
9. Medium and Small Business Information Security Measures Promotion Project: What is
Malware. http://www.jnsa.org/ikusei/03/08-01.html
10. Malware Information Bureau: What is a botnet? How do you prevent it? https://eset-info.
canon-its.jp/malware_info/trend/detail/150120_3.html
11. Malware information Bureau: C&C server. https://eset-info.canon-its.jp/malware_info/term/
detail/00062.html
12. Internet Initiative Japan Internet Infrastructure Review, vol. 33. https://www.iij.ad.jp/dev/
report/iir/033/01_01.html
13. Livnat, Y., Agutter, J., Moon, S., Erbacher, R.F., Foresti, S.: A visualization paradigm for
network intrusion detection. In: Proceedings of 6th Annual IEEE Systems, Man and
Cybernetics Information Assurance Workshop, SMC 2005, pp. 92–99 (2005)
14. Habe, H.: Random forest. IPSJ Technical report, 2012-CVIM-182(31), 1–8 (2012)
15. The UCI KDD Archive Information and Computer Science. University of California Irvine:
KDD Cup 1999 Data. http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html
16. Salunkhe, H.S., Jadhav, S., Bhosale, V.: Analysis and review of TCP SYN flood attack on
network with its detection and performance metrics. Int. J. Eng. Res. Technol. 6(1), 250–256
(2017)
 Service Oriented Architecture
for Interconnecting LoRa Devices
with the Cloud

Konstantinos Tsakos and Euripides G. M. Petrakis(B)

School of Electrical and Computer Engineering, Technical University of Crete (TUC),

Chania, Crete, Greece
[email protected], [email protected]

Abstract. The contribution of this work is two-fold: First, we show

how Low Power Wide Area Networks (LPWANs) can be interconnected
with the cloud; second, leveraging on PaaS functionality, we develop
LoRaWare, a Service Oriented Architecture (SOA) and system that
allows developers to enhance the capabilities of LoRa enabled applica-
tions using advanced cloud services such as, selective publication and
subscription to data, IoT connectivity using MQTT protocol, persis-
tent storage etc. We experimented with LoRa and LoRaWAN, the lat-
est successful representative of LPWAN protocols. We applied a typical
experimental setup with LoRa environmental sensors transmitting mea-
surements over long distances to gateways. LoRa gateways receive LoRa
packets from sensors in range and re-transmit them to the cloud using
an IP protocol (typically UDP). In this work, we opt for MQTT, a more
elaborate lightweight publish-subscribe IP protocol offering advanced
security, better routing control and visibility of the communication (i.e.
easier handling and control of data packets). To support connection of
LoRa IoT networks with the cloud, we developed the Network Server,
a cloud service that encompasses the necessary functionality for port-
ing LoRa packets to applications (i.w. encrypts/decrypts, de-duplicates,
authenticates LoRa packets and converts LoRa payloads to JSON). We
developed our solution in Fiware, the cloud infrastructure of the Euro-
pean Union. The reason for our selection is that, currently, Fiware sup-
ports interconnection with LoRa only via the network of another provider
(i.e. The Things Network). The Network Server is the only solution for
connecting LoRa networks directly to Fiware. We run an exhaustive set
of experiments in order to study system response time and scalability as
well as, the practical range efficiency of LoRaWAN protocol.

1 Introduction

The idea of Internet of Things (IoT) combined with cloud computing, opens new
horizons in the field of real time data collection and analysis [2]. Cloud comput-
ing emerges as the key platform for IoT data storage, processing and analytics
due to its simplicity, scalability and affordability (i.e. no up-front investment, low
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1082–1093, 2020.
https://doi.org/10.1007/978-3-030-15032-7_91
 LoRaWare 1083

operation costs). IoT manufacturers have developed products, which typically

use short range protocols such as Bluetooth and later Bluetooth Low Energy
(BLE) that connect IoT devices to a gateway (e.g. a smartphone or tablet) and
then to the cloud via a cellular network and the internet. However, the commer-
cial success of this solution is questionable as the number of IoT devices (e.g.
sensors) that can be connected to a gateway using BLE is limited and the energy
toll on gateways can be huge, imposing that the gateway must be connected to
a sustainable power source and be close to the IoT devices. Bluetooth and BLE,
the same as WiFi and ZigBee, are not suited for long-range transmission, while
cellular networks are costly, consume a lot of power, and are expensive.
Low Power Wide Area Network (LPWAN) technologies fill the gap between
mobile (3G, 4G, LTE) and short-range wireless (e.g. Bluetooth, WiFi and Zig-
Bee) networks [6]. LPWAN technology falls short in terms of QoS compared to
cellular standards and this means that an operator cannot use it to provide the
kind of Service Level Agreements (SLAs) that are critical for customers in cer-
tain application domains (e.g. remote health monitoring) [1]. However, LPWAN
specifications fit well the requirements of many IoT applications (e.g. smart
cities, home automation, industrial automation, environmental monitoring) who
need to transmit small quantities of data periodically over a long range, while
maintaining long battery life (e.g. an environmental sensor).
LoRaWAN is the networking protocol and network architecture that uses
LoRa modulation1 . LoRaWAN supports 2–5 Km ranges in urban areas (entire
cities can be covered with a few LoRa gateways) and up to 15 Km in suburban
areas. It works in the unlicensed spectrum below 1 Ghz which come at no cost. It
is an asynchronous protocol, which is optimal for battery lifetime and cost. There
are no royalty issues except of the LoRA modulation chip which is produced by
Semtech2 .
The present work is the technological bridge between the two important tech-
nologies referred to above namely, LoRaWAN and Cloud computing. The focus
of this work is on interconnecting LoRa gateways with the cloud. We develop a
Network Server whose purpose is to receive LoRa packets from gateways and port
LoRa payloads to other cloud services. The Network Server applies a sequence of
operations (i.e. decrypts, de-duplicates, authenticates LoRa packets) and, finally
transforms LoRa packets to JSON format. For outgoing packets the same solu-
tion is applied in reverse order: Packets are encrypted and transmitted to target
gateways.
Leveraging on advanced FiWare services we developed LoRaWare, an inno-
vative IoT platform that supports a complete sequence of services for IoT data
in the cloud that includes, selective publication and subscription of users and
services to data, processing of events based on rules, persistent storage, data
analytics functions and application specific services. The platform is secure by
design allowing access to data and services based on authorization and user
roles. To show proof of concept and support the LoRa claims of range efficiency,

1
https://www.link-labs.com/blog/what-is-lorawan.
2
https://www.semtech.com.
1084 K. Tsakos and E. G. M. Petrakis

we developed an application with LoRa sensors transmitting temperature and

humidity measurements to LoRa gateways and from there to the Network Server
in the cloud.
As of July 2018, Fiware announced availability of a LoRaWAN IoT Agent
developed by The Things Network3 (TTN) and Loriot.io4 . The same as our Net-
work Server, the new IoT Agent’s ambition is to provide an interface with LoRa
networks. However, it requires that application owners register their devices,
gateways and applications to the network of a third party (i.e. the TTN net-
work5 ); from where, LoRa payloads are transmitted to Fiware using an IP pro-
tocol. For applications owners and developers who need to install and maintain
a LoRa IoT network on their own and connect to Fiware directly (i.e. not via
TTN network), our Network Server is the only solution.
We run an exhaustive set of experiments using real and simulated (but real-
istic) data in order to study the system response time and system scalability.
We report average end-to-end processing times (i.e. from the time IoT data are
received by the network server to the time they are stored in the database) and
also, average time spent on each service in the processing sequence in the cloud.
Our experimental results demonstrate that our system is capable of perform-
ing in real - time (or close to real time) for up to thousands of requests. In a
different experiment, we study the practical range of LoRa transmission in a
complex terrain with two gateways placed 5.7 Km apart from each other. The
experimental results reveal that the error rate for packages captured by any of
the two gateways increases drastically with the distance from the gateways.
The rest of this paper is structured as follows: Sect. 2 describes IoT networks
with LoRaWAN support. Section 3 presents the architecture of the cloud back-
end that includes the Network Server. Implementation issues are discussed in
Sect. 4. Section 5 presents our experimental results followed by conclusions in
Sect. 6.

2 Sensing Environment
Figure 1 highlights LoRaWare ecosystem consisting of (a) the sensing platform,
(b) the back-end (cloud) implementing (among other services) a Network Server
and, (c) the applications which run in the cloud and allow consumers (end-users)
to receive information from selected devices (sensors).
Sensors are not associated with a specific gateway. Instead, data transmitted
by a sensor can be received by multiple gateways (i.e. all gateways in range).
Each gateway will forward the received packets to the Network Server. The
intelligence and complexity of the LoRa sensing network is pushed to the network
server which manages the active connections.
The LoRa nodes (i. e. sensors) transmit LoRa modulated packets which are
captured by one or more gateways (up-link transmission). A gateway receives
3
https://www.thethingsnetwork.org.
4
https://www.loriot.io.
5
https://github.com/Fiware/iot.IoTagent-LoraWAN/blob/master/README.md.
 LoRaWare 1085

Fig. 1. LoRaWare ecosystem

LoRa packets from sensors in range and re-transmits them to the cloud (i.e. to
the Network server) using an IP protocol (e.g. UDP). This operation is carried-
out using the packet forwarder6 , an open source software by Semtech which runs
on every gateway. The packet forwarder applies UDP for the communication
between the LoRa Gateways and the Network Server. Conversely, LoRa gateways
receive down-link messages from the Network server using an IP protocol from
where they are forwarded to LoRa devices using LoRaWan. Figure 2 illustrates
this process.

Fig. 2. LoRa gateway

Other protocols higher in the internet stack can be also applied for the com-
munication between a gateway and the network server (e.g CoAP, MQTT). In
this work, we opt for MQTT7 , an elaborate lightweight publish-subscribe IP pro-
tocol (i.e. a LoRa payload can be sent to many MQTT clients simultaneously
each serving a different purpose or application). Besides, secure one-to-many
communication (i.e. UDP is more susceptible to spoofing and denial of service
attacks), MQTT offers better routing control and visibility of the communica-
tion (i.e. easier handling and monitoring of the packets sent by the gateways).
Relying on TCP, MQTT applies TLS (Transport Layer Security) for data encryp-
tion and secure communication. Overall, MQTT should be preferred over UDP
unless speed of processing is a key factor and error correction or flow control is
not necessary.
6
https://github.com/kersing/packet forwarder.
7
http://mqtt.org.
1086 K. Tsakos and E. G. M. Petrakis

We developed the “LoRa gateway bridge” service that receives LoRa packets
from the packet forwarder and converts UDP to MQTT/TCP prior to transmis-
sion to Network Server. For the publish-subscribe communication between the
network server and the LoRa gateway bridge we relied on Eclipse Mosquitto mes-
sage broker8 supporting protocol versions 3.1 and 3.1.1. The gateway then allows
one-to-many communication subscribing to different topics (e.g. areas of inter-
est where data are published in order to serve different applications or users).
At the same time, the Network Server can serve many clients (subscribing to
many topics simultaneously). When a client subscribes to a topic, it is entitled
to receive data published on this topic.

3 Back-End Services

LoRaWare implements in the cloud the services illustrated in Fig. 3. Important

advantages of LoRaWare design are (a) adoption of the NGSI9 (i.e the Fiware
data exchange model based on JSON) for subscribing LoRa devices to the cloud
and (b) the use of the Publish-Subscribe service of Fiware for making IoT data
available to users and for users to subscribe to this information, and get notified
on changes of this information or when new information becomes available. The
entry point to the cloud is the Network Server.

Fig. 3. Cloud platform

Network Server: The Network Server manages active sessions (i.e. devices that
have joined the network), serves new LoRa nodes when they join the network,
decodes and decrypts the physical LoRa payload, de-duplicates the received
packets (which may be captured by multiple gateways), authenticates the packets
8
https://mosquitto.org.
9
https://orioncontextbroker.docs.apiary.io/#introduction/preface/status.
 LoRaWare 1087

Fig. 4. Network server

to make sure that they are not due to attacks (e.g. denial of service attacks).
Also, it manages the state of each node using mac-commands (e.g. to change data
rate, channels, etc.). Finally, it maps LoRa context (i.e. sensor names, types)
and content information (i.e. measurements) to NGSI. Figure 4 illustrates this
process as a sequence of steps which are explained below.
MQTT Broker: It subscribes to payload published on LoRa gateway bridge. In
analogy with the MQTT broker on gateway, it is implemented using Eclipse
Mosquitto. The MQTT broker subscribes to one or more gateways and con-
versely, a payload which is published on a gateway can be sent to subscribed
MQTT Broker clients on the Network Server.
LoRa Server: It is responsible for managing active node sessions (i.e. nodes which
have joined the network). For the active sessions, it (a) de-duplicates the received
data, (b) authenticates this data, (c) decrypts/encrypts LoRa packets using the
network key, (d) forwards (encrypted) data to the LoRa application server, (e)
asks the application server for feedback. Finally it is responsible for managing
the state of a LoRa node through so called mac-commands (e.g. to change the
data-rate, channels, etc.).
LoRa Application Server: It is compatible with the LoRa Server component
(above) and is responsible for (a) the device “inventory” (i.e. a database for
devices, gateways and device data), (b) handling of join-requests and (c) encryp-
tion and decryption of application payloads using the application key. It is
responsible for handling users, nodes, applications and access rights (i.e. via
user interface or API). An application is a collection of devices with the same
purpose or of the same type and can be associated with one more gateways. It
responds to requests issued by LoRa server for which nodes are allowed to join
the network and if so, which settings to use. When a new node joins the network,
permission is granted by LoRa Application Server. It offers node management
per application, per organization and gateway management per organization. It
also offers functionality for user and sensor management (e.g for assigning users
or sensors, to organizations or applications). For authentication and authoriza-
tion, users can be created in LoRa Application Server. A user itself can be an
administrator or a regular user. Administrators are authorized to perform any
action (e.g. add new network servers). LoRa Application Server is entitled to
connect to one or multiple LoRa Server network-server instances. Finally, LoRa
Application Server is responsible for handling users, nodes and applications.
1088 K. Tsakos and E. G. M. Petrakis

LoRa Server and LoRa Application Server are parts of the open-source
LoRaWAN network server project10 which provides many components for build-
ing LoRa networks. Both services can be accessed using a RESTful or gRPC
API and implement a storage for information pertinent to these services (e.g.
nodes, users, subscriptions, access rights etc.).
JSON Parser: It parses the MQTT payload, filters-out metadata of LoRa pay-
loads, keeps sensor values and subscribes this information to the MQTT IoT
agent of IDAS service (beloq). The agent accepts a different payload format from
the one that LoRa Application Server publishes. It is implemented in PyThon
using Paho Client11 client class. This class provides functionality to make pub-
lishing one off messages to an MQTT server very straightforward. Our script
runs as a daemon and its role is twofold: (a) it listens to the topic where the
LoRa Application Server publishes data and, (b) it publishes the new JSON
payload on a topic where the IoT Agent of IDAS subscribes to receive data.
IDAS Backend Device Management Service12 : This service is provided by
Fiware. It collects data from the Network Server and translates them to NGSI
format. Devices running a different protocol are served by a separate IoT Agent
(i.e. IDAS Backend Device Management is a composition of IoT Agents). An
IoT Agent listens to an IoT protocol (MQTT in our case) and transforms the
LoRa payload to NGSI. Developers should know in advance which protocol they
will be using to connect their devices to the cloud and select the right IoT Agent
to use. IoT Agents publish IoT values to the Publish-Subscribe Context Broker;
users or other services can subscribe to this information and get notified on value
changes or, every time a new value becomes available (e.g. for triggering actions
when new values become available). Figure 5 illustrates this process.

Fig. 5. Connecting IoT devices to FIWARE

10
https://www.loraserver.io.
11
https://www.eclipse.org/paho/.
12
https://catalogue-server.fiware.org/enablers/backend-device-management-idas.
 LoRaWare 1089

Publication and Subscription Service: This service acts as a mediator for

the data sent to the back-end and the end-users or applications. It is implemented
using Publish Subscribe Context Broker13 service of FIWARE. Services or users
can subscribe to data produced in the front-end. Sensors are listed as “public
entities” and users or other services are subscribers to these entities. Each time
a new sensor is registered to the system or a new measurement becomes avail-
able, this component is updated. When a new sensor is used, a new entity is
created in the broker. When a new measurement becomes available by a sensor,
a notification is sent to all entities subscribed to this information.
Event Management Service: It is a rule engine that gets input from the
publication and subscription service. The main purpose of this component is to
analyse data that become available to the publish and subscribe service (above).
When a sensor creates a new measurement, its corresponding entities in the
publication and subscription services are updated as well. The event processing
service subscribes to this information and gets a notification about the change.
This triggers the execution of rules that govern the application: If the value is
within the limits of these rules no action is taken; otherwise (e.g. the temperature
exceeds its threshold), the service notifies the subscribed user to take action. In
turn, rule violations may triggers events (e.g. an alert is generated like sending
an SMS to subscribed users). Complex Event Processing (CEP)14 of FIWARE
is a reference implementation of this service.
User Identity and Authorization Management Service: Implements user
management mechanisms and access control based on user roles (e.g. admin-
istrator, user) and access policies using FIWARE Keyrock Identify manage-
ment (IDM)15 service of FIWARE. It provides a Single Sign On (SSO) service
for secure access to services, data and networks and, applies (a) identification
services for users, (b) management of their profiles, (c) authorization services
based on OAuth2.016 (i.e. applications accessing protected resources are checked
whether they are authorized to do so using their OAuth2.0 credentials).
Storage Service: It is a shared database where data for user actions, sensors
(history data), messages, rules, events (e.g. rule violations) are permanently
stored. It is implemented as a non-SQL database using MongoDB.
Application Logic: Its purpose is to orchestrate, control and execute services
running in the cloud. Threshold violations (reported by the Event Management
service below) are typically managed by this component. When a request is
received (from a user or service), it is dispatched to the appropriate service.
For example, services regarding user accounts and access rights are dispatched
(from application logic) to user management service. It is tightly related with

13
https://catalogue-server.fiware.org/enablers/publishsubscribe-context-broker-
orion-context-broker.
14
http://fiware-iot-stack.readthedocs.io/en/latest/cep/index.html.
15
https://catalogue-server.fiware.org/enablers/identity-management-keyrock.
16
https://oauth.net/2/.
1090 K. Tsakos and E. G. M. Petrakis

the connections and the publication and subscription services which form the
communication channel with the gateway. Finally, application logic performs
basic security controls (e.g. checking if a session between the mobile application
and the cloud is active).
Additional services can also be imported to the cloud. For example, a Mashup
Editor (e.g. NodeRed17 ) to support composing new applications with minimal
effort, Business Support Services (BSS) dealing with subscription, system mon-
itoring, billing and accounting services required for commercial exploitation of
the platform, data analytics services etc.

4 LoRaWare Implementation
To show proof of concept, we developed an example application (use case) where
humidity and temperature measurements produced by LoRa nodes are moni-
tored in real - time on the cloud. The sensor measurements are updated asyn-
chronously (using AJAX calls). Humidity and temperature charts as a function
of time are displayed on a Web interface. The application subscribes to the Pub-
lication and Subscription service (above) which is responsible for updating the
database service and the User Interface when new sensor measurements become
available.
LoRaWare Sensing Environment: It is implemented based on Ideetron prod-
ucts (gateways and LoRa nodes). The Lorank818 is a LoRa gateway with pro-
fessional specifications. According to specifications, it enables connection with
many thousands end nodes and communication up to many Kms if mounted on
a high point with free sight. Each LoRa node is implemented on Ideetron Nexus
boards19 . Every Node has one Nexus Board with a microcontroller and a PCB
antenna for the transmission of RF packets. It connects to a Nexus Demoboard
which has temperature, humidity and motion sensors mounted on it.
LoRaWare Back-End: It is implemented as a composition of autonomous
RESTful services [4,7] communicating with each other over HTTP (even-though
these can be deployed on the same VM). Individual services or groups of ser-
vices are deployed on the same or different Virtual Machines (VMs) in the cloud.
Network delays are expected due to the nature of this design. However, the exper-
imental results (Sect. 5) demonstrate that the back-end is capable for responding
in real time even under heavy workloads.
The back-end is implemented on a FIWARE cloud running in OpenStack.
The services communicate asynchronously using Slim PHP (for handling REST
requests), cURL for communicating data and requests between services and,
finally, MongDB for the database. The user interface of the Web application was
implemented using HTML and CSS/jQuery. The back-end is deployed on five
VMs, all running locally (in the FIWARE node of TUC) with the exception of
17
https://nodered.org.
18
https://www.ideetron.nl/lora/?lang=en.
19
https://webshop.ideetron.nl/LoRa/.
 LoRaWare 1091

identification and authorization service that runs on a shared instance of Keyrock

Identity Management installed on the FIWARE node of Spain (FIWARE is a
federation of cloud infrastructures distributed across the EU). The four VMs
that run locally are, the Network Server, the Database (MongoDB), one VM
running the Publish-Subscribe service services (IDAS Device Management and
Publish-Subscribe Context Broker) and, one VM running application logic, event
management and application logic services. Each VM has one processor (x86
64 processor architecture, 2,800 MHz, first level cache size 32 KB, second layer
4,096 KB cache size), 2,048 MB RAM, 20 GB hard drive capacity, runs Ubuntu
14.04 and an Apache HTTP server (that handles PHP requests).

5 Performance Evaluation

We run an exhaustive set of experiments using real and simulated (but realistic)
data in order to study the time performance of the cloud solution and the range
efficiency of LoRa protocol using as a use case the Web application that monitors
the humidity and temperature measurements from sensors.
Back-End System Evaluation: The purpose of this experiment is to evaluate
(a) the response time and (b) the scalability of the back-end system (i.e. how
system response times increase with the number of connected users). We measure
overall system response time (i.e. from the time a measurement is received by
the cloud to the time it is stored in the database), average response time of each
service and also, average workload (i.e. CPU, memory usage) of each running
VM.
The response time for the system to process 1 request is 170.5 ms from which
16.8 ms account for network delays (which is low since all VMs are running in
the same cloud Infrastructure). Table 1 reports the time spent on each service
to process a single request.
Table 1. Response time for each service

Service MQTT LoRa LoRa JSON IoT Pub-Sub Application Storage

Broker Server App. Parser Agent CB
Server
Time (ms) 1 1 22 4 2 2 40 100

In order to study system scalability the system is stressed with many con-
current requests simulating the workload of a network with thousands of LoRa
sensors. We used ApacheBench20 to issue multiple simultaneous requests to each
VM. In ApacheBench we are opted to define the total number of requests and
how many of them will be executed simultaneously. In this experiment we report
response time of three running VMs: The Network Server, the Publish - Subscribe
services and the Use Case (i.e. application logic and event processing services).
The performance of MongoDB database (fourth VM) has been studied elsewhere
20
https://httpd.apache.org/docs/2.4/programs/ab.html.
1092 K. Tsakos and E. G. M. Petrakis

and is known to scale-up well [5]. The Identity Management service (fifth VM) is
not expected to cause any performance bottlenecks as it is addressed only once
by each user at login and, for authorizing access to resources.
Table 2 reports average response times (per request) for various values of con-
currency ranging from 1 through 300 (i.e. number of requests executed simultane-
ously). All measurements of time account also for the time spent for the commu-
nication between VMs or between services within the same VM. We notice a very
low resource usage when concurrency = 1. This is expected, as only one request
is executed at any time. Response times improve slightly with the simultaneous
execution of 50 requests (i.e. the Apache HTTP server switches to multitasking).
Processing capacities may increase or raise restrictions for space or bandwidth
for concurrency > 100. The best values are obtained with concurrency values
between 50 and 150, hence we conclude that the system optimal performance is
for up to 150 concurrent users.

Table 2. Response time for each service

VM1: Network Server VM2: IDAS & Pub/Sub CB VM3: Application

Concurency 1 100 300 1 100 300 1 100 300
Time (ms) 22.7 21 22.1 29.9 9.4 5.7 14.5 6.3 11.9
CPU (%) 23–35 45–100 57–100 31–45 34–58 56–100 23 100 100
RAM(MB) 228–247 239–357 253–377 228–570 239–593 253–604 247 257 273

LoRaWare may potentially produce big amounts of data or receive many

requests which can surpass the capacities that this experimental system set-
up is able to provide. An obvious solution would be to spawn additional VMs
implementing the same service thus having more than on VM sharing the load.
LoRa Network Evaluation: Reliable communication of LoRa devices depends
on number of transmission parameters [3]. The following parameter values are
defined (in Arduino sketches): Spread factor = 12, bandwidth = 125 kHz and
transmission power = 17 dbm. The code rate was set to 4/5 (the value was fixed
by Ideetron). These values would maximize the effective range of LoRa (although
for these values power consumption will be maximum as well).
We run the experiment in a complicated urban and sub-urban environmental
terrain in the city of Chania, Crete. We placed two Lorank 8 gateways 5.7 Km
apart from each other. The first one is placed in an urban area and the second one
in a sub-urban area of the city of Chania. Two mobile LoRa nodes transmitted 74
temperature and humidity measurements (37 each node). The longest effective
range was 923 m and 1,23 Km for the gateway placed in the urban and sub-urban
area respectively. Table 3 shows how error rate varies with distance. Despite the
high error rate, its worth noting that more than 80% of transmitted packages
were received successfully up to 350 m and 700 m from the gateway in the urban
and sub-urban area respectively. These results are below our expectations. This
could be due to the noise interference and the non-uniform ground terrain with
many obstacles that hinder transmission in the area of the experiment.
 LoRaWare 1093

Table 3. Error Rate (ER) as a function of the distance from gateway

Distance (m) 100 300 500 800 1,000 1,500

ER Urban Area (%) 0 15 30 85 100 100
ER Sub-Urban Area (%) 0 0 20 35 65 90

6 Conclusions

We propose and implement the Network Server, a service that enables porting of
LoRa networks to the cloud. As a result, the cloud supports unified management
of LoRa payloads with minimum additional effort. In addition, we designed and
implemented LoRaWare, an experimental infrastructure for studying the perfor-
mance of LoRa networks. The response times reported for all cloud services are
a good support to our claims of efficiency: LoRaWare is capable of responding
in real-time (or close to real-time) even under heavy workloads. However, the
effective range of LoRa is way lower than expected. Although these results are
characteristic for a worst case environment (with many buildings, metal struc-
tures, rough terrain with many obstacles and gateways not optimally placed
at high altitude), the results indicate that lots of experimentation is necessary
for tweaking LoRa parameters in order to achieve closer to the optimal (i.e.
theoretical) performance.

References
1. Adelantado, F., Vilajosana, X., Tuset-Peiro, P., Martinez, B., Melia-Segui, J., Wat-
teyne, T.: Understanding the limits of LoRaWAN. IEEE Commun. Mag. 55(9),
34–40 (2017)
2. Biswas, A.R., Giaffreda, R.: IoT and cloud convergence: opportunities and chal-
lenges. In: IEEE World Forum on Internet of Things (WF-IoT), Seoul, South Korea,
pp. 375–376, March 2014
3. Bor, M., Roedig, U.: LoRa transmission parameter selection. In: International Con-
ference on Distributed Computing in Sensor Systems (DCOSS), pp. 27–34, May
2017
4. Douzis, K., Sotiriadis, S., Petrakis, E.G.M., Amza, C.: Modular and generic IoT
management on the cloud. Future Gener. Comput. Syst. (FGCS) 78(1), 369–378
(2018)
5. Petrakis, E.G.M., Sotiriadis, S., Soultanopoulos, T., Tsiachri-Rentaa, P., Buyya,
R., Bessis, N.: Internet of things as a service (iTaaS): challenges and solutions for
management of sensor data on the cloud and the fog. Internet Things 3–4(10),
156–174 (2018)
6. Raza, U., Kulkarni, P., Sooriyabandara, M.: Low power wide area networks: an
overview. IEEE Commun. Surv. Tutor. 19(2), 855–873 (2017)
7. Schreier, S.: Modeling RESTful applications. In: ACM International Workshop on
RESTful Design (WS-REST 2011), Hyderabad, India, pp. 15–21, March 2011
 Self-adaptive RFID Authentication
for Internet of Things

Bacem Mbarek, Mouzhi Ge(B) , and Tomas Pitner

Faculty of Informatics, Masaryk University, Brno, Czech Republic

[email protected], [email protected], [email protected]

Abstract. With the development of wireless Internet of Things (IoT)

devices, Radio frequency identification (RFID) has been used as a
promising technology for the proliferation and communication in IoT net-
works. However, the RFID techniques are usually plagued with security
and privacy issues due to the inherent weaknesses of underlying wireless
radio communications. Although several RFID authentication mecha-
nisms have been proposed to address the security concerns in RFID, most
of them are still vulnerable to some active attacks, especially the jam-
ming attack. This paper therefore proposes a novel self-adaptive authen-
tication mechanism with a robust key updating, in order to tackle the
security vulnerabilities of key updating algorithms and their inability to
jamming attacks. Furthermore, we assess the performance and applica-
bility of our solution with a thorough simulation by taking into account
the energy consumption and authentication failure rate.

Keywords: Internet of Things · RFID · Authentication ·

Jamming attack · Security

1 Introduction
Internet of Things (IoT) has rapidly become a critical research subject due to
the significant development and innovation of smart technologies such as RFID
technology, embedded sensors and Near Field Communication (NFC) devices.
These smart devices, along with the communication infrastructure, are usually
referred as the IoT network [1,2]. IoT involves an extensive presence of a vari-
ety of connected things or objects including RFID tags, wireless sensors, mobile
phones, and other devices [3,4]. Since RFID technology is affordable and effec-
tive to prevent illegitimate users from unauthorized usage of the IoT devices
resources, the RFID technology has become a prevalent and important infras-
tructural technology in IoT.
In a RFID system, data are transmitted to a portable device, named as tag,
which is read by an RFID reader and processed according to the requirements
of a particular application. The data transmitted by the tag may provide identi-
fication or location information or specifications about the tagged product, such
as price, color, date of purchase, etc. RFID aims to identify objects remotely,
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1094–1105, 2020.
https://doi.org/10.1007/978-3-030-15032-7_92
 Self-adaptive RFID Authentication for Internet of Things 1095

with neither physical nor visual contact. Although RFID for tracking applica-
tions first appeared during the 1980s, RFID technology is more recently well
developed mainly due to the standardization and development of low-cost tags.
This new generation of RFID tag has facilitated various applications, such as
digital supply chains, location-based tracking, detecting counterfeit of expensive
items, tracing livestock as well as labeling books in libraries [5]. Among others,
RFID technology has been widely deployed in IoT networks [6] and Smart City
applications [7].
One of the advantages of RFID technology is the automatic authentication
and identification technique between the RFID components. On the other hand,
if an attack occurs on the tags, the authentication techniques issues may lead
to mis-identification. For example, security attacks, such as denial of service,
spoofing, impersonation of tags or channel eavesdropping, are rendered more
practicable because of lacking of computational and storage capacity. One of the
well-known RFID attack is to intercept the keys used between the tags. If the
attack activity compromises some tags, it may obtain several paths to perform
the root authorization of those paths. Meanwhile, since keys are never changed
in the static authentication architecture, the captured keys will still be used
by uncompromised tags. This will in turn cause more security problem in IoT
networks.
In this paper, we therefore propose a reliable and lightweight RFID authen-
tication protocol with a robust Key updating scheme. We focus on the secu-
rity vulnerabilities of key updating algorithms and their inability to jamming
attacks. Our proposed protocol is able to effectively defend against both pas-
sive and active attacks, including compromising, DoS and jamming attacks. The
intuition of our solution is that to resolve the problem of jamming attacks, the
reader appends MACs with different keys (with the updated key and the previous
keys of each tag) so that a tag can verify the received MACs authenticity with
the updating key or with the N previous keys. If the reader fails to authenticate
with the updated key, the reader still can use the previous keys.
The remainder of the paper is organized as follows. Section 2 describes and
revisits a typical RFID protocol and its security concerns. Section 3 discusses
the state-of-the-art solutions that have been proposed to solve the RFID secu-
rity breach. Based on the related work, Sect. 4 proposes a new solution that
can dynamically authenticate the RFID keys. In order to evaluate the proposed
solution, Sect. 5 conducts a security analysis simulation by considering authenti-
cation failure rate and energy consumption. Finally, Sect. 6 concludes the paper
and outlines future works for this paper.

2 RFID Protocol and Security Concerns

RFID device is an essential infrastructure in an IoT network to remotely identify

wireless objects without physical or visual contact. A RFID system typically
consists of two electronic chips, a reader and a tag [8]. RFID tags support the
reception within a range of a few meters and high-speed communication. The
1096 B. Mbarek et al.

tags are usually constructed as a network to work collaboratively. A typical

communication between a reader and a tag is described as follows. A reader
queries a tag to obtain the tag’s contents though RF (Radio frequency) interface.
Moreover, by authenticating a tag to a reader, a tag should be ready to open
its information to a reader. We consider that the typical RFID authentication
protocol as shown in Fig. 1 so that it can be applied easily to existing systems.

Fig. 1. A reader queries a tag with RFID authentication protocol

RFID tags are usually largely deployed in IoT and thus, an adversary could
inject misleading information in the network and even fool other nodes, making
them believe it is an authentic device. Therefore, it is important to protect
RFID tags against all types of attacks. In RFID systems, an adversary may
monitor all the messages transmitted between a reader and a tag. The adversary
can also infringe upon a user’s privacy using various methods. Therefore, RFID
systems must be designed to be protected from attacks such as eavesdropping
and impersonation. We have summarized three typical RFID security concerns
in IoT applications as follows.
Confidentiality: An attacker may try to access the information to IoT devices
without authorizations. Confidentiality should be ensured by using encryption
algorithms. Cryptography encrypts message to generate an illegible text. Key
confidentiality is to protect the key so that it can only be recovered by authorized
tags.
Authentication: Key authentication is to provide assurance to authorized tags
that the key is distributed by certain reader but not from an attacker. Jamming
attack can be a critical problem for authentication procedure between the reader
and the tag. Jamming is one of many exploits to compromise the wireless envi-
ronment. It works by denying service to authorized users as legitimate traffic,
which is jammed by the overwhelming frequencies of illegitimate traffic.
Data Integrity: Physical attack is a viable method of compromising the
integrity of a device, hence an attacker upon compromising a tag may be able
to read its secret values like keys and link this tag with past actions performed
on the tag. A Key freshness should be ensured by tags and a compromised key
cannot be reused for authentication request. Forward privacy ensures that mes-
sages transmitted now will still be secure in the future, even after compromising
the tag.
 Self-adaptive RFID Authentication for Internet of Things 1097

3 Related Work
Since RFID authentication enables a reliable and efficient communication chan-
nel between the components in IoT network, over the last decade many efforts
have been made to achieve efficient RFID authentication. To the best of our
knowledge, the most efficient protocols are tree-based [9,10]. They provide an
efficient key search scheme with logarithm complexity. In such approaches, each
tag holds multiple keys instead of a single key. A virtual hierarchical tree struc-
ture is constructed by the reader to organize these keys. Every node in the tree,
except the root, stores a unique key. Each tag is associated with a unique leaf
node. Keys in the path from the root to the leaf node are then distributed to
this tag. A tag encrypts the authentication message by using d keys from the
tree. During authentication, the reader performs a depth search in the key tree.
To simplify the algorithm applicability, [11] propose the HashLock scheme:
The tag carries a key K and a metaID which is simply the hash value h(K). When
a reader queries the tag, the tag responds with its metaID which is forwarded
to the back-end database. The database recognizes the tag from its metaID, the
secret key K is forwarded to the reader which eventually reports back this value
to the tag. At this point the tag hashes the value received and if it matches the
stored metaID it unlocks itself. Although this scheme is simple and reliable it
suffers from the obvious drawback that a tag can still be tracked by its metaID.
In order to solve the tracking problem with single metaID, [12] presented
a randomized version where tag responses change with every query made. In
particular, the tag sends a pair <r, h(ID, r)>, where r is a random nonce. On
one hand, this scheme solves the tracking problem mentioned above, on the
other hand this scheme introduced a new problem: As the tag responds with
different values every time, the database must exhaustively search through its
list of known IDs until it finds one that matches h(ID, r), for the given r. In other
hand, a few works use and implement key update operations at protocol level for
full-fledged tags. For example, in [8], authors proposed a security enhancements
of an existing mutual authentication protocol (TAP Protocol). Then, security
enhancements of the TAP Protocol based on key update. However, the tag,
the reader and the server updates its seed si and key ki after each successful
authentication.
Another variation of the previous scheme is proposed in [13], in which a
hash-chain mechanism is used to protect against tracking and ensure forward
security in tag transactions. Again, the tag replies back with different values,
but its secret ID changes with every response through the application of a hash
function. However, just as in the previous scheme scalability is problematic as
it requires exhaustive search in the back-end database to locate the ID of the
tag. Although in [10] authors propose a challenge-response scheme that allows
for more efficient identification. The protocol involves three messages per round
and allows for mutual authentication of both the tag and the reader. However,
this protocol uses O(log N) communication rounds, where N is the number of
possible tags in the system. Hence, one drawback of this scheme is that it cannot
be applied to legacy systems in which a reader makes only one request and the
1098 B. Mbarek et al.

tag responds with a simple value. A performance enhancement of randomization-

based authentication is proposed in recent work [14], a simple protocol has been
presented in which the both the tag and the reader use a random number gen-
eration and only a bitwise operations, i.e., bitwise XOR and rotation operations
which makes it an ultra-lightweight solution for mutual authentication between
reader and the RFID tag.

Table 1. RFID authentication protocols

Scheme Description Advantages Limitations

The tree based Dynamic key-updating Defend against the Vulnerable with
authentication [9, 10, 15] using a key-tree based compromise attacks and jamming attacks
scheme resolve the scalability
problem
Randomized Access The reader generates a Defend against the Vulnerable with
control [8, 11–14] random number R and compromise attacks jamming attacks
uses it to updates the
shared key with the tag
Third-party A third part allows the Defend against the Vulnerable with
authentication [16] RFID-tag to verify a compromise attacks jamming attacks
valid authentication

More recently, authors in [17] proposed an effective approaches to providing

a certificateless authentication protocol which is rooted in a novel ECC-based
pairing free certificateless signature scheme. However, ECC protocol is vulnerable
to jamming attacks and DoS attacks. However, due to the fact that limited
resources can be used to protect the transmitted data at tag side, the wireless
channel between the reader and the tag may be attacked by different kinds of
existing and potential methods.
Based on our literature review, we found that the RFID authentication
schemes in IoT networks can be classified into three categories, which are tree
based authentication, randomized access control, and third-party authentication.
For each category, in Table 1, we have summarized the advantages as well as
limitations of the very scheme. It can be seen that one fundamental issue of the
previously proposed authentication approaches appear to be vulnerable with
jamming attacks. Therefore this paper tackles RFID authentication by consid-
ering the jamming attack, which is a critical security problem in IoT research,
because the IoT devices can be breached by jamming attacks.

4 Self-adaptive RFID Authentication

This section first describes how the jamming attack occurs, and then provides a
detailed description of our proposed authentication solution, which includes the
self-adaptive authentication mechanism as well as the authentication algorithms
for the reader and tag.
 Self-adaptive RFID Authentication for Internet of Things 1099

4.1 Jamming Attack

RFID tags encrypts authentication messages to RFID reader, and the reader
searches the key space to locate the tags. Due to the lack of efficient key updating
algorithms, previous schemes are vulnerable to jamming attacks. For example, an
attacker generates a jamming signal, so that the tag cannot receive the message
from the reader and does not update its secret information while the reader
updates the tag’s secret information. After this attack, the secret information
will be inconsistent between the reader and the tag. Therefore, the authentication
will fail. Hence, the individual keys will be updated dynamically, with respect to
the defense against jamming attacks. Suppose the reader keeps only the current
keys per tag. Then, the authentication will be failed. In Fig. 2, we present an
example of jamming attack which can be seen by the update of RFID keys.

4.2 Self-adaptive Authentication Mechanism

We propose a self-adaptive Authentication Mechanism (SAM) capable of authen-
ticating entities in IoT network through an RFID tags technology, that aims to
resolve the problem of jamming attack. Then, the reader only needs to store
the previous commitment keys for each tag. If the reader fails to authenticate
with the updated key, the reader can use the previous keys. Since N previous
tags-keys are stored in the reader. For each authentication, the reader generates
a MAC (message authentication code) encrypted with the updated key ad gen-
erates N MACs encrypted with N previous keys. The format of each send query
is as follows.

Fig. 2. RFID protocols with a key update phase that suffer from jamming attack
1100 B. Mbarek et al.

S = <M AC(kupdated , M )||M AC(ki , M ), M AC(ki−1 , M ), ...., M AC(ki−N +1 , M )>

4.3 Authentication Algorithm

At the beginning, the reader and the tag have the same shared key k. The ID
of each tag is stored on the reader and on the memory of tag. The reader gen-
erates a code MAC M encrypted with the updated key kupdated (M ) and gener-
ates N M ACs encrypted with N previous keys ki (M ), ki−1 (M ), ...., ki−N +1 (M ).
Algorithms 1 and 2 describe the proposed authentication procedure between the
reader and the tag.
In the following algorithms, the authentication is conducted in two ways. By
authenticating a reader to a tag and by authenticating a tag to a reader. The
first algorithm describes the different authentication steps done by the reader.
The reader updates the sharing common secret key by xored the current key Ki
with a random value R, and checks whether the new key is unique. If it is not
unique, the reader regenerates R until it becomes unique. Then, after updating
and sending the key. The reader will communicate with the tag with the new
updated key and N previous keys. The readers have the property that if the
updated keys are lost, they can be recomputed using previous keys. Therefore,
even if some disclosed keys are lost due to packet loss or jamming attacks, the
tag still can recover the key from the previous keys and check the authenticity
of messages. The second algorithm describes the authentication procedure in
the tag after updating the key, whereby the tag can verify the reader with the
updated key or with the N previous sharing common keys.
In the most IoT RFID systems, the communication complexity refers to
the amount of data transmitted between the reader and the tag in the RFID
authentication algorithm [18]. Let C(n) be the communication complexity of
RFID-algorithm, CR(n) is the reader-side communication complexity, CT (n)
is the tag-side communication complexity, and they satisfy the following rela-
tionship: C(n) = CR(n) + CT (n). Our authentication algorithm reduces the
effect of jamming attacks as soon as possible using two manners for providing

Algorithm 1. Authentication Procedure on the Reader

The reader generates a random value R and computes k = k ⊕ R
if k ⊕ R is not unique then
the reader regenerates R until k ⊕ R becomes unique.
else if k ⊕ R is unique then
Server encrypts the updated key with the previous key Ki (kupdated ) and sends it
to the tag.
if update keys are lost due to packet loss or jamming attacks. then
The reader can communicate with the tag using N previous keys.
The reader creates and sends the following structure to the tag.
S = <kupdated (M )||ki (M ), ki−1 (M ), ...., ki−N +1 (M )>
end if
end if
 Self-adaptive RFID Authentication for Internet of Things 1101

Algorithm 2. Authentication Procedure on the Tag

The Tag checks Kupdated (M )
if it is correct then
The reader is authenticated.
else if it is not correct then
Tag checks ki (M ), ki−1 (M ), ...., ki−N +1 (M ).
if the previous keys are authenticated then
The reader is authenticated
else if failed then
The message will be considered forged and the authentication will failed.
The tag would need to inform the reader and the reader would need to retrans-
mit the updated key.
end if
end if

authentication (current and previous keys). This later method reduces the com-
munication overhead in the presence of jamming attacks compared to the other
typical RFID-algorithms.

5 Evaluations
In order to evaluate the self-adaptive authentication protocol, We used the NS-
2.35 simulator [19] to analyze the performance of the SAM scheme in terms
of energy consumption and percentage of authentication failure. We run our
simulations 50 times on a large scale network with 800 tags and one reader that
initiates the communication in every iteration with all the settings provided in
the Table 2. We assume that all tags have a fixed position throughout the period
of simulation.

Table 2. Simulation parameters

Parameter Value
Simulation time 100 s
Run times 50 times
Number of tags 800
Number of reader 1
Mobility None

We compare our proposed approach with an enhanced mutual TAP protocol

proposed in [8] that uses a randomized access control to update the shared key
with the tag to defend against compromise attacks. Also, through prototype
implementation, we observe that our proposed protocol is easy to deploy and
scalable in existing RFID infrastructures.
1102 B. Mbarek et al.

5.1 The Average of Authentication Failure Rate

We evaluate the average of authentication failure rate between the reader and the
tag, with varying the number of jamming attacks. Figure 3 shows the percentage
of authentication failure versus the number of jamming attacks. We found that
SAM have a significantly low percentage than the TAP protocol. The main
reason is that SAM protocol uses the two authentication ways (with current
updated key and the with previous keys), while with TAP protocol the reader
keeps only the current keys per tag. When a jamming attacks occurs, the tag
will not receive the updated key and the authentication will fail.

Fig. 3. Evaluation of authentication failure between the reader and the tags

Figure 3 depicts the results for the percentage of authentication failure

between the SAM and TAP approaches. In particular, it shows that the per-
centage of authentication failure is lower with SAM, especially when the number
of jamming attack is high i.e., above 10. For instance, with TAP, we can detect
around 80% of the forged packets while only 60% are failed with SAM because
it uses two authentication ways which are current and N previous keys.

5.2 Energy Consumption

We also evaluate the energy consumption of SAM and TAP protocols by varying
the number of jamming attacks. Figure 4 reports the impact of this attack on
energy consumption. From this figure, we obtained three interesting outcomes.
The first outcome is that by increasing the number of jamming attacks, the
energy consumption increases for all the studied protocols. For instance, when
the number of jamming attacks is equal to 6, SAM and TAP consume more than
30 J. The second outcome is that SAM has a low energy consumption compared
to TAP, when increasing the number of forged packets. One explanation to this
 Self-adaptive RFID Authentication for Internet of Things 1103

phenomenon is that authentication in our solution will be done with the current
and the previous keys in case of jamming attack, and the authentication process
will continue by the decryption of N encrypted MACs with N previous keys;
while for TAP the authentication will not succeed in the presence of jamming
attack and the reader needs to repeat the procedure many times until the receiver
gets the updated key. The third outcome is that without attack, SAM incurs
higher communication overhead. This may be explained by the fact that in order
to encrypt message, the SAM protocol needs to search N keys (previous keys)
and use them to encrypt N MACs; while with TAP protocol the reader creates
and sends only one MAC with the current key.

Fig. 4. Evaluation of energy consumption

6 Conclusion

In this paper, we have proposed a new self-adaptive RFID authentication proto-

col, named as SAM, to provide a secure and efficient tag-to-reader transaction
in IoT applications. By using dynamic key-updating algorithms, our proposed
solution enhances the key updating system based on enabling different ways to
authenticate packets, which significantly reduces the impact of jamming attacks.
One important advantage of our protocol is that it can be seamlessly deployed
to existing systems for increasing the security of tag identification while at the
same time maintaining the system efficiency.
Our simulation results show that SAM protocol outperforms the TAP proto-
col in terms of energy consumption and authentication failure rate. For energy
consumption compared to TAP, although SAM consumes slightly more energy
in the absence of attacks, it consumes significantly less energy in the presence
of attacks. This indicates that when the security is more prioritized in an IoT
system, SAM can be used when the system is started. Furthermore, SAM can
1104 B. Mbarek et al.

also be dynamically invoked when certain jamming attack is detected. As future

work, we plan to deploy SAM for real-world IoT network and further consolidate
the performance of proposed authentication algorithms.

References
1. Ge, M., Bangui, H., Buhnova, B.: Big data for internet of things: a survey. Future
Gener. Comput. Syst. 87, 601–614 (2018)
2. Mbarek, B., Meddeb, A., Ben Jaballah, W., Mosbah, M.: A secure electric energy
management in smart home. Int. J. Commun. Syst. 30(17) (2017)
3. Memos, V.A., Psannis, K.E., Ishibashi, Y., Kim, B.-G., Gupta, B.B.: An efficient
algorithm for media-based surveillance system (EAMSUS) in IoT smart city frame-
work. Future Gener. Comput. Syst. 83, 619–628 (2018)
4. Luo, H., Wen, G., Su, J., Huang, Z.: SLAP: succinct and lightweight authentication
protocol for low-cost RFID system. Wirel. Netw. 24(1), 69–78 (2018)
5. Adame, T., Bel, A., Carreras, A., Melià-Seguı́, J., Oliver, M., Pous, R.: CUIDATS:
an RFID-WSN hybrid monitoring system for smart health care environments.
Future Gener. Comput. Syst. 78, 602–615 (2018)
6. Bangui, H., Ge, M., Buhnova, B.: Exploring big data clustering algorithms for
internet of things applications. In: Proceedings of the 3rd International Conference
on Internet of Things, Big Data and Security, IoTBDS 2018, Madeira, Portugal,
pp. 269–276 (2018)
7. Stepanek, P., Ge, M.: Validation and extension of the smart city ontology. In: Pro-
ceedings of the 20th International Conference on Enterprise Information Systems,
ICEIS 2018, Madeira, Portugal, pp. 406–413 (2018)
8. Naija, Y., Beroulle, V., Machhout, M.: Security enhancements of a mutual authen-
tication protocol used in a HF full-fledged RFID tag. J. Electron. Test. 34(3),
291–304 (2018)
9. Dimitriou, T.: A secure and efficient RFID protocol that could make big brother
(partially) obsolete. In: Pervasive Computing and Communications (2006)
10. Molnar, D., Wagner, D.: Privacy and security in library RFID: issues, practices,
and architectures. In: Proceedings of the 11th ACM Conference on Computer and
Communications Security, pp. 210–219 (2004)
11. Weis, S.A., Sarma, S.E., Rivest, R.L., Engels, D.W.: Security and privacy aspects
of low-cost radio frequency identification systems. In: Security in Pervasive Com-
puting, pp. 201–212 (2004)
12. Lu, L., Han, J., Hu, L., Liu, Y., Ni, L.M.: Dynamic key-updating: privacy-
preserving authentication for RFID systems. In: Pervasive Computing and Com-
munications, pp. 13–22 (2007)
13. Ohkubo, M., Suzuki, K., Kinoshita, S., et al.: Cryptographic approach to “privacy-
friendly” tags. In: RFID Privacy Workshop, vol. 82 (2003)
14. Tewari, A., Gupta, B.: Cryptanalysis of a novel ultra-lightweight mutual authenti-
cation protocol for iot devices using RFID tags. J. Supercomput. 73(3), 1085–1102
(2017)
15. Molnar, D., Soppera, A., Wagner, D.: A scalable, delegatable pseudonym protocol
enabling ownership transfer of RFID tags. In: International Workshop on Selected
Areas in Cryptography, pp. 276–290 (2005)
16. Baashirah, R., Abuzneid, A.: Survey on prominent RFID authentication protocols
for passive tags. Sensors 18(10) (2018)
 Self-adaptive RFID Authentication for Internet of Things 1105

17. Shen, J., Gui, Z., Ji, S., Shen, J., Tan, H., Tang, Y.: Cloud-aided lightweight certifi-
cateless authentication protocol with anonymity for wireless body area networks.
J. Netw. Comput. Appl. 106, 117–123 (2018)
18. Liu, B., Su, X.: An anti-collision algorithm for rfid based on an array and encoding
scheme. Information 9(3), 63 (2018)
19. Downard, I.T.: Simulating sensor networks in ns-2. Technical report
 Limiting the Influence to Vulnerable
Users in Social Networks:
A Ratio Perspective

Huiping Chen1 , Grigorios Loukides1(B) , Jiashi Fan1 , and Hau Chan2

1
King’s College London, London, UK
{huiping.chen,grigorios.loukides,jiashi.fan}@kcl.ac.uk
2
University of Nebraska-Lincoln, Lincoln, USA
[email protected]

Abstract. Influence maximization is a key problem in social networks,

seeking to find users who will diffuse information to influence a large
number of users. A drawback of the standard influence maximization is
that it is unethical to influence users many of whom would be harmed,
due to their demographics, health conditions, or socioeconomic charac-
teristics (e.g., predominantly overweight people influenced to buy junk
food). Motivated by this drawback and by the fact that some of these
vulnerable users will be influenced inadvertently, we introduce the prob-
lem of finding a set of users (seeds) that limits the influence to vulner-
able users while maximizing the influence to the non-vulnerable users.
We define a measure that captures the quality of a set of seeds, as an
additively smoothed ratio between the expected number of influenced
non-vulnerable users and the expected number of influenced vulnerable
users. Then, we develop greedy heuristics and an approximation algo-
rithm called ISS for our problem, which aim to find a set of seeds that
maximizes the measure. We evaluate our methods on synthetic and real-
world datasets and demonstrate that ISS substantially outperforms a
heuristic competitor in terms of both effectiveness and efficiency while
being more effective and/or efficient than the greedy heuristics.

1 Introduction

There has been an increased interest from the public and private sectors and
organizations in leveraging social networks to spread information of adopting
certain behavior (e.g., buying ipads or alcoholic beverages). A typical method-
ology of an organization is to influence a selected few users (seeds), through
free gifts, discounts, and information sessions, to adopt the desirable behavior.
The hope is that these seeds will influence other users in their social circles to
adopt the same behavior, and the subsequent influenced users will influence oth-
ers in their respective social circles. As the information propagates throughout
the social network, eventually some number of users will adopt the desirable
behavior.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1106–1122, 2020.
https://doi.org/10.1007/978-3-030-15032-7_93
 Limiting the Influence to Vulnerable Users in Social Networks 1107

As a result, the organization’s goal is to solve the problem of selecting a set of

k seeds which maximize the largest expected number of adoptions of all the users
in the social network (spread ). This problem is known as the influence maximiza-
tion problem in social networks [10] and has been widely studied in the recent
decade [14]. A main drawback of influence maximization is that it is unethical
to influence users many of whom could be harmed due to their demographics,
health conditions, or socioeconomic profile [7]. The users who could be harmed
are referred to as vulnerable and are identified based on domain knowledge (e.g.,
user message content and sentiment analysis) [15,21]. For example, when an
organization aims to promote alcoholic beverages, it should avoid influencing
users many of whom have drinking problems. Similarly, when it aims to pro-
mote junk food, it should avoid influencing users many of whom are overweight.
This is important for performing socially responsible influence maximization [1],
which benefits not only the vulnerable users but also the companies, because
most users are often willing to pay more for products marketed in a socially
responsible way [19]. Motivated by the presence of vulnerable users, we initiate
the study of influence maximization in social networks with both vulnerable and
non-vulnerable users. In particular, due to the diversity of social networks and
that some vulnerable users will be influenced inadvertently, we consider the prob-
lem of limiting the influence to vulnerable users while maximizing the influence
to the non-vulnerable users in social networks.
Contribution. Our work makes the following specific contributions.
(1) Influence Measure. To deal with influence maximization in our setting,
we need a measure to quantify the quality of a set S of seeds (seed-set). The
measure should ideally consider both vulnerable and non-vulnerable users, limit
influencing users many of whom are vulnerable, and allow obtaining a seed-set
with guaranteed quality. We examine the following natural measures and show
that they are inappropriate to be used for influence maximization in our setting:
(a) the difference σN (S) − σV (S) and (b) the ratio σσNV (S)
(S)
, where S is a seed-set
and σN (S) and σV (S) is the expected number of influenced non-vulnerable users
and vulnerable users, respectively. Then, we propose an additive smoothing ratio
(ASR) measure σσNV (S)+c
(S)+c
, where c > 0 is a specified constant. We show that
ASR satisfies all the aforementioned properties and examine the impact of c in
our influence maximization setting. Thus, our problem becomes finding a seed-
set S of size at most k that maximizes ASR. This is a challenging problem
because ASR is not monotone and neither submodular nor supermodular, which
implies that it cannot be approximated through algorithms for submodular or
supermodular maximization [4,18,23].
(2) Baseline Heuristics for Finding an ASR-Maximizing Seed-set. Since ASR
is a ratio of submodular functions, we develop a natural greedy heuristic (GR)
that finds a seed-set of size at most k and large ASR iteratively. In each iteration,
GR selects as seed a non-vulnerable node which influences a large number of
additional non-vulnerable nodes for a small number of additional vulnerable
nodes. We then develop GRM B , a variation of GR that estimates the spread
efficiently.
1108 H. Chen et al.

(3) Approximation Algorithm for Finding an ASR-Maximizing Seed-set. We

design ISS , an efficient approximation algorithm for finding a seed-set to max-
imize ASR. Since ASR is not submodular, ISS cannot maximize it directly.
Instead, ISS constructs three candidate seed-sets (one with ASR, another with
a submodular lower bound function of ASR, and a third with a submodular
upper bound function of ASR) and selects the best candidate seed-set. This is
performed iteratively, with different bound functions that aim to increase the
ASR of the final seed-set. Our experiments show that ISS outperforms a heuris-
tic that is based on the difference σN (S) − σV (S) [20] as well as GR and GRM B ,
with respect to ASR and the spread of non-vulnerable and/or vulnerable nodes,
while it is efficient and also scalable with respect to the seed-set size k.

2 Background

Submodular Functions. Let U be a universe of elements and 2U be its power

set. A function f : 2U → R is monotone, if f (X) ≤ f (Y ) for all subsets X ⊆
Y ⊆ U , and non-monotone otherwise. A function f : 2U → R is submodular, if it
satisfies the diminishing returns property f (X ∪{u})−f (X) ≥ f (Y ∪{u})−f (Y ),
for all X ⊆ Y ⊆ U and any u ∈ U \ Y [12]. If the property holds with equality,
then f is modular. A function f : 2U → R is supermodular if and only if −f
is submodular [12]. A modular function f : 2U → R is both submodular and
supermodular. For brevity, we may write f (X|u) for the marginal gain f (X ∪
{u}) − f (X).
Let f : 2U → R be a submodular function. For any Y ⊆ U , the modular
upper bound f Y (X) of f (X) is a modular function [9]

 
f
Y (X) = f (Y )+ (f ({u}) − f ({})) − (f (Y ) − f (Y \ {u})) (1)
u∈X\Y u∈Y \X

and the modular lower bound fY,πY (X) of f (X) is a modular function [9]


fY,πY (X) = fY,πY (u). (2)
u∈X

Y is referred to as the parameter of the bound, π Y is a random permutation of

the elements of Y (i.e., one-to-one mapping of Y onto itself), and


f (πuY ) − f (πuY− ), if u ∈ Y
fY,πY (u) = (3)
0, otherwise

where πuY− is the prefix of π Y comprised of all elements of π Y that appear before
u in π Y , and πuY is the prefix of π Y comprised of all elements of πuY− and u.
 Limiting the Influence to Vulnerable Users in Social Networks 1109

Independent Cascade (IC) Model. To model influence, we consider the

classical IC model [10,20,24]. The model views the social network as a weighted
directed graph G(V, E), where V and E is the set of nodes and edges of G,
respectively. In our setting, V is partitioned into N and V, comprised of all
non-vulnerable and vulnerable nodes, respectively. We assume that N = ∅,
otherwise no seed can be selected, and that V is selected by the organization
performing influence maximization based on domain knowledge [15,21]. The set
of in-neighbors (respectively, out-neighbors) of a node u is denoted with n− (u)
(respectively, n+ (u)), and its size is referred to as the in-degree (respectively,
out-degree) of u. In the IC model, each newly activated node u tries to activate
each inactive out-neighbor u ∈ n+ (u ) once with probability p((u , u)), which
is modeled as a weight of the edge (u , u) in G and is typically set to |n−1(u)|
[24]. If multiple nodes have the same out-neighbor, they all try to activate it
in an arbitrary order independently. The diffusion process starts from a set S
of nodes (seeds), which are active at time 0. Each seed tries to activate its out-
neighbors at time 0, each activated out-neighbor stays active and tries to activate
its own inactive out-neighbors at time 1, and the process proceeds similarly
and ends when no new node becomes active. A seed-set S activates a node
u with probability
 PS (u), and the spread
 of S over V , N , and V is defined
as σ(S) = u∈V PS (u), σN (S) = u∈N PS (u), and σV (S) = u∈V PS (u),
respectively. For any seed-set S, σN (S) and σV (S) are monotone submodular
functions [10]. We may omit the argument and value of σN and σV when it is
clear from the context (e.g., write a seed-set with zero σN instead of a seed-set
S with σN (S) = 0).

3 Measures and Problem Definition

To study influence maximization in our setting, we need a measure that quantifies

the quality of a seed-set and can be incorporated into methods to construct a
high quality seed-set. The measure should favor a seed-set S that influences
many non-vulnerable but few vulnerable nodes and also satisfy the following
properties:

1. It should consider the influence of vulnerable and non-vulnerable nodes. In

fact, we observed experimentally that constructing S based on only σN (S)
(resp., σV (S)) results in large σV (S) (resp., small σN (S)), which is undesir-
able.
2. It should consider what fraction of all influenced users are vulnerable. This
is important to penalize seed-sets that influence a large expected number of
users many of whom are vulnerable.
3. It should allow constructing a seed-set with guaranteed quality (e.g., not “too
far” from the optimal seed-set in the worst case) [10].
1110 H. Chen et al.

Natural Measures. A first measure is the difference σN (S) − σV (S) given a

seed-set S (i.e., the measure used in [20], with vulnerable nodes being treated as
non-target nodes). This measure does not consider what fraction of all influenced
users are vulnerable. Therefore, it may lead to constructing seed-sets with a
large expected number of influenced users many of whom are vulnerable. For
example, this measure would favor promoting an alcoholic beverage to 140 users
out of whom 40 have drinking problems, instead of 59 users with no drinking
problems, since (140 − 40) − 40 > 59 − 0. In addition, σN (S) − σV (S) cannot be
approximately maximized [9]. Thus, to construct a seed-set S, one has to settle
with heuristics, such as [20], which offer no approximation guarantees.
Another natural measure is the ratio σσNV (S)
(S)
. The ratio considers what fraction
σ(S)−σV (S)
of all influenced users are vulnerable, because it can be rewritten as σV (S) =
σ(S)
σV (S) − 1 and the constant can be removed when it is maximized. However, it
is undefined for every seed-set S with σV (S) = 0 (i.e., S that does not influence
vulnerable nodes). Thus, it cannot distinguish between any two seed-sets S1 ,
S2 such that σV (S1 ) = σV (S2 ) = 0 and σN (S1 ) > σN (S2 ) (e.g., it cannot favor
promoting an alcoholic beverage to 59 users with no drinking problems vs. 2 users
with no drinking problems) and also it is not clear how it can be approximately
maximized. For example, the GreedRatio framework [3] for maximizing a ratio
of two monotone submodular functions would result in a seed-set of unbounded
size, which is not useful for influence maximization. The inverse ratio σσNV (S)
(S) is
defined for σV (S) = 0 but it cannot be used to distinguish between the seed-
sets S1 and S2 above, and it is equally difficult to minimize (minimizing it is
equivalent to maximizing σσNV (S)
(S)
). Thus, it cannot be used to find a seed-set with
small or zero σV (S) and large σN (S), which helps our goal (to attract many
users few of whom are vulnerable).
Our Proposed Measure. To retain the benefits of the ratio σσNV (S)(S)
, while fixing
the issues caused by seed-sets that do not influence any vulnerable nodes, we
apply additive smoothing [16] to the ratio. This leads to our additive smoothing
ratio (ASR) measure, defined as ASR(S, c) = σσNV (S)+c
(S)+c
, where S is a seed-set
and c > 0 is a constant determined by the organization performing influence
maximization. ASR is well defined (and larger than zero) when σV (S) = 0.
Furthermore, among the seed-sets S1 and S2 mentioned above, it favors the
seed-set S1 , which influences a larger expected number of non-vulnerable nodes.
In ASR, the constant c can be seen as a weight whose addition to σN (S) and
to σV (S) changes their ratio and determines seed selection. The impact of c on
seed selection will be discussed in Sects. 4 and 6. Given our measure ASR, we
define our influence maximization problem below.
 Limiting the Influence to Vulnerable Users in Social Networks 1111

Problem Definition. Given u 1 u

1 4 Seed-set S σN (S) σV (S) ASR(S, 1)
a graph G whose nodes are 1
S0 = {u2 , u4 } 2 1 3/2
partitioned into N and V 1
v1 S0 ∪ {u1 } 3 2 4/3

and parameters k and c, find S1 = {u2 , u3 , u4 } 3 2 4/3

a seed-set S ⊆ N of size u2 1 v2
S1 ∪ {u1 } 4 3 5/4
S2 = {u3 } 1 2 2/3

at most k that maximizes 1 S2 ∪ {u2 } 2 2 1

S3 = {u1 , u3 }
ASR(S, c). u3 1 v3 S3 ∪ {u2 }
3
4
3
3
1
5/4
Our problem is NP-hard (a) (b)
(by reduction from the stan-
dard influence maximization Fig. 1. (a) Example graph. N = {u1 , . . . , u4 }, V =
problem [10]) and cannot {v1 , v2 , v3 }, and each edge probability is equal to 1.
be approximated using algo- (b) The spread over non-vulnerable nodes, the spread
rithms for submodular [4,18] over vulnerable nodes, and ASR for different seeds-
or supermodular [23] max- sets with c = 1.
imization. This is because
ASR is non-monotone and is neither submodular nor supermodular, as we show
below.

Example 1. Consider the graph of Fig. 1a, whose set of nodes is partitioned into
N = {u1 , . . . , u4 } and V = {v1 , v2 , v3 }, and the ASR of the seed-sets in Fig. 1b.
ASR(S, c) is: (I) non-monotone, because for S0 ⊆ S0 ∪{u1 }, ASR(S0 , 1) = 3/2 >
ASR(S0 ∪{u1 }, 1) = 4/3; (II) not submodular, because for S0 ⊆ S1 and u1 ∈ N \
S1 , ASR(S0 ∪{u1 }, 1)−ASR(S0 , 1) = −1/6 < ASR(S1 ∪{u1 }, 1)−ASR(S1 , 1) =
−1/12, and (III) not supermodular, because for S2 ⊆ S3 and u2 ∈ N \ S3 ,
ASR(S2 ∪{u2 }, 1)−ASR(S2 , 1) = 1/3 > ASR(S3 ∪{u2 }, 1)−ASR(S3 , 1) = 1/4.

4 Baselines: Greedy Heuristics for Maximizing ASR

We explore two greedy baseline methods for constructing a seed-set S with size
at most k and large ASR(S, c). The first is, GR, a natural heuristic for limiting
the influence to vulnerable nodes. GR performs k iterations. In each iteration i
(steps 3 to 6), it adds into the subset Si the node u with the maximum ratio
between: (I) the sum of the marginal gain in σN , caused by adding u, and the
constant c, and (II) the sum of the marginal gain in σV , caused by adding u, and
the constant c. Since ASR is non-monotone, a subset constructed in an iteration
before i may have a larger ASR than Si . Therefore, in step 7, GR considers the
subsets constructed in all iterations and returns the one with the largest ASR.
1112 H. Chen et al.

We now discuss how GR deals with

a non-vulnerable node v that influences
no vulnerable nodes. Adding v into Si
makes the objective function of GR equal
to σN (Sci |v)+c (see step 4), since Si does
not influence more vulnerable nodes after
the addition of v (i.e., σV (Si |v) = 0). If
σN (Si |v) is small, it is better to add a dif-
ferent node v  which influences few vulner-
able nodes but “through” these vulnerable
nodes reaches out to many more non-vulnerable nodes than v. In fact, GR adds
(Si |v  )+c
v  instead of v if σσNV (S 
i |v )+c
> σN (Sci |v)+c , and uses the parameter c to control
the bias towards nodes such as v  , which influence a small number of vulnerable
nodes but many more non-vulnerable nodes than u, as shown in Example 2 and
experimentally in Sect. 6.
Example 2. In iteration i = 0, the non-vulnerable nodes u1 to u4 in Table 1a are
considered and the node u ∈ {u1 , . . . , u4 } with the largest σσNV (S
(S0 |u)+c
0 |u)+c
is added
into S0 = {}. As shown in Table 1b, c determines the added node. For c = 0.01,
u1 that influences no vulnerable and few non-vulnerable nodes is added, for c = 1,
u3 that influences one vulnerable and many non-vulnerable nodes is added, and
for c = 10, u4 that influences more vulnerable and non-vulnerable nodes than
u2 is added.

Table 1. (a) Non-vulnerable nodes u that are To improve the efficiency
considered for addition into S0 = {}, and of GR, we propose a variant,
the expected number of vulnerable and non-
GRM B (MB is for MIA Batch-
vulnerable nodes they influence. (b) The node
update). Unlike GR which com-
that is added into S0 for different values of c.
putes spread exactly by adapt-
ing the method of [8] to the
IC model, GRM B estimates the
spread efficiently using the MIA
(Maximum Influence Arbores-
cence) method [24]. MIA estimates the probability PS (u) for a node u and
seed-set S based on the union of paths from S that have the highest proba-
bility to influence u, instead of all paths. Consequently, GRM B is two orders of
magnitude faster on average than GR.

5 The ISS Approximation Algorithm for Maximizing

ASR
This section presents ISS (Iterative Subsample with Spread bounds), starting
from the bound functions of ASR that ISS employs.
Lower and Upper Bound Function of ASR. ASR(S, c) is non-monotone
non-submodular for any subset S (see Sect. 3) and, thus, it is difficult to approx-
 Limiting the Influence to Vulnerable Users in Social Networks 1113

imate directly. Our ISS algorithm finds a seed-set S with approximately maxi-
mum ASR(S, c), using two submodular functions ASRL and ASRU that bound
ASR from below and from above, respectively. These functions are defined as
follows:
L σN (S) + c σN (S) + c
ASR (S, c, Y ) = =  
σ
V,Y (S) + c σV (Y ) + σV ({u}) − (σV (Y ) − σV (Y \ {u})) + c
u∈S\Y u∈Y \S

σN (S) + c σN (S) + c

ASRU (S, c, π Y ) = = 
σV,πY (S) + c (σV,Y,πY (u)) + c
u∈S

where Y ⊆ N is the parameter in each bound function, and


σV (πuY ) − σV (πuY− ) , if u ∈ Y
σV,Y,πY (u) =
0 , otherwise

ASRL is obtained by replacing σV (S) in ASR with its modular upper bound
σ 
V,Y (S) (see Eq. 1) and using the fact that σV ({}) = 0. ASRU is obtained by
replacing σV (S) in ASR with its modular lower bound σV,πY (S) (see Eq. 2).
ASRL , as well as ASRU , is submodular with respect to a seed-set S, because:
(a) Its numerator is monotone submodular, as a sum of the monotone submod-
ular function σN (S) and the constant c [12], (b) its denominator is a modular
function, as a sum of a modular bound function and the constant c, and (c) the
ratio between a submodular function and a modular function is clearly submod-
ular (see Sect. 2). However, ASRL , as well as ASRU , is non-monotone, as shown
in Example 3.

Example 3 (continuing from Example 1). Let c = 1 and S  = {}. Since

ASRL (S1 , 1, S  ) = 4/4 > ASRL (S1 ∪ {u1 }, 1, S  ) = 5/6, ASRL is non-

monotone. Let S  = {u2 , u3 } and its permutation π S = (u3 , u2 ). Since
S  U S 
ASR (S3 , 1, π ) = 4/3 > ASR (S3 ∪ {u2 }, 1, π ) = 5/4, ASRU is non-
U

monotone.

ISS Algorithm. The algorithm works iteratively, as can be seen from the pseu-
docode. In each iteration, it creates a seed-set Scur in three phases: (a) dummy
element creation, (b) construction of three candidate seed-sets (one using ASR,
a second using ASRL and a third using ASRU ), and (c) selection of the best
candidate seed-set and removal of dummy elements from it. The iterations stop
when Scur is not better than the previously created seed-set Spr in terms of ASR
(steps 21–22). This guarantees that the algorithm terminates [9].
1114 H. Chen et al.

Phase I (steps 4 to 7): A set D of k dummy elements whose addition into any
seed-set S does not change σN (S) and σV (S), are created. Then, a dummy
element u ∈
/ D is added into a subset N  (initially containing all non-vulnerable

nodes), until |Nk | is an integer.

Phase II (steps 8 to 18): A random sample of |Nk | elements from N  and a
dummy element is created. Next, the candidate subset SiO , SiU and SiL is
extended with a node in the sample causing the largest marginal gain with
respect to ASR, ASRL , and ASRU , respectively. The parameter of ASRL is
the seed-set Spr , constructed in the previous iteration and that of ASRU is a
random permutation π Spr of Spr .
Phase III (steps 19 to 23): The best candidate subset with respect to ASR is
selected as Scur , and all dummy elements are removed from it. If Scur is not
better than Spr in terms of ASR, the while loop in step 3 is terminated and Scur
is returned. Otherwise, another iteration is performed with the aim of generating
a seed-set with larger ASR, due to the use of different (and often better [9])
bounds.
 Limiting the Influence to Vulnerable Users in Social Networks 1115

Theorem 1. ISS constructs a seed-set S such that:

 
σV (S ∗ ) + c c 1 1 ∗
E[ASR(S, c)] ≥ max ∗
, · · (1 − ) · ASR(S , c)
σ
V,Spr (S ) + c c + k · maxu∈N σ
V,Spr ({u}) e e

where S ∗ = arg maxS⊆N ,|S|≤k ASR(S, c), σ V,Spr is the modular upper bound used
in ASRL (step 14) in the last iteration of ISS, and the expectation is over every
possible S constructed by ISS.

Proof. Let SkL,j (respectively, Sprj ) denote the subset SkL (respectively, Spr ) in
step 19 of an iteration j of ISS (j-th execution of the while loop in step 3). Since
ASRL bounds ASR from below, we have ASR(SkL,j , c) ≥ ASRL (SkL,j , c, Sprj ).
Also, from the monotonicity of expectation, this inequality can be written as

E[ASR(SkL,j , c)] ≥ E[ASRL (SkL,j , c, Sprj )], (4)

where each expectation is over every SkL,j . We now observe that

1 1
E[ASRL (SkL,j , c, Sprj )] ≥ · (1 − ) · ASRL (S ∗,j , c, Sprj ), (5)
e e
where S ∗,j = arg maxS⊆N ,|S|≤k ASRL (S, c, Sprj ). Equation 5 holds because SkL,j
is constructed based on the Sub-sample Greedy algorithm [17] with ASRL in
each iteration (execution of the while loop in step 3) of ISS . Thus, we obtain:

1 1
E[ASR(SkL,j , c)] ≥ · (1 − ) · ASRL (S ∗,j , c, Sprj )
e e  
1 1 σN (S ∗,j ) + c σN (S ∗,j ) + c σV (S ∗,j ) + c
≥ · (1 − )· · ·
e e σ V,Sprj (S
∗,j ) + c σV (S ∗,j ) + c σN (S ∗,j ) + c
σV (S ∗,j ) + c 1 1
≥ · · (1 − ) · ASR(S ∗,j , c).
∗,j ) + c e
σ
V,Sprj (S e

The first inequality holds from Eqs. 4 and 5, the second from the definition
of ASRL and because we multiply by 1 (in square brackets), and the third by
the definition of ASR. Since the third inequality holds for every iteration j of
ISS :

σV (S ∗ ) + c 1 1
E[ASR(S L , c)] ≥ · · (1 − ) · ASR(S ∗ , c),
∗) + c e
(6)
σ
V,Spr (S e
where S L is the subset SkL,j constructed in step 19 of the last iteration of ISS .
Let SkU,j (respectively, Sprj ) denote the subset SkU (respectively, Spr ) in step
19 of an iteration j of ISS (j-th execution of the while loop in step 3). We first
prove:
U,j

σV,S
 pr (Sk
j
)≤ σV,S
 pr ({u}) ≤ k ·
j
max σV,S

U,j
pr ({u}) ≤ k · max σV,S
j

u∈N
pr ({u})
j
(7)
U,j u∈S
u∈S k
k
1116 H. Chen et al.

The first inequality holds because σ

V,Sprj is submodular, the second because

|SkU,j | ≤ k and σ U,j

V,Sprj is non-negative, and the third because Sk ⊆ N . From
Eq. 7 and σV (SkU,j ) ≤ σ U,j
V,Sprj (Sk ) (as σ
V,Sprj bounds σV from above), we get:

σV (SkU,j ) + c ≤ c + k · maxu∈N σ
V,Sprj ({u}), which implies:
 
σ Spr (SkU,j ) + c σ Spr (SkU,j ) + c
V,π j V,π j c
≥ ≥ . (8)
σV (SkU,j ) + c c + k · maxu∈N σV,S
 pr ({u}) j
c + k · maxu∈N σV,S
 pr ({u}) j

because σV,πSprj (SkU,j )
The second inequality is  ≥ 0 by

definition.
σ (SkU,j )+c σ (SkU,j )+c
σN (SkU,j )+c Spr
σN (SkU,j )+c Spr
ASR(SkU,j , c) =  
j j

σV (SkU,j )+c
· σV,π (SkU,j )+c
= σ U,j · V,π
σV (SkU,j )+c
,
Spr Spr (Sk )+c
V,π j V,π j

which implies:

σN (SkU,j ) + c σV,πSprj (SkU,j ) + c
E[ASR(SkU,j , c)] = E[ · ]
σV,πSprj (SkU,j ) + c σV (SkU,j ) + c

σN (SkU,j ) + c c
≥ E[ · ]
σV,πSprj (SkU,j ) + c c + k · maxu∈N σ
V,Sprj ({u})

from Eq. 8, where the expectation is over each SkU,j . Also, λj =

c
c+k·maxu∈N σV,S
 ({u})
is constant in iteration j, so
prj

σN (SkU,j ) + c
E[ASR(SkU,j , c)] ≥ λj · E[ U,j
] = λj · E[ASRU (SkU,j , c, π Sprj )].
σV,πSprj (Sk ) + c
(9)

We now observe that

1 1
E[ASRU (SkU,j , c, π Sprj )] ≥ · (1 − ) · ASRU (S ∗,j , c, π Sprj ), (10)
e e
where S ∗,j = arg maxS⊆N ,|S|≤k ASRU (S, c, π Sprj ). Equation 10 holds because
SkU,j is constructed based on the Sub-sample Greedy algorithm [17] with ASRU
in each iteration (execution of the while loop in step 3) of ISS . Thus, we obtain:

1 1
E[ASR(SkU,j , c)] ≥ λj · · (1 − ) · ASRU (S ∗,j , c, π Sprj )
e e 
1 1  ∗,j
σN (S ) + c σ S
V,π prj
(S ∗,j ) + c
≥ λj · · (1 − ) · ·[ ]
e e σ Sprj (S ∗,j ) + c σV (S ∗,j ) + c
V,π
c 1 1
≥ · · (1 − ) · ASR(S ∗,j , c)
c + k · maxu∈N σ
V,Sprj ({u}) e e
 Limiting the Influence to Vulnerable Users in Social Networks 1117

The first inequality holds from Eqs. 9 and 10, the second because the term
in square brackets is at most 1 (since σV,πSprj bounds σV from below), and the
third from the definition of λj and ASR. Since the third inequality holds for
each iteration j of ISS , we obtain:

c 1 1
E[ASR(S U , c)] ≥ · · (1 − ) · ASR(S ∗ , c) (11)
c + k · maxu∈N σ
V,Spr ({u}) e e

where S U is the subset SkU,j constructed in step 19 of the last iteration of ISS .
The proof follows from Eqs. 6 and 11 and step 19 in ISS .

Although the guarantee in Theorem 1 holds irrespectively of the number of

iterations of ISS , we observed, in our experiments, that more iterations result
in seed-sets with larger ASR. This is because the parameter Spr and π Spr of the
bound functions ASRL and ASRU , respectively, is updated in every iteration
which often improves the bounds [9]. We also observed that ISS needed at most
4 iterations to terminate. ISS is faster than GR and scales better with respect to
k, because it selects seeds from a sample of N of size approximately |Nk | , instead
of the entire set N , and because it performs a small number of iterations.

6 Experimental Evaluation
In this section, we evaluate GR, GRM B , and ISS , in terms of effectiveness and
efficiency, by comparing them against TIM [20], a heuristic for finding a seed-set
S with size at most k and large σN (S) − σV (S), and two baselines that employ
Greedy [18]: RB , which applies Greedy [18] to the subset of non-vulnerable nodes
that do not influence vulnerable nodes, and RB , which applies Greedy with the
objective function σN . RB creates a seed-set S with σV (S) = 0 and was used to
see whether S can have large σN (S). RB creates a seed-set S with large σN (S)
and was used to see whether S can have small σV (S). RB found seed-sets that
influenced many more vulnerable nodes than those of all other methods, thus,
we omit its results.
All algorithms were
Table 2. Characteristics of datasets.
implemented in C++ and
Dataset # of # of Avg Max # of vuln. θ applied to the Wiki-vote
nodes (|V |) edges (|E|) in-degree in-degree nodes (|V|) (WI ), Twitter (TW ), and
WI 7115 103689 13.7 452 100 0.01
TW 235 2479 10.5 52 25 0.01
PolBlogs (POL) datasets
POL 1490 19090 11.9 305 100 0.003 (see Table 2). POL is avail-
AB 840 10008 11.9 137 10 0.01
able at http://www-person
al.umich.edu/∼mejn/ and all other datasets at http://snap.stanford.edu/data.
We also used synthetic datasets, generated by the Albert-Barabasi model, as in
[15], with a varying number of edges in [500, 10000]. We refer to the dataset with
10000 edges as AB. We set p(u , u) = |n−1(u)| for each edge (u , u) as in [5,15]. We
also set the maximum probability threshold for a path to θ = 0.01, so that all
methods achieve a good accuracy/efficiency trade-off by discarding paths that
1118 H. Chen et al.

have low probability to influence a node, as in [6]. The default value for k was
5 and for c was 1. The vulnerable nodes were selected randomly. To improve
the efficiency of ISS , we used the CELF optimization [10] for the submodular
bound functions (steps 14 and 16). Also, the results for ISS were averaged over
10 runs. All experiments ran on an Intel Xeon CPU E5-2640 @2.66 GHz with
64 GB RAM. Due to space limitations, we omit some results that were qualita-
tively similar to the reported ones (e.g., results for varying |V| in W I).
40

RB σV

600
RB σV RB σV RB σV

150
50 100 150 200

GR σV GR σV GR σV GR σV
Spread σV and σN

Spread σV and σN

RB σN

Spread σV and σN

Spread σV and σN
30

RB σN RB σN RB σN
GR σN

400
GR σN GR σN GR σN

100
20

200

50
10
0

0
0.01 0.1 0.5 1 5 0.01 0.1 0.5 1 5 5 10 25 50 100 5 10 20 30 40 50
c c k k
(a) POL (b) TW (c) POL (d) TW

Fig. 2. Spread of vulnerable and non-vulnerable nodes: (a) P OL vs c, (b) T W vs c,

(c) P OL vs k, and (d) T W vs k.

Comparison to RB. GR constructs seed-sets that influence at least 5.5 and

up to 38 times more non-vulnerable nodes than those constructed by RB , for
different values of c (see Figs. 2a and b) and k (see Figs. 2c and d). The reason
is that, for all c and k values, vulnerable nodes were distributed across the
graph. So, the seed-sets constructed by RB that did not influence vulnerable
nodes did not influence many non-vulnerable nodes, while those constructed
by GR influenced a small number of vulnerable nodes but could reach to and
influence many more non-vulnerable nodes. Moreover, TIM , GRM B , and ISS
outperformed RB (the results for them are omitted). Thus, in all subsequent
experiments, we omit results for RB , since it does not construct practically
useful solutions and set c = 1 because this allows constructing seed-sets with
good σN /σV trade-off.
ASR with c = 1. All our algorithms substantially outperform TIM in terms
of ASR for varying k (see Figs. 3a, b, and c) and varying number of vulnerable
nodes |V| (see Fig. 3d). ISS outperformed all other methods, being 3, 1.7, and
2 times better than TIM , GR, and GRM B on average (over all datasets and
k values), respectively. ISS was also 8.9, 3.3, and 1.9 times better than TIM ,
GR, and GRM B on average (over all |V| values in Fig. 3d), respectively. We omit
the results for GR and TIM for the largest dataset WI from all subsequent
experiments, since GR and TIM did not finish within 3 days.
Spread of Vulnerable and Non-vulnerable Nodes. We demonstrate that
all our algorithms substantially outperform TIM in terms of σN and/or σV .
First, we report Figs. 4a and b, where each point (x, y) corresponds to the values
(1 − σV|V|
(S) σN (S)
, |N | ), referred to as protection and utility of a seed-set S. ISS
 Limiting the Influence to Vulnerable Users in Social Networks 1119

10 15 20 25 30 35

150
GR ISS GR ISS GRMB GR

80
GRMB TIM GRMB TIM
ISS GRMB
30
ISS

ASR(S,1)
100
ASR(S,1)

ASR(S,1)

ASR(S,1)
60
TIM
20

40

50
20
10

0
5 10 25 50 100 5 10 20 30 40 50 5 10 25 50 100 100 200 300 500
k k k Number of vulnerable nodes
(a) POL (b) TW (c) WI (d) POL

Fig. 3. ASR with c = 1 vs k for (a) P OL, (b) T W , and (c) W I. (d) ASR with c = 1
vs |V|.

GR GR ISS

600
GRMB GR GR ISS
0.15

140
GRMB TIM
GRMB TIM
0.3

ISS GRMB

Spread non−vuln.

Spread non−vuln.
TIM ISS
σN(S) / |N|

σN(S) / |N|

400
TIM

100
0.10

0.2

200
0.1
0.05

60
0.0

20
0

0.88 0.92 0.96 1.00 0.70 0.80 0.90 5 10 25 50 100 5 10 20 30 40 50

1 − σV(S) / |V| 1 − σV(S) / |V| k k
(a) POL (b) TW (c) POL (d) TW

Fig. 4. Map for (a) P OL and k = 5, (b) T W and k = 10. Spread of non-vulnerable
nodes vs k for (c) P OL, and (d) T W .
7
GR ISS GR ISS GRMB
14

GRMB ISS
40

GRMB TIM
6
250

GRMB TIM ISS

Spread non−vuln.

2 3 4 5
Spread vuln

Spread vuln

Spread vuln
30

6 8 10
150

20 10

4
50

1
2
0

0
0

5 10 25 50 100 5 10 25 50 100 5 10 20 30 40 50 5 10 25 50 100

k k k k
(a) WI (b) POL (c) TW (d) WI

Fig. 5. Spread of non-vulnerable nodes vs k for (a) T W , and (b) W I. Spread of vul-
nerable nodes vs k for (c) P OL. Spread of vulnerable nodes for (d) T W , and (e) W I.

outperformed TIM with respect to both protection and utility, achieving overall
better protection than GR and better utility than GRM B . We also report σN and
σV in Figs. 4c to 5d. GR and TIM constructed seed-sets that influence too many
vulnerable nodes. GRM B performed inconsistently (e.g., its seed-sets influenced
few vulnerable nodes in Fig. 5b and too many vulnerable nodes in Fig. 5c). ISS
influenced few vulnerable nodes and a moderate number of non-vulnerable nodes,
achieving a good σN /σV trade-off.
Efficiency. All our methods are much faster than TIM for varying k (see Figs. 6a
and b). TIM required 10 h when k = 50 in the case of T W which only has 235
nodes, and 17 days when k = 25 in the case of P OL. GR was faster but did not
terminate within 3 days in the case of W I, and GRM B was the fastest due to
1120 H. Chen et al.

106
GR ISS GR ISS GR ISS
GRMB GR ISS GRMB
101 102 103 104 105 106 107
TIM TIM GRMB TIM

100 101 102 103 104 105

GRMB TIM

100 101 102 103 104 105

104
Runtime (s)
Runtime (s)

Runtime (s)

Runtime (s)

102
100
5 10 25 50 100 5 10 20 30 40 50 100 200 300 500 103 5 ⋅ 103 104
k k Number of vulnerable nodes Number of edges
(a) POL (b) TW (c) POL (d) AB

Fig. 6. Runtime vs k for (a) P OL, and (b) T W . Runtime vs (c) number of vulnerable
nodes for P OL, and (d) number of edges for AB.

its efficient spread estimation function [24]. ISS was significantly faster than GR
and TIM and the most scalable method with respect to k. Figure 6c shows the
runtime for varying |V|. All our algorithms become faster with |V|, since fewer
nodes can be selected as seeds and are at least three orders of magnitude faster
than TIM on average. Figure 6d shows the runtime for varying number of edges.
Our algorithms were faster than TIM by up to three orders of magnitude.

7 Related Work

No existing work addresses influence maximization when there are vulnerable

nodes. The most related works are [20] and [15]. [20] aims to maximize the
difference between the expected number of influenced users who belong to a
target group and the expected number of all other influenced users. Our work
differs from [20] along three dimensions. First, [20] can select target nodes as
seeds, but we cannot do the same for vulnerable nodes, as this would harm them.
Second, our ASR measure has desired properties unlike the measure σN (S) −
σV (S) in [20] (see Sect. 3). Third, our methods are substantially more effective
and efficient than the heuristic in [20]. [15] is applied after influence maximization
(i.e., considers a given seed-set) and seeks to delete edges in order to limit the
activation probability of vulnerable nodes in the Linear Threshold model [10].
Thus, it is orthogonal to our work.
There are many works on targeted viral marketing (e.g., [11,13,20,22,25]).
For example, [13] considered influence maximization when each target node has
a constant profit, and [22] considered the impact of the location and login time
of target nodes. Unlike ours, the works in [11,13,20,22,25] do not consider vul-
nerable nodes.
There are also works on influence maximization considering nodes with neg-
ative impact on the influence diffusion process [2,5]. [5] studied influence max-
imization under a model where each node can diffuse information of opposite
content to the information that is being spread from the seed-set. [2] studied
influence maximization, when some nodes reject the diffused information. Differ-
ent from these works, no node negatively impacts the influence diffusion process
in our approach.
 Limiting the Influence to Vulnerable Users in Social Networks 1121

8 Conclusion
In this paper, we study influence maximization when there are vulnerable nodes.
We first propose a measure for limiting the influence to vulnerable nodes, which
is obtained by applying additive smoothing to the ratio between the expected
number of influenced non-vulnerable nodes and the expected number of influ-
enced vulnerable nodes. Based on the measure, we define a new influence maxi-
mization problem that seeks to find a seed-set of size at most k that maximizes
the measure. We propose two greedy baseline heuristics, and the ISS approxi-
mation algorithm to solve our influence maximization problem. We evaluate our
methods on synthetic and real-world datasets and show that ISS outperforms
the method of [20] and our baselines in terms of effectiveness and efficiency.

References
1. http://www.conecomm.com/research-blog/2017-csr-study
2. Abebe, R., Adamic, L., Kleinberg, J.: Mitigating overexposure in viral marketing
(2018)
3. Bai, W., Iyer, R., Wei, K., Bilmes, J.: Algorithms for optimizing the ratio of sub-
modular functions. In: ICML, pp. 2751–2759 (2016)
4. Buchbinder, N., Feldman, M., Naor, J., Schwartz, R.: Submodular maximization
with cardinality constraints. In: SODA, pp. 1433–1452 (2014)
5. Chen, W., et al.: Influence maximization in social networks when negative opinions
may emerge and propagate. In: SDM, pp. 379–390 (2011)
6. Goyal, A., Lu, W., Lakshmanan, L.V.S.: SIMPATH: an efficient algorithm for
influence maximization under the linear threshold model. In: ICDM, pp. 211–220
(2011)
7. Gupta, S.: A conceptual framework that identifies antecedents and consequences
of building socially responsible international brands. Thunderbird Int. Bus. Rev.
58(3), 225–237 (2016)
8. Gwadera, R., Loukides, G.: Cost-effective viral marketing in the latency aware
independent cascade model. In: PAKDD, pp. 251–265 (2017)
9. Iyer, R., Bilmes, J.: Algorithms for approximate minimization of the difference
between submodular functions, with applications. In: UAI, pp. 407–417 (2012)
10. Kempe, D., Kleinberg, J., Tardos, E.: Maximizing the spread of influence through
a social network. In: KDD, pp. 137–146 (2003)
11. Khan, A., Zehnder, B., Kossmann, D.: Revenue maximization by viral marketing:
a social network host’s perspective. In: ICDE, pp. 37–48 (2016)
12. Krause, A., Golovin, D.: Submodular function maximization. In: Tractability
(2013)
13. Li, F., Li, C., Shan, M.: Labeled influence maximization in social networks for
target marketing. In: PASSAT/SocialCom 2011, pp. 560–563 (2011)
14. Li, Y., Fan, J., Wang, Y., Tan, K.: Influence maximization on social graphs: a
survey. TKDE 30(10), 1852–1872 (2018)
15. Loukides, G., Gwadera, R.: Preventing the diffusion of information to vulnerable
users while preserving pagerank. Int. J. Data Sci. Anal. 5(1), 19–39 (2018)
16. Manning, C., Raghavan, P., Schütze, M.: Introduction to Information Retrieval
(2008)
1122 H. Chen et al.

17. Mitrovic, M., Bun, M., Krause, A., Karbasi, A.: Differentially private submodular
maximization: data summarization in disguise. In: ICML, pp. 2478–2487 (2017)
18. Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for
maximizing submodular set functions. Math. Program. 14(1), 265–294 (1978)
19. Nielsen: Sustainable selections: how socially responsible companies are turning a
profit. https://bit.ly/2DW99pE
20. Pasumarthi, R., Narayanam, R., Ravindran, B.: Near optimal strategies for tar-
geted marketing in social networks. In: AAMAS, pp. 1679–1680 (2015)
21. Shaw, G., Karami, A.: Computational content analysis of negative tweets for obe-
sity, diet, diabetes, and exercise. Proc. Assoc. Inf. Sci. Technol. 54(1), 357–365
(2017)
22. Song, C., Hsu, W., Lee, M.L.: Targeted influence maximization in social networks.
In: CIKM, pp. 1683–1692 (2016)
23. Svitkina, Z., Fleischer, L.: Submodular approximation: sampling-based algorithms
and lower bounds. SIAM J. Comput. 40(6), 1715–1737 (2011)
24. Wang, C., Chen, W., Wang, Y.: Scalable influence maximization for independent
cascade model in large-scale social networks. DMKD 25(3), 545–576 (2012)
25. Wen, Y.T., Peng, W., Shuai, H.: Maximizing social influence on target users. In:
PAKDD, pp. 701–712 (2018)
 IoT Based Wide Area Road Surface State
Sensing and Communication System
for Future Safety Driving

Yoshitaka Shibata1(&), Akira Sakuraba1, Goshi Sato2,

and Noriki Uchida3
1
Iwate Prefectural University, Takizawa, Japan
{shibata,a_sakuraba}@iwate-pu.ac.jp
2
Resilient ICT Research Center, Sendai, Japan
[email protected]
3
Fukuoka Institute Technology, Fukuoka, Japan
[email protected]

Abstract. In order to realize future safe automotive driving environment, road

state condition technology is very important. In this paper, a new generation
wide area road surface state information platform based on IoT based sensing
and V2X technologies is introduced. In IoT sensing, various environmental
sensors camera and GPS are integrated on vehicles to precisely detect the var-
ious road surface states and determine the dangerous locations on GIS. Those
road information are transmitted the neighbor vehicles and road side server in
realtime using V2X communication network. Driver can receive the road sur-
face status information from the vehicle in opposite direction or road side server
and eventually pay attentions to his/her driving before encountering the danger
location. This technology can also lead to automotive driving by automatically
controlling the car speed and direction.

1 Introduction

In our social life, mobility is the most important means for economic activity to safely
and reliably carry persons, loads, feeds and other materials around world. Huge number
of vehicles are being produced day by day and their qualities are developed year and
year in the developed countries. Furthermore, in recent, self-driving cars have been
emerged and running on highways and public roads in well developed countries.
However, on the other hand, the road infrastructures in many developing countries
are not well maintained compared with improvement of vehicles. In particular, as
shown in Fig. 1, the road conditions in developing countries are so bad and dangerous
due to luck of regular road maintenance, falling objects from other vehicles, overloaded
trucks or buses. Therefore, in order to maintain safe and reliable auto driving, the
vehicles have to detect those obstacles in advance and avoid when they pass through.

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1123–1132, 2020.
https://doi.org/10.1007/978-3-030-15032-7_94
1124 Y. Shibata et al.

Fig. 1. Road condition in local roads and heavy rain

On the other, in the cold or snow countries, such as Japan and Northern countries
[1], most of the road surfaces are occupied with heavy snow and iced surface in winter
and many slip accidents occurred even though the vehicles attach snow specific tires. In
fact almost more 90% of traffic accidents in northern part of Japan is caused from
slipping car on snowy or iced road and whiteout as seen in Fig. 2 [2].

Fig. 2. Road condition in snow country

In both cases, traffic accidents are rapidly increased. Therefore, safer and more
reliable road monitoring and warning system which can transmit the road condition
information to drivers and new self-driving method before passing through the dan-
gerous road area is indispensable. Thus, once a traffic accidents or disaster occurred,
information collection, transmission and sharing are delayed or even cannot be made.
Eventually the resident’s lives and reliabilities cannot be maintained. More robust and
resilient information infrastructure and proper and quick information services with road
environmental conditions are indispensable.
In order to resolve those problems, we introduce a new generation wide area road
surface state information platform based on crowd sensing and V2X technologies. In
road sensing, many data from many vehicles with various environmental sensors
including accelerator, gyro sensor, infrared temperature sensor, quasi electrical static
sensor, camera and GPS are integrated to precisely detect the various road surface
states and determine the dangerous locations on GIS. Those road information are
transmitted to the neighbor vehicles and road side server in realtime using V2X
communication network.
In V2X communication on the actual road, both the communication distance and
the total size of data transmission must be maximized at the same time when vehicles
are running on the road. The 5G networks are expected as V2X communication
infrastructure. However, it is far away to realize 5G service in a country.
 IoT Based Wide Area Road Surface State Sensing 1125

On the other hand, in recent, the next generation wireless networks such as
IEEE802.11ac/ad/ah/ai/ay and LPWA technologies are getting popular. However sin-
gle network of those is not enough to realize highspeed and long distance communi-
cation at the same time. In order to resolve such problems, N-wavelength cognitive
wireless communication method is newly introduced in our research.

2 Road State Information Platform

We introduce a new generation wide area road surface state information platform based
on crowd sensing and V2X technologies as shown in Fig. 3 [3]. The wide area road
surface state information platform mainly consists of multiple road side wireless nodes,
namely Smart Relay Shelters (SRS), Gateways, and mobile nodes, namely Smart
Mobile Box (SMB). Each SRS or SMB is furthermore organized by a sensor infor-
mation part and communication network part.
The sensor information part includes various sensor devices such as semi-
electrostatic field sensor, an acceleration sensor, gyro sensor, temperature sensor,
humidity sensor, infrared sensor and sensor server. Using those sensor devices, various
road surface states such as dry, rough, wet, snowy and icy roads can be quantitatively
decided.
On the other hand, the communication network part integrates multiple wireless
network devices with different N-wavelength (different frequency bands) wireless
networks such as IEEE802.11n (2.4 GHz), IEEE802.11ac (5.6 GHz), IEEE802.11ah
(920 MHz) and organizes a cognitive wireless node. The network node selects the best
link of cognitive wireless network depending on the observed network quality by
Software Defined Network (SDN). If none of link connection is existed, those sensing
data are locally and temporally stored until approaches to another mobile node or road
side node, and starts to transmit sensor data by DTN Protocol. Thus, data communi-
cation can be attained even though the network infrastructure is not existed in chal-
lenged network environment such as mountain areas or just after large scale disaster
areas.
In our system, SRS and SMB organize a large scale information infrastructure
without conventional wired network such as Internet. The SMB on the car collects
various sensor data including acceleration, temperature, humidity and frozen sensor
data as well as GPS data and carries and exchanges to other smart node as message
ferry while moving from one end to another along the roads.
On the other hand, SRS not only collects and stores sensor data from its own
sensors in its database server but exchanges the sensor data from SMB in vehicle nodes
when it passes through the SRS in road side wireless node by V2X communication
protocol. Therefore, both sensor data at SRS and SMB are periodically uploaded to
cloud system through the Gateway and synchronized. Thus, SMB performs as mobile
communication means even through the communication infrastructure is challenged
environment or not prepared.
This network not only performs various road sensor data collection and transmis-
sion functions, but also performs Internet access network function to transmit the
various data, such as sightseeing information, disaster prevention information and
1126 Y. Shibata et al.

Fig. 3. Road surface state information platform

shopping and so on as ordinal public wide area network for residents. Therefore, many
applications and services can be realized.

3 Crowed Sensing System

In order to detect the precise road surface conditions, such as dry, wet, dumpy, showy,
frozen roads, various sensing devices including accelerator, gyro sensor, infrared
temperature sensor, humidity sensor, quasi electrical static sensor, camera and GPS are
integrated to precisely and quantitatively detect the various road surface states and
determine the dangerous locations on GIS in sensor server [4]. The sensor server
periodically samples those sensor signals and performs AD conversion and signal
filtering in Receiver module, analyzes the sensor data in Analyzer module to quanti-
tatively determine the road surface state as shown in Fig. 4 and learning from the
sensor data in AI module to classify the road surface state as shown in Fig. 5. As result,
the correct road surface state can be quantitatively and qualitatively decided. The
decision data with road surface condition in SMB are temporally stored in Regional
Road Condition Data module and mutually exchanged when the SMB on one vehicle
approaches to other SMB. Thus the both SMBs can mutually obtain the most recent
road surface state data with just forward road. By the same way, the SMB can also
mutually exchange and obtain the forward road surface data from road side SRS.

Fig. 4. Analyzer module Fig. 5. AI module

 IoT Based Wide Area Road Surface State Sensing 1127

4 V2X Communication System

In Fig. 6 shows V2X communication method between the SMB of a vehicle and the
SRS of road side server. First, one of the wireless networks with the longest com-
munication distance can first make connection link between SMB and SRS using SDN
function. Through the this connection link, the communication control data of other
wireless networks such as UUID, security key, password, authentication, IP address,
TCP port number, socket No. are exchanged [5–7]. As approaching each other, the
second wireless network among the cognitive network can be connected in a short time
and actual data transmission can be immediately started. This transmission process can
be repeated during crossing each other as long as the longest communication link is
connected. This communication process between SMB and SRS is the same as the
communication between SMB to other SMB except for using adhoc mode.
On the other hand, in the V2x communication between vehicle and the global cloud
server on Internet, the sensor data from SMB is transmitted to the SRS in road side
server. Then those data are sent to the gateway function unit and the address of those
data from local to global address and sent to the global cloud server through Internet.
Thus, using the proposed V2X communication protocol, not only Intranet communi-
cation among the vehicle network, but also Intranet and Internet communication can be
realized [8–10].

Fig. 6. V2X communication method between the SMBS and SRS

5 Prototype System and Evaluation

In order to verify the effects and usefulness of the proposed system, a prototype system
is constructed and those functional and performance are evaluated. Figure 7 and
consists of sensor server system and Smart Mobile Box or Smart Relay Shelter. Fur-
thermore, sensor server system is organized by 10 axis mechanical sensor, infrared
temperature sensor, humid sensor, quasi electric sensor. Those analog sensor signals
1128 Y. Shibata et al.

are converted to digital data by A/D convertors and processed for noise reduction and
data synchronization by sensor server. Those data are sent to the sensor decision server
to process various realtime spectral, correlation and machine leaning.

Fig. 7. Prototype system

On the other hand, the prototype system for two-wavelength communication as

SMB or SRS is shown in Fig. 8. Currently WI-U2-300D of Buffalo Corporation for
Wi-Fi communication of 2.4 GHz as the prototype of two-wavelength communication,
and OiNET-923 of Oi Electric Co., Ltd. for 920 MHz band communication respec-
tively. WI-U2-300D is a commercially available device, and the rated bandwidth in this
prototype setting is 54 Mbps. On the other hand, the OiNET-923 has a communication
distance of 1 km at maximum and a bandwidth of 50 kbps to 100 kbps.

Fig. 8. Crowed sensing system

(a) Evaluation of 10 Axis Mechanical Sensor Data

First the roughness of road surface sate is evaluated by 10 axis mechanic sensor is
evaluated. The 10 axis mechanic sensor is fixed to the driving car with 40 km/h and
sampled xyz acceleration, gyroscope, magnetic field data in xyz axises with 10 ms.
sample rate. In derive the rough vertical deviation, only y axis acceleration data are
 IoT Based Wide Area Road Surface State Sensing 1129

Fig. 9. XYZ acceleration data Fig. 10. Vertical acceleration data

Fig. 12. Compensated vertical acceleration

data
Fig. 11. Spectral of vertical acceleration

extracted. Then those extracted data are compensated to reduce the effect of the satent
of sensor. After that, those compensated data are processed by FFT. Finally the real
roughness of the road surface can be derived in realtime (Figs. 9, 10, 11 and 12).
(b) Evaluation of Quasi Electric Sensor
Next, the dry and wet, snowy road conditions can be evaluated using quasi electric
sensor. The quasi electric sensor were sampled for three different road surface condi-
tions as the same manner in previous experiment and converted to frequency spectra
density as shown in Figs. 13, 14, 15 and 16. From the analysis of the results, the dry
and wet, snowy road conditions can be identified based on the relative spectral density.

Fig. 13. Spectral for dry road Fig. 14. Spectral density for wet road
1130 Y. Shibata et al.

Fig. 15. Spectral for snowy road Fig. 16. Amplitude for three states

(c) Evaluation of V2X Communication

Next performance evaluation of V2X communication is performed as shown in
Fig. 17. Two networks transmission cases including single Wi-Fi network communi-
cation and a combined communication with Wi-Fi and Sub-Giga band with 920 MHz
considered to compare their performance. It is known that Wi-Fi transmission provides
higher transmission data rate such as 54 Mbps for IEEE802.11n, but it communication
distance between the vehicle and the roadside server is limited only up to 60 m at most.
On the other hand, Sub-G band provides longer communication distance with more
than 1–2 km, but the transmission data rate is limited only 100 Kbps for LORA.
Therefore, in order to provide both long distance communication and higher total
transmission rate while driving the vehicle, both Wi-Fi is combined. The combined
network, the UUIDs of both servers, SSID and its key, Socket ID for TCP and metadata
of sensor DB are priori exchanged through the Sub-Gia band. When the vehicle
approaches enough to the road side server to communicate by Wi-Fi network, the larger
data can be immediately exchanged in a short time. Eventually a larger total data
transmission can be attained.

Fig. 17. V2X Field experimentation Fig. 18. Total Amount of data transmission

Figure 18 shows the total transmission data for nine times of filed experiments.
From this experimental result, the combination network provided very stable total data
transmission of 5.0 MB while Wi-Fi network provide unstable total data transmission
of average 3.0 MB.. Thus, the combination network can attain 50% better performance
 IoT Based Wide Area Road Surface State Sensing 1131

in data transmission. In this time, only two different wireless networks were combined
into a cognitive wireless network in this experiment. Therefore, by combining with n
(n > 2) numbers of wireless networks with different wavelength, better performance
such as, larger total transmission data with longer communication distance can be
expected.

6 Conclusions and Remarks

In order to recover the problems of poor road conditions and challenged information
communication infrastructure in aging society, IoT based mobility information network
is proposed. Road side wireless nodes and mobile nodes with various sensors and
different wireless communication devices are introduced to organize a large scale
information infrastructure without conventional wired network and with very economic
and low priced mobility network environment can be realized by V2V and V2X
communication protocols. The system concept and architecture are precisely explained.
The expected application and services for residents are also discussed. As future works,
we will design and build a prototype system of our proposed mobility Information
infrastructure in rural road between a city and a town in coast areas. We will design and
implement Realtime Road Surface Data Exchange System, Dig Data Road Condition
Sharing system and Tourist and Disaster Information to evaluate their functions and
performance.

Acknowledgement. The research was supported by Strategic Information and Communications

R&D Promotion Program Grant Number 181502003 by Ministry of Affairs and Communication.

References
1. A 2016 Aging Society White Paper. http://www8.cao.go.jp/kourei/whitepaper/w-014/zenbun/
s1_1_1.html
2. A 2016 Decling Birthrate White Paper. http://www8.cao.go.jp/shoushi/shoushika/
whitepaper/measures/english/w-2016/index.html
3. Shibata, Y., Sato, G., Uchida, N.: A new generation wide area road surface state information
platform based on crowd sensing and V2X technologies. In: The 21th International
Conference on Network-Based Information Systems (NBiS2018). Lecture Notes on Data
Engineering and Communications Technologies book series (LNDECT), vol. 22, pp. 300–
308 (2018)
4. Ito, T., Sakuraba, A., Shibata, Y.: Development of decision algorithm for road surface
conditions by crowd sensing technology. In: The 13th International Conference on Advances
on Broadband and Wireless Computing, Communication and Applications. Lecture Notes on
Data Engineering and Communications Technologies book series (LNDECT), vol. 25,
pp. 361–366 (2018)
5. Ito, K., Hirakawa, G., Shibata, Y.: Experimentation of V2X communication in real
environment for road alert information sharing system. In: IEEE AINA2015, pp. 711–716,
March 2015
1132 Y. Shibata et al.

6. Ito, K., Hirakawa, G., Shibata, Y.: Estimation of communication range using Wi-Fi for V2X
communication environment. In: The 10th International Conference on Complex, Intelligent,
and Software Intensive Systems, (CISIS2016), pp. 278–283. Fukuoka Institute of
Technology, July 2016
7. Ito, K., Hashimoto, K., Shibata, Y.: V2X communication system for sharing road alert
information using cognitive network. In: The 8th International Conference on Awareness
Science and Technology (iCAST 2017) (2017)
8. Uchida, N., Ito, K., Hirakawa, G., Shibata, Y.: Evaluations of wireless V2X communication
systems of for winter road surveillance systems. In: The 19th International Conference on
Network-Based Information Systems (NBiS 2016), pp. 58–63. Technical University of
Ostrava, September 2016
9. Shibata, Y., Sato, G., Uchida, N.: IoT based mobility network information infrastructure in
challenged network environment toward aging society. In: The 9th International Workshop
on Disaster and Emergency Information Network Systems, (IWDENS 2017). Tamkang
University, Taiwan, March 2017
10. Uchida, N., Takeuchi, S., Ishida, T., Shibata, Y.: Mobile traffic accident prevention system
based on chronological changes of wireless signals and sensors. J. Wirel. Mob. Netw.
Ubiquit. Comput. Dependable Appl. (JoWUA) 8(3), 57–66 (2017)
 Expressing Trust with Temporal
Frequency of User Interaction
in Online Communities

Ekaterina Yashkina1 , Arseny Pinigin1 , JooYoung Lee1 , Manuel Mazzara1 ,

Akinlolu Solomon Adekotujo2 , Adam Zubair2 , and Luca Longo3(B)
1
Innopolis University, Innopolis, Russia
{e.yashkina,a.pinigin,j.lee,m.mazzara}@innopolis.ru
2
Lagos State University, Ojo, Nigeria
[email protected], [email protected]
3
School of Computer Science, Technological University Dublin,
Dublin, Republic of Ireland
[email protected]

Abstract. Reputation systems concern soft security dynamics in diverse

areas. Trust dynamics in a reputation system should be stable and adapt-
able at the same time to serve the purpose. Many reputation mechanisms
have been proposed and tested over time. However, the main drawback
of reputation management is that users need to share private informa-
tion to gain trust in a system such as phone numbers, reviews, and
ratings. Recently, a novel model that tries to overcome this issue was
presented: the Dynamic Interaction-based Reputation Model (DIBRM).
This approach to trust considers only implicit information automatically
deduced from the interactions of users within an online community. In
this primary research study, the Reddit and MathOverflow online social
communities have been selected for testing DIBRM. Results show how
this novel approach to trust can mimic behaviors of the selected repu-
tation systems, namely Reddit and MathOverflow, only with temporal
information.

Keywords: Reputation management · Computational trust ·

Online communities · Social media · Social networks

1 Introduction

Numerous approaches haven been proposed to capture trust level of users in

a system [6,7,9,18,19] and their propagation [22]. Among others, reputation
mechanisms are widely adopted by online social media to enable secure and
effective interactions among users [2,20,27,30]. Trust dynamics in a reputation
system should be stable and adaptable at the same time to serve the purpose
[1,17]. Typically, reputation mechanisms consider factors such as feedback on
previous interaction, frequencies of interactions, the presence of a user in an
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1133–1146, 2020.
https://doi.org/10.1007/978-3-030-15032-7_95
1134 E. Yashkina et al.

underlying system or community [25,28]. However, the main drawback of rep-

utation management is that users need to share private information to gain
trust in a system such as phone numbers, reviews, ratings, and other pieces
of information. This might impact the system itself because some users are
not willing to share their sensitive information [8]. A recent work proposed a
new reputation mechanism that only considers the frequency of users interac-
tions in order to compute their reputation value [21]. This approach, named
the Dynamic Interaction-Based Reputation Model (DIBRM), was preliminary
tested with data from the StackOverflow online community. Findings suggested
that by only considering information related to the frequency of the interactions
of users, DIBRM could closely mimic the original reputation scores proposed by
StackOverflow. Therefore, DIBRM can have a competitive advantage over other
reputation mechanisms since it is fully automated without require intervention
and explicit information from users [14].
The goal of this research is to further test and validate the DIBRM mecha-
nism when applied to other social networks, namely Reddit and MathOverflow.
The remained of the paper is organized as follows. Section 2 introduces related
work on reputation and trust emphasizing some limitations. Section 3 builds on
such limitations and describe a novel primary research experiment employing
the aforementioned DIBRM mechanism. Results are presented in Sect. 5 and
discussed in Sect. 5.1. Eventually, Sect. 6 summarizes the contribution to the
body of knowledge and recommend future research.

2 Related Work

Reputation systems are essential in many application domains such as online

communities [10]. They exist as a form of soft security measure to overcome the
issues generated by those individuals who legitimately cheat the system. Dif-
ferent types of reputation mechanisms have been introduced in the last years
[23]. These mechanisms usually sees an online community as a directed graph
whereby nodes either represent a user or a piece of information, and an arrow
models the interaction between two nodes [4]. From a graph, topological infor-
mation can be extracted such as the position of each node or the interactions
between nodes [12]. Here, a reputation model, called ReMSA (Reputation Man-
agement for Social Agents) is built upon users’ feedback and voting as well as
the time decaying necessary to update the reputation of users every time a new
interaction is added to the graph. The voting mechanism is recursive and a
node can collect pieces of feedback, about the target node, from remote nodes
that are not directly connected to it. Brown et al. [3] recently proposed a rep-
utation system for Non-Player Characters (NPC) in online games. The authors
focused on how the reputation of users can be realistically propagated by consid-
ering for instance, their communication, relationships and physical constraints.
It is interesting to see that the resulting communities, based on interactions of
players, also match the environments (cities or communities) in online games.
Although many reputation mechanisms are means for controlling systems and
 Expressing Trust with Temporal Frequency 1135

therefore centralized, some distributed reputation algorithm exists as well. For

example, a decentralized reputation mechanism was presented in [29]. Here, each
user in a community is involved in the subjective assessment of the reputation
of other users. This is useful when there is no implicit reputation scheme as it
always occurs in peer-to-peer systems. However, this mechanism is explicit and
its scalability represents the main drawback. In many social networks or other
online communities containing information in the form of news articles, technical
contents or comments from users, an explicit or implicit mechanism for rating
contents or users does not exist [14]. To overcome this limitation, a body of lit-
erature is devoted to assign reputation values to entities (users of information)
automatically [5,13,16]. For example, the algorithm presented in [11] can auto-
matically extract the reputation of a user from published textual content using
natural language analysis.
The work by Longo et al. [15] tested the hypothesis that temporal-based
factors, such as the activity, frequency, regularity of interactions of an entity
and its presence in a community, can be used as evidence of its trustworthiness.
This hypothesis was tested considering Wikipedia and its content (12000 users,
94000 articles). The authors successfully demonstrated how their algorithm could
identify top Wikipedia contributors similarly to the explicit ‘barnstars’ mecha-
nism1 employed by Wikipedia. The main drawback of using this approach is the
computational time required to quantify each temporal factors for each users
which exponentially increases in the number of users and their interactions.
To overcome this limitation, a simpler mechanism was introduced by Melinkov
et al. [21]. Here, the frequency of interactions among users is uniquely considered
to compute their reputation. This approach was tested by employing information
collected from StackOverflow2 . Findings show how this mechanism can closely
approximate the reputation scores inferred by the StackOverflow mechanism.

3 Design and Methodology

A primary research study has been conducted to investigate the capability of
the Dynamic Interaction-based Reputation Model (DIBRM) [21] for assessing
the reputation of users in online communities. In details, two social online com-
munities, namely Reddit3 and MathOverflow4 , have been selected for such an
investigation. The remainder of this section is devoted to the formal description
of the DIBRM model, its application to the two selected online platforms and
the strategy employed for its evaluation.

3.1 The Dynamic Interaction-Based Reputation Model

Interactions In in DIBRM are modeled as:
In = Ibn + Icn
1
https://en.wikipedia.org/wiki/Wikipedia:Barnstars.
2
https://stackoverflow.com/.
3
https://www.reddit.com/.
4
https://mathoverflow.net/.
1136 E. Yashkina et al.

where n ∈ 0 . . . N is the index of the interaction and N is the total number of

interactions of a user. In contains a time stamp indicating that moment in time
when an interaction took place and a value that describes the contribution to
the reputation. These values can be enumerated by the associated time stamp
to form a chain of historical user’s activity. Interactions have different effects
on the trust of a user. Each interaction has a basic value Ibn . For example,
in MathOverflow, an interaction could be ‘asking’, ‘answering’, and ‘editing’.
Each interaction has a fixed contribution to the user’s reputation value and we
consider it as a basic value. Depending on the state of communication between a
user and the system characterized by activity and frequency, an interaction can
be perceived differently. Icn captures the cumulative part of an interaction. It is
defined as:
1
Icn = Ibn ∗ α ∗ (1 − )
An + 1
where α is the weight of the cumulative part. This indicates how much Icn can
grow (if α = 1 then Icn ∈ 0 . . . Ibn ). An is the number of activities for a user.
Online social communities have different contexts and features that can affect
the properties of the trustworthiness of its entities. One of these properties is the
frequency of the communication of an user which is defined as the period of time
between the last two In activities performed. DIBRM models this property as
ta . As an example, ta for Wikipedia, on one hand, can be set to one week, time
period within which a user can create or edit some article. On the other hand,
for StackOverflow, it can be set to one day, indicating the interval of time within
which a user can answer to a certain online post (question in this context). The
number of periods between the 2 last interactions (tn and tn−1 ) of a user can be
formalised as:  
tn − tn−1
Δn =
ta
If the difference between tn and tn−1 is less than ta , then the amount of activity
periods will increase by one. It means that a user interacts frequently within the
online community. Eventually, the final degree of trust for a certain user can be
formalized as:
Tn = Tn−1 ∗ β Δn + In , β ∈ [0, 1]
where β is the forgetting factor, the rate reputation decays, that is chosen by
each system individually. The more β approximates to 1, then more the trust
value of a user decreases. Another parameter considered by DIBRM is the sum of
the previous trust values computed for a given user. We refer to this parameter
as historical reputation. In the evaluation section, we use this parameter to
compare DIBRM against other reputation models because this value accumulates
historical information about a users reputation. In other words, if a user currently
has a low reputation but was very active in the past, having interacted within
the community multiple times, its historical reputation can still be high.
 Expressing Trust with Temporal Frequency 1137

3.2 Reddit and MathOverflow

Reddit is an American social news aggregator, web-content rating and discussion
platform [26]. Registered users can submit entries to it as texts, images and links.
These entries can be up or down voted by other users. Since the official reputation
algorithm employed by Reddit is not publicly available, although speculations
exist [24], its detailed description cannot be provided. However, what is known
is that the time of submission of an entry, in the Reddit online platform, is an
important element for rankings posts. In fact, those posts with a similar number
of ‘upvotes’ and ‘downvotes’ are ranked lower compared to other posts with
‘upvotes’ only. Reddit also sorts comments in a different manner than posts, and
it does not consider temporal information, displaying the most popular first [24].
Additionally, two other types of information are computed: ‘post karma’ and
‘comment karma’ which is how much a user has contributed to the community.
The score assigned to a particular post depends mostly on the date of submission.
Newer publications have higher scores when compared to older publications even
if the amount of votes is the same.
MathOverflow is a question and answer online platform for mathematicians.
It allows its users to ask questions, submit answers, rate them and collect points
for their activities. A user who asked a question can only accept one answer to
be rewarded with points even if there are several correct answers. However, users
can ‘upvote’ multiple answers. According to MathOverflow, a user’s reputation
depends on three activities: asking, answering and editing. Restrictions exist to
avoid malicious behaviors. Accepting own answer does not increase own reputa-
tion. Deleted posts do not affect reputation and if a user reverses a vote, so the
corresponding reputation loss/gain.

4 Evaluation
To measure the efficiency of DIBRM, a comparative analysis of its inferences, in
form of a ranking of its users by their trustworthiness, is compared against the
rankings of users produced by the Reddit and MathOverflow models. In detail,
an evaluation metric is defined:
N  D 
1  1 
μD = 1 − 2 ∗ ∗ |RReij − RDij |
N i=1
D j=1

where N is the number of users, D the number of days between first and last
dates, RReij is the Reddit reputation value of user i on day j (analogously use
RM aij for MathOverflow) and RDij is the DIBRM reputation value of user i
on day j. |RReij − RDij | is the absolute difference between rating places of
individual i on particular day j. This value shows how close DIBRM rating is
to Reddit
D and MathOverflow. Then the average difference of ratings for user
i D1
∗ j=1 |RReij − RDij | (also for RM aij ) in all-days period is calculated. It
allows to avoid focusing on one estimation and analyse the general behavior of
the model. After that, the average difference of all users is estimated. The last
1138 E. Yashkina et al.

step focuses on subtracting the average difference from 1, which is then divided
by the number of rating places N . This result gives information about how much
the DIBRM rating model is close to Reddit and MathOverflow.
A second approach is to measure the ratings of users via an historical rep-
utation value. The formula remains the same but instead of RDij (reputation
rating place of user i on day j) RHij (historical reputation rating) is used.

N D
1  1 
μH =1− 2 ∗ ( ∗ |RReij − RHij |)
N i=1
D j=1

Moreover, an error metric should be estimated to have a clear picture of DIBRM.

This is performed by calculating the standard deviation of the metric μ. For
reputation, it is σD , for historical reputation it is σH .

4.1 Analysis of Data

Each system defines its own user reputation metrics. As previously mentioned,
the procedure used by Reddit for computing user reputation is not public, there-
fore it is not possible to fully assess the similarities with the DIBRM model.
MathOverflow system works similarly to StackOverflow. To evaluate DIBRM,
‘votes’, ‘posts’, ‘comments’ and users are all necessary pieces of information.
The total number of users collected from MathOverflow over a period of one
year is 4793. Reddit data was collected for a period of three months only. These
amounts are deemed sufficient for comparative purposes. Data was converted
into a csv-format for subsequent processing and tuples of the following form
have been generated:

• posts: <PostId, UserId, CreationDate, Vote>

• comments: <CommentId, UserId, CreationDate, PostId, Vote, ParentId>.

In details:

1. PostId, CommentId: identifier of an entity (integer)

2. UserId: identifier of the author of an entity (integer)
3. CreationDate: date of creation of an entity (date)
4. Vote: amount of entity’s approvals (integer)
5. ParentId: entity’s identifier to which a comment was left to (integer).

For the Reddit data, information on ‘posts’ and ‘comments’ has to be divided
since the Reddit rating (karma) consists of two parts: post karma and comments
karma, these being computed differently. The overall rating is the sum of these
two values. For MathOverflow, ‘posts’ and ‘comments’ can instead be stored
together. This organisation using tuples allows to exploit both the Reddit and
the MathOverflow reputation models.
 Expressing Trust with Temporal Frequency 1139

5 Experimental Results
In this section, a comparison is done between the DIBRM inferences and those
from StackOverflow, Reddit and MathOverflow.
DIBRM. The DIBRM model was tested with StackOverflow covering 4-year of
data including 15, 000 users, 8, 000, 000 posts and 33, 000, 000 votes. The histor-
ical reputation approximation computed out of this data was 0.85 with different
values of α, β and ta . This is considered as a very good approximation. A possible
interpretation of the results from Tables 1, 2, 3 and 4, is given below.

Table 1. Historical reputation scores Table 2. Reputation scores for different

for different β (forgetting factor) α (cumulative factor)

# ta β μH σH # α μD σD
1 2 0.90 0.8803 0.0024 1 1 0.8313 0.0936
2 2 0.99 0.8805 0.0026 2 2 0.8441 0.0774
3 8 0.90 0.8808 0.0023 3 4 0.8426 0.0814
4 8 0.99 0.8808 0.0021 4 8 0.8515 0.0723

Table 3. Reputation scores for differ- Table 4. Historical reputation

ent ta (period) scores for different ta (period)

# ta μD σD # ta μH σH
1 1 0.8122 0.1100 1 1 0.8816 0.0128
2 2 0.8313 0.0936 2 2 0.8805 0.0026
3 4 0.8510 0.0744 3 4 0.8813 0.0086
4 8 0.8605 0.0604 4 8 0.8808 0.0021

• by increasing ta (period) makes trust scores slightly larger because users have
a wider window for communications and interactions. However, if the repu-
tation change of two users over time is compared, the larger the ta gets, the
larger the difference between two users is;
• by increasing the forgetting factor (β) also the growth of the reputation scores
increases because it gives more importance to past interactions (best results
are achieved with β = 0.99);
• the cumulative factor (α) shows how much the interaction value can grow as
well how the I value can increase.

In summary, the DIBRM model succeeded in approximating the StackOverflow

ratings with accuracy of 0.88. The only disadvantage of DIBRM is parameter
tuning.
1140 E. Yashkina et al.

Table 5. DIBRM & Reddit scores (α = 1, β = 0.9, ta = 1 day).

Date posts post users comments comment users Post results Comment results
2018.05.27 1000 800 14000 12000 −2305.657 −1600.453
2018.05.30 2500 2350 17000 15000 −172.952 −100.525
2018.06.06 4775 4000 33500 28000 −21.163 −0.304
2018.06.13 8468 5129 57556 32008 −9.539 0.126
2018.06.20 13367 7699 91225 47883 −5.278 0.455
2018.06.27 17495 9737 121326 59870 −3.544 0.584
2018.07.02 19282 10553 133349 64236 −3.047 0.620

Reddit. From Fig. 1a it is possible to observe that the difference between

DIBRM and Reddit ratings is significant (about 1.5 million). According to
Table 5, this is similar to the average result for most of the users.
Figure 1b depicts an exception: this approximation is the overall best result,
although the difference is still significant. Additionally, the fluctuation of the
DIBRM ratings resembles the Reddit ratings, showing a good approximation.
The inferences with ‘comment’ ratings are significantly better than those
with ‘post’ ratings. Figure 1c shows the approximation of DIBRM for ‘comment’
ratings for the user SergioVamos which is very accurate, except for a few days. In
Fig. 1d, a typical trend of approximation is depicted with the two ratings close to
each other with marginal differences. The main reason why the accuracy is low
with the ‘post’ ratings is likely due to the limited size of the dataset considered.
When DIBRM was tested with the StackOverflow data, it covered roughly 3

(a) user Meldroit1 (post) (b) user Gondala123 (post)

(c) user SergioVamos (comment) (d) user SurfingSilverSurfer (comment)

Fig. 1. Rating changes for different users for Reddit vs DIBRM.

 Expressing Trust with Temporal Frequency 1141

years but, with the Reddit data, (manually collected in a short time frame), only
1 month was considered. From the results of Table 5, it can be observed that,
with additional data, they can become more accurate (when the value is 1, then
the two results are similar, implying a perfect approximation). As a consequence,
if the sample size is small, the difference between the actual and the DIBRM
ratings is significant (Fig. 1a) as well as the ratio of the difference compared to
the number of users. Additionally, when increasing α or β individually, results
dot not change much (about 0.0001). However, if both parameters are set to
large values, then the results improve on a magnitude of 0.1. The most suitable
time period for Reddit was found to be ta = 1 day and with departure from this,
inferences get worse.
MathOverflow. An open dataset by MathOverflow was downloaded5 . It con-
tains information about votes, comments, posts and users from the launch of

(a) Average rating change over a month.

(b) Monthly difference between the average users reputation. The approximation was μD
= 0.68 for this period.

Fig. 2. Dynamic reputation graphs for MathOverflow vs DIBRM.

5
https://archive.org/details/stackexchange.
1142 E. Yashkina et al.

(a) Average rating change, ta = 8 ∗ 104 days.

(b) Difference between the average users reputation, ta = 8 ∗ 104 days.

Fig. 3. Dynamic reputation graphs for MathOverflow vs DIBRM.

(a) Monthly Average rating change. β = 0.9 was used. μD is lower than 0.6. The
difference grows larger as in Figure 2a (ta = 1).

(b) Monthly Difference between the average reputations of users with β = 0.9
(similar behavior with Figure 2b).

Fig. 4. Dynamic reputation graphs for α = 2, ta = 2 days and Ib = 4.

 Expressing Trust with Temporal Frequency 1143

the platform, 28 October 2009, to 12 December, 2017. A portion was selected

including ‘votes’, ‘comments’ and ‘posts’ (from 28 October 2009 to 28 Novem-
ber 2010, 4793 users). One month data was used for measuring the difference
between the DIBRM and MathOverflow models. From Fig. 2a, the estimation by
DIBRM differs more over time because of the dynamic properties. While with
the MathOverflow model users tend to increase their reputation over time, in the
DIBRM model users tend to lose reputation values due to the β Δn factor. This
difference decreases with a larger amount of months in the historical DIBRM
model. As in Fig. 3a, the two lines become parallel (Fig. 4).

5.1 Discussion
In general, the experiments conducted with MathOverflow data showed better
results when compared to those executed with Reddit data. In fact, there was no

(a) Average rating change over a month. ta = 2 days, β = 0.99 and Ib = 4. Unlike
other cases, difference is decreasing over time.

(b) The best set of parameters that makes DIBRM and MathOverflow ratings
closer is: ta = 2 days, β = 1, α = 1.4 with approximation, μD 0.95.

Fig. 5. Dynamic reputation graphs for the best scenario.

1144 E. Yashkina et al.

negative μD values and the difference between the DIBRM ratings and the actual
rating was smaller with data from MathOverflow. This is likely due to the size
of the dataset: larger for MathOverflow containing more temporal information.
Unfortunately, with Reddit it was not possible to collect a significant amount
of data. Moreover, the Reddit rating algorithm is officially not made publicly
available, contrarily to MathOverflow. Results showed how the DIBRM model
can approximate other rating systems accurately and with minimal information.
Therefore it can be suitable as an inferential system of reputations of online
users when available data is limited (Fig. 5).

6 Conclusion
Several studies have been conducted on reputation models, with such models
having a draw back with the anonymity of users, an important concept in mod-
ern research. In this paper, a comparative research has been performed using
an existing reputation model, the Dynamic Interaction-based Reputation Model
(DIBRM). This model expresses trust of users by employing temporal frequency
of their interaction in a given online community. In detail, the DIBRM model
was applied to other popular social networks, namely Reddit and MathOverflow,
extending its previous application with StackOverflow. Through a set of exper-
iments, it has been observed that DIBRM, without any rating and user profile
information, can mimic the behaviors of reputation algorithms used by differ-
ent online social platforms with a good degree of accuracy. It also shows how
its accuracy can be greatly improved when enough interaction data is available.
This implies that DIBRM captures the modern essence of reputation mecha-
nisms without accounting for the underlying structure of online platforms and
the nature of interactions. Future work can focus on building an automatic tun-
ing mechanism for the parameters required within DIBRM, giving it the ability
to become truly versatile. Additionally, further empirical validation of DIBRM
can be done by running similar comparisons with data from other interaction-
based online platforms.

References
1. Abdul-Rahman, A., Hailes, S.: Supporting trust in virtual communities. In: 33rd
Annual Hawaii International Conference on System Sciences 2000, pp. 9–pp. IEEE
(2000)
2. Bharadwaj, K.K., Al-Shamri, M.Y.H.: Fuzzy computational models for trust and
reputation systems. Electron. Commer. Res. Appl. 8(1), 37–47 (2009)
3. Brown, J.A., Lee, J., Kraev, N.: Reputation systems for non-player character inter-
actions based on player actions. In: In 13th AAAI Conference on Artificial Intelli-
gence and Interactive Digital Entertainment, pp. 1–7 (2017)
4. Buchegger, S., Mundinger, J., Le Boudec, J.Y.: Reputation systems for self-
organized networks. IEEE Technol. Soc. Mag. 27(1), 41–47 (2008)
 Expressing Trust with Temporal Frequency 1145

5. Dondio, P., Longo, L.: Trust-based techniques for collective intelligence in social
search systems. In: Next Generation Data Technologies for Collective Computa-
tional Intelligence, pp. 113–135. Springer (2011)
6. Dondio, P., Longo, L.: Computing trust as a form of presumptive reasoning. In:
2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI)
and Intelligent Agent Technologies (IAT), vol. 2, pp. 274–281. IEEE (2014)
7. Gal-Oz, N., Yahalom, R., Gudes, E.: Identifying knots of trust in virtual communi-
ties. In: IFIP International Conference on Trust Management, pp. 67–81. Springer
(2011)
8. Hoffman, K., Zage, D., Nita-Rotaru, C.: A survey of attack and defense techniques
for reputation systems. ACM Comput. Surv. (CSUR) 42(1), 1 (2009)
9. Hsu, M.H., Chang, C.M., Yen, C.H.: Exploring the antecedents of trust in virtual
communities. Behav. Inf. Technol. 30(5), 587–601 (2011)
10. Jøsang, A., Ismail, R., Boyd, C.: A survey of trust and reputation systems for
online service provision. Decis. Support Syst. 43(2), 618–644 (2007)
11. Lee, J., Duan, Y., Oh, J.C., Du, W., Blair, H., Wang, L., Jin, X.: Automatic
reputation computation through document analysis: a social network approach.
In: 2011 International Conference on Advances in Social Networks Analysis and
Mining, pp. 559–560 (2011)
12. Lee, J.Y., Oh, J.C.: A model for recursive propagations of reputations in social
networks. In: 2013 IEEE/ACM International Conference on Advances in Social
Networks Analysis and Mining, ASONAM 2013, pp. 666–670. ACM (2013)
13. Longo, L., Barrett, S., Dondio, P.: Information foraging theory as a form of collec-
tive intelligence for social search. In: International Conference on Computational
Collective Intelligence, pp. 63–74. Springer (2009)
14. Longo, L., Barrett, S., Dondio, P.: Toward social search - from explicit to implicit
collaboration to predict users’ interests. In: 5th International Conference on Web
Information Systems and Technologies, Lisbon, Portugal, pp. 693–696 (2009)
15. Longo, L., Dondio, P., Barrett, S.: Temporal factors to evaluate trustworthiness
of virtual identities. In: 3rd International Conference on Security and Privacy in
Communications Networks and the Workshops 2007, SecureComm 2007, pp. 11–19.
IEEE (2007)
16. Longo, L., Dondio, P., Barrett, S.: Enhancing social search: a computational col-
lective intelligence model of behavioural traits, trust and time. In: Transactions on
Computational Collective Intelligence II, pp. 46–69. Springer (2010)
17. Longo, L., Dondio, P., Bresciani, R., Barrett, S., Butterfield, A.: Enabling adapta-
tion in trust computations. In: Future Computing, Service Computation, Cognitive,
Adaptive, Content, Patterns 2009. Computation World, pp. 701–706. IEEE (2009)
18. Massa, P.: A survey of trust use and modeling in real online systems. In: Trust in
E-services: Technologies, Practices and Challenges, pp. 51–83. IGI Global (2007)
19. Massa, P., Avesani, P.: Controversial users demand local trust metrics: an experi-
mental study on epinions.com community. In: 20th National Conference on Artifi-
cial Intelligence, 17th Innovative Applications of Artificial Intelligence Conference,
pp. 121–126 (2005)
20. Matthews, P.: Going meta: norm formation and enactment on the stack exchange
network. In: 25th International Conference Companion on World Wide Web, pp.
781–785 (2016)
21. Melnikov, A., Lee, J., Rivera, V., Mazzara, M., Longo, L.: Towards dynamic
interaction-based reputation models. In: 32nd IEEE International Conference on
Advanced Information Networking and Applications, AINA 2018, pp. 422–428
(2018)
1146 E. Yashkina et al.

22. Ruan, Y., Alfantoukh, L., Fang, A., Durresi, A.: Exploring trust propagation
behaviors in online communities. In: 2014 17th International Conference on etwork-
Based Information Systems (NBiS), pp. 361–367. IEEE (2014)
23. Sabater, J., Sierra, C.: Review on computational trust and reputation models.
Artif. Intell. Rev. 24(1), 33–60 (2005)
24. Salihefendic, A.: How Reddit ranking algorithms work. Hacking Gonzo 23 (2010)
25. Sherchan, W., Nepal, S., Paris, C.: A survey of trust in social networks. ACM
Comput. Surv. (CSUR) 45(4), 47 (2013)
26. Singer, P., Flöck, F., Meinhart, C., Zeitfogel, E., Strohmaier, M.: Evolution of
Reddit: from the front page of the internet to a self-referential community? In:
Proceedings of the 23rd International Conference on World Wide Web, pp. 517–
522. ACM (2014)
27. Tausczik, Y.R., Kittur, A., Kraut, R.E.: Collaborative problem solving: a study
of mathoverflow. In: 17th ACM Conference on Computer Supported Cooperative
Work & Social Computing, pp. 355–367 (2014)
28. Tormo, G.D., Mármol, F.G., Pérez, G.M.: Dynamic and flexible selection of a
reputation mechanism for heterogeneous environments. Future Gener. Comput.
Syst. 49, 113–124 (2015)
29. Yu, B., Singh, M.P.: An evidential model of distributed reputation management.
In: Proceedings of the First International Joint Conference on Autonomous Agents
and Multiagent Systems: Part 1, pp. 294–301. ACM (2002)
30. Zhang, L., Tan, C.P., Li, S., Fang, H., Rai, P., Chen, Y., Luthra, R., Ng, W.K.,
Zhang, J.: The influence of interaction attributes on trust in virtual communities.
In: International Conference on User Modeling, Adaptation, and Personalization,
pp. 268–279. Springer (2011)
 Hot Spot Tracking by Time-Decaying
Bloom Filters and Reservoir Sampling

Kai Cheng(B)

Department of Information Science, Kyushu Sangyo University,

Higashi-ku, Fukuoka 813-8503, Japan
[email protected]

Abstract. In most networking applications such as IoT (Internet of

Things), data are being generated at a high rate so that the long–term
storage cost outweighs its benefits. Such streams of data are stored
temporarily, and should be mined fast before they are lost forever.
In a previous work, we have presented Time–decaying Bloom Filters
(TBF) for maintaining time–varying frequency statistics in data streams.
TBF extends the standard Bloom Filters (for approximate membership
queries) by replacing the bit-vector with an array of small counters,
whose values decay periodically with time. In this paper, we consider
hot spot tracking problem for data streams. To this problem, we propose
a novel scheme by integrating TBF and online sampling technology. Data
streams are sampled in an online manner using a reservoir. Items newly
sampled are passed to a TBF where frequency statistics are maintained
for hot spot reporting.

1 Introduction
The exponential increase of devices connected to the Internet known as IoT
(Internet of Things) in recent years poses new challenges for efficient, secure
and scalable access to massive data generated by different devices. Most data
in this context can be modeled as data stream, a real–time, continuous, ordered
(implicitly by arrival time or explicitly by timestamp) sequence of data items.
Often it is impossible to control the order in which items arrive, nor is it feasible
to store a stream in its entirety. Network traffic data, telephone call records,
sensor readings, web access logs are some examples of data streams.
Hot spot tracking is a typical problem in the data stream applications which
requires to efficiently maintain frequency statistics for the underlying data items.
For example, it is important to maintain hot spot items on EC sites so as to
predict customer interest and to optimize supply chain management. It is also
important to maintain hot spots of web page views to optimize web site orga-
nization, improve search engine performance or reorganize content delivery net-
works. Hot spots are also known as heavy hitters and hot spot tracking problem
is also called as heavy–hitter problem. Network operators are interested in the
heavy hitters of a network, i.e., the traffic flows responsible for most bytes or
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1147–1156, 2020.
https://doi.org/10.1007/978-3-030-15032-7_96
1148 K. Cheng

packets. This information is useful for a number of common applications such

as identifying denial of service (DoS) attacks, monitoring traffic growth trends,
provisioning network resources and link capacities, and warning heavy network
users.
To achieve efficient hot spot tracking, it is important to maintain frequency
statistics for potentially frequent items. A brute–force approach is to use a list
of counters for all data items arrived. Each counter is a pair of (item, count),
giving the frequency count of the indicated item. But this is infeasible for large–
scale applications, such as busy portal sites and general–purpose search engines,
because each occurrence of an item incurs O(n log n) amortized cost to maintain
the corresponding counter (n is the number of distinct items).
Bloom Filters are space-efficient data structures that maintain a very com-
pact inventory of the underlying data, supporting O(1) time membership queries
over a given data set [1]. The space requirements of Bloom filters fall significantly
below the information theoretic lower bounds for error–free data structures. This
efficiency is at the cost of a small false positive rate (items not in the set have a
small probability of being recognized as in the set), but have no false negatives
(items in the set are always recognized as being in the set). Bloom filters are
widely used in practice when storage is at a premium and an occasional false
positive is tolerable.
Standard Bloom Filters, although useful, have several limitations. First, they
do not support deletes because simply turning off the corresponding bits may
introduce false negative errors (some bits of other items, although still in the set,
may be turned off). Second, they cannot handle multi–set (set with duplicates
of items) to answer multiplicities of items. In many applications, e.g., iceberg
queries [8], data mining, such multiplicity information is crucial.
In a previous work [3], we have proposed Time–Decaying Bloom Filters
(TBF) to address the challenges in maintaining time–varying frequency statistics
over data streams. In that work, we developed a basic scheme, called basic TBF,
for maintaining time–decaying aggregates over data streams. For data streams
with skewed distributions, we proposed further optimization by leveraging the
skewness where only a few “large” items are associated with bigger counters.
In this paper, we propose a scheme for hot spot tracking by integrating
TBF and online sampling technology. In this scheme, data streams are sampled
in an online manner using a reservoir without long–term storage. Items newly
sampled are passed to a TBF where frequency statistics are maintained for hot
spot reporting.

2 Sampling Data Streams with a Reservoir

Sampling is a process of choosing a suitable representative subset from a popula-
tion. Uniformity is a basic requirement of representativeness. A sampling process
produces a uniform random sample when all possible samples of the same size
have the same probability. A simple random sampling is to choose each element
of the population uniformly and entirely at random, such that each individual
Hot Spot Tracking by Time-Decaying Bloom Filters and Reservoir Sampling 1149

has the same probability of being chosen at any stage during the sampling pro-
cess, and each subset of the same size has the same probability to be chosen.
On the other hand, a systematic sampling scheme starts by selecting an element
from the list at random and then every kth element in the frame is selected. The
sampling interval k = N/n, where n is the sample size, and N is the population
size. Using this procedure each element in the population has a known and equal
probability of selection.
The sampling schemes however are not suitable for data streams because
either population or sampling probability cannot be completely known before-
hand. Sampling of data streams has to work in an online manner, where sampling
decisions are made while data arriving one by one with out knowing the popu-
lation. Reservoir is a data structure that maintains a fixed number k of samples
from a large population so that at any time, any element of the population has
the same probability being selected. The basic idea behind reservoir algorithms
is to select a sample of size k. The first step of any reservoir algorithm is to
put the first k elements of the file into a “reservoir”. The rest of the elements
are processed sequentially and elements can be selected for the reservoir only
as they are processed. An algorithm is a reservoir algorithm if it maintains the
invariant that after each element is processed a true random sample of size k
can be extracted from the current state of the reservoir.
The process to maintain a uniform reservoir is simple: save the first k items
in an array of size k; for each item n, n > k, choose a random integer r from 1 to
n (inclusive); if r ≤ k, replace item r of the array with item n and discard item
n otherwise. This process can guarantee the uniformity of sample, that is, when-
ever an item is processed it has the equal probability to be included in sample
among others no matter when and in which order they are processed. In fact,
Reservoir sampling has another advantage that it can break correlation in the
input stream of nodes (edges) since consecutive items are selected independently
and uniformly at random.
Reservoir can also support sampling with unequal probabilities. A weighted
reservoir is a data structure for k weighted items selected randomly in a sequen-
tial order from a population of N items (N may be unknown in advance) so that
each of the n items processed so far has an inclusion probability propositional
to its weight. That is, at the time when the nth item is processed, the inclusion
probability of item t ≤ n with a weight wt , where:
wn
wt = n (1)
1 wi

The process to maintain a weighted reservoir is as follows: save the first k

items in an array of size k; for each item t > k, generate a uniform random value
ut from (0, 1) and use the following value as the priority of item t:
1
pt = − log ut (2)
wt
1150 K. Cheng

3 Time-Decaying Bloom Filters (TBF)

In this section, we summarize Time-decaying Bloom Filters (TBF) data struc-
ture. A basic TBF is a counting Bloom filter (Bloom filter with the bitmap
replaced by an array of counters) whose counters decay with time. Time–
decaying counters are introduced to handle aging of data items.
A time-decaying counter or simply decaying counter is a counter whose value
decays periodically with time. The fashion of how a counter decays with time is
determined by a special non–increasing, non–negative function φ(t), called time–
decaying function (tdf ). A tdf function should satisfy the following conditions:
1. φ(0) = 1
2. φ(t) is non–increasing
3. 0 ≤ φ(t) ≤ 1 for all t ≥ 0.
Let v(e) be the value of item e in S up to the current time,  tn . Let vi (e) be
the value of item e up to the time ti since ti−1 . That is, v(e) = i vi (e). Then,
the decayed value of item e is
n

v∗ = φ(ti ) · vi
i=1

Cohen et al. [4] give a variety of tdf functions. In this paper, we adopt expo-
nential tdf, one of the most widely used tdf’s in practice.
φ(t) = λt/T , 0 ≤ λ < 1, T >0
T and λ are parameters that control the behavior of how fast the counter will
decay. T is the period between two consecutive applications of φ. In other words,
it controls the granularity of time–sensitivity. A time period of T time units is
referred to as an epoch. The parameter λ ∈ [0, 1], called exponential decaying
factor or simply decaying factor, controls the speed of exponential decaying.
Although our proposal is primarily based on the exponential tdf, later we
will discuss the applicability of our method to other tdf’s.
Given a data stream S, and the tdf φ(t), our goal is to design a succinct
data structure such that at any time instant an estimate v̂ of v for item e can
be answered in time O(1), i.e., query cost is independent of the length of S.
1. v̂ ≥ v, i.e., we never underestimate the occurrences of item e
2. Error of the estimate is bounded to an allowable level.
3. O(1) time complexity for adding an item, or answering a query of counter
value for each item.

3.1 Bloom Filters with Decaying Counters

In this section, we describe the basic Time-decaying Bloom Filters (TBF). A
basic TBF is a counting Bloom filter (Bloom filter with the bitmap replaced
by an array of counters) whose counters decay with time. Concretely, a TBF
consists of,
Hot Spot Tracking by Time-Decaying Bloom Filters and Reservoir Sampling 1151

1. A set of independent k hash functions, h1 , · · · , hk , mapping each item to

[1..m]
2. An array of m counters, C1 , · · · , Cm
3. A real–valued time–decaying function, tdf φ(t).
Initially, all counters are initialized to 0. While an item x arriving, increment
each of the k counters Ch1 (x) , · · · , Chk (x) . Whenever an epoch arrives after T
time units elapsed, decay all counters by applying φ(t) to them (Fig. 1).

Fig. 1. Time–decaying bloom filter

Dependent on the form of φ(t), the last operation can be complicated. In

[4], there is given a abundance of candidate tdf ’s (they call decay functions),
e.g., exponential decay, sliding window decay, polynomial decay, polyexponential
decay and chordal and polygonal decay. As in Bloom Filters, space efficiency is
a tough requirement, we prefer such tdf’s that can be maintained incrementally,
particularly, we prefer the exponential tdf, which is the most commonly used in
practice and as described in the following decaying counters using this tdf can
be maintain very efficiently.

3.2 Maintaining Time–Decaying Counters

Given vi , the real value obtained at time instant ti after a period of T , and
vi∗ , the exponentially decayed count value at ti , can be recursively computed as
follows:

vi∗ = (1 − λ)vi + λ · vi−1

∗
, λ ∈ [0, 1) (3)
Let gi∗ = vi∗ /(1 − λ), we have the following form.

gi∗ = vi + λ · gi−1
∗
(4)
1152 K. Cheng

Using the above equation, we can increment a counter directly after the decay
∗
function was applied, i.e., λ · gi−1 . After gi∗ is available, the result of vi∗ can be
obtained by the following equation.

vi∗ = (1 − λ) · gi∗ (5)

There are two problems with the basic TBF. First, it is difficult to give
a priori on how many bits are enough for the counters so that overflows can
be avoided. As a result we are forced to allocate all counters the maximal bits.
Second, when distribution of the data items is highly skewed, it is not economical
to use uniform-sized counters for all items.

Fig. 2. Counter pool of variable–length counters with a Trie index

3.3 TBF with Variable–Length Counters

When data distribution is highly skewed, there are only a few counters with
high values and most counters only hold relatively small values. To leverage this
feature, in [3], we proposed to maintain variable–length counters by a linked lists.
At first, a small elementary counter is allocated to each slot. When the basic
counter gets overflowed, an extra counter from the free counter pool is allocated
for the carried bits.
In this paper, we propose another approach to handling counters of variable–
size. As shown in Fig. 2, the counter pool is an array of M elementary counters.
Without loss of generality, assume M = 2w . An elementary counter is a counter
Hot Spot Tracking by Time-Decaying Bloom Filters and Reservoir Sampling 1153

Fig. 3. TBF with variable–length counters

of bit size b. Any counter can be of size 2i b bits (i = 0, 1, · · · ). The indexes of

counter in the counter pool are organized as a Tie where a larger counter has a
short address. When necessary, two adjacent counters can join to form a larger
counter. For example, 010 corresponds to cell 2 and 10 to cells 4 and 5.
The algorithm for maintaining a TBF with variable–length counters is shown
in Fig. 3. Each slot in a TBF comes with an elementary counter. On insertion
of an item x, a minimum increment policy is applied. Those with minimal value
among k counters x mapped to will increment by one. When all the k counters
become overflowed, a larger counter from free counter pool is allocated and its
address (a binary string) instead of the counter value is stored in each of the k
slots. To distinguish a counter and an address, we use an additional flag bit for
each slot.
To add a new item q into the optimized TBF filter, we simply add the item
into the TBF, incrementing each of its counters hi (i = 1, · · · , k) by 1 unless
the counter becomes overflowed. When a counter gets overflowed, we allocate an
additional counter. An optimization here is to delay the allocation of additional
counters till the minimum counters go beyond the scope. When the minimum
1154 K. Cheng

counter is not yet beyond its scope, we just skip the increment operation for
larger counters because in the query step, we are only interested in the minimum
one. The exponential decaying operations take place whenever a epoch starts.
Assume the data stream S has N item occurrences. Let n be the number of
distinct items in S. Then, a standard Bloom Filter with a bitmap of m bits, k
independent hash functions, has a error rate
Er = (1 − e−kn/m )k , (6)
which is minimized when k = ln 2 · ( m
n ).

Claim 1. For any item e ∈ S, fˆe ≤ fe . That is, the time-decayed frequency counts
estimated by TBF are no more than the real time-decayed frequency counts.
Proof: Since each counter was set at least by one item, the at any time instance,
the accumulated count values are no less than real frequency counts. TBF and
Eq. 4 are decayed at the same speed and at the same time instance (same T ),
thus fˆe ≥ fe .
Now we analyze the improvement of space efficiency under Zipfian distribu-
tion. The multiplicity ni of the ith most frequent ni = C/iz , where 0 < z < 1.
Suppose there are m
m distinct items in the streamand N > m is the total length of
m
the stream. Then i=1 ni = N . Let Hm (z) = i=1 (1/iz ) then C = N/Hm (z).
m

m log N − log ni = m · log Hm (z) + z · log m! (7)
i=1

Hm (z) is a decreasing function, Hm (0) = m, Hm (1) = Hm = ln m + Θ(1),

Hm is harmonic number.

4 Discussion on Hot Spot Tracking

In this section, we discuss the approach to applying reservoir sampling and TBF
for hot spot tracking. First, a weighted reservoir is ideal tool for sampling data
items that are likely to be hot spots in an online manner. To do this, each item
in the reservoir is represented as an entry of the form (e, f ), where f is the
frequency of an item e. After saving the first k items, on the arrival of a item
e, it will be decided whether to enter the reservoir as a new entry (e, 1), or
cause an existing entry whose frequency updated. Since frequency is used as the
weight, items with larger frequency, that is, the hot spots, tend to stay in the
reservoir. Second, TBF can be used for hot spot tracking by tracking those items
whose counter is overflowed. By increasing the counter size of the B-TBF, we
can control the lower bound of frequency.

5 Related Work
Maintaining statistics over data streams has been one of the most fundamental
topics in the research of data stream management systems. One related prob-
lem is the heavy hitter or frequent item finding problem [2,7,10], which aims at
Hot Spot Tracking by Time-Decaying Bloom Filters and Reservoir Sampling 1155

tracking and reporting items in a data set whose multiplicities exceed a given
threshold. Lossy counting [10] is an efficient and well-known algorithm for find-
ing heavy hitters. The algorithm has the following high-level operation. It splits
an input stream of elements into fixed-size windows and processes each window
sequentially. For each element in a window, it inserts an entry into a table, which
monitors the number of occurrences of the elements, or, if the element is already
in the table, it updates its frequency. At the end of each window, the algorithm
removes elements of small frequency from the table. The table maintains a small
number of entries and can, at any given point in time, determine the heavy
hitters of the input data stream.
In [11], the authors gave detailed survey on the applications of Bloom filters
in distributed systems. A another recent survey [9] reviewed dozens of Bloom
filter variants as well as related probabilistic data structures.
Counting is important in many applications and a few Bloom filter variants
were developed, such counting bloom filers, spectral bloom filters and stable bloom
filters. However, counters in these work were mostly introduced to support dele-
tions rather than for frequency statistics. Our work shares some idea with Cohen
[5], where variable-length counters are used to represent counters compactly.
Because their proposed scheme is not so suitable to update-dominated case, it
has to rebuild the whole data structure completely after a series of updates. Our
scheme is simple to implement and suitable for frequent updates.
Counter aging was also dealt in a couple of work. In [6], counters do not
represent the multiplicities of the items but rather their age. Initially all counters
are set to their maximum value. Then when inserting an item, they decrement d
of them chosen uniformly at random. [12] uses two single-bit vectors and switches
between them by activating one and deactivating another one so that old data
were discarded (flushed).

6 Concluding Remarks

We have discussed approach to applying Time–decaying Bloom Filters, an effi-

cient data structure and reservoir sampling algorithms in hot spot tracking over
data stream. In our previous work, we extended standard Bloom Filters and
developed Time-decaying Bloom Filters, TBF that leverages skewed distribu-
tions and can deal with time–sensitivity. A commonly-used time–decaying func-
tion, exponential decay is used. TBF focus on recent occurrences of a data item
because they are more important than older ones. Since data streams exhibit
strong skewness, with a small fraction of data items occurrence far more fre-
quently the most others, so we do not need allocate large counters for all items.
Instead, we allocate small counters for the first round, then for those necessary
to extend, large counters are allocated to replace the small ones. In this paper,
we extended our previous work by introducing a Tire index for variable–length
counters.
1156 K. Cheng

References
1. Bloom, B.H.: Space/time trade-offs in hash coding with allowable errors. Commun.
ACM 13(7), 422–426 (1970)
2. Charikar, M., Chen, K., Farach-Colton, M.: Finding frequent items in data streams.
In: Proceedings of the International Colloquium on Automata, Languages and
Programming (ICALP), pp. 693–703 (2002)
3. Cheng, K., Xiang, L., Iwaihara, M.: Time-decaying bloom filters for data streams
with skewed distributions. In: 15th International Workshop on Research Issues in
Data Engineering: Stream Data Mining and Applications (RIDE-SDMA 2005), pp.
63–69, April 2005
4. Cohen, E., Strauss, M.: Maintaining time-decaying stream aggregates. In: PODS
2003, pp. 223–233 (2003)
5. Cohen, S., Matias, Y.: Spectral bloom filters. In: SIGMOD Conference, pp. 241–252
(2003)
6. Deng, F., Rafiei, D.: Approximately detecting duplicates for streaming data using
stable bloom filters. In: Proceedings of the 2006 ACM SIGMOD International
Conference on Management of Data, SIGMOD 2006, pp. 25–36. ACM, New York,
NY, USA, (2006)
7. Estan, C., Varghese, G.: New directions in traffic measurement and accounting:
focusing on the elephants, ignoring the mice. ACM Trans. Comput. Syst. (TOCS)
21(3), 270–313 (2003)
8. Fang, M., Shivakumar, N., Garcia-Molina, H., Motwani, R., Ullman, J.D.: Comput-
ing iceberg queries efficiently. In: Proceedings of the Twenty-fourth International
Conference on Very Large Databases, pp. 299–310 (1998)
9. Luo, L., Guo, D., Ma, R.T.B., Rottenstreich, O., Luo, X.: Optimizing bloom filter:
challenges, solutions, and comparisons. CoRR, abs/1804.04777 (2018)
10. Manku, G., Motwani, R.: Approximate frequency counts over data streams. In:
Proceedings of 28th International Conference on Very Large Data Bases, VLDB
2002, pp. 346–357 (2002)
11. Tarkoma, S., Rothenberg, C.E., Lagerspetz, E.: Theory and practice of bloom filters
for distributed systems. IEEE Commun. Surv. Tutorials 14(1), 131–155 (2012)
12. Yoon, M.K.: Aging bloom filter with two active buffers for dynamic sets. IEEE
Trans. Knowl. Data Eng. 22(1), 134–138 (2010)
 A Fuzzy Logic Based Trust-ABAC Model
for the Internet of Things

Hamdi Ouechtati(B) , Nadia Ben Azzouna, and Lamjed Ben Said

LR11ES03 SMART Lab, Universite de Tunis ISG, Tunis, Tunisia

[email protected], [email protected],
[email protected]

Abstract. The Internet of Things (IoT) integrates a large amount of

everyday life devices from heterogeneous network environments, bringing
a great challenge into security and reliability management. In order to
cope with certain challenges posed by device capacity and the nature
of IoT networks, a lightweight access control model is needed to resolve
security and privacy issues. In this paper, we present Fuzzy logic based
Trust-ABAC model, an access control model for the Internet of Things.
Our model for the IoT is an improvement of our previous work Trust-
ABAC [3] by a new Fuzzy logic-based model in which we consider an
evaluation of trust based on recommendations and social relationship
that can deal effectively with certain types of malicious behavior that
intend to mislead other nodes. Results prove the performance of the
proposed model and its capabilities to detect the collision and singular
attacks with high precision.

1 Introduction
A future Internet of Things (IoT) system connects the physical world into
cyberspace via radio frequency identification (RFID) tags, sensors, and smart
objects, which allows us to exchange information and receive intelligent ser-
vices [19]. However, the IoT environment can include several malicious entities
and devices who aim to break its basic functionality (e.g. service composition)
through the broadcast of erroneous information about their position, identity,
reputation, etc. Therefore, the number of threats and their sources increases
proportionally. Although the IoT benefits are unlimited, there are many secu-
rity issues. One of the main elements to address IoT security issues is the access
control. This latter is an important part of today systems. Whether it is about
physical access control or software access control, it has become an essential
and critical element for most businesses. Over the years, multiple models have
emerged, like the relatively recent model called Attribute-Based Access Control
(ABAC). The concept is not especially innovative, but with the rise of IoT sys-
tems, the ABAC model could turn out to be very interesting due to the use of
multiple and diverse attributes in the policies.
However, without trustworthy solutions to ensure user’s honesty, the IoT will
not reach enough popularity to be considered as a well established paradigm.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1157–1168, 2020.
https://doi.org/10.1007/978-3-030-15032-7_97
1158 H. Ouechtati et al.

Accordingly, trust management becomes a major challenge to ensure enhanced

security, qualified services and especially determining the trust level of objects.
Accordingly, in this paper, we put more focus on solving the access control
problem in the IoT. Therefore, our Fuzzy logic based Trust-ABAC model for
the IoT is an improvement of our previous work Trust-ABAC [3] by a new
Fuzzy logic-based model in which we consider an evaluation of trust based on
recommendations and social relations that can deal effectively with certain types
of malicious behavior that intend to mislead other nodes. In this work, we address
only the collusion attack because they interrupt the trust-based mechanism. For
example, in the collusion attack, some objects having social relationships may
cooperate together in order to reach the same interests either by decreasing or
increasing the object’s trust value [21].
The remainder of this paper is organized as follows. The second section
gives an overview of some of the related works on access control. The third
section presents the description of our Fuzzy logic based Trust-ABAC model. In
this section, we will take a closer look on the architecture and the interactions
between the various components of our Fuzzy logic based Trust-ABAC model.
The Sect. 4 focuses on the description of the new Fuzzy logic based trust eval-
uation based on recommendations and social relations. Section 5 presents the
description of the experimental study and the results that are used to evaluate
our model. In the Sect. 6, the conclusion is given and the future work is pointed
out.

2 Related Works
In order to ensure the controllability, security and legitimacy, the access control
model must take into account security factor, environment factor, etc. [4]. A
large number of access control models has been proposed in order to ensure data
confidentiality. In [11], the authors propose an Attribute-Based Access Control
(ABAC)-based cross-domain access control system, which applies the thoughts
of Attribute-Based Access Control in cross domain access control. The system
eliminated the defects in the process of the Role-Based Access Control applying
in SOA. Trust-based access control systems have been substantially explored
in the literature. Sicari et al. [13] discussed key issues associated with security,
privacy, and trust in IoT in terms of challenges and potential solutions. In [12],
Nitti et al. proposed an access control model with trust management is proposed.
This model takes node user’s behavior trust into consideration and adapts to the
characteristics of dynamism and distribution in IoT system, but it just considers
single trust as the authorization constraint.
Smari et al. [15] proposed an attribute-based access control with trust and
privacy. The model associates the subjects and objects defined with sets of
respective attributes. Those attributes represent the various properties of the
considered entities, such as name, role, type, location, purpose, etc. The model
facilities the development of an access control policy that consists of a set of
access control rules between subjects and objects in a particular context. Fur-
thermore, the classical ABAC model is extended to incorporate various aspects
 A Fuzzy Logic Based Trust-ABAC Model for the Internet of Things 1159

of trust into the model. The proposed extension contains a function that com-
putes the required trust of a requesting subject with regard to the accessed
object attributes. The trust may cover of subjects, objects, or policies. The
trust attribute is determined based on the basic trust level, the reputation of
the user, and the recommendations made by other users or systems. Chen et
al. [16] proposed role-based access control model, R2BAC, based on the concept
of trust in order to evaluate the behavior of the nodes and to infer their roles.
This model uses only QoS metrics to calculate trust values, which are assigned
to roles. Moreover, authors in [17] propose FTBAC, a trust-based access control
solution by using the linguistic values of experience, knowledge and recommen-
dation as inputs. Then, these fuzzy trust values are mapped to access privileges.
Although scalability and energy consumption are simulated, a real implementa-
tion of the solution is not provided. Armando et al. [18] presented a novel access
control framework that combines trust with risk and supports access control in
dynamic contexts through trust enhancement mechanisms and risk mitigation
strategies. This allows to strike a balance between the risk associated with a
data request and the trustworthiness of the requester. If the risk is too large
compared to the trust level, then the framework can identify adaptive strategies
leading to a decrease of the risk (e.g., by removing/obfuscation a part of the
data through anonymization) or to increase the trust level (e.g., by asking for
additional obligations to the requester).
Although the above works leveraged the concept of trust in IoT environments
to realize effective access control, these trust models only consider a common
small set of parameters to evaluate the trustworthiness of IoT devices (like QoS
or reputation). In contrast, our proposed trust model considers most of these
parameters altogether, adding even a new dimension of trust regarding security.
This security dimension allows considering security evidences, which are inferred
from security mechanisms being employed in a specific transaction, as part of
the trust model which quantifies the trustworthiness. Furthermore, to cope with
the vagueness of information in pervasive environments, the model uses fuzzy
logic to infer the trust values. In addition, our trust model has been integrated
into Trust-ABAC [3], with the aim of making IoT devices aware of other devices’
trust scores and drive their access control decisions accordingly.

3 Fuzzy Logic Based Trust-ABAC Model

In this paper, we introduce an access control model called Fuzzy logic based
Trust-ABAC for the Internet of Things. This model is an improvement of our
previous work Trust-ABAC [3] by a new Fuzzy logic-based trust evaluation in
order to enhance the access control level in the IoT environment. The improve-
ment that we will introduce consists of two main dimensions: (i) the evalua-
tion of the trust level of the recommendation message and (ii) the detection of
the collision attack in order to filter the inappropriate recommendations. The
increasing number of existing heterogeneous objects and the complexity related
to managing the wealth of information provided by these objects, represent one
1160 H. Ouechtati et al.

of the crucial challenges in the IoT. Furthermore, in an open environment such as

the IoT, objects are sometimes unknown. Therefore, we introduce a new Fuzzy
logic-based model in which we consider an evaluation of trust based on rec-
ommendations and social relations that can deal effectively with certain types
of malicious behavior and ensure access control in such an open environment.
Furthermore, the ease of reasoning and the integration of different attributes in
decision-making based on linguistic variables instead of numerical ones that we
can adopt the fuzzy logic theory in this work since it is a promising artificial
intelligence technology with good performance.

3.1 Architecture Components

The architecture of our Fuzzy logic based Trust-ABAC model for the IoT is com-
posed of a set of features and functional components that enforce authorization
decisions based on attributes and policies. One of the strengths of our model is
the possibility to use a set of multiple and diverse attributes in the policies. It is
even more interesting if this set can be supplied with data from an IoT system,
because obtaining this information is easier in the case of IoT thanks to the
sensors. The attributes’ values are the set of properties that may be associated
with a given entity. An entity may be a subject, resource or environment that is
considered relevant to the interaction between a user and an application. In our
architecture, there are three main components (Fig. 1):

Fig. 1. Fuzzy logic based Trust-ABAC architecture for the Internet of Things.

• Trust-ABAC model: This is our previous access control model for the IoT
which is a combination between the access control model based on attributes
(ABAC) and the trust mechanism [3]. In our Trust-ABAC model, the autho-
rization decision is based on the attributes of the various entities involved in
making access control decisions and the trust value.
 A Fuzzy Logic Based Trust-ABAC Model for the Internet of Things 1161

• The broker: It is a mediator between the Trust-ABAC model and the recom-
mendation manager. This component should implement three main functions.
First, it manages the requests for trust values of the Trust ABAC model. Sec-
ond, it is responsible for backing up feedback rating received concerning the
objects that request access after each transaction. This feedback rating are
used by the broker in order to updates the trust value of the object that
request access stored in its local database. Third, it sends the recommenda-
tions’ requests to the recommendation manager.
• The recommendation manager: This component acts as an interface
between our model and the IoT network. It is responsible for filtering the
received recommendations. Therefore, this component should implement two
main functions. First, it evaluates the trust level of the recommendation mes-
sage and detects the collision attack in order to filter the inappropriate recom-
mendations. Second, it sends only the trusted recommendations to the broker.
The steps of filtering recommendations and the detection of certain types of
malicious behavior will be explained and discussed in detail subsequently.

3.2 Trust-ABAC Model for the IoT

Trust-ABAC model for the IoT is our previous access control model for the IoT
which is a combination between the access control model based on attributes
(ABAC) and the trust mechanism [3]. An overview of the main components of
the architecture of our Trust-ABAC model is listed below:

• PEP (Policy Enforcement Point): The PEP receives the access requests,
sends an authorization request to the PDP and applies the decisions of the
PDP.
• PDP (Policy Decision Point): This module is in charge of taking the
access decision based on (i) the attributes obtained from the request for
authorization, (ii) the policies and the rule sets obtained from the policy
authority.
• Policy Authority: This component allows defining access policies, and feeds
the access control rules in the PDP.
• Context handler: This component should implement two functions: (i) it is
a repository for all the definitions of the attributes and their implementations
in our model, and (2) it makes conversion between the format used outside
the system and the one used inside it.
• Resource, environment and subject authorities: These components
define and provide resource, environment and subject attributes necessary
for the authorization decision.

In our Trust-ABAC model, the authorization decision is based on the environ-

mental attributes, subject attributes, and resource attributes. When an object
sends a request for access, the system evaluates this request based on the corre-
sponding access control policy, the attributes’ values and the trust value of the
object in question.
1162 H. Ouechtati et al.

4 A New Fuzzy Logic-Based Trust Evaluation

Upon receiving recommendations regarding the object requesting access, the
recommendation manager starts by grouping the recommendations according
to the existing social relationships between the senders and the recommended
object. Afterward, it evaluates the similarity between the recommendation val-
ues based on the: (1) internal similarity that represents the degree of similarity
of each recommendation value sent by each object compared to the recommen-
dation values of the community to which it belongs, and (2) external similarity
that represents the degree of similarity of each recommendation value sent by
each object compared to the recommendation values of the other communities.
Simultaneously, our proposed model calculates the degree of the social relation-
ship that measures the strength of the relations that exist between objects. In
this step, the goal of calculating similarity and the degree of the social relation-
ship is to detect the appearance of the collision attack and evaluate the trust
level of the received recommendations. Each step will be explained and discussed
in detail subsequently.

4.1 Classification of Recommendations

The IoT is a vital source of recommendations resulting from the interaction
between objects having social relationships. The objects that have the same
social relationships form a community. These latter are formed by sets of objects
that interact (social interaction) with each other more than others. They are
therefore groups of objects that have forged stronger links or have common
affinities. To this end, our proposed model collects the recommendations from
objects with which the object requesting access has social relationships. There-
fore, we use the communities to which the object belongs to classify the received
messages or recommendations respectively [22]:

1. The OOR recommendations: Set of recommendations sent by several objects

having ownership relationship with the recommended object. This kind of
social relationships is created between heterogeneous objects that belong to
the same owner.
2. The C-LOR recommendations: Set of recommendations sent by several
objects having co-location relationship with the recommended object. This
kind of social relationships is created between stationary objects in the same
place, and the same environment.
3. The C-WOR recommendations: Set of recommendations sent by several
objects having C-WOR relationship with the recommended object. This kind
of social relationships is created between objects that meet each other in the
owner’s workplace.
4. The SOR recommendations: Set of recommendations sent by several objects
having social relationship with the recommended object. This kind of social
relationships is created as a consequence of frequent sporadically or continu-
ously encountering between objects.
 A Fuzzy Logic Based Trust-ABAC Model for the Internet of Things 1163

The interest of this classification is multiple: (i) can be used to determine

the similarity between the recommendation values in order to identify and limit
the sources of attacks and, (ii) calculate the degree of the social relationship in
order to evaluate the trust level of the received recommendations.

4.2 Evaluation of the Similarity Between Recommendations

Based on the classification which we proposed in Subsect. 4.1, the recommen-
dation manager evaluates the similarity between the recommendation values in
order to detect the appearance of the collusion attack, in which some objects
may cooperate together to achieve the same interest (eg. increase or decrease the
reputation value of another object.). Therefore, the recommendation manager
compares each recommendation value sent by each object: (i) with the median
of the recommendations received from the community to which it belongs in
order to evaluate its internal similarity, and (ii) with the average of the medians
of the other communities in order to evaluate its external similarity. Notes that
objects that have the same social relationships form community. These latter
are formed by set of objects who interact with each other more than others. The
purpose of the evaluation of the internal and external similarity is to assess the
compatibility and credibility of each recommendation value sent by each object
both internally and externally. Thereafter, the internal and external similarities
are combined in the following in order to evaluate the recommendation value
credibility. The latter enables us to evaluate the resemblance between the inter-
nal and external similarities on the aggregate level. For example, if the internal
similarity is very high and the external similarity is very low then the recommen-
dation value credibility is very low: which means that the recommendation value
is not accurate because this recommendation is very similar to the community
to which it belongs and completely different to the other communities. Conse-
quently, this community collude to reach the same purpose and has carried out
a collusion attack. Actually, the recommendation value credibility provides an
exact evaluation of recommendations when the number of the false recommen-
dations coming from each community does not exceed the number of the exact
recommendations.

4.3 Evaluation of the Degree of the Social Relationship

In order to apperceive the impact of the social relationships between objects
on the recommendation values. We introduce the degree of the social relation-
ship which measures the strength of the relationship that exists between objects.
Based on the classification which we proposed in Subsect. 4.1, the recommenda-
tion manager attributes to each object belonging to a particular type of com-
munity an initial degree of social relationship that will evolve according to the
evolution of the type of the social relationship and the increase in the number of
transactions. Thereafter, the recommendation manager will combine the recom-
mendation value credibility with the degree of the social relationship in order to
take into account the degree of the social relationship in the trust level evaluation
1164 H. Ouechtati et al.

of the recommendation. Therefore, more the degree of the social relationship is

strong, more the level trust increases.

4.4 Evaluation of the Trust Level of the Recommendations

and Detection of the Collusion Attack
Based on the classification of recommendations which we proposed in Sub-
sect. 4.1, the recommendations are grouped according to the existing social rela-
tionships between the senders and the object that requests access to form com-
munities. The clustering technique of recommendations is not the objective of
this paper and it can be done using the existing work such as [23,24]. After that,
we use the fuzzy inference to evaluates the trust level of received recommen-
dations and detect the appearance of the collision attack. For this purpose, we
employ the internal and external similarities in order to evaluate the recommen-
dation value credibility. The latter enables us to ascertain the exactitude of the
received recommendations and detect the appearance of the collision attack. As
illustrated in Fig. 2, the membership functions named very low, low, medium,
high and very high are used to represent the internal similarity, the external
similarity, and the recommendation value credibility.

(a) Input 1: Internal Similarity (IS) (b) Input 2: External Similarity (ES)

(c) Output: Recommendation value credibil- (d) Correlation between inputs and output
ity(RVC)

Fig. 2. The membership functions and the correlation between inputs and output for
the recommendation value credibility (RVC).

Figure 2d illustrates the correlation between the inputs (Internal Similarity

(IS) and External Similarity (ES)) and output (Recommendation value credibil-
ity (RVC)). We notice that, if IS and ES vary from 0 to 0.4 and from 0 to 0.6
 A Fuzzy Logic Based Trust-ABAC Model for the Internet of Things 1165

respectively, then the RVC is between 0 and 0.4. Therefore, more the IS and ES
is very high (which means that the recommendation value is very similar to the
community to which it belongs and to the other communities), then the recom-
mendation value credibility is very high (which means that the recommendation
value is very accurate, so no risk of collusion attack). However, if IS and ES vary
from 0 to 0.4 and from 0.6 to 1 respectively, then the RVC is between 0.8 and
1. Therefore, more the IS is very high and ES is very low (which means that is
very similar to the community to which it belongs and completely different to
the other communities), then the recommendation value credibility is very low
(which means that the recommendation value is not accurate and the commu-
nity to which it belongs colludes to reach the same purpose and has carried out
a collusion attack).
To be more accurate in the evaluation of the trust level of the received recom-
mendations, we combine the recommendation value credibility with the degree
of the social relationship. Therefore, two inputs and an output parameters are
fuzzified which are the recommendation value credibility (RVC), the degree of
the social relationship (DSR) and the Trust level as illustrated in Fig. 3.

(a) Input 1: Recommendation value credibility (b) Input 2: Degree of the social relationship
(RVC) (DSR)

(c) Output: Trust Level (d) Correlation between inputs and output

Fig. 3. Fuzzy classes and membership functions for the trust level of recommendation
and the correlation between inputs and output.

Figure 3d depicts the correlation between the inputs (Recommendation value

credibility (RVC) and Degree of the social relationship (DSR)) and output (Trust
Level). This figure shows that if the RVC and the DSR vary from 0 to 0.4 and
from 0.8 to 1 respectively, then the trust is between 0.7 and 1. Therefore, more
the RVC and DSR are very high (which means that the recommendation value
is very accurate, and the sender of this recommendation having a high degree of
1166 H. Ouechtati et al.

the social relationship caused by the big number of transactions), then the Trust
Level of the recommendation is very high. Conversely, if any sender having a
low recommendation value credibility relative to other senders and having a low
degree of the social relationship caused by the limited number of transactions.
Then, the trust level of the recommendation sent by this object is very low.
In brief, the proposed Fuzzy logic-based trust evaluation consists of two main
phases: (i) the evaluation of the trust level of the received recommendations
and (ii) the detection of the collusion or singular attacks in order to filter the
inappropriate recommendations.

5 Performance Evaluation
The performance of the proposed Fuzzy logic based Trust-ABAC model was
evaluated using a fuzzy logic-based inference engine created over MATLAB. In
addition, we used MATLAB to randomly generate the received recommendation
values. We also conduct simulations for 100 s where each step takes around 1s.
Regarding the number of received recommendations, it varies from n = 10 to
n = 100. As shown in Table 1, we perform a series of experiments for different
percentages of false recommendations (P = 10%, 40%) in order to evaluate the
precision of the proposed solution in detecting the collusion attack.

Table 1. Performance of our proposed solution versus the collusion attack.

Number of recommendations Percentages of false recommendations Precision

n = 10 10% 96%
n = 10 40% 94%
n = 100 10% 95%
n = 100 40% 91%

The results obtained after performing such experiments show that our pro-
posed solution performs well to detect the misbehaving objects that provide a
wrong recommendation in order to decrease or increase the trust value of a vic-
tim object that requests access. As illustrated in Table 1, we observe in the worst
case, where 40% of objects send false recommendations, our proposed solution
is able to achieve a precision of around 94%. With the increase of the number of
received recommendations (n = 100), in the worst realistic scenario, when 40%
of objects send false recommendations, our proposed solution is able to keep
almost the same precision which, is around 91%. Therefore, it is obvious that
the proposed solution resists to the collusion attack and achieves a high precision
regardless the number of the received recommendations.
 A Fuzzy Logic Based Trust-ABAC Model for the Internet of Things 1167

6 Conclusion and Future Work

In this paper, a Fuzzy logic based Trust-ABAC model for the Internet of Things
was proposed. This latter is an improvement of our previous work Trust-ABAC
[3] by a new Fuzzy logic-based model in which we consider an evaluation of
trust based on recommendations and social relationships in order to enhance
the access control level in the IoT environment. The improvement that which we
presented consists of two main dimensions: (i) the evaluation of the trust level
of the recommendation message and (ii) the detection of the collision and sin-
gular attacks in order to filter the inappropriate recommendations. Simulations
prove the effectiveness of the proposed model to detect the collision and singular
attacks with high precision regardless the number of the received recommenda-
tions. As future work, our proposed model will be extended in order to be more
adaptive to better meet user’s needs by taking into account his context and his
behavior.

References
1. Palattella, M.R., Dohler, M., Grieco, A., Rizzo, G., Torsner, J., Engel, T., Ladid,
L.: Internet of things in the 5G era: enablers, architecture, and business models.
IEEE J. Sel. Areas Commun. 34(3), 510–527 (2016)
2. Akpakwu, G.A., Silva, B.J., Hancke, G.P., Abu-Mahfouz, A.M.: A survey on 5G
networks for the internet of things: communication technologies and challenges.
IEEE Access 5(12), 3619–3647 (2017)
3. Ouechtati, H., Azzouna, N.B.: Trust-ABAC towards an access control system for
the internet of things. In: International Conference on Green, Pervasive, and Cloud
Computing, pp. 75–89. Springer, Cham (2017)
4. Li, F.-H., Su, M., Shi, G.-Z., et al.: Research status and development trends of
access control model. Dianzi Xuebao (Acta Electronica Sinica) 40(4), 805–813
(2012)
5. Blackstock, M., Lea, R.: IoT interoperability: a hub-based approach. In: 2014 Inter-
national Conference on the Internet of Things (IOT), pp. 79–84. IEEE (2014)
6. Lampson, B.W.: Protection. In: 5th Princeton Symposium on Information Sciences
and Systems, vol. 8, pp. 18–24 (1974)
7. Sandhu, R.S.: Lattice-based access control models. IEEE Comput. 26, 9–19 (1993)
8. Sandhu, R.S., Coynek, E.J., Feinsteink, H.L., Youmank, C.E.: Role-based access
control models. IEEE Comput. 29, 38–47 (1996)
9. Damiani, M.L., Bertino, E., Catania, B., Perlasca, P.: GEO-RBAC: a spatially
aware RBAC. ACM Trans. Inf. Syst. Secur. 10, 1–34 (2006)
10. Feng, F., Lin, C., Peng, D., Li, J.: A trust and context based access control model
for distributed systems. In: The 10th IEEE International Conference on High Per-
formance Computing and Communications, pp. 629–634 (2008)
11. Dan, N., Hua-Ji, S., Yuan, C., Jia-Hu, G.: Attribute based access control (ABAC)-
based cross-domain access control in service-oriented architecture (SOA). In: Pro-
ceedings of the International Conference on Computer Science and Service System
(CSSS 2012) (2012)
12. Nitti, M., Girau, R., Atzori, L.: Trustworthiness management in the social internet
of things. IEEE Trans. Knowl. Data Eng. 26(5), 1253–1266 (2014)
1168 H. Ouechtati et al.

13. Sicari, S., Rizzardi, A., Grieco, L.A., et al.: Security, privacy and trust in internet
of things: the road ahead. Comput. Netw. 76, 146–164 (2015)
14. Saied, Y.B., Olivereau, A., Zeghlache, D., et al.: Trust management system design
for the internet of things: a context-aware and multi-service approach. Comput.
Secur. 39, 351–365 (2013)
15. Smari, W.W., Clemente, P., Lalande, J.-F.: An extended attribute based access
control model with trust and privacy: application to a collaborative crisis manage-
ment system. Future Gener. Comput. Syst. 31, 147–168 (2014)
16. Chen, D., Chang, G., Sun, D., et al.: Modeling access control for cyber-physical
systems using reputation. Comput. Electric. Eng. 38(5), 1088–1101 (2012)
17. Mahalle, P.N., Thakre, P.A., Prasad, N.R., Prasad, R.: A fuzzy approach to
trust based access control in internet of things. In: 2013 3rd International Con-
ference on Wireless Communications, Vehicular Technology, Information Theory
and Aerospace and Electronic Systems (VITAE), pp. 1–5. IEEE (2013)
18. Armando, A., Bezzi, M., Di Cerbo, F., et al.: Balancing trust and risk in access con-
trol. In: OTM Confederated International Conferences on the Move to Meaningful
Internet Systems, pp. 660–676. Springer, Cham (2015)
19. Xia, F., Yang, L.T., Wang, L., et al.: Internet of things. Int. J. Commun. Syst.
25(9), 1101–1102 (2012)
20. Douceur, J.R.: The sybil attack. In: International Workshop on Peer To Peer Sys-
tems. Springer, Berlin, Heidelberg (2002)
21. Bhuiyan, M.Z.A., Wu, J.: Collusion attack detection in networked systems. In:
Proceedings of the 14th IEEE International Conference on Dependable, Autonomic
and Secure Computing (IEEE DASC 2016), p. 18 (2016)
22. Atzori, L., Iera, A., Morabito, G., et al.: The social internet of things (SIoT) when
social networks meet the internet of things: concept, architecture and network
characterization. Comput. Netw. 56(16), 3594–3608 (2012)
23. Gong, S.: A collaborative filtering recommendation algorithm based on user clus-
tering and item clustering. JSW 5(7), 745–752 (2010)
24. Pham, M.C., Cao, Y., Klamma, R., et al.: A clustering approach for collaborative
filtering recommendation using social network analysis. J. UCS 17(4), 583–604
(2011)
 Decision Support System Using Trust
Planning Among Food-Energy-Water
Actors

Suleyman Uslu1 , Davinder Kaur1 , Samuel J. Rivera2 , Arjan Durresi1(B) ,

and Meghna Babbar-Sebens2
1
Indiana University-Purdue University Indianapolis, Indianapolis, IN, USA
{suslu,davikaur}@iu.edu, [email protected]
2
Oregon State University, Corvallis, OR, USA
{sammy.rivera,meghna}@oregonstate.edu

Abstract. In today’s world, it is challenging to find an application in

any field that does not necessitate or cannot benefit from a sophisticated
decision-making mechanism. This arises from the trend for more com-
plex approaches to problems and their functionalities. The process of
decision making becomes more demanding and complicated as the num-
ber of parameters is on the rise. One example of such a field that the
decisions have a major influence on a large number of people is Food-
Energy-Water where many parameters and stakeholders are included in
decision makings. In this paper, we presented a Decision Support System
using a Trust Management Framework which calculates and anticipates
trust among participants using their direct pairwise interactions and help
them reach consensus. Results showed that our system can detect unfair
ratings by participants and knowledge by an actor that would be widely
accepted by other stakeholders.

1 Introduction
Decision making has become a very significant process in many fields and appli-
cations. Its importance escalates as the results of the decisions have more dimen-
sions and essential effects on critical applications. Food-Energy-Water (FEW) is
one of the fields that require prudent decisions since the effects of the decisions
made on this field can have catastrophic results on the sustainability of natu-
ral resources. There are several ways that actors in the FEW fields can benefit
from an enhanced Decision Support System (DSS). First, it can democratize and
coordinate the development and evaluation of feedbacks. Second, it can help in
sharing and emphasizing approval of suggested solutions. Third, it can visualize
the consequences of suggested solutions on water, land, and energy resources.
In this paper, considering the recent decision making approaches [4,8,11–
13], we presented a Decision Support System which benefits from our measure-
ment theory-based Trust Management Framework [19]. This DSS is designed
for actors in FEW systems and gives them the opportunity to propose sophisti-
cated solutions, rate each other’s solutions, and rate each other’s ratings. Trust
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1169–1180, 2020.
https://doi.org/10.1007/978-3-030-15032-7_98
1170 S. Uslu et al.

is built between two actors based on their ratings to each other’s ratings which
we believe is a strong assumption for trust calculation. Results have shown that
human factor which is disclosed by trust management framework can play an
essential role in detecting unfair actors and actors having strong knowledge of
the field in FEW decision makings.
The paper is organized as follows: In Sect. 2, we present the background and
related work on decision making and trust. In Sect. 3, we present the details
of our models and algorithms of our Decision Support System using our Trust
Management Framework. In Sect. 4, we present the details of the simulations,
the scenarios, and the results. Then, we conclude the paper in Sect. 5.

2 Background and Related Work

In this section, we discuss previous work on decision making and trust.

2.1 Decision Making

Today’s most influential fields, such as environment, society, and economics

require critical decisions to be made. It is very crucial to have strong decision-
making mechanisms in these fields. There are events which consequently resulted
in huge losses such as Global Financial Crisis in 2008 and BP oil spill in the Gulf
of Mexico in 2010. Such events can be considered as the impacts of decisions that
are becoming more complex as the world itself is becoming a more complicated
place. One person is not making them; instead, multiple stakeholders are involved
in the decision making. These stakeholders can be from a different background
or even have conflicting interests [8].
There can be several reasons for decision failures as mentioned in [8]. More-
croft [11] claims that one reason is reductionist thinking. When a problem is
approached simplistically and divided into smaller pieces, information which
can be critical and reside in the connections of these smaller pieces can be lost.
He also mentions other possible reasons such as incomplete or uncertain infor-
mation, quick fixes, narrowing goals, and culture and tradition.
Kambiz [8] claims that today’s decisions are made by multiple stakeholders
due to the complexity level of the problem. Besides, none of the individuals
who are possible candidates to make the decision can decide on their own or
have all the information required for the decision making. Some other important
aspects of multi-stakeholder decision making are the desire of stakeholders to
reach the best outcome, their willingness of learning and participation in the
process. There are also several other approaches for participant conversation
and communication where multiple participants are involved such as interviews,
surveys, focus groups, scenario analysis, stakeholder workshops, and role-playing
games [5].
Dong et al. [4] claim that as the stakeholders might have conflicting or com-
peting interests, they can also be experts in the field participating in the decision
 Decision Support System Using Trust Planning 1171

making the process by expressing their opinions. Therefore, the aim of the par-
ticipants can be considered as an attempt to reach a consensus regarding the
solution to the problem. This attempt can be seen as an iterative and dynamic
mechanism that can have various rounds which include expression, discussion,
and modification of ideas until the participants reach an agreement. Dong
et al. [4] proposed an approach to minimize the alterations of the proposed
solutions by the experts at each round.
Babbar-Sebens et al. [3] proposed innovative designs such that a computer
optimization algorithm can make use of the feedback from the participants.

2.2 Trust
Trust is a widely used but a context-dependent concept which can influence
decision making either directly or indirectly [20–22]. Some of the difficulties of
using trust in decision making are that there can be a large number of stake-
holders and historical calculations may be required because of the nature of the
trust which can deviate over time. Therefore, the measurement of trust needs to
be computerized. There are several approaches for measuring the trust among
participants [7,10,14]. Ruan et al. [19] proposed a measurement theory-based
Trust Management Framework to measure and anticipate trust among entities
in a community. There are several applications of this framework such as stock
market forecast using Twitter data and trust management in cloud and Internet
of Things [15–18,24,25].

2.3 Decision Making Using Trust

There are several studies using trust in decision making. Jøsang [6] proposed
methods for trust-based decision making for transactions over the internet. Kim
et al. [9] introduced a trust-based consumer decision-making model to be used in
e-commerce. Alfantoukh et al. [1,2] proposed a model for decision making among
multiple stakeholders in water allocation systems and also a generic framework
for consensus reaching using a trust. We proposed a Decision Support System
integrated with Trust Management Framework and measured trust among actors
of FEW systems where they are sharing a limited amount of natural resource
and showed that when stakeholders acknowledge trust and act according to that,
they can reach consensus in a fewer number of rounds in decision making [23].

3 Trust-Based Decision Support System

In this paper, we used the Trust Management Framework proposed in [19] to
measure trust among participants and reduce the number of rounds of nego-
tiation using realistic scenarios which are not only based on sharing a natural
resource but deciding several interrelated parameters in FEW. In this section,
we explain the multi-parameter solutions and algorithms for rating and trust
measurement.
1172 S. Uslu et al.

3.1 Solutions

A solution is a proposal offered by a participant during the decision making

which can be considered as a negotiation among the participants to reach a
consensus. Sharing a finite amount of a resource such as water could be a simple
case for negotiation among actors in FEW. In [23], we proposed a trust-based
decision-making algorithm where actors negotiate for a split of a finite amount
of a resource.
In more advanced scenarios, the negotiation is not constrained by one
resource; instead, there are other parameters which can influence the actors’
interests and therefore their proposals. In our model, we decided to include
four parameters: groundwater, surface water, crop choice, and fertilizer choice.
Groundwater and surface water parameters can have continuous values in [0, 100]
whereas crop choice, and fertilizer choice can be discrete integer values in [1, 3].
In our model, the number of actors in the decision making is 5. Considering
the 4 parameter mentioned, a solution will provide total of 20 values as shown
in Table 1. Solutions were generated using a genetic algorithm. Then, a clus-
tering algorithm is used to cluster generated solutions into the desired number
of clusters. Therefore, besides restricting the number of solutions, we generated
solutions which are distinct from each other with some distance to better repre-
sent possible proposals by the actors.

Table 1. Sample solution for 5 actors with 4 parameters

Parameters Actor 1 Actor 2 Actor 3 Actor 4 Actor 5

Ground water 11.4798125 35.5564315 12.8204529 21.5142238 10.1051691
Surface water 42.8265841 12.7084777 13.3468875 16.9873113 4.7743236
Crop choice 2 3 2 2 1
Fertilizer choice 2 1 3 1 1

3.2 Algorithms

Algorithms section consists of two subsections which are rating functions and the
measurement of trust. They are explained in detail in the subsequent subsections.

3.2.1 Rating Functions

Rating is the second step in a round after participants propose their solutions.
Each actor attending the negotiation rates each other’s solutions. Actors can
rate solutions based on their own merits; however, we decided to use a predefined
function for normal cases. As shown in Eq. 1, participant A rates participant B’s
solution in round i based on the ratio of his profit in B’s solution to his profit
in his first proposal. riAB is the rating of A to B’s solution in round i, P roBA
i
 Decision Support System Using Trust Planning 1173

is the profit of A in B’s solution in round i, and P roAA

1 is the profit of A in his
first solution.
P roBA
i
riAB = (1)
P roAA
1
The next step after each actor rates other actors’ solutions is the rating of
ratings. Similarly, actors are free to rate others’ ratings; however, we used the
ratio of their actual rating to their expected rating. Equation 2 shows the rating
of actor C to the rating of A to B’s solution in round i where ri∗AB is the actual
and riAB is the expected rating.

ri AB ri∗AB
rrC = (2)
riAB

Since the ratings are calculated as a ratio, they sometimes exceed 1 when the
person whose rating is being rated is very generous. Therefore, the rating values
are limited to 1 to keep the ratings in [0, 1].

3.2.2 Trust Calculation

Flexible, modular, and highly adaptable Trust Frameworks, such as [19], require
trust definition from the user. Since trust is defined between two entities, the
best approach for input for trust measurement would be using the interactions
between these entities. For example, Ruan et al. [17] used tweets from one user
to another. In [23], we previously proposed to use solution ratings from one
participant to another for trust measurement. In this paper, we improved the
input for trust measurement by introducing the notion of rating of ratings. In
this version, actors rate each other’s solution as usual. Then, they rate each
other’s ratings. The reason we introduced rating of ratings is that it is a more
direct interaction from one actor to another compared to the rating of solutions.
We believe that measuring trust using a more direct interaction or engagement
would increase the credibility of trust measurements.
At each decision making, there are multiple rounds in which actors interact
with each other where each interaction is considered as one measurement for
trust. At each round, stakeholders rate each other’s proposals. Therefore, there
are n − 1 ratings given to a stakeholder’s solution since they do not rate their
solutions, where n is the number of stakeholders similarly, a stakeholder rates n−
1 solutions. Considering rating of ratings, a rating is rated by n − 1 stakeholders
since the owner of the rating does not rate his rating. As a result, each stakeholder
interacts with n − 1 other stakeholders, and there is also n − 1 unidirectional
interaction between every pair of stakeholder at each round.
When calculating the impression from one stakeholder to another, we take the
average of ratings of ratings which are the measurements for trust. As can be seen
in Eq. 3, impression from stakeholder A to stakeholder B is the average of all rating
values of A to all ratings given by B, where riAr is a rating of A to a rating of B
B

and N is the total number of aforementioned ratings of ratings. After calculating

1174 S. Uslu et al.

impression, denoted as m, confidence, represented as c, is calculated. As shown in

Eq. 4, confidence is inversely related to the standard error of the mean.

N
riAr
B

i=1
mAB = (3)
N


 N
 (mAB − riAr )2
B

 i=1
cAB = 1 − 2e where e = (4)
N (N − 1)
Impression m and confidence c is considered as the parameters of the trust
between two stakeholders. Also, to trust parameters, Trust Management Frame-
work proposed by Ruan et al. [19] can infer trust between stakeholders who have
not had any interaction yet. The framework assigns default trust values which
are 0.5 for both m and c at the beginning. Using inference methods, which are
transitivity and aggregation, we can anticipate trust between two stakeholders
who do not participate in the same decision-making process.
There are several transitivity, and aggregation formulas proposed in [19].
We selected TP1 which is based on the multiplication of two impression values.
Transitive impression mSD SD
T and transitive error eT from source S to destination
D can be calculated as shown in Eqs. 5 and 6.
mSD
T = mST ⊗ mT D = mST mT D (5)


eSD
T = eST ⊗ eT D = (eST )2 (mT D )2 + (eT D )2 (mST )2 (6)
For aggregation, we used the first operator AP1 in [19] which is based on
average values of two impression values. Equations 7 and 8 show the calculation
of aggregated impression value mSD SD
T1 and its error eT1 from source entity S to
destination D.
mSD
T1 + mT2
SD
mSD
T1 ⊕ m SD
T2 = (7)
2

SD SD 1
eT1 ⊕ eT2 = ((eSD )2 + (eSD 2
T2 ) ) (8)
22 T1

4 Test and Results

In this section, we discuss the details of the simulation, scenarios and results.

4.1 Simulations
As a naive approach, we explained the simulations and presented the results of
decision-making scenarios where stakeholders share a natural resource in [23]. In
this paper, they are not only sharing a resource but propose solutions regarding
four different parameters for all participants which are discussed in Sect. 3.1.
 Decision Support System Using Trust Planning 1175

Each stakeholder can gain different profits from different solutions. Crop
choice which directly affects sales prices also results in different amounts of yield
for a different amount of irrigation. In addition to crop type, fertilizer choice
also affects the amount of yield which can also be considered as a function
of irrigation. Also, similar to crop choice, each fertilizer has different costs to
farmers.
To calculate the profit for each actor in FEW sectors, we first calculate the
revenue of a farmer based on the crop sale as shown in Eq. 9 where M Y denotes
Max Yield and P M Y denotes Percent of Max Yield. Also, C denotes crop type,
SW and GW denote surface and groundwater amount respectively, and SPcrop
denotes sales price of the crop. Then, we calculate costs which are fertilizer cost
and water cost for each water type, SW and GW , in Eq. 10 which differ for
each farmer and are shown in Eqs. 11 and 12. Lastly, profit is calculated by
subtracting the cost from the revenue as in Eq. 13.

Revenue = M Y (C, SW + GW ) ∗ P M Y (SW + GW ) ∗ SPcrop (9)

Cost = Costf ertilizer (SW + GW ) + CostSW + CostGW (10)

CostSW = [1, 0.5, 0.2, 0.1, 0.05] (11)

CostGW = [0.1, 0.2, 0.12, 1.2, 1] (12)

P rof it = Revenue − Cost (13)

After profits of farmers for each solution is calculated, they are normalized
to [0, 1]. Then, solutions are sorted based on a weighted average of profits where
each stakeholder considers their profit having 60% weight and every other actors’
profits having 10% weight. Table 2 shows the first four lines of sorted solution list
for Actor 1 where P roAn is the profit of Actor n, Wavg is the weighted average
of profits, and Ravg is the presumed average ratings of other actors.

Table 2. Normalized profits of solutions sorted for Actor 1

P roA1 P roA2 P roA3 P roA4 P roA5 Wavg Ravg

1.000 0.110 0.249 0.592 0.019 0.697 0.243
0.736 0.501 0.426 0.416 0.701 0.646 0.511
0.754 0.451 0.010 0.942 0.516 0.644 0.480
0.782 0.681 0.022 0.694 0.044 0.613 0.360

Participants start with proposing the first solution which has the maximum
weighted averaged profit. At each round, they select and propose another solution
1176 S. Uslu et al.

considering a negotiation rule which states that they cannot propose a solution
that has lower equally averaged profit compared to the solution they proposed
in the previous round. This rule is implemented to establish a meaningful nego-
tiation.
After actors propose their solutions, the next step is that each actor rate
every other actor’s solution. They rate a solution based on how much they are
satisfied with their profit in the solution compared to their profit in their first
proposal as explained in Sect. 3.2.1. After rating solutions, participants rate the
ratings of other stakeholders based on Eq. 2 which is based on the fairness of the
rating that uses the rating formula in Eq. 1. We implemented stopping condition
of the decision making in such a way that when every participant proposes the
same solution, not necessarily in the same round, the negotiation ends and the
solution that is proposed by everyone is accepted.
The rules for solution selection from the solution pool, rating of solutions, and
rating of ratings are for normal participants. In a perfect scenario, all stakehold-
ers behave according to those rules and reach a consensus. Scenario 1 is simulated
to show the results of a decision making where all 5 participants behave nor-
mally. Furthermore, we investigated when there is an actor who behaves in an
unfair way to keep his profit high in Scenario 2. Lastly, we presented Scenario
3 where there is a stakeholder who is rating other’s solutions and ratings low
based on his knowledge on the field such as an environmental effect of a solution.

4.2 Scenarios
We run our simulation in 3 different scenarios. Our Trust Management Frame-
work measured trust between each stakeholder pair at each round. A stakeholder
rates n − 1 stakeholders and gets n − 1 ratings for each of his ratings. There-
fore, there is n − 1 unidirectional input from other stakeholders which are all
used for trust calculation at each round. We also believe that trust needs to
be expressed directly between participants; therefore, we used rating of ratings
instead of rating of solutions to measure trust.

4.2.1 Scenario 1
In this scenario, all stakeholders follow the rules defined for a normal player.
They select and propose the next solution where there is an improvement for
the average profit of other stakeholders even though their profit decreases. Since
all stakeholders followed the rules, their expected profit from their proposed
solutions decreased until they stop proposing a new solution from the pool.
However, since the average profit of other stakeholders increased, they received
higher ratings. Also, they received perfect rating of ratings since they exactly
used the defined formula to rate a solution. This resulted in increasing trust
values for everyone as shown in Fig. 1 where An denotes Actor n. Since they
have a history of trust, the increase became gradual, and the effect of new ratings
decreased as the history was building up and their trust was getting stronger.
 Decision Support System Using Trust Planning 1177

Fig. 1. Global trust values increase for each stakeholder when they rate each other
fairly and do not stop negotiating prematurely

4.2.2 Scenario 2
In this scenario, stakeholder 4 stopped earlier than where he was supposed to,
which means he insisted on some solution to keep his profit higher. In his last
solution, profit of stakeholder 3 was significantly lower than the average profit
of stakeholders in the solutions. Therefore, stakeholder 3 rated this solution low.
In response to his low rating, Stakeholder 4 rated his solution low too and also
his rating. However, since the community believed that stakeholder three was
right downvoting the stakeholder 4, they were on the side of stakeholder 3 and
rated his rating high compared to stakeholder 4. This resulted in a decrease
in the trust of stakeholder four as shown in Fig. 2. Since the community mostly

Fig. 2. The participant who is rating unfairly is detected by the trust framework using
the feedback from the community which is ratings of ratings
1178 S. Uslu et al.

consisted of fair people, our trust model was able to identify the person behaving
unfairly using ratings of ratings among participants.

4.2.3 Scenario 3
In this scenario, stakeholder 1 has some knowledge regarding the solutions and
both proposes and rates the solutions based on his knowledge. When all stake-
holders proposed their solutions, he downvoted solutions which he believes have
some hidden negative effect and should not be chosen. After the discussion among
stakeholders, others agreed with the stakeholder 1 and did not rate his ratings
low. This resulted in a gradual increase in his trust instead of a decrease. Since
other participants did not rate each other’s solutions low, their ratings were
downvoted by stakeholder 1. This resulted in a slowdown in the increase of their
trust as shown in Fig. 3.

Fig. 3. When the participant who is giving lower ratings than expected rating based
on his knowledge is able to convince the community, he does not lose the trust of the
community

5 Conclusion
In this paper, we proposed a decision support system using a trust manage-
ment framework and simulated participants’ behaviors where they select a multi-
variable solution from solution pool, propose it, rate solutions from others, and
then rate each other’s ratings at each round. Our trust framework was able to
successfully measure the trust among participants, detect unfair behaviors, and
strengthen the trust of a person that community agrees within three different
scenarios using the feedback from the community.
 Decision Support System Using Trust Planning 1179

Acknowledgements. This work was partially supported by the National Science

Foundation under Grant No. 1547411 and by the U.S. Department of Agriculture
(USDA) National Institute of Food and Agriculture (NIFA) (Award Number 2017-
67003-26057) via an interagency partnership between USDA-NIFA and the National
Science Foundation (NSF) on the research program Innovations at the Nexus of Food,
Energy and Water Systems.

References
1. Alfantoukh, L., Ruan, Y., Durresi, A.: Trust-based multi-stakeholder decision mak-
ing in water allocation system. In: International Conference on Broadband and
Wireless Computing, Communication and Applications, pp. 314–327. Springer
(2017)
2. Alfantoukh, L., Ruan, Y., Durresi, A.: Multi-stakeholder consensus decision-
making framework based on trust: a generic framework. In: 2018 IEEE 4th Interna-
tional Conference on Collaboration and Internet Computing (CIC), pp. 472–479.
IEEE (2018)
3. Babbar-Sebens, M., Minsker, B.S.: Interactive genetic algorithm with mixed ini-
tiative interaction for multi-criteria ground water monitoring design. Appl. Soft
Comput. 12(1), 182–195 (2012)
4. Dong, Y., Xu, J.: Consensus Building in Group Decision Making. Springer (2016)
5. Hamilton, S.H., ElSawah, S., Guillaume, J.H., Jakeman, A.J., Pierce, S.A.: Inte-
grated assessment and modelling: overview and synthesis of salient dimensions.
Environ. Model. Softw. 64, 215–229 (2015)
6. Jøsang, A.: Trust-based decision making for electronic transactions. In: Proceedings
of the fourth Nordic Workshop on Secure Computer Systems (NORDSEC 1999),
pp. 496–502 (1999)
7. Jøsang, A., Ažderska, T., Marsh, S.: Trust transitivity and conditional belief
reasoning. In: IFIP International Conference on Trust Management, pp. 68–83.
Springer (2012)
8. Kambiz, M.: Multi-Stakeholder Decision Making for Complex Problems: A Systems
Thinking Approach with Cases. World Scientific (2016)
9. Kim, D.J., Ferrin, D.L., Rao, H.R.: A trust-based consumer decision-making model
in electronic commerce: the role of trust, perceived risk, and their antecedents.
Decis. Support Syst. 44(2), 544–564 (2008)
10. Kim, Y.A., Song, H.S.: Strategies for predicting local trust based on trust propa-
gation in social networks. Knowl.-Based Syst. 24(8), 1360–1371 (2011)
11. Morecroft, J.D.: System dynamics: portraying bounded rationality. Omega 11(2),
131–142 (1983)
12. Morecroft, J.D.: A systems perspective on material requirements planning. Decis.
Sci. 14(1), 1–18 (1983)
13. Morecroft, J.D., Larsen, E.R., Lomi, A., Ginsberg, A.: The dynamics of resource
sharing: a metaphorical model. Syst. Dyn. Rev. 11(4), 289–309 (1995)
14. Ruan, Y., Durresi, A.: A survey of trust management systems for online social
communities-trust modeling, trust inference and attacks. Knowl.-Based Syst. 106,
150–163 (2016)
15. Ruan, Y., Durresi, A.: A trust management framework for cloud computing plat-
forms. In: 2017 IEEE 31st International Conference on Advanced Information Net-
working and Applications (AINA), pp. 1146–1153. IEEE (2017)
1180 S. Uslu et al.

16. Ruan, Y., Durresi, A., Alfantoukh, L.: Trust management framework for internet
of things. In: 2016 IEEE 30th International Conference on Advanced Information
Networking and Applications (AINA), pp. 1013–1019. IEEE (2016)
17. Ruan, Y., Durresi, A., Alfantoukh, L.: Using Twitter trust network for stock market
analysis. Knowl.-Based Syst. 145, 207–218 (2018)
18. Ruan, Y., Durresi, A., Uslu, S.: Trust assessment for internet of things in multi-
access edge computing. In: 2018 IEEE 32nd International Conference on Advanced
Information Networking and Applications (AINA), pp. 1155–1161. IEEE (2018)
19. Ruan, Y., Zhang, P., Alfantoukh, L., Durresi, A.: Measurement theory-based trust
management framework for online social communities. ACM Trans. Internet Tech-
nol. (TOIT) 17(2), 16 (2017)
20. Shambour, Q., Lu, J.: A hybrid trust-enhanced collaborative filtering recommen-
dation approach for personalized government-to-business e-services. Int. J. Intell.
Syst. 26(9), 814–843 (2011)
21. Shambour, Q., Lu, J.: A trust-semantic fusion-based recommendation approach
for e-business applications. Decis. Support Syst. 54(1), 768–780 (2012)
22. Sutcliffe, A.G., Wang, D., Dunbar, R.I.: Modelling the role of trust in social rela-
tionships. ACM Trans. Internet Technol. (TOIT) 15(4), 16 (2015)
23. Uslu, S., Ruan, Y., Durresi, A.: Trust-based decision support system for plan-
ning among food-energy-water actors. In: Conference on Complex, Intelligent, and
Software Intensive Systems, pp. 440–451. Springer (2018)
24. Zhang, P., Durresi, A., Barolli, L.: Survey of trust management on various net-
works. In: 2011 International Conference on Complex, Intelligent and Software
Intensive Systems (CISIS), pp. 219–226. IEEE (2011)
25. Zhang, P., Durresi, A., Ruan, Y., Durresi, M.: Trust based security mechanisms for
social networks. In: 2012 Seventh International Conference on Broadband, Wire-
less Computing, Communication and Applications (BWCCA), pp. 264–270. IEEE
(2012)
 iHome: Smart Home Management
as a Service in the Cloud and the Fog

George Myrizakis and Euripides G. M. Petrakis(B)

School of Electrical and Computer Engineering, Technical University of Crete (TUC),

Chania, Crete, Greece
[email protected], [email protected]

Abstract. We present iHome, a smart home management service in the

cloud. The service addresses the need of users to monitor and control
their homes remotely provided that the home devices are “smart” them-
selves (i.e. they can be connected to the internet and operated remotely).
Home devices transmit their identifier, measurements and status to a
fog node and from there to the cloud. To mitigate concerns in regards
to data protection, communication delays in delivering large amounts of
data to the cloud, all services for the home are realized within a fog node
installed at home. This information becomes available to registered users
in the cloud based on subscriptions (i.e. to users authorized to review
and respond to this information). User access rights are defined based
on user roles (i.e. cloud administrators, home moderators and residents).
Besides data publication and subscription services, an innovative fea-
ture of iHome, is a rule-based event management service which forwards
alerts to subscribed users for responding to critical events (i.e. incidents
of fire, malfunctioning appliances at home). iHome is implemented based
on principles of Service Oriented Architecture design as a composition
of RESTful services.

1 Introduction

Smart homes are the building blocks of smart cities. E-government, Green devel-
opment and Energy Management policies for large residential areas may be bene-
fit from the evolution of smart home technologies. Smart policies and solutions at
smart home level may influence policies applying at a larger scale (i.e. smart city
level) and the reverse: Smart city policies may drive the development of smart
home solutions for more efficient resource management (e.g. water, energy, inter-
net etc.). Engineering and architectural solutions for smart homes are reviewed
in [3,5].
The idea of Internet of Things (IoT) combined with cloud computing, opens
new horizons in the field of smart home technology. Smart home systems are now
designed as two-fold solutions involving two interoperable and interconnected
components: The IoT side (front-end) to support the network of home devices
and their connection with the Internet (via some gateway or edge node) and,

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1181–1192, 2020.
https://doi.org/10.1007/978-3-030-15032-7_99
1182 G. Myrizakis and E. G. M. Petrakis

the cloud side (back-end) where home control functionality is implemented and
where data are collected for permanent storage and analysis.
The use of sensors installed in smart appliances and their capability for Inter-
net connectivity provides significant benefits in applications areas that require
fast and continuous monitoring from anywhere. In real-life smart home and smart
city applications, huge amounts of data are collected and analyzed (e.g. improved
home management solutions or better business policies can be designed based on
the results of this analysis). These solutions have to be scalable (to deal with the
ever-increasing number of smart homes and users, and finally, with the increased
size of data), cost-effective, respond within reasonable time (e.g. considering the
time constraints of each application) and, address concerns related to users pri-
vacy and data safety. While smart home solutions are deployed for the home
and the cloud, security concerns arise at both places. Although cloud systems
are considered to be more secure for deploying IoT applications, users and data
are exposed to many risks as IoT at homes operates in the periphery of the
cloud, it is open to many users and, as such, it is generally less protected than
the cloud itself. The security concerns for the front-end part are addressed in
[8]. In order to guarantee security at the back-end, data must be transferred and
stored securely to the back-end, user activities and system operations must be
monitored at all times and, access to data and services must be allowed based
on user roles and authorization [7].
The cloud is the ideal environment for implementing the back-end component
of smart home applications due to reasons related to its affordability (no up-front
investment, low operation costs), ease of deployment (taking advantage of IaaS
and PaaS solutions already available in the market by many cloud vendors), low
maintenance costs (i.e. easy software updates and enhancements), scalability (i.e.
computational resources can be added on demand) and accessibility (i.e. smart
home services can be accessed anytime from anywhere over the Web).
Cloud computing has some inherent disadvantages and limitations. Even
though the cloud may offer virtually unlimited computational resources, inter-
net bandwidth may impede application performance or, the cost of transferring
and processing large amounts of data to the cloud call for hosting smart home
services at the front-end side (rather than hosting all of them at the back-end
side) or, closer to the place where data is generated (e.g. a server operating at
home). The distance between the front and back-end is also the main reason
for long delays in responding to critical events occurring at home. To address
these limitations, the paradigm of fog computing emerged lately, starting from
Cisco [2]. Fog computing can be assumed as an extension of cloud computing,
bringing virtualized services closer to the edge of the network (e.g. closer to
IoT devices). Fog brings the benefits of low latency (due to its close proximity
with the IoT devices), location awareness and increased security, privacy and
availability. Efforts to standardize architectures and platforms for extending the
cloud to support functional edge nodes are currently underway [4].
 iHome 1183

The concept of smart home is interesting and trendy1 . Existing solutions

relying on the idea sensors and appliances connected to the Internet are now
becoming available as commercial services2 supporting functionalities ranging
from home automation, intelligent home control, ambient or assisted living and
health monitoring, to security control (e.g. anti-burglary systems). Off-the shelve
smart devices with the desired functionality are currently becoming available in
the market at affordable prices and can be integrated into smart home applica-
tions. An important feature of all systems (and also of iHome) is user interaction
with a mobile device and intelligent user interfaces (using e.g. voice interfaces)
over the Internet. Commercial applications promise fast responses, high avail-
ability and, (most important) high reliability. iHome in not intended to compete
commercial solutions in terms of performance but rather, to show how a cost
effective smart home system based on innovative services can be designed and
deployed in the cloud and fog using well established, open-source technologies
and principles of service oriented design.
iHome, is a service that runs in the cloud and the fog and implements a
typical smart home scenario: Smart devices installed at home can transmit mea-
surements or status information to gateways and from there to the cloud. To
mitigate concerns in regards to data security and in regards to delays in deliv-
ering large amounts of data to the cloud, services for the home are realized
within a fog node installed at home. Access to this information is allowed only
for users registered to iHome in the cloud based on their role and authorization.
(i.e. cloud administrators, home administrators, residents). Permission to access
homes is granted by cloud administrators. Home users (i.e. home administrators
and residents) may access home information based on subscriptions. Besides
measurements, alerts are a special type of information which is generated when
events take place. These can be simple events (e.g. room temperature exceeds a
threshold) or critical events (e.g. incidents of security breach or fire). The events
are described in terms of rules involving measurements and threshold values.
The analysis of history (log) data collected from a large number of homes
(eventually this information becomes big) can lead to important conclusions in
relation to people habits and behavior, their needs, the causes of events or, in
relation to energy consumption at peak and non-peak hours. The analysis of
this information may provide the means for policy makers, energy managers to
improve their plan for more efficient, safer and profitable management of a smart
city.
iHome solution, is based on smart devices for the home IoT (front-end or fog)
and on a composition of RESTful micro-services for the cloud side (back-end).
Because a large number of smart devices for the front-end is not available to
us, we decided to rely on software simulating the operation of smart devices at
homes. This allowed us to shift focus from device specific functionality (e.g. IoT
transmission protocols and vendor specific device functionality), to the design
and implementation of the front and back-end solutions. We run an exhaustive

1
https://www.qulix.com/industries/internet-of-things-apps/smart-home/.
2
https://thinkmobiles.com/blog/best-smart-home-apps/.
1184 G. Myrizakis and E. G. M. Petrakis

set of experiments using simulated (but realistic) data aiming to evaluate both,
iHome response time and scalability. The results demonstrate that the system
can handle up to a large number of users and data in real time.
The rest of this papers is organized as follows: Issues related system design
and its implementation are discussed in Sect. 2. Evaluation results are presented
in Sect. 3 followed by conclusions and issues for future research in Sect. 4.

2 Design and Architecture

iHome is a novel Future Internet (FI) service for data collection from IoT devices
in an automatic, generalized and modular way. Building upon principles of Ser-
vice Oriented Architectures (SOA) design and driven by the key requirements
of today’s IoT systems for adaptability, low-cost and scalability, iHome archi-
tecture is modular and expandable. iHome is implemented in OpenStack and
FIWARE3 , an open-source distributed cloud infrastructure funded by the EU.
iHome encompasses IoT-A [1,10] design principles in an attempt to develop
an innovative IoT platform that supports generic services and IoT devices (i.e.
independent of connectivity and not coupled to specific IoT protocols). IoT-A
proposes an Architecture Reference Model (ARM) defining the principles and
guidelines for generating IoT architectures, providing the means to connect ver-
tically closed systems both, at the communication layer (i.e. where IoT devices
interact with the system) and the service layer (i.e. where services are provided).
We followed a valid design approach that identified the functional and non-
functional requirements of the system and specifically, (a) the functional com-
ponents and their interaction, (b) the information that is managed and how
this information is acquired, transmitted, stored and analysed, (c) the physical
software entities that support the functional and information activities, (d) the
requirements for assuring data, network and user security and privacy. iHome
(reference) architecture is described by a set of UML diagrams [6] including
(a) information (class) diagram describing information that is handled by the
system, (b) deployment diagram showing interaction of system components, (c)
activity diagram describing flowcharts for several types of user actions the most
important of them being, system login request (user authentication and autho-
rization), request for new account, request to access a device, event handling
and, (d) an architecture diagram.

2.1 User Groups

The following user groups and the functional requirements associated with each
group are identified:
Cloud Moderators (Infrastructure Administrators): Except their com-
petence to providing cloud services, they are responsible for providing services

3
https://www.fiware.org.
 iHome 1185

to users subscribing to iHome. In particular, they are responsible for perform-

ing Create, Read, Update, Delete (CRUD) operations on (a) homes (i.e. they
can register new homes to the system), (b) they can register new home users
(i.e. home administrators and residents) and define their access rights (i.e for
associating residents and devices to homes or rooms).
Home Moderators (Home Administrators): They are the managers of the
home: (a) They configure, maintain and monitor iHome services, (b) they are
responsible for performing CRUD operations on rooms, devices and events, (c)
they are responsible for granting access rights to residents and of monitoring
their operations, (d) they have access to status information for rooms, home
devices and events; (e) they have access to history (log) data, (f) they define
rules for handling events.
Residents: They are entitled to accessing information pertinent to their role
and perform read (only) operations on information regarding devices and r rooms
they are subscribed and based on access rights assigned to them by the home
administrator.

2.2 Architecture

iHome architecture is modular, expandable and generic (i.e. it can be adapted

to diverse use case scenarios). The implementation relies on modular services
that run in the cloud and implement fundamental functionalities. They offer
important benefits such as, connectivity of smart devices, scalability, openness,
re-usability, multi-tenancy, increased accessibility and security. Some of the ser-
vice modules foreseen are available online as re-usable Generic Enablers (GEs)
on FIWARE catalogue4 . All others have been developed by the authors. Each
module is designed as a RESTful service [9] which is able to interact with other
RESTful services over HTTPS. This design highlights significant advantages that
include (in addition to the comprehensive structure and re-usability of services),
easy replacement of modules and flexible system configuration in order to adapt
to future needs of the application. Figure 1 illustrates the iHome solution with
its two system nodules namely, front-end and back-end respectively.

2.3 Front-End

In addition to being a gateway (whose role is to support real-time IoT data

collection from home devices and their communication with the cloud) it imple-
ments additional services that transform the gateway to a fog node. In iHome
set-up, a home is a composition of any number of rooms with devices installed
in each room (i.e. each device is associated with a room). iHome implements
also device simulators producing temperature, humidity and luminosity values.
In addition, a room or device accepts change of mode of operation commands

4
https://www.fiware.org/developers/catalogue/.
1186 G. Myrizakis and E. G. M. Petrakis

Fig. 1. iHome architecture

(i.e. on/off, heating/cooling etc.) and read status commands (i.e. normal or mal-
function). The communication between services and between the front and back-
end is implemented in HTTPS. The front-end is implemented as a composition
of RESTful services:
Data Collector: Accommodates any kind of sensor protocol (e.g. Bluetooth,
Zigbee) that can be supported by smart devices and collects measurements from
all home devices. It implements a two-way communication with home devices;
it decodes and decrypts device physical payloads (i.e. physical devices transmit
encrypted data); translates physical device payloads in JSON format; it transfers
control commands received from authorized users (e.g. turn on/off commands,
change mode of operation commands etc.) to specific devices. It is the only
component that is affected by the property of a device to apply a specific protocol
(e.g. Bluetooth). The rest of the system is protocol agnostic since all data are
communicated and processed in JSON.
Database (DB): It implements a temporary storage for device data and users.
Device data are transmitted to the cloud at predefined time intervals (i.e. every
minute). In the mean time, data are stored in this temporary storage. In cases of
Internet loss, data are cached and transmitted to the back-end when connection
is up again. The identifiers of all authorized users (i.e. home moderators and
residents) are stored as well. An authorized user has received an OAuth2.0 key
(“token”) encoding user roles which is generated at the back-end (see Sect. 2.4).
The SSL certificate (i.e. asymmetric key) is also stored. It is used to establish an
encrypted HTTPS connection between the front and the back-end. It is used to
create the session symmetric key of an HTTPS session. Each time a new HTTPS
session between the front and the back-end is initiated, a new symmetric key is
generated.
 iHome 1187

Connectivity Service: It implements the HTTPS connection between the

front-end and the cloud. Data forwarded to the cloud are encrypted using the
symmetric SSL key (or decrypted upon receipt). Similarly, data received at the
back-end are decrypted with the symmetric key (or encrypted prior to transmis-
sion).
User authorization: Services running at both system ends are protected by
an OAuth2.0 mechanism5 . Users receive an OAuth2.0 token upon login at the
back-end (see Sect. 2.4). This token encodes the users’ role and authorization to
access other services. In iHome, the token of a user who is authorized to access a
home (with the role of home moderator or resident) is added to the database. If a
user issues a request to access a service or device at the front-end, the OAuth2.0
token is added to the header of the request. User Authorization service at the
front-end compares this token with the ones stored in the database. If there is a
match, the request is executed (i.e. it is forwarded to the device specified in the
request; otherwise the request is aborted.

2.4 Back-End
The back-end is a composition of autonomous RESTful services communicat-
ing with each other using HTTPS. Individual services or groups of services are
deployed in the same or different Virtual Machines (VMs) in the same or in a
different cloud. Network delays are expected due to the nature of this design.
However, the experimental results (Sect. 3) demonstrate that iHome is capable
for responding in real time even under heavy workloads. Access to data and
services is protected by an OAuth2.0 mechanism. iHome back-end implements
the following services:
Connectivity Service: In line with connectivity service running in the front-
end, it implements secure communication between the front and back-end
(decoded data are formatted in JSON).
User identity management: Implements (a) identification services for users
and manages their profiles, (b) authorization services supporting access control
based on user roles and access policies. It is provided by Keyrock Identify Man-
agement (IdM) (IdM)6 service of FIWARE. Keyrock IdM offers Single Sign On
(SSO) service for users and applications and implements an OAuth2.0 mecha-
nism. Applications accessing protected resources are checked whether they are
authorized to do so using their OAuth2.0 credentials. All users must be reg-
istered to FIWARE cloud and have a username and password in order to log
into the application. Keyrock IdM issues a “session key” (token) encoding users’
authorization to access other services.
User Authorization: Although Keyrock IdM is entitled to issue a session key,
it does not perform any authorization check itself. This operation resorts to user
5
https://oauth.net/2/.
6
https://catalogue-server.fiware.org/enablers/identity-management-keyrock.
1188 G. Myrizakis and E. G. M. Petrakis

authorization service. This service is typically invoked by application logic (see

below) which is responsible for issuing requests to other services and for calling
services in order. Every (user) request must have the session key in its header.
Once a request reaches application logic and prior to invoking the target service,
the session key is forwarded to authorization service. From there, a request is
send to Keyrock IdM which responds with the users’s identity information. User
authorization service is entitled to approve or reject the request based on this
information. Application logic executes requests only if they are approved by
this service.
Publication and Subscription (PS) Service: It acts as a media broker (i.e.
passing information to users subscribed to services). It mediates between devices
publishing measurements or receiving control commands to specific devices. Each
time a new sensor is registered to iHome, or a new measurement becomes avail-
able, this component is updated (i.e. a new entity is created in this service or
its value is updated, respectively) and a notification is sent to all entities sub-
scribed to this information. Subscribed users can also retrieve entity values from
this service. This service is implemented using the Publish-Subscribe Context
Broker7 service of FIWARE.
Event Processing (EP) Service: The home moderator may create rules for
handling events. All rules are stored in the cloud and are operated by the appli-
cation logic service (see Sect. 2.4). For each rule, application logic service sends
a subscription request to PS service. The role of PS service is to trigger the
execution of the rule each time room environment measurements change (e.g.
temperature, humidity, luminosity measurements in iHome scenario). If the con-
ditions of the rule are met (e.g. the measurement exceeds the pre-defined thresh-
old value), an alert is generated which is forwarded to application logic service
which is responsible for handling the event. Home users subscribed to this event
get a notification and, in cases of critical events, application logic service trig-
gers an action (e.g. alarm sound or phone calls to authorities for handling the
event in the case of fire or security breach at home). Figure 2 illustrates the EP
sequence diagram. In a future implementation, an EP service is foreseen for the
home (i.e. the front-end) as well. The role of this EP service is to provide faster
responses for events occurring at home. All rules defined at the back-end must
be downloaded to the front-end or, two different sets of rules can be defined for
front and back-end respectively.
Application Logic: Orchestrates, controls and executes services in order. When
a request is received, it is dispatched to the appropriate service. For example,
services regarding user accounts and access rights are dispatched (through appli-
cation logic) to user authorization service. It performs basic security controls (e.g.
checking if a session between the front-end and the cloud or between the Web
application and the cloud has been initiated). Threshold violations (reported by
EP service) are managed by this component. It creates notifications when a rule
7
https://catalogue-server.fiware.org/enablers/publishsubscribe-context-broker-orion-
context-broker.
 iHome 1189

Fig. 2. Event Processing (EP) sequence diagram

triggers actions (e.g. alerts when a device is malfunctioning). Such notifications

and alerts are forwarded to the Web application.
History Database: It implements a data storage service for log data (i.e. device
measurements, actions, events), users (i.e. user profiles, access rights), rules trig-
gered by the even processing service each time a device value is updated. It is
implemented as a non-SQL database (i.e. a JSON storage service) using Mon-
goDB8 .
Web Application (UI): Implements the user interface and runs on a Web
browser. A user can access the system by providing user name and password
after he/she registers to FIWARE cloud. This request is forwarded to Keyrock
IdM which issues the OAuth2.0 token (session key). This token is added to the
header of every request made by the user to iHome. The request is approved or
denied based on user role and authorization. The Web application is implemented
using HTML and CSS/jQuery.
Data Analytics: This component is particularly useful as data become big. The
big data processing service handles analytics for useful feedback for improving
system responses, enhancing users experience and applying better business plans.
It is described here for completeness of the discussion but it is left as future work.

3 Performance Evaluation
We run an exhaustive set of experiments and we analyze the performance limits
of the services running in the cloud. We study also system scalability (i.e. how
8
https://www.mongodb.com.
1190 G. Myrizakis and E. G. M. Petrakis

system response time increases with the number of connected users). iHome is
deployed in three Virtual Machines (VMs). The first two VMs are deployed in the
FIWARE node of TUC (FIWARE is a federation of distributed cloud infrastruc-
tures running OpenStack). These are small flavor VMs with one virtual processor
(x86 64 processor architecture, 2,800 MHz, first level cache size 32 KB, second
layer 4,096 KB cache size), 2,048 MB RAM, 20 GB hard drive capacity. Each VM
runs Ubuntu Operating System 14.04 and Apache HTTP server. The first VM
runs application logic, user authorization, EP, database and connectivity ser-
vice. The second VM runs PS service alone. Finally, the third VM runs Keyrock
IdM service which is provided by a shared VM installed in a remote FIWARE
node (e.g. in Spain). All VMs may receive up to a large number of simultaneous
requests as the number of users and requests increase. All measurements of time
below account also for the time spent for the communication between VMs or
between services within the same or different VMs.
For each experiment we report average response time (over 2,000 requests) for
the most common operations. Each operation can be executed either sequentially
(i.e. one after the other) or in parallel. We use ApacheBench9 (Apache HTTP
server benchmarking tool) to send multiple simultaneous requests to iHome. In
ApacheBench we are opted to define the total number of requests and how many
of them will be executed simultaneously. We report average response time for
2,000 requests executed in a sequence (concurrency = 1) and also for increasing
values of concurrency (i.e. concurrency >> 1). All service requests address the
user authorization and Keyrock IdM service at the back-end which checks if
the user or service issuing the request has the necessary access rights. Table 1
summarizes the performance of the following basic operations executed in iHome.
A home posts room values to the connectivity service (at the back-end)
and published in PS service (i.e. the corresponding entities are updated) and
the database. Users subscribing to this information get a notification about this
change on the Web application (through application logic service). The EP ser-
vice subscribes always to this information and triggers the execution of rules that
govern the monitoring of a home. If these values are within the pre-defined lim-
its, no action is taken; otherwise (e.g. temperature exceeds a limit), EP service
notifies the home moderator to take action.
A user requests room values from a home. The request is issued on the
Web application. From there, it is forwarded to PS service which always holds
the most recent values of all home entities. Finally, the Web application receives
the requested information (again through application logic which is responsible
for orchestrating operations running in iHome).
Insert a device at a home. The home moderator needs to start moni-
toring a new device in a home. The request is issued on the Web Application
and its forwarded to application logic service. This operation will insert a new
entity in PS service with the three default values (i.e. temperature, humidity
and luminosity). The request is forwarded (through connectivity service) to the
front-end. After decoding, user authorization service in the front-end will check
if the user has the necessary access rights. If access is granted, the new device is
9
https://httpd.apache.org/docs/2.4/programs/ab.html.
 iHome 1191

created (data collector and database services are updated as well). The device
is published at PS service (i.e. a new entity is created) and the home moderator
can define users subscribing to the new PS entity. The user receives a notification
upon successful completion of this process.

Table 1. Performance of basic iHome services

A home posts room values to the cloud:

POST localhost:8228/getRoomValues
Concurrency 1 50 100 150 200 300
Time (ms) 88.29 67.23 63.29 66.22 70.29 77.71
CPU (%) 97.94 99.43 98.86 98.14 96.24 99.39
RAM (%) 69.67 75.65 80.96 87.10 93.96 95.88
Get room values from a home:
GET localhost:8448/homes/homeId/rooms/roomId
Concurrency 1 50 100 150 200 300
Time (ms) 67.22 61.91 63.93 65.83 67.29 73.59
CPU (%) 36.16 74.96 78.26 77.66 73.08 68.63
RAM (%) 73.29 75.52 77.39 80.51 83.03 84.75
Insert a new device at a home:
POST localhost:8448/homes/homeId/devices
Concurrency 1 50 100 150 200 300
Time (ms) 370.5 100.1 101.3 103.1 104.4 109.8
CPU (%) 21.84 58.68 53.72 51.65 53.02 49.87
RAM (%) 66.43 68.96 70.74 72.86 74.90 77.76

Response times improve with the simultaneous execution of requests (i.e.

the Apache HTTP server switches to multitasking) reaching their lowest values
for concurrency between 50 and 150. Even with concurrency = 300 the average
execution time per request is close to real-time and resource usage is well below
100% in most cases.
iHome may produce big amounts of data and requests, requiring large pro-
cessing capabilities which can surpass the capacities that our experimental sys-
tem set-up is able to provide. In this iHome set-up, core services are implemented
in three VMs. Presumably, overloading the VMs might occur in an application
with a much larger number of concurrent users. An obvious solution to dealing
with performance would be to employee additional VMs each running a single
service (or a small group of services). Alongside, we can allocate additional VMs
implementing the same service (or groups of services) thus having more than on
VM sharing the load.

4 Conclusions and Future Work

Leveraging PaaS functionality we show how a smart home management system
is designed and implemented as a service in the cloud. iHome exhibits a highly
1192 G. Myrizakis and E. G. M. Petrakis

modular SOA design based on micro-services running both, at a home (i.e. a

fog node) and in the cloud allowing users to monitor and control their homes
remotely in real-time. iHome has been tested in a realist scenario with up to
three hundred concurrent users. The experimental results reveal that, iHome is
capable of responding in real time even under heavy workloads. Besides being fast
and modular, iHome innovative design relies on secure cloud services permitting
access to services and information based on access rights and authorization (i.e.
all services are protected by a security mechanism). At the same time, iHome
implements secure communication for data and services over HTTPS. As a future
extension, iHome must be tested using a large number of physical devices and
appliances installed at homes. Implementation of data analytics functionality in
a public cloud, in a use case with many homes connected to iHome, is also an
interesting direction for future work.

References
1. Bassi, A., Bauer, M., Fiedler, M., Kramp, T., van Kranenburg, R., Lange, S.,
Meissner, S.: Enabling Things to Talk: Designing IoT Solutions with the IoT Archi-
tectural Reference Model. Springer, Heidelberg (2013)
2. Bonomi, F., Milito, R.A., Zhu, J., Addepalli, S.: Fog computing and its role in the
internet of things. In: Proceedings of the First Edition of the MCC Workshop on
Mobile Cloud Computing (MCC 2012), pp. 13–16, Helsinki, Finland, 8 2012
3. Ghosh, S.: Smart homes: architectural and engineering design imperatives for smart
city building codes. In: Technologies of Smart-City Energy Security and Power
(ICSESP 2018), pp. 1–4, Bhubaneswar, India, March 2018
4. OpenFog Architecture Working Group. Openfog architecture overview, February
2016
5. Liu, L., Liu, Y., Wang, L., Zomaya, A., Hu, S.: Economical and balanced energy
usage in the smart home infrastructure: a tutorial and new results. IEEE Trans.
Emerg. Top. Comput. 3(4), 556–570 (2015)
6. Myrizakis, G.: iHome: smart home management as a service in the cloud. Technical
report TR-TUC-ISL-01-2019, Diploma thesis, School of Electrical and Computer
Engineering, Technical University of Crete (TUC), Chania, Crete, January 2019
7. Petrakis, E.G.M., Sotiriadis, S., Soultanopoulos, T., Renta, P.T., Buyya, R., Bessis,
N.: Internet of things as a service (iTaaS): challenges and solutions for management
of sensor data on the cloud and the fog. Internet Things 3–4(9), 156–174 (2018)
8. Rehman, S., Gruhn, V.: An approach to secure smart homes in cyber-physical
systems/internet-of-things. In: Software Defined Systems (SDS), pp. 126–129,
Barcelona, Spain, May 2018
9. Schreier, S.: Modeling restful applications. In: ACM International Workshop on
RESTful Design (WS-REST 2011), pp. 15–21, Hyderabad, India, 3 2011
10. Sotiriadis, S., Stravoskoufos, K., Petrakis, E.G.M.: Future internet systems design
and implementation, cloud and IoT services based on IoT-a and FIWARE. In:
Angelakis, V., Tragos, E.Z., Kapovits, A., Poehls, H.C., Bassi, A. (eds.), Designing,
Developing and Facilitating Smart Cities: Urban Design to IoT Solutions, pp. 193–
207. Springer, October 2017. Chap. 10
 Design of Robot Service Functions
for a Framework Establishing
Human-Machine Trust

Fumi Ito, Eri Ozawa, and Yuka Kato(B)

Tokyo Woman’s Christian University,

2-6-1 Zempukuji, Suginami-ku, Tokyo 1678585, Japan
[email protected]

Abstract. In recent years, there has been an increasing interest in

constructing a platform on information flow, which highly integrates
robotics, IoT devices, and new generation networks and provides safely,
securely and widely distribution and utilization of IoT data (i.e., infor-
mation flow). From this background, we have been studying a framework
establishing and enhancing human-machine trust for secure IoT services.
This paper reports the design result of a robot application for the frame-
work. The feature of the application is its function that system users can
control the disclosure level of privacy information to service providers
by themselves. In this paper, we show the usage scenario of the robot
application and examine the feasibility of the framework functions by
using it.

1 Introduction
In recent years, there has been an increasing interest in constructing a platform
on information flow, which highly integrates robotics, IoT devices, and new gen-
eration networks and provides safely, securely and widely distribution and uti-
lization of IoT data (i.e., information flow) [6]. To realize services using such IoT
data, around the source of the information flows, it is necessary to build a good
relationship between humans and nearby IoT devices (i.e., machines) which get
into our living spaces, and to create feelings of security and trust to use IoT ser-
vices. Therefore, we have been studying a framework establishing and enhancing
human-machine trust for secure IoT services [5]. An image of the framework is
shown in Fig. 1.
This framework acquires and visualizes personal data collected from various
sensors in daily life and indicates the acquisition level of the data (i.e., the
degree of the acquisition) to users. Moreover, the framework classifies users into
several types according to user profiles, and sets a suitable disclosure level (to
machines) of personal data to each type. The disclosure level can be also adjusted
appropriately by the interaction with users on the framework. Until now, an
effective assistant method of the privacy-level adjustment has been proposed [3].
The method adjusts the level by recommending appropriate disclosure settings
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1193–1204, 2020.
https://doi.org/10.1007/978-3-030-15032-7_100
1194 F. Ito et al.

Fig. 1. Image of framework establishing human-machine trust.

to system users depending on their information literacy levels and preferences.

In the literature, a simulation experiment on the adjustment was conducted
for the evaluation. However, the control of personal data distribution in real
environments is achieved by extremely complicated elements as compared to
simulation. Therefore, we need to verify the effectiveness and usefulness of the
framework by demonstration experiments.
From this background, in this paper, we design an application service (a robot
application), which uses an embodiment communication robot and combines a
wide variety of IoT devices with the robot, for the purpose of the verification
of the framework. First, we extract information data that can be acquired in a
daily life of general household and that can become privacy information. Next,
we design a robot application with the function that can adjust the disclosure
level of personal data by designating the range of data search. After that, by
showing a usage scenario of the designed application, we examine the validity of
setting disclosure level of privacy information and adjusting the level.
The contribution of this paper include:

• Design a robot application for the verification of the framework.

• Verification of feasibility on the framework functions by assuming a usage
scenario.

The remaining part of the paper proceeds as follow: Sect. 2 gives a brief
overview of our target framework “Framework Establishing Human-Machine
Trust”; Sect. 3 analyses privacy information for the framework and describes the
classification result; Sect. 4 designs a robot application to validate the framework;
Sect. 5 is concerned with the validation result by using an application scenario.;
Sect. 6 concludes the paper.
 Design of Robot Service Functions 1195

2 Framework Establishing Human-Machine Trust

Framework Establishing Human-Machine Trust is the target of this paper. In
order to establish trust between humans and machines (IoT devices), we set two
subjects below and design framework functions to overcome the difficulties [5].

• Present privacy information (the degree and the category) acquired by IoT
devices to framework users in a natural way.
• Adjust the disclosure level of privacy information already acquired by inter-
acting naturally with the users.

To solve the former one, we need to develop methodologies to acquire pri-

vacy information, to determine the category and degree of the information, and
to present the determined results to users in a natural way. To solve the latter
one, we need to develop a dialogical interface that makes it possible to set the
disclosure level of privacy information while talking with IoT devices, like an
interpersonal conversation. In addition, the function is required that the disclo-
sure level is automatically set according to the previous interaction records and
so on, even though a user does not do anything.
For the subjects, the framework has two types of functions. One is the func-
tion of acquiring and visualizing privacy information. The other is the interaction
function to adjust the disclosure level. The details are explained as follows.

2.1 Function Acquiring and Visualizing Privacy Information

The function extracts privacy information from IoT data, gives the scores of
the information, and visualizes them. The target IoT data are video and sound
data of daily life collected from web cameras and microphones, vital information
collected from wearable devices, text information posted to SNS and so on. Until
now, to set the adequate disclosure level of personal data (PD) on using various
Internet services, we have proposed a method recommending suitable settings
according to information literacy of users as well as users’ preferences on the
disclosure level [3]. The method analyses records of the settings, and regards
system users who can set similar disclosure levels to similar service providers as
consistent users. Then, it compares the consistency level relatively with those of
other users and estimates information literacy level based on that. After that,
it recommends appropriate settings according to the information literacy level.
The research validated the effectiveness of the proposed method by comparing
the existing ones [1,4].
This method realizes a semi-automated setting, but it is necessary to adjust
the level adaptively with a solid interaction between users and the framework in
order to conduct such sensitive judgment as disclosure/closure of PD according
to the conditions without fail. Therefore, with this visualization function, the
framework achieves the adjustment by combining with an interactive technology
for adjusting the disclosure level described below.
1196 F. Ito et al.

2.2 Interactive Function to Adjust Disclosure Level

The function conducts the interaction adjusting disclosure level of privacy infor-
mation, which are grasped by machines, based on user profiles, and provides the
user interface for the adjustment. User’s intention on privacy can be reflected
IoT systems by the function. Specifically, the function considers the trade-off
between security risks occurring by privacy information and service benefits
resulting from providing the information, and deletes the information or controls
the permissions as maximizing the profits of minimizing the risks. In addition,
it makes system users to be able to grasp the following three points.

• What kinds of privacy information do IoT devices acquire, and how much
risks are there for that?
• Who can access the acquired privacy information, and how much risks are
there for that?
• How are the acquired information abused, and how much risks are there for
that?

Since the degree of risks for privacy information diverges for individuals, a
personalized calculation method of the degree according to the following proce-
dure has been proposed [5]. First, on the set of information types Id acquired
by an IoT device d, the method defines a function ∀i ∈ Id , R(d, i, Ci ) which
calculates the degree of risks. Here, Ci is a set of coefficients of weights for
personalizing the degree on information i. Next, it updates the set of the coeffi-
cients of weights Ci to Ci∗ using ∀i ∈ Id , R(d, i, Ci∗ ) = R∗ (d, i). Here, d∗ is an IoT
device that a user changes the degree of risks by the interaction, and R∗ (d, i) is
the changed degree of risks for the information i.

3 Classification of Privacy Information

For the verification of the trust framework, in this paper, we design robot service
functions (for a communication robot with a physical embodiment) which can
interactively execute the two functions mentioned in the previous section (i.e.,
the function acquiring and visualizing privacy information and the interactive
function to adjusting the disclosure level). This is because a communication
robot equipped with various sensors and actuators as the interface is suitable
for achieving multimodal interaction between humans and machines in a natural
way for users.
In designing the application, we firstly extract information that is poten-
tial privacy information and that can be acquired at a place of daily life in an
ordinary household (in this case, we use only information that can be acquired
with noninvasive sensors). We explain the outline in this section. Actually, there
have been many types of research about such topics before. For example, in
the literature [2], the authors classified privacy information into ID information,
ID-related information, personally identifying information and personal inner
 Design of Robot Service Functions 1197

Fig. 2. Classification result of potential privacy information. Data in the top layer are
sensing data directly acquired by sensors. Reasoning information directly from certain
information are connected with solid arrows, and reasoning information indirectly from
certain information (reasoning by data integration, data analysis etc.) are connected
with dotted arrows.

information, and expressed the relevance between these information and infor-
mation processing technologies. We, in this paper, classify privacy information
based on emotional and social information in addition to the pieces of infor-
mation in [2], and connect each information with a directional branch if one
information can be inferable from the other. Thus, we clarify the relationships
among the information. The result of the classification is shown in Fig. 2. It is
classified into hierarchical layers, and connected with each other by solid arrows
in a directly inferable case and by dotted arrows in an indirectly inferable case.
Here, we classify privacy information into five layers (sensing data, ID-related
information, personally identifying information, personal inner/vital information
and social information), include several data types into each layer and include
some privacy information into each data type. Inferable information from other
information would be single information or comprehensive information in a data
type. Moreover, there is a possibility that certain information is inferred from
the combination of these pieces of information. For example, we can directly
infer external information such as voice and the way of speaking from the result
of sound sensing with a microphone. In the same manner, if a moving trajec-
tory can be estimated by analyzing the distance between the microphone and
the target, we can indirectly obtain human behavior information by the sensing
data as well. In addition, we can infer social information such as human rela-
tionships by combining human behavior information (e.g., behavior contents and
1198 F. Ito et al.

patterns) with other information. As a result, the attributes of individuals, such

as occupations, might be able to be inferred.
Like this, by using the analysis result, we can find acquirable privacy informa-
tion on following the arrows, and can designate the range of privacy information
provided to machines by selecting the range of searching on the graph. There-
fore, the graph makes it possible to set the disclosure levels for each privacy
information, each data type, and each hierarchical layer.

4 Design of Robot Application

In this section, we design a robot application with functions that can collect
the privacy information extracted in the previous section and can adjust the
disclosure level of the information by designating the searching range.

4.1 System Architecture

Figure 3 shows the system architecture to realize the two functions. The system
consists of three parts, which are IoT devices as the interface between humans
and machines (these include communication robots, tablet PCs and IoT devices
installed in environments), cloud-edge services to provide IoT services to users
and an adjustment service of the disclosure level to set providing privacy infor-
mation to machines by interacting with users in a natural way. By combining
function modules in each part, the framework executes the functions of providing
services to users, acquiring and visualizing privacy information, and interacting
with users. Here, since the purpose of the paper is to design a robot applica-
tion to verify the framework, our scope is the part in a red frame in Fig. 3. We
push aside the implementation of the interaction for the adjustment and the
adjustment model in this paper.
The process flows between function modules are shown in Fig. 4. First, in
the case that a system user uses IoT services, sensing results of IoT devices and
a robot are analyzed according to the adjustment model, these data are stored
in the framework, and then the IoT services are provided (in the upper part of
Fig. 4). At that time, the privacy information is used/stored on an appropriate
disclosure level for the user by following the adjustment model. The stored pri-
vacy information is given the score in reference to user features, and the score is
visualized to present the result to the user (in the middle part of Fig. 4). Here,
the user updates the adjustment model by interacting with machines in a natu-
ral way. The framework conducts profiling of the user features in the interaction
process, and reflects the result to the adjustment model (in the lower part of
Fig. 4).
In this paper, under the assumption of the existence of the adjustment model
that is updated by the result of the interaction, we design a robot application,
which provides services according to the process of IoT services shown in the
upper part of Fig. 4. The classification result described in Sect. 3 is incorporated
into the parts of presenting privacy information and adjustment model. We make
 Design of Robot Service Functions 1199

Fig. 3. Structure of framework. It consists of three parts, which are IoT devices as the
interface between humans and machines, cloud-edge services to provide IoT services
to users and an adjustment service of the disclosure level to set providing privacy
information to machines by interacting with users in a natural way.

it possible to set available disclosure range of privacy information easily and

intuitively, such as shared in family members, shared in friends or neighbors and
shared in public.

4.2 Outline of Services

As for an example of IoT services for the verification of the framework, we adopt
an application that system users freely interact with a robot installed in a home
via speech dialogue. We use a tabletop-sized popular robot platform for the
communication robot. The robot functions are as follows:

• Speech dialogue function with humans

• Acquiring environmental information with cameras and microphones
• Emotional expression by LED lights
• Eight-degree-of-freedom (one axis for a body, two axes for arms and three
axes for a neck)
• Connecting with cloud servers and external devices

In this case, we combine the robot with a tablet PC and use it for the interface
between humans and machines. For a specific service content, we consider an
application that the robot conducts an adequate dialogue with a user depending
on the user situation recognized by IoT devices. The service functions are as
follows:

• The robot starts a service with the user’s speaking as a trigger.

• The robot identifies the user by face recognition (it is omitted when face
recognition is not permitted).
1200 F. Ito et al.

Fig. 4. Process flow between function modules. Upper part shows process in using IoT
services, middle part shows visualization process on privacy information, and lower
part shows interaction process for the adjustment.

• The robot responds to the user according to the user information and the
environmental conditions.

At that time, the content that the robot responds to the user is different
depending on the disclosure level of privacy information. For example, in the
case that the level is low, general information, such as weather forecast and
news, is provided to the user. On the contrary, in the case that the user discloses
the records of personal inner data and vital data, a service content according to
the health condition is provided.
 Design of Robot Service Functions 1201

4.3 Acquired Privacy Information

Privacy information acquired in the service is differ depending on the combina-
tion of using IoT devices. Here, we assume to use information acquired with a
camera and microphone installed in a robot, temperature and humidity sensors,
and vital sensors (pulse, breath, and temperature). In addition, we also intend
to use online information via cloud services (schedule and SNS information etc.).
At that time, the following information is acquired by machines when we set the
disclosure level of privacy information in each hierarchical layer illustrated in
Fig. 2. Note that these items will not always be acquired by the disclosure, but
it is a situation that “acquisition becomes possible”.
• Sensing data: None
• ID-related information: ID information, pulse rate, breath, body temperature
• Personally identifying information: voice, the way of speaking, appearance,
face, expression, body language, position, location, behavior pattern
• Personal inner/vital information: condition of stress, emotion, health condi-
tion
• Social information: human relationships.

5 Validation Using Service Scenario

5.1 Content of Scenario
In this section, we validate the feasibility of the framework functions by using
a service scenario for the application designed in Sect. 4. Here, we consider the
situation that a user uses several IoT services by conducting a conversation with
a communication robot before going out in the morning. Specifically, we consider
the service scenario as below:
The user speaks “Good morning” to the robot. With that triggered, the robot
recognizes the user based on the face image acquired by the camera, and replies
“Good morning, Mr./Ms. Name” to the user. At the same time, the robot
acquires environmental information such as temperature and humidity, pulse
and body temperature of the user from IoT devices installed in the home and
user’s attached vital sensors. In addition, the robot acquires online information
from cloud services such as calendar information. The robot generates various
information providing services using these data and the information presumed
therefrom, and provides information to the user interactively.

Depending on the situation of the user, a service accompanying the simple motion
of the robot may be provided, and the exchange of information via the tablet
PC may be performed.
As users of the application, we assume three types. These are strict users for
the disclosure who has high information literacy, active users for the disclosure
who has high information literacy, and users who are not conscious of the dis-
closure because of having low information literacy. If the information literacy is
high, we will increase the degree of personalization at the disclosure level. On
the contrary, if it is low, we will recommend average settings.
1202 F. Ito et al.

5.2 Adjustment Method

Strict Users for the Disclosure. Based on the user information, the frame-
work judges that it is a strict user for the disclosure of privacy information, and
decides the content of the provided service according to the corresponding adjust-
ment model. For example, if the system is set up to acquire sensing data and
ID-related information, the robot will provide the service to the user identified,
so it is possible to call the username. However, no environmental information is
acquired, and the robot generates a reply based on general information such as
weather forecast.
If the user wants to receive more detailed service according to the situation,
the service is added by users request to the system. At that time, a change in
the disclosure level of privacy information due to the addition to the service
is visualized, and the user judges whether or not to update the adjustment
model by the interaction with the system. For example, when adding a service
to ventilate in accordance with the CO2 level in the room, the possibility that
the behavior information and the physical information are estimated and the
degree of the risk is presented to the user, and the user decides to add the
service according to the contents. For users with high information literacy, in
order to reduce the complexity of setting, we base on common settings for strict
users for the disclosure. Meanwhile, according to the addition and deletion of the
IoT service, a detailed setting for each service is also realized. To determine the
extent of information literacy, we use the method described in Sect. 2. Here, the
disclosure level for each service can also be set for active users for the disclosure
described in the next section.
Active Users for the Disclosure. In the same way as above, the content of
the provided service is determined by the corresponding adjustment model. For
example, in the case where the system is set to acquire social information, it is
possible to provide services that combine environmental information, vital infor-
mation, and personal information of users. The system can provide a service that
suggests meals for breakfast according to the schedule of the day, health/mental
condition, preferences and those records.
Even in this case, the user can confirm the disclosure level of privacy infor-
mation as necessary, and change the information not to disclose separately and
individually. At that time, the relation between the currently provided service
and the privacy information is presented, and the user judges the disclosure or
not according to the contents. For example, when changing to a setting that
does not provide health conditions, a list of services using health conditions is
presented.
For users with high information literacy, in order to reduce the complex-
ity of setting, we base on common settings for active users for the disclosure.
Meanwhile, according to the addition and deletion of the privacy information,
a detailed setting for each information is also realized. Here, the disclosure level
for each privacy information can also be set for strict users in the disclosure
described in the previous section.
 Design of Robot Service Functions 1203

Users Who Are not Conscious of the Disclosure. In the same way as
above, the content of the provided service is determined by the corresponding
adjustment model. For users who are not conscious of the disclosure of pri-
vacy information, the system provides average services according to the average
disclosure level. For example, if the system is set up to acquire sensing data,
ID-related information and personally identifying information, it is possible to
provide services such as changing robot response contents according to the place
where the user is (bedroom, kitchen, etc.).
The user can confirm the disclosure level of privacy information as necessary,
but the degree of freedom in changing the level is limited. Here, we base on
common settings for users who are not conscious of the disclosure, and allow
changes in about three levels.

5.3 Future Works

As described above, by setting the user type and the requested disclosure level,
it becomes possible to update the adjustment model according to the user fea-
tures. Although the updating method of the adjustment model (the method of
determining the degree of disclosure considering its convenience and risk) and
the concrete method of interaction (in a natural way) are future works, the
possibility of the adjustment as a framework can be verified in this paper.

6 Conclusion
This paper designed a robot application premised on use in a framework estab-
lishing human-machine trust, and verified the feasibility of the framework func-
tions by considering the service scenario of the application. The result of the
paper is as follows:
• Designing an application service in which a robot properly interact with a
person via speech dialogue as a robot application to verify the framework
functions.
• Verifying the adjustment by developing usage scenario of the application,
and verifying the possibility of the adjustment by natural interaction with
a robot by specifying the disclosure level for each hierarchy layer of privacy
information classification.
In the future, using the designed robot application, we will demonstrate how
to live with a robot while adjusting the disclosure level of privacy information
in an environment where users live at home. Specifically, we will set up various
sensors and a communication robot at Smart Home Experimental Facility (living
environment of a one-bedroom home) at NAIST in Japan, and construct an
experimental environment. Then, we will design a scenario where several subjects
actually live in. After that, we will investigate whether the subjects can grasp
how much privacy information is sensed, how they can adjust the disclosure level
of sensed privacy information in a natural way (with a small burden) and how
much trust relationship with the robot can be constructed.
1204 F. Ito et al.

Acknowledgements. This work was supported by JSPS KAKENHI Grant Number

17KT0080 and the Cooperative Research Project Program of the Research Institute of
Electrical Communication, Tohoku University.

References
1. Agarwal, Y., et al.: Protectmyprivacy: detecting and multigating privacy leaks on
iOS devices using crowdsourcing. In: MobiSys 2013, pp. 97–100 (2013)
2. Babaguchi, N., Nishio, S.: Sensing and privacy protection of network robots. J. Inst.
Electron. Inf. Commun. Eng. 91(5), 380–386 (2008)
3. Kayaba, K., Oide, T., Abe, T., Suganuma, T.: A proposal for personal data control
assistant base on user’s preference clearness. IEICE Technical report, 117(205):1–6
(2018, in Japanese)
4. Liu, R., Cao, J., Yang, L., Zhang, K.: Recommendation for privacy settings of mobile
apps based on crowdsourced users’ expectations. In: IEEE MS 2015, pp. 150–157
(2015)
5. T. Suganuma, K. Yasumoto, and Y. Kato. A concept of a human-machine trust
generating framework for secure IoT services. IPSJ Technical report (DPS175), pp.
1–7 (2018, in Japanese)
6. Yasumoto, K., Yamaguchi, H., Shigeno, H.: Survey of real-time processing technolo-
gies of IoT data streams. J. Inf. Process. 24(2), 195–202 (2016)
 A Proposal for a Dynamic Digital Map
to Prevent Heatstroke Using IoT Data

Kanae Matsui(B) and Keiya Sakai

Tokyo Denki University, 5 Senju Asahi-cho, Adachi-ku, Tokyo 120-8551, Japan

[email protected]

Abstract. This paper presents a dynamic digital map displaying user’s

position and his/her environmental and health status for prevention of
heatstroke, which is one of worst natural disaster in Japan. We use Inter-
net of Things (IoT) data of Global Positioning System (GPS) and tem-
perature surrounding people. Victims of heatstroke in Japan have been
increasing in summer, and the heatstroke causes a large number of deaths
annually. To avoid this situation when people are walking outside in sum-
mer, our proposed digital map provides the information of their walking
locus created by their GPS data along with their heartbeat rate and sur-
rounding temperature collected by IoT devices. To develop our proposed
system, we conducted an experiment, and the results of the dynamic
digital map are promising.

1 Introduction
Heatstroke during summer is a common occurrence in Japan. In 2017 alone,
there were 49,583 emergency cases and 635 deaths [1,2]. Japan Meteorological
Agency (JMA) has announced the heatstroke to be one of the major risks to
people’s health [3]. The season in which the risk of heatstroke is highest, usually
lasts three months. Outside locations are considered as 35.3% of heatstroke in
Japan occur when people are outside [4]. Moreover, people are likely to use a
digital map to reach their destination, e.g., Google Maps [5] and Yahoo! map [6]
while they are outside [7]. So, showing both statuses i.e. of their environment
and of their health by plotting the data in a dynamic digital map can be an
efficient means of notification of an imminent heatstroke.
With this background, a smart city consists of IoT devices, applications and
systems to make peoples’ life smarter and safer [8]. To prevent disasters, which
include heatstroke, is also one of main purposes in a smart city [9]. This study
proposes a smart walkable environment with a dynamic digital map. The map
shows a user’s heartbeat and her surrounding temperature collected by IoT
devices. These information would help for users to understand their environ-
mental and health conditions. The proposed dynamic digital map utilizes three
types of IoT data; GPS, people’s surrounding temperature, and their heartbeats,
to display the risk of a heatstroke while they are walking outside. In order to

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1205–1216, 2020.
https://doi.org/10.1007/978-3-030-15032-7_101
1206 K. Matsui and K. Sakai

test the proposed digital map, we conducted a 1 day experiment with eight par-
ticipants, in northern Tokyo, Kita-Senju, Adachi Ward on August 29th, 2018.
The results show the efficiency of the proposed map.

2 Related Works
Research on IoT data utilization for display on a dynamic digital map is
described in this section. Historically, a car navigation system has been a good
example and continues to be one the most widely installed systems in the world.
It is used to show a user’s position and road traffic information with a digital
map [10]. Thanks to the spread of smart phones, many digital map applications
like Google Maps, have many users, and location-based web applications, e.g.,
TripAdvisor [11], which provides the information of trip attractions, and yelp
[12], which indicates reputable restaurants around users, have bloomed [13].
Further, taxi service applications, e.g., Uber [14] and Lyft [15], provide a user
interface to display user’s and called driver’s locations [16]. The driver’s location,
indicated as icons in their user interfaces (UI), updates with their movements,
which means that their interface handles mobile data collected by IoT devices.
In addition, User Interface (UI)/User Experience (UX) part of the applications
should process massive data in near real-time. Towards this end, Uber Engineer-
ing has developed a platform, called kepler.gl [17], for location-based applications
to map the massive amounts of data into a digital map [18]. They utilize open
source mapping platforms like Mapbox for custom designed maps. Other location
based businesses also utilize open source mapping platforms.
Owing to the spread of customized maps for different purposes, evolution of
digital maps and their platforms including Application Programming Interfaces
(APIs) and Software Development Kits (SDKs), IoT data can be utilized for
developing our own customized maps. In this paper we propose a customized
digital map to reduce the risk of heatstroke using IoT data.

3 Proposed Dynamic Digital Map

The proposed dynamic digital map aims to display user’s position from GPS data
along with her heartbeat and surrounding temperature to indicate her risk of
getting a heatstroke. Main usage of this system is that user check this map during
their walking and would be careful when statuses of their environment, which
indicates temperature, and their heartbeats shows higher values. To display this,
a system, which processes the digital map, has two main functions; one is to
collect the data using IoT devices, and the other is to plot collected data onto
the digital map. Below, we describe the steps of the proposed digital map creation
in (A) what types of devices and data were used, (B) data calibration, and (C)
how to process those data and plot into digital map.

3.1 Data Collection

For the proposed map, three types of data are utilized; data of GPS, user’s
surrounding temperature, and his/her heartbeats.
 A Proposal for a Dynamic Digital Map to Prevent Heatstroke 1207

3.1.1 GPS Data Collection

To collect GPS data, a function of GPS data detection in Android based
smartphone (M04 by Fujitsu Limited) was used. In addition, we used devices
(sub meter-class positioning augmented service compatible receiver named QZ1
by NEC Corporation) for communicating with Quasi-Zenith Satellite System
(QZSS), paired with smartphones for calibrating GPS data precisely. Further,
the QzssProveTool smartphone application, should be installed, to access the
collected and calibrated data from QZSS.

3.1.2 People Surrounding Temperature Data Collection

People surrounding temperature defines as temperature collected by an IoT
device necked by users in this paper. So, values indicates middle of body’s out-
door temperature. To provide each user’s danger to heatstroke, temperature
surrounded by users is effective, because the range of the weather forecast is too
wide and sometimes not fit to user’s locations of each person. Therefore, IoT
device (ATR TSND151 by ATR Promotions.Inc.) which collects temperature
around the device, is used, and lowered from user’s necks of users (shown in
Fig. 1).

Fig. 1. Overview of data collection.

3.1.3 Heartbeat Data Collection

To collect heartbeat data, a wearable devices (Fitbit iconic by Fitbit, Inc.) is
used. This device was developed for collecting human’s health conditions as data
of heartbeats, number of steps, walking distance, burned calories, and so on, but
only heartbeat data is used for the system. This device has own server and the
1208 K. Matsui and K. Sakai

data collected by the devices automatically sends to their server. Therefore, the
system needs to access the server and collect the data from there via OAuth,
which is one of authorization open standard [19].
Figure 1 illustrates an overview of the data collection. The devices of GPS
and heartbeats should be paired with smart phones via Bluetooth Low Energy
(BLE). These data with timestamp are sent to a central database and stored.
However, each device has different timing and rules to add timestamps. Next
step of “data calibration” is to cleaning data with same timestamp.

3.2 Data Calibration

As above-mentioned, used devices have different timing and rules to add times-
tamp when they create measured data. For example, GPS data is measured by
1 s intervals, the surrounding temperature data is measured by 1 s intervals, and
heartbeat data is measured by 5 s intervals, which means measurement intervals
are different. Therefore, data calibration is necessary to unify data intervals as
time series data. For this study, heartbeat data calibrated as 5 s intervals with
spline interpolation, which is one of methods of interpolation method on time
series data [20].
Other data are used their mean values of 5 s intervals from data of 1 s inter-
vals. After the data collection, the data in the database is calibrated by spline
interpolation, and then the calibrated data will be used for the digital map
creation.

3.3 Digital Map Creation

With the calibrated data, this step creates the digital map. To create the map,
Google Fusion Tables, which is a free web application service, can create cus-
tom map with other data, which contains of longitude and latitude, in Google
Maps [21]. Figure 2 illustrates a sample of the proposed map; left side of the
figure shows mapping with GPS data and temperature data; right side of the
figure shows mapping with GPS data and heartbeat data. A line indicates a
walking path, and a color indicates degrees of both temperature and heartbeat;
red indicates higher temperature and heartbeat, yellow indicates neutral, and
green indicates lower.
When the path indicates red, users must be careful to avoid danger of heat-
stroke by behavioral changes, e.g., stop walking and go to shadow, drinking
water, and ask someone to help.
 A Proposal for a Dynamic Digital Map to Prevent Heatstroke 1209

Fig. 2. Samples of proposed digital map.

4 Experiment

To test processing proposed digital map with actual data, we conducted an

experiment in August 29th 2018 with eight participants. Detailed information,
which are data, participants, and a method how to walk for data collection and
map creation, as followings.

4.1 Date

The experiment was conducted from 14:30 to 16:00, in August 29th, 2018. The
outdoor temperature announced by JMA were 27.4◦ at 14:00, 27.4◦ at 15:00,
and 28.4◦ at 16:00 at the day. Figure 3 illustrates averages of 24-h temperature
degrees on August and the temperature at the experiment day.

Fig. 3. Weather condition of experiment day.

1210 K. Matsui and K. Sakai

The average temperature for three hours was 30.1◦ , so the temperature of
experiment day was lower than the average. In addition, heatstroke risk would
be higher when outdoor temperature is over 31.0◦ , so the day had lower risk.
Figure 4 shows the indicator of heatstroke risk according to temperature [22].

Fig. 4. Guideline on everyday life to avoid heatstroke

4.2 Participants
To develop each person’s digital map, eight participants were selected. Table 1
shows detailed information of participants. The participant’s health condition
would be affected to heartbeat rates, but every participant’s conditions were
neutral at the day.

Table 1. Detailed information of participants

No. Gender Age range Health condition

1 Male 10’s Neutral
2 Male 20’s Neutral
3 Male 20’s Neutral
4 Male 20’s Neutral
5 Male 20’s Neutral
6 Female 30’s Neutral
7 Male 30’s Neutral
8 Male 50’s Neutral

4.3 Place
As a field of experiment, northern side of Tokyo, Kita-senju, Adachi Ward,
Japan, was selected. One street in shopping district, where named Shukuba-
machi Road, and is along the former Nikko Highway and the old Nikko Kaido
Road, about 200 m north of the intersection of Kita-Senju Station West, was
selected as the target street. In the experiment, a task of the participants is to
move from Tokyo Denki University 600 m away from the target. Figure 5 shows
pictures of the target street, where participants went to and spent their time.
 A Proposal for a Dynamic Digital Map to Prevent Heatstroke 1211

Fig. 5. Pictures of target street.

4.4 Methods

To conduct the experiment, the participants start to walk from the university
to Shukubamachi Road, and spend there for 30 min without any orders with
the devices. After that, they return to the university. Total movement time was
about 1.5 h, and movement distance was about 3 km.

5 Results

As the results, (A) data availability is given to check whether calibrated data
would be useful by comparison of row data the calibrated data. Hence, (B) map
creation is given to check whether created map would be matched for partici-
pant’s movement.

5.1 Data Availability

In this part, data availability, which indicates accuracy of calibrated data, is

checked. Table 2 shows comparison of row and calibrated data by line interpola-
tion, which is one of calibration methods for time series data. RD and CD in the
table indicate “Raw data” and “Calibrated data”. Deviation rate in the table
indicates the difference of raw data, which indicates the data measured by Fitbit
iconic, and calibrated data. Their difference is 42.7% as average, but values of
mean, minimum, and max do not have difference. So, the calibration would be
succeeded. Also, all heartbeat values were within the range, could be assumed.
Data of GPS and people’s surrounding temperature were already calibrated
by software that installed to their devices. The surrounding temperature showed
higher temperature than whether forecast temperature. Higher temperature
would be understandable because devices necked from participant must get affec-
tion of sunlight reflection. To check the data precisely, other measurement, e.g.,
1212 K. Matsui and K. Sakai

Table 2. Comparison with heartbeat rate data of raw and calibration

RD CD Deviation rate(%)
1 Number of data 2102 3011 43.2
Mean value 101.6 101.6 0
Min value 73.0 74.5 0
Max value 126.0 125.3 0
2 Number of data 2057 2966 44.9
Mean value 96.5 96.3 0
Min value 79.0 79.0 0
Max value 113.0 113.0 0
3 Number of data 4194 5103 21.6
Mean value 96.5 99.0 0
Min value 79.0 83.0 0
Max value 113.0 125.0 0
4 Number of data 2057 2966 44.9
Mean value 99.5 94.1 0
Min value 83.0 94.1 0
Max value 123.0 119.7 0
5 Number of data 1679 946 43.6
Mean value 104.1 102.5 0
Min value 81.0 79.0 0
Max value 124.0 121.0 0
6 Number of data 1538 2447 59.1
Mean value 94.8 93.7 0
Min value 77.0 77.5 0
Max value 128.0 125.2 0
7 Number of data 2164 3073 42.0
Mean value 104.6 104.2 0
Min value 88.0 88.0 0
Max value 137.0 134.0 0
8 Number of data 2057 2966 44.1
Mean value 99.5 102.5 0
Min value 83.0 81.5 0
Max value 123.0 123.0 0

measurement road heat by other devices, should be taken into next step. In next
part, GPS data will be shown as the digital maps, and some of data should
be wrong as walking path. That point should be improved by using calibration
software such as Road API by Google.Inc [23] or collecting various types of data
and calibrated them [24].
 A Proposal for a Dynamic Digital Map to Prevent Heatstroke 1213

5.2 Map Creation

Figure 6 illustrates No. 4, 5, 7, and 8 participants’ digital maps digital maps

as representative; left side indicates their status of surrounding temperature,
and right side indicates the heartbeat rates. Also, graphs in bottom side show
time series data of the temperature and heartbeat to show changes according
to people’s movement. The graphs in bottom side show time series data of the
temperature and heartbeat to show changes according to people’s movement.
As mentioned above about GPS data, some of walking paths were not
matched even though these were calibrated using QZSS (shown as black frames
in Fig. 6). Also, GPS data were missing when the participants went through a

Fig. 6. Proposed dynamic digital map for participants

1214 K. Matsui and K. Sakai

point of underground road in the station. As is characteristic of GPS data, data

calibration is necessary to account for missing data from underground roads.

5.3 Heatstroke Prevention

To prevent heatstroke during walking in summer, people’s surrounding temper-
ature and heartbeat rate could be indicators to determine ranges of risk for
individual.

5.3.1 People Surrounding Temperature

From the view point of heatstroke, there were no strong correlation between the
people’s surrounding temperature and their heartbeat, which collected data in
the experiment. However, their surrounding temperature were higher than the
outdoor temperature, which JMA announced, the average of the surrounding
temperature was 35.6◦ . As Fig. 4 showed, 31◦ in summer indicates danger of
heat stroke, the participant’s surrounding temperature was much higher than
the indicator.

5.3.2 Heartbeat Rate

The risk would be higher when people have higher heartbeat, e.g., over 120 bpm.
So, the results of experiment did not show the participant’s heat stroke risk, but
the proposed map showed promising to be useful to avoid the heatstroke. Because
the maps showed customized information for each participant to understand their
environment and health conditions.

6 Discussion
With the results, following points should be improved.

6.1 Data Accuracy

According to the results of data availability, GPS data should be calibrated
for providing precise digital maps. Using algorithms or API for walking path
collected by GPS could be one of solutions.

6.2 Information Processing

This work used people’s surrounding temperature, but mechanism of heat bal-
ance is not affected by outdoor temperature. To sense heatstroke risk precisely,
it is better to use the heat index like Wet Bulb Globe Temperature (WBGT),
which focuses on the heat exchange and balance between the human body and
the outside air [25]. To process WBGT data, humidity, thermal environment
such as solar radiation and radiation are necessary to collect. In order to change
to use WGBT, utilization of other IoT device should be considered.
 A Proposal for a Dynamic Digital Map to Prevent Heatstroke 1215

7 Conclusion and Remarks

In this study, we proposed the dynamic digital map, which display their envi-
ronmental and physical data to prevent heatstroke during waking in daytime
of summer. To test a system, which consists of three types of IoT devices for
data collection of GPS, people’s surrounding temperature, and heartbeat, and
smartphones, creates proposed maps, we conducted the 1 day experiment with
participants with different background, and created maps shows their walking
path with their surrounding temperature and heartbeat. In addition, we checked
each data availability, and the data could be utilized our purposes. The results
showed our proposed map can be utilized for one of information provision ways
to avoid heatstroke by telling user’s environmental and health conditions.
As the next step of this study, the results as the digital map and the infor-
mation how to avoid heatstroke when they face to the risk will be shown to
participants and check their behavior would be changed according to provided
information.

Acknowledgements. This work was supported in part by the R&D project “Design
of Information and Communication Platform for Future Smart Community Services”
by the Ministry of Internal Affairs and Communications (MIC) of Japan, and was sup-
ported by Funds for the Integrated Promotion of Social System Reform and Research
and Development, MEXT, Japan, and by MEXT/JSPS KAKENHI Grant (B) Number
17K12789 and SECOM Science and Technology Foundation.

References
1. Ministry of Health, Labour and Welfare: “Number of deaths due to heat
stroke (demographics statistics).” https://www.mhlw.go.jp/stf/seisakunitsuite/
bunya/0000121413.html. Accessed 25 Nov 2018. (Japanese Only)
2. Ministry of Internal Affairs and Communications: “Emergency transportation situ-
ation due to heat stroke.” http://www.fdma.go.jp/neuter/topics/houdou/h29/10/
291018 houdou 3.pdf. Accessed 25 Nov 2018. (Japanese Only)
3. Japan Meteorological Agency: “Call for attention of heat stroke in abnormal
weather early warning information.” https://www.jma.go.jp/jma/kishou/know/
kurashi/soukei netsu.html. Accessed 25 Nov 2018. (Japanese Only)
4. Jun, M., Fumiaki, F., Hideo, T.: Urban climate in the Tokyo metropolitan area in
Japan. J. Environ. Sci. 59, 54–62 (2017)
5. Google: Google Maps. https://www.google.com/maps. Accessed 25 Nov 2018
6. Yahoo Japan Corporation: “Yahoo!Map (Japanese ver). https://map.yahoo.co.jp/.
Accessed 25 Nov 2018
7. Zanella, A., et al.: Internet of things for smart cities. IEEE Internet Things J. 1(1),
22–32 (2014)
8. Sakhardande, P., Hanagal, S., Kulkarni, S.: Design of disaster management system
using IoT based interconnected network with smart city monitoring. In: Interna-
tional Conference on Internet of Things and Applications (IOTA), pp. 185–190.
IEEE (2016)
1216 K. Matsui and K. Sakai

9. Mcmahon, D., Don, D., et al.: Effects of digital navigation aids on adults with
intellectual disabilities: comparison of paper map, Google maps, and augmented
reality. J. Spec. Educ. Technol. 30(3), 157–165 (2015)
10. Gerla, M., et al.: Internet of vehicles: from intelligent grid to autonomous cars and
vehicular clouds. In: 2014 IEEE World Forum on Internet of Things (WF-IoT),
pp. 241–246. IEEE (2014)
11. TripAdvisor, Inc.: TripAdvisor (Japanese ver). https://www.tripadvisor.jp/.
Accessed 25 Nov 2018
12. Yelp, Inc.: yelp. https://www.yelp.com/. Accessed 25 Nov 2018
13. Xie, K., et al.: Use of real-world connected vehicle data in identifying high-risk
locations based on a new surrogate safety measure. Accid. Anal. Prev. (2018).
https://doi.org/10.1016/j.aap.2018.07.002
14. Uber Technologies Inc.: Uber. https://www.uber.com/. Accessed 25 Nov 2018
15. Lyft, Inc.: Lyft. https://www.lyft.com/. Accessed 25 Nov 2018
16. Koopman, C., Mitchell, M., Thierer, A.: The sharing economy and consumer pro-
tection regulation: the case for policy change. J. Bus. Entrepreneurship L. 8, 529
(2014)
17. Uber Engineering: “kepker.” https://kepler.gl/. Accessed 25 Nov 2018
18. Gomes, G.A., Santos, E., Vidal, C.A.: Interactive visualization of traffic dynam-
ics based on trajectory data. In: 2017 30th SIBGRAPI Conference on Graphics,
Patterns and Images (SIBGRAPI), pp. 111–118. IEEE (2017)
19. Internet Engineering Task Force (IETF): “The OAuth 2.0 Authorization Frame-
work.” https://tools.ietf.org/html/rfc6749. Accessed 25 Nov 2018
20. Scargle, J.D.: Studies in astronomical time series analysis. II-Statistical aspects of
spectral analysis of unevenly spaced data. Astrophys. J. 263, 835–853 (1982)
21. Google: Google Fusion Tables. https://fusiontables.google.com/. Accessed 25 Nov
2018
22. Ministry of the Environment: “Heat stroke prevention information site.” https://
fusiontables.google.com/. Accessed 25 Nov 2018
23. Google Maps Platform: “Roads API.” https://developers.google.com/maps/
documentation/roads/intro. Accessed 25 Nov 2018
24. Wu, J., et al.: Fast complementary filter for attitude estimation using low-cost
MARG sensors. IEEE Sens. J. 16(18), 6997–7007 (2016)
25. Budd, G.M.: Wet-bulb globe temperature (WBGT)—its history and its limitations.
J. Sci. Med. Sport. 11(1), 20–32 (2008)
 Analysis of Consumers Perceptions
of Food Safety Risk in Social Networks

Alessandra Amato1 , Walter Balzano1 , Giovanni Cozzolino1(B) ,

and Francesco Moscato2
1
University of Naples Federico II, via Claudio 21, Naples, Italy
[email protected],
{walter.balzano,giovanni.cozzolino}@unina.it
2
Department of Scienze Politiche, University of Campania “Luigi Vanvitelli”,
Caserta, Italy
[email protected]

Abstract. Data analysis is increasingly becoming a fundamental com-

petence for the success of companies, but also in many social phenomena.
Particular attention deserves the health sector because compared to oth-
ers, and especially compared to the economic one, it has fewer case stud-
ies in which the social network analysis has been applied and, therefore,
its potential has not yet been fully understood and adequately exploited.
Taking into account this consideration, in this paper we investigate a
methodology for semantic analysis of textual information obtained from
social media streams, in order to perform an early identification of food
contaminations. As a case study, we consider a set of reviews gathered
from the social network Yelp [1], on which we perform the textual anal-
ysis foreseen in the proposed methodology.

1 Introduction
Social Media Analytics is a digital monitoring activity that allows you to check
the effectiveness of one or more social networks through a detailed analysis of the
data flow. In addition, Social Media Analysis collects and analyses the metrics
not only of the most famous social networks, but also of all social media on the
web, such as online newspapers, forums and blogs.
Social Media Analytics, also called Social Network Analysis (SNA), is there-
fore a young methodology that is widely used in the field of social sciences and
has inevitably developed in the field of economics and marketing. In the digital
age, carrying out this activity is of fundamental importance for those involved
in digital marketing, since, once the metrics are analysed, it is possible to under-
stand how to exploit the enormous potential of social media.
Thanks to the media analysis it is possible to obtain many data and infor-
mation on its visitors, customers, potential customers or simple followers. Here
are some of the features:

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1217–1227, 2020.
https://doi.org/10.1007/978-3-030-15032-7_102
1218 A. Amato et al.

• get information on users’ place;

• get information on the sex and age of visitors;
• identify the most searched and used keywords;
• identify the most used keywords for a specific item and service on sale, or on
certain posts, links, etc.;
• check times and days when the social is most visited by users;
• monitor the general behaviour of users.

The data obtained can be of various kinds and in particular we can identify
the five most important: textual data (comments, post...), data related to the
analysed network (Facebook friendships, Twitter or Instagram following sys-
tem...), actions (shares, reactions, “like”...), hyperlinks, useful data for SEO
(Search Engine Optimization).
Knowing how to communicate in healthcare is important and social media
is an indispensable tool today, but in Italy there are not many Health Authori-
ties to have understood it. According to the OASI 2018 Report of the Bocconi
University in collaboration with CEDAS [2], in a survey conducted on 51 com-
panies, there is no analysis and inclusion of information and feedback collected
through social media within the business decision-making processes. No com-
pany among those examined makes use of structured analytics and reporting
systems that allow the collection and analysis of information coming from social
media in a complete and systematic way. Only 23% of companies that declare
to perform data analysis uses special tools, such as Excel processing, or ad hoc
tools such as Google Analytics or Hootsuite, while the remaining 77% do not use
any dedicated tools. 50% of the companies interviewed declare to analyse only
some information and 21% all the information collected on social networks, but
in both cases in an unstructured way, 4% analyze only some information, but in
a structured way, while the remaining 25% uses social media without analysing
any of what it collects, making it impossible, therefore, to evaluate the quality
and effectiveness of using the tool as an added value. And it’s a huge missed
opportunity.
However, the researches already carried out confirms the possible usefulness
of SNA use by decision-makers in the public health field. In fact, the first anal-
ysis of real-time data for the purpose of revealing the spread of a virus was
made in 2009 with the H1N1 influenza. This is also the conclusion of a study
[3] conducted by a team of bioinformatics researchers of the Computer Science
Department of Columbia University (New York) and published on the Journal of
the American Medical Informatics Association. They proposed a program able to
identify, analysing Yelp posts, possible food-borne outbreaks among restaurants
of New York.
Moreover, they proved that SNA approach was faster and more reliable than
the report to the Department of Health and Mental Hygiene, because people
consider more practical and comfortable to post messages on social media rather
then to report any accidents to the public health institutions. This trend is
occurring not only in the case of food infections: according to a recent estimate,
 Analysis of Consumers Perceptions of Food Safety Risk in Social Networks 1219

nowadays the first reports of epidemics, for example influenza, are made in 70%
of cases via social media.
On these basis, in this paper we propose a methodology, based on semantic
technologies, aiming to extract hot topics from social network streams in order
to perform an early identification of food contamination events. The proposed
methodology can be also applied for social and political opinion mining cam-
paigns [4–6]. As a case study, we consider a set of reviews gathered from the
social network Yelp [1], on which we perform the textual analysis foreseen in the
proposed methodology.
Text Mining
Information retrieval and knowledge management activities on data gathered
form social network requires to adopt Big Data techniques and efficient process-
ing algorithms [7–9]. Big Data approaches adopt graph algorithms and parallel
and distributed architectures that make use of platforms like Apache Hadoop
[10], Spark, Flink and other platforms for data-intensive distributed applications
[11]. These platforms implement popular programming models, such as MapRe-
duce, in which a master node splits input datasets into subsets, assigned to
parallel and independent worker nodes, which outputs are used to compute the
result of the original job.
Many morpho-semantic approaches have been proposed in medical domain
for text analysis. Pratt [12] dealt with the identification and the transformation
of terminal morphemes in the English medical dictionary; Wolff [13] worked on
the classification of the medical lexicon based on formative elements of Latin and
Greek origin; Pacak and Pratt [12] classified the diseases words ending in -itis;
Norton and Pacak [14] classified the surgical operation words ending in -ectomy
or -stomy; Dujols et al. treated the suffix -osis.
Lovis et al. [15], proposed an approach to solve the problem of automatic
population of thesauri, deriving the meaning of the words from the morphemes
that compose them; Hahn et al. [16] segmented the sub-words in order to recog-
nise medical documents; in [17] authors exploit SNOMED thesaurus applying
machine learning methods on it to analyse morphological data.
Many other works deal with the definition and the implementation of an
architecture for information structuring, by mapping, through the exploitation
of semantic technologies, the entities extracted from different corpora on a set
of shared concepts [18–20], in order to ensure the interoperability.
Many studies have been conducted for the enhancement of semi-automatic
medical information extraction from structured repositories or unstructured
texts. In [21] author proposes a clinical information system, that, through
Information Extraction techniques, transform medical unstructured informa-
tion retrieved on clinical records into a structured format. Rink [22] proposes
a method that exploits machine learning algorithms, in conjunction with some
language resources (such as Wikipedia, WordNet, etc.) and processing resources
(POS tagger, chunk parser, etc.) to perform a semi-automatic automatic feature
extraction from electronic medical reports. In [23] the authors adopt NLP tools
to automatically extract medications and related information from discharge
1220 A. Amato et al.

summaries. Finally, several modelling approach have been proposed [24–26] to

define a proper data model but also hardware approaches to improve perfor-
mances [27].
The rest of the paper is organised as follows. In Sect. 2, we discuss the graph
model, used in social network analysis, and we introduce Text Analysis Tools
adopted in our methodology. Then, in Sect. 3, we report the design of performed
experimentation. Computational results are summarised in Sect. 4. Conclusions
and remarks are discussed in Sect. 5.

2 A Social Network Analysis Methodology

Nowadays, social networks analysis techniques mostly exploit the graph theory
to model the network. Through graph theory, a single user or groups of users of a
social network can be represented as a vertex of the graph and their relationships
as edges. In this way, network data can be represented in a matrix form, directly
analysable without the need of drawing the graph, that could be unfeasible for
large social network data sets [28–30]. The edges can have a direction and a
weight, that represent respectively which vertex influences the other and the
strength of the relation.

2.1 Text Analysis Tools

2.1.1 TaLTac
TaLTaC (Trattamento automatico Lessicale e Testuale per l’analisi del Con-
tenuto di un Corpus) [31] is a software for natural language processing (Lexical
and Textual), able to analyse the content of a Corpus, performing many different
operations on documents like Text Analysis, Text Mining and Corpus Analysis.
Text Mining is one of the main features of TaLTaC that extract relevant informa-
tion from the Corpus, by exploiting endogenous and exogenous resources. The
endogenous resources are composed by a number of fragments and by categorical
variables, associated to the text in order to identify logically related fractions of
the corpus. The exogenous resources allows to identify peculiar language of the
text by computing frequency of a term or lexical unit in a corpus.

2.1.2 GATE
GATE (General Architecture for Text Engineering) [32] is a family of software
built around a common specific framework (GATE Embedded) with which it
is possible to implement text analysis and information extraction applications.
GATE comes with many language and processing resources that can be com-
bined into running applications. For example, ANNIE is one of GATE’s built-in
application, that is an information extraction system constituted by a set of lan-
guage resources and processing resources structured in a pipeline specialised for
information extraction tasks.
 Analysis of Consumers Perceptions of Food Safety Risk in Social Networks 1221

2.2 Methodology
In our methodology we foresee multiple phases performing different textual elab-
oration, in order to achieve a fine grain analysis of different data sources.

2.2.1 Text Pre-treatment

A text pre-treatment has to be performed in order to obtain better results in
the subsequent phases. It consists of the following two steps.
Normalization normalizes the writings of names, acronyms and other entities
by executing the following operations:
• changing apostrophes into stresses in order to determinate the right word;
• annotate words or sequence of words with labels reporting their right mean-
ing (i.e. a name can be mistaken for a noun, ‘Rose’ or ‘rose’);
• capitalize letters, according to the eventual labels associated with the word
(i.e. if ‘Rose’ is a proper name it must have the capital letter, not in other
cases).
Spelling errors correction that involves the comparison of misspelled words with
the system dictionary in order to correct and analyse them in the right way.

2.2.2 Lexical Analysis

Lexical analysis performs the segmentation of the Corpus (i.e. the division of the
text into segments, sequences of graphic forms separated by a strong divider like
punctuation) and the estimation of various lexical parameter of each segment,
starting from single elements that compose them. Other important operations are
the Tagging and the Lexation: Tagging links a description of the grammatical or
semantic characteristics [33,34] to each word, while Lexation identifies sequence
of words defined as a unique entity during the pretreatment. Finally, we define
the Corpus’ key words by comparing their frequency, assuming that the ones
that have a noticeable standard deviation are more meaningful.

2.2.3 Textual Analysis

Textual Analysis is mainly composed by the study of the Concordances and by
co-occurrences identification.
The study of the Concordances examines the context of every chosen word or
segment. We adopt TF-IDF index to rank the results by computing the frequency
and distribution of the query keywords in the documents provided, according to
Eq. (1):
N
TF-IDF = ft,d · log (1)
Dt
where:
ft,d is the frequency of term t in the resource d;
N is the number of resources of the corpus;
Dt is the number of resources containing the term t.
Through the identification of co-occurrences, the set of near elements that
repeats in the text are treated as single entities, thus enhancing the definition
of primary concepts contained in the Corpus.
1222 A. Amato et al.

3 Experimental Design
We design an experimental campaign in order to analyse on-line social networks
data sets and to derive information regarding contamination and food-related
risks. The data set is constituted of tuples collected from the social network
Yelp, where users publish reviews about local businesses [1]. As a simplified
experimentation, we extracted from collected tuples an anonymized subset of
fields, i.e. user name, reviewed place, date of the review and the textual review.

3.1 TaLTac Analysis

Firstly, we perform a text pre-treatment, in order to normalize and compute the
sub-occurrences in the Corpus (see Fig. 1).

Fig. 1. Text normalization window

Through the Textual Analysis we perform the identification of the segments,

that are saved in two files:
1. the “List of Segments (with index IS)” which contains the segments with their
number of occurrences, number of elements forming the segment and the IS
index;
2. the “List of Significant Segments” containing significant segments.
Then, we compute TF-IDF index and we analyse the specificity of our Cor-
pus. Last, we compute the concordances and co-occurrences (see Fig. 2).
 Analysis of Consumers Perceptions of Food Safety Risk in Social Networks 1223

Fig. 2. Co-occurences window

3.2 GATE Analysis

We exploited the functionalities offered by a Gate plugin for Italian language
available at https://github.com/sercanto/GATE-plugin-Lang Italian, that pro-
vides a complete application, named Italian NE, with a set of processing
resources for the Italian language treatment: Italian Grammar, Italian Gazetteer,
Italian Tokeniser, Adapt Tokeniser to Tagger. Moreover, in our processing
pipeline we add the Italian POS Tagger and ANNIE Sentence Splitter plug-
ins.
Then we proceed with the composition and initialization of the Information
Retrieval pipeline, customizing the configuration on the basis of ANNIE pipeline
(see Fig. 3):

• Document Reset PR: remove all the annotation sets and their contents
made on document, resetting it to its original state;
• ANNIE Italian Tokeniser: splits the text into very simple tokens such as
numbers, punctuation and words of different types;
• ANNIE Sentence Splitter: segments the text into sentences through a
cascade of finite-state transducers;
• Adapt Tokeniser to Tagger
• Italian POS Tagger: assigns a part-of-speech tag to each word or symbol;
1224 A. Amato et al.

Fig. 3. ANNIE pipeline for Italian language

• Italian Gazetteer: identifies entity names in the text exploiting a set of

predefined lists (such as Organizations, Cities, Names, etc.);
• Italian Grammar: ANNIE named entity grammar;

4 Computational Results
We show the computational results obtained on a subset of 50 tuples taken from
the initial data set, made up of more than 5000 tuples collected from Yelp social
network, regarding a specific place reviewed by the users.

4.1 TaLTac Results

Through TaLTac we obtain the peculiar lexicon (shown in Fig. 4), that is the list
of words with highest occurrences and TF-IDF. Since the lexicon is composed
by words like “games”, “beer”, “good”, “great”, “food”, “place”, we can assume
that the analysed tuples mostly contain reviews about a place where you can
eat, drink beer or play some games.

Fig. 4. Peculiar lexicon by occurences

 Analysis of Consumers Perceptions of Food Safety Risk in Social Networks 1225

Fig. 5. Concordances of the word “food”

Fig. 6. Annotation sets produced by GATE for an Italian corpus.

The co-occurrences analysis, reported in Fig. 2, strengthen the good senti-

ment about the place reviewed, with expressions such as “a good”, “the bar”,
“beer selection”, “a place”, “the games”. Last, to improve the contextualization
of extracted tuples, we studied the concordances of a meaningful word, “food”,
through all the Corpus (see Fig. 5), in order to understand the context of the
lemma.

4.2 GATE Results

By using GATE processing resources (Fig. 6) we can assert more grammatical

features of our data and, thus, we can deduce even more meaning respect to the
analysis described in Subsect. 4.1.
1226 A. Amato et al.

5 Conclusions
In this work, we focused on the analysis of the content of the text obtained
from social media streams. Through our study it is proposed a methodology
for early detection of food diseases gathering information from different social
sources, such as reviews. From these basis it can be performed further analysis
in health domain (i.e. the first reports of epidemics, like influenza), but also
on different domains such as marketing (through an analysis of the feedback of
customers) or cyber-security (detection of text containing malicious messages),
crawling information from social networks data streams.

References
1. The Free Encyclopedia Wikipedia. Yelp
2. SDA Bocconi: Rapporto oasi 2018 - osservatorio sulle aziende e sul sistema sanitario
italiano (2018)
3. Effland, T., Lawson, A., Balter, S., Devinney, K., Reddy, V., Waechter, H., Gra-
vano, L., Hsu, D.: Discovering foodborne illness in online restaurant reviews. J.
Am. Med. Inform. Assoc. ocx093 (2018)
4. Amato, F., Moscato, V., Picariello, A., Piccialli, F.: SOS: a multimedia recom-
mender system for online social networks. Future Gener. Comput. Syst. 93, 914–
923 (2017)
5. Piccialli, F., Chianese, A.: The Internet of Things supporting context-aware com-
puting: a cultural heritage case study. Mob. Netw. Appl. 22(2), 332–343 (2017)
6. Amato, F., Moscato, V., Picariello, A., Sperlı̀, G.: Multimedia social network mod-
eling: a proposal. In: 2016 IEEE Tenth International Conference on Semantic Com-
puting (ICSC), pp. 448–453. IEEE (2016)
7. Balzano, W., Murano, A., Stranieri, S.: Logic-based clustering approach for man-
agement and improvement of VANETs. J. High Speed Netw. 23(3), 225–236 (2017)
8. Balzano, W., Vitale, F.: DiG-park: a smart parking availability searching method
using V2V/V2I and DGP-class problem. In: 2017 31st International Conference
on Advanced Information Networking and Applications Workshops (WAINA), pp.
698–703. IEEE (2017)
9. Balzano, W., Murano, A., Vitale, F.: Hypaco–a new model for hybrid paths com-
pression of geodetic tracks. In: The International Conference on Data Compression,
Communication, Processing and Security, CCPS-2016 (2016)
10. The Apache Hadoop project. Apache hadoop
11. Javanmardi, S., Shojafar, M., Shariatmadari, S., Ahrabi, S.S.: FR trust: a fuzzy
reputation-based model for trust management in semantic P2P grids. Int. J. Grid
Utility Comput. 6(1), 57–66 (2015)
12. Pratt, A.W., Pacak, M.: Identification and transformation of terminal morphemes
in medical English. Methods Inf. Med. 8(2), 84–90 (1969)
13. Wolff, S.: The use of morphosemantic regularities in the medical vocabulary for
automatic lexical coding. Methods Inf. Med. 23(4), 195–203 (1984)
14. Norton, L.M., Pacak, M.G.: Morphosemantic analysis of compound word forms
denoting surgical procedures. Methods Inf. Med. 22(1), 29–36 (1983)
15. Lovis, C., Baud, R., Rassinoux, A.-M., Michel, P.-A., Scherrer, J.-R.: Medical dic-
tionaries for patient encoding systems: a methodology. Artif. Intell. Med. 14(1),
201–214 (1998)
 Analysis of Consumers Perceptions of Food Safety Risk in Social Networks 1227

16. Hahn, U., Honeck, M., Piotrowski, M., Schulz, S.: Subword segmentation–leveling
out morphological variations for medical document retrieval. In: Proceedings of the
AMIA Symposium, p. 229. American Medical Informatics Association (2001)
17. Grabar, N., Zweigenbaum, P.: Automatic acquisition of domain-specific morpho-
logical resources from thesauri. In: Proceedings of RIAO, pp. 765–784. Citeseer
(2000)
18. Xhafa, F., Barolli, L.: Semantics, intelligent processing and services for big data.
Future Gener. Comput. Syst. 37, 201–202 (2014)
19. Moore, P., Xhafa, F., Barolli, L.: Semantic valence modeling: emotion recognition
and affective states in context-aware systems. In: Proceedings - 2014 IEEE 28th
International Conference on Advanced Information Networking and Applications
Workshops, IEEE WAINA 2014, pp. 536–541 (2014)
20. Xhafa, F., Asimakopoulou, E., Bessis, N., Barolli, L., Takizawa, M.: An event-
based approach to supporting team coordination and decision making in disaster
management scenarios. In: 2011 Third International Conference on Intelligent Net-
working and Collaborative Systems (INCoS), pp. 741–745. IEEE (2011)
21. Fette, G., Ertl, M., Wörner, A., Kluegl, P., Störk, S., Puppe, F.: Information
extraction from unstructured electronic health records and integration into a data
warehouse. In: GI-Jahrestagung, pp. 1237–1251 (2012)
22. Rink, B., Harabagiu, S., Roberts, K.: Automatic extraction of relations between
medical concepts in clinical texts. J. Am. Med. Inf. Assoc. 18(5), 594–600 (2011)
23. Doan, S., Bastarache, L., Klimkowski, S., Denny, J.C., Xu, H.: Integrating existing
natural language processing tools for medication extraction from discharge sum-
maries. J. Am. Med. Inf. Assoc. 17(5), 528–531 (2010)
24. Amato, F., Cozzolino, G., Moscato, F., Moscato, V., Picariello, A., Sperli, G.: Data
mining in social network. Smart Innov. Syst. Technol. 98, 53–63 (2019)
25. Fusella, E., Cilardo, A.: Minimizing power loss in optical networks-on-chip through
application-specific mapping. Microprocess. Microsyst. 43, 4–13 (2016)
26. Fusella, E., Cilardo, A.: H2ONoC: a hybrid optical–electronic NoC based on hybrid
topology. IEEE Trans. Very Large Scale Integration (VLSI) Syst. 25(1), 330–343
(2017)
27. Cilardo, A.: Exploring the potential of threshold logic for cryptography-related
operations. IEEE Trans. Comput. 60(4), 452–462 (2011)
28. Amato, F., Moscato, V., Picariello, A., Piccialli, F., Sperlı́, G.: Centrality in hetero-
geneous social networks for lurkers detection: an approach based on hypergraphs.
Concurrency Comput.: Pract. Experience 30(3), e4188 (2018)
29. Chianese, A., Marulli, F., Piccialli, F., Valente, I.: A novel challenge into mul-
timedia cultural heritage: an integrated approach to support cultural informa-
tion enrichment. In: 2013 International Conference on Signal-Image Technology &
Internet-Based Systems (SITIS), pp. 217–224. IEEE (2013)
30. Piccialli, F., Jung, J.E.: Understanding customer experience diffusion on social
networking services by big data analytics. Mob. Netw. Appl. 22(4), 605–612 (2017)
31. Adolfo Morrone Sergio Bolasco, Francesco Baiocchi. Taltac
32. The GATE project team. Gate
33. D’Acierno, A., Moscato, V., Persia, F., Picariello, A., Penta, A.: iWIN: a sum-
marizer system based on a semantic analysis of web documents. In: Proceedings
- IEEE 6th International Conference on Semantic Computing, ICSC 2012, pp.
162–169 (2012)
34. Albanese, M., D’acierno, A., Moscato, V., Persia, F., Picariello, A.: Modeling rec-
ommendation as a social choice problem. In: Proceedings of the 4th ACM Confer-
ence on Recommender Systems, RecSys 2010, pp. 329–332 (2010)
 Implicit and Continuous Authentication
of Smart Home Users

Noureddine Amraoui(B) , Amine Besrour, Riadh Ksantini,

and Belhassen Zouari

Higher School of Communications of Tunis, 2083 Ariana, Tunisia

[email protected],
{amine.besrour,riadh.ksantini,belhassen.zouari}@supcom.tn

Abstract. This paper presents a security framework that continuously

authenticates smart homes users in order to make sure that only autho-
rized ones are allowed to control their Internet of Things (IoT) devices
while, at the same time, preventing them in case of performing abnormal
and dangerous control actions. To do so, control commands under normal
operation of both users and devices, are first implicitly traced to build a
One Class Support Vector Machine (OCSVM) model as a baseline from
which deviations (i.e., anomalous commands) should be detected and
rejected, while normal observations (i.e., normal commands) should be
considered as legitimate and allowed to be executed. Experiments con-
ducted on our artificial datasets show the efficiency of such user behavior-
based approach achieving at least 95.29% and 4.12% of True Positive
(TP) and False Positive (FP) rates, respectively.

1 Introduction
As Internet of Things (IoT) devices are being deployed in homes making them
smart and more convenient, smart homes owners are now able to manage and
control their intelligent devices (e.g., door lock) either from the inside at home
or from any outside location. While bringing such convenience, IoT devices are
susceptible to be maliciously controlled. Many possible ways may allow an adver-
sary to take control over the home owner’s control application. Examples of
such threats include compromising the user account (e.g., stealing username and
password) or simply finding the owner’s smartphone dropped somewhere. Con-
sequently, the adversarial user will be fully privileged to control any device with
no way to verify his/her identity. For example, a burglar will be able to turn-off
the surveillance camera, then unlocking the main door lock to enter the house.
On the other hand, even authorized users could intentionally/unintentionally
perform dangerous control actions that may cause serious problems. For exam-
ple, a kid may use the home control panel to raise the heating degree on the
thermostat which may cause a fire. So the two important questions that one
would ask are: (1) how to ensure the real identity of a smart home user in order
to protect his/her IoT devices even when credentials and the controlling applica-
tion have been compromised? and (2) how to detect abnormal control commands
even if the user is authorized?
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1228–1239, 2020.
https://doi.org/10.1007/978-3-030-15032-7_103
 Implicit and Continuous Authentication of Smart Home Users 1229

Authentication is a fundamental security aspect that enables computer sys-

tems to recognize specific users. Ordinary, this identity verification can take the
form of a simple username and password, or involve more complex means such
as biometrics. However, these techniques mainly have two drawbacks for smart
home users. First, they only provide verification at the time of login. Any future
change in user identity will go undetected. This is very risky considering that
most smartphone applications, which a user usually uses to control smart home
devices, only provide a login-time authentication. That is, no verification will be
required once the app has been closed and then opened another time. Addition-
ally, conventional authentication heavily rely on an explicit user interaction and
cooperation (e.g., entering a password) which makes re-authentication, at each
IoT device control request, a tedious task. On the other hand, generally, smart
home users usually control their intelligent devices in some unique and frequent
patterns. These patterns can be described by many behavioral features such as
rate (i.e., number of devices controlled in a time window), sequence (i.e., order
of control), etc. In contrast to classic techniques, authenticating users based on
their behaviors has many advantages [1]. In fact, such behavioral features could
non-obtrusively be collected. That is, neither authorized nor unauthorized users
are aware of being forensically monitored. Moreover, identity verification based
on these features could be continuously performed throughout the entire user’s
control session.
Tracking, collecting, and assessing users activities of any cyber-system is bet-
ter known as User Behavior Analytics (UBA) [3]. This paradigm assumes that
users tend to follow frequent patterns when interacting with and using com-
puting devices and resources. UBA technologies are mainly enabled by Machine
Learning (ML) techniques which are better known as Anomaly Detection (AD).
The basic idea behind AD is to first build a baseline model over the normal data.
Then, deviations from this baseline could be further detected accordingly.
In this paper, we leverage UBA and AD paradigms to devise a framework
that continuously authenticate users control commands of smart homes devices,
based on their implicitly collected behaviors. The framework is distributed on
two levels of a smart home system architecture. First, in the Cloud backend, the
framework builds normal models that define legitimate user data which consists
of control commands traced through an access log as well as devices usage data
describing the status of IoT devices in different times of the day. More specifi-
cally, a One Class Support Vector Machines (OCSVM) model is trained on a set
of normal behavioral scores which have been calculated using this data. Then,
according to the nature of control (i.e., locally from a control panel, or remotely
from a smartphone app), the framework continuously analyzes every requested
control command either in the Hub or in the Cloud backend, respectively, using
the trained OCSVM model. This latter works as a baseline from which, anoma-
lous commands are detected and rejected, while normal ones are confirmed and
executed.
The rest of this paper is organized as follows. We first present the related
works in Sect. 2. The design of our proposed framework will be explained in
1230 N. Amraoui et al.

Table 1. Related works comparison

Domain Reference Threat(s) Behavioral Features AD Technique(s) Data Source

Web applications [4] Application Requests number per SVD, RM Web logs
layer DDoS time window, duration
between requests
[7] Request rate PCA
Databases [5] Abuse of Summarizing queries Clustering Audit logs
privileges result tuples in clusters
for each user
[6] User account Rarity, volume, new Anomaly scores
compromise, object accessed assignment
abuse of
privileges
OSNs [11] Fake, Number of likes per day, PCA UGC
compromised number of likes in
Facebook different categories
identities
[9] User account First activity, action Euclidian distance,
compromise latency, visit duration, self-variance
browsing sequence, etc.

details in Sect. 3. In Sect. 4, we present and discuss our experimental results.

Finally, we conclude this work and determine some future perspectives in Sect. 5.

2 Related Work

Typically, a User Behavior Analytics (UBA) system based on Anomaly Detection

(AD) includes two main processes [12]. (1) User Profiling: in which, the user
is characterized by his/her historical normal behavioral data in order to build
usage profiles. (2) User Identification: in which, the observed user behavior is
compared to the pre-established normal profiles to take decisions accordingly. As
summarized in Table 1, this approach has been extensively leveraged to secure
different applications and computing systems.
Web Applications. Web logs may be used to extract a set of features describ-
ing the user behavior when requesting Web resources. Consequently, any anoma-
lous request could be then detected based on normal requesting patterns that
has been built on previous users sessions. In this context, several methods
has been proposed to detect Application layer Distributed Denial of Service
(DDoS) attacks. Liao et al. used the number of normal user requests in every
sub-time window in addition to the duration between them [4]. These fea-
tures are extracted based on the idea that users access and spend more time
on pages of their interests. To detect anomalies, a hybrid classification algo-
rithm based on Sparse Vector Decomposition and Rhythm Matching method
was used. Najafabadi et al. applied another anomaly detection technique based
on the Principle Component Analysis (PCA) to detect DDoS attacks [7]. The
idea was to identify the N-top principal components that better describe the
normal user behavior. Then, the data projection on the remaining components
captures anomalies and noise in the data.
 Implicit and Continuous Authentication of Smart Home Users 1231

Databases. By learning the behavior of clients when requesting DB resources

(i.e., Tables), any changes detected in this behavior could be a sign of an abnor-
mal access. Mathew et al. proposed a clustering-based AD technique to detect
the abuse of privileges of DB users [5]. The approach consists of first summarizing
query’s result tuples in a fixed vector. Then, all user’s past queries can be thought
as one cluster. When a new query arrives, if it belongs to the user’s cluster, it
will be classified as normal, or abnormal otherwise. More recently, Mazzawi et
al. presented a novel approach to determine whether a DB user activity is mali-
cious or not [6]. The idea consists of assigning high anomaly scores to actions or
groups of actions with a low probability of occurrence. Three main behavioral
features has been used to describe users activities: (1) rarity: represented by the
probability of appearance of the action in a new timeframe, (2) volume: repre-
sented by the number of occurrences of an action given that it appeared in the
timeframe, and (3) new object: amounts of new objects being accessed.
Online Social Networks (OSN). Users can be distinguished according to
their social behavioral profiles which could be built from their User Generated
Content (UGC) such as commenting, updating status, etc. Viswanath et al.
used PCA-based AD to detect fake and colluding Facebook identities [11]. Users
social behaviors were characterized by their Like activity using three types of
features: (1) temporal: number of Likes per day, (2) spatial: number of Likes
in different categories (e.g., sports, politics), and spatio-temporal: summary of
the distribution of Like categories using entropy. Ruan et al. extracted a set of
behavioral features that both characterize a user’s extroversive activities (e.g.,
action latency) and introversive activities (e.g., browsing sequence) [9]. To dif-
ferentiate between different users profiles, two measures was used: (1) Euclidian
distance: which quantify the difference between two features vectors of two users
profiles, and (2) self-variance: which is the mean difference for each pair in a
collection of profiles for a user.
Nevertheless, securing the smart home cyber system and authenticating users
based on their interactions with IoT devices is poorly addressed by the research
community. We can consider the work proposed in [8] as the only one we have
found in this context. In this work, authors proposed an access control model
to secure the controlling of smart home devices in case of user account compro-
mise. This is similar to our objective but the proposed solution is very basic.
First, authors only used association rules learning as a mean to learn normal
user behavior. Additionally, they did not use any behavioral features that may
efficiently describe a smart home user behavior. Finally, their experiments have
been conducted on a simple generated user access log with no artificial patterns.
In contrast, our solution is based on OCSVM which is a very robust AD tech-
nique. Moreover, user activity is described by many behavioral features such
as control rate, latency, and control sequence. Finally, our experiments was con-
ducted on an artificial datasets that have been generated under many parametric
conditions to simulate the user frequent patterns. To the best of our knowledge,
we are the first to investigate in the use of AD techniques in authenticating
smart homes users.
1232 N. Amraoui et al.

Fig. 1. High-level architecture of implicit-continuous authentication framework

3 Framework Design
In this section, we discuss the design of our proposal. We first present an overview
of the framework by explaining its architecture and the data collection process.
Then, we discuss the framework operation in details.

3.1 Overview

3.1.1 High-Level Architecture

Figure 1 shows the three different levels of a smart home architecture. Our
authentication framework is distributed in the top two levels of this architec-
ture where the enrollment modules on are deployed in the Cloud backend, while
the continuous authentication modules are both implemented in the Cloud and
in the Hub:

• Enrollment: an offline process aims to build normal user behavioral patterns

based on historical user access log and IoT devices usage data that have been
collected under normal operation. When the enrollment stage is finished, the
learned models are locally saved in the cloud backend and also migrated to
the hub to be used in the local continuous authentication.
• Continuous Authentication: an online process that analyzes devices control
commands requested by users to continuously authenticate them. If a control
command is locally requested (e.g., using a control panel), the analysis is done
on the Hub. However, if a control command is remotely requested (e.g., using
a smartphone), the analysis is done on the Cloud backend.

3.1.2 Data Collection

To build normal patterns of users, two different types of data must first be
collected:
 Implicit and Continuous Authentication of Smart Home Users 1233

Fig. 2. Enrollment process

• User Access Log: an access log file is a common way to provide information
about who has controlled what, and at what time. Both locally and remotely
requested control commands are traced and saved in one storage file.
• IoT Devices Status Data: this data describes the status of the smart home
system. Each record in this data describes the status of each IoT device
connected to the Hub(s) at a given time of the day.

3.2 Enrollment
Figure 2 describe the enrollment stage including four sub-processes.

3.2.1 Data Preparation

Sessions Extraction
A user session can be defined as a group of interactions that one user takes within
a given time frame. The features describing a user session of a smart home sys-
tem are: (1) Start: timestamp of the first command, (2) Wd/We: specify the type
the day (i.e., weekday or weekend), (3) Rate: number of devices controlled, (4)
Sequence: order of control, and (5) Context: the nature of control (i.e., locally
or remotely).

Devices Usage Data Preparation

Assuming the existence of a user frequent pattern, IoT devices states are some-
how repeated at a given time during a period of the day in both weekdays and
weekends. Consequently, devices status data has to be cleaned where the year
and month are removed from the timestamp and only time and the type of the
day are retained. Then, the resulting data is sorted by time.

3.2.2 Probabilistic Models Construction

Sequential Model
In the sessions log extracted, during each session, a user controls a sequence of
devices in a particular order. We construct a sequential model on this set of
1234 N. Amraoui et al.

sequences by calculating the two following parameters: (1) Initialization Vector:

containing a set of probability values that a user starts with each one of the
controllable devices at a given period of the day for both weekdays and week-
ends. (2) Transition Matrix: containing the probability values that a user transits
between each two devices.

Devices Frequent Usage

From the prepared devices usage data, constructing this model consists of calcu-
lating the probability for each device to be in a specific state at a specific period
of the day for both weekdays and weekends. For example, during the weekend,
a user 36% frequently set the thermostat between 23 and 26◦ .

3.2.3 Behavioral Scores Retrieval and Normalization

First Command Scores These scores describe a user behavior when he/she
starts a session by requesting a specific device:

• Initialization Probability: the probability that the user usually starts with
the requested device, extracted from the initialization vector of the sequential
model.
• Device Usage Probability: the frequent usage probability of the requested
device with the specific action and time, extracted from devices frequent
usage model.
• Inter-Sessions Latency: the delay between the first command of a given session
and last command of the previous one.

Activity Command Scores

These scores describe a user behavior when he/she performs a control command
preceded by other commands belonging to the same session:

• Last Transition Probability: the probability that the previously controlled

device would be followed by the device requested in this control command in
the same session.
• Current Sequence Probability: the sum of transition probabilities between all
devices requested in the current session divided by the number of transitions.
• Device Usage Probability: the frequent usage probability of the requested
device with the specific action and time, extracted from devices frequent
usage model.
• Intra-Session Latency: the delay between the requested control command and
its antecedent in the same session.
• Control Rate: the number of current controlled devices divided by the current
session duration.

To give better training and detection results, behavioral scores need be scaled
in one common range (e.g., [0, 1]). The following formula will be used for this
task, where X’ is the new value, X is the given score to be normalized, and
 Implicit and Continuous Authentication of Smart Home Users 1235

min(X) and max(X) are the smallest and the biggest values of all existing scores,
respectively.
X  = (X − min(X))/(max(X) − min(X)) (1)

3.2.4 Machine Learning Models Training

From a ML perspective, discriminating authorized control commands from unau-
thorized ones can be seen as a binary classification. However, in the Enrollment
stage, only normal commands are in our hand. One Class Classification (OCC) is
an unsupervised machine learning used in situations where only positive samples
are available which is our case. In our work, we use the One Class Support Vector
Machines (OCSVM) [10]. This model has shown high performances in detecting
anomalies in many other application domains compared to other AD algorithms
[2]. In particular, two OCSVM models will be trained on the retrieved behav-
ioral scores of user control commands. The first one is built over first commands
scores, while the second is built over activity commands scores.

3.3 Continuous Authentication

Figure 3 describes the process of the Implicit-Continuous User Authentication
based on Anomaly Detection which is executed as follows. When opening the
controlling application, a user first must log in using his/her credentials. If login
succeeded, the control interface shows up and the user can start manipulating
the smart home devices. When a control command is requested by the user,
the behavioral scores are calculated for this command according to its nature
(i.e., first or activity command). These scores will accordingly be given to the
trained OCSVM anomaly detection model for each type of command outputting
an anomaly decision i.e., (+1) normal or (−1) abnormal. Depending on this
output, the command is either executed and saved in the access log, or a re-
authentication is demanded from the user to verify his/her identity. We call

Fig. 3. Process of implicit-continuous authentication

1236 N. Amraoui et al.

an anomaly score the parameter used to prevent a user, that can successfully
pass the re-authentication while continuously showing an anomalous behavior,
from keep using the control interface. This may be the case of an adversarial
person hijacking the legitimate user credentials. To do so, the anomaly score
is first initialized after the login, and then increased after each successful re-
authentication. If this score exceeds a predefined threshold (e.g., five times), the
requested command is rejected and the user is completely logged out from the
control interface. However, if this score is still below the threshold, the requested
command is executed and saved in the access log.

4 Experimental Results
In this section, we present an experimental analysis and results to examine the
effectiveness of our proposed framework.

4.1 Setup and Datasets

The data used to train our probabilistic and ML models in the enrollment stage
are the access log and IoT devices status data. Due to the lack of such data
of real-world deployments, our experiments will be conducted on an artificially
generated datasets. The artificial datasets has been generated under the following
assumptions:

• Hub/IoT Devices: six IoT devices connected to and coordinated by one single
Hub. This latter will be the data collection point where control commands
and devices status are traced. The devices are: Thermostat, Front Door Lock,
Garage Door Lock, Living Room Bulb, Outside Camera, and Television.
• Periods of the Day: a day is partitioned on four periods: Morning (M), After-
noon (A), Evening (E), and Night (N) with a particular range of time for
each one as follows: { ‘M’: [‘08:00:00’,‘11:59:59’], ‘A’: [‘12:00:00’,‘17:59:59’],
‘E’: [‘18:00:00’,‘23:59:59’], ‘N’: [‘00:00:00’,‘07:59:59’]}.

4.2 Experimental Model

Assuming that users tend to follow different behaviors, we have created five
access logs of adversarial users (AU) that deviate from a predefined normal
user. To make sure that the performance evaluation results will not be biased
or coincidental, we conduct a 5-fold cross-validation process. Precisely, about
80% of the normal user behavioral scores was randomly selected to train each
one of the two OCSVM models. The remaining 20% is then split into five (5)
blocks which are then added to each one of the five (5) adversarial users that we
have created. This process is repeated four (4) more times and the result is five
(5) trained First and Activity OCSVM models against five (5) adversarial First
and Activity behavioral scores containing randomly selected instances from the
normal behavioral scores. The resulting test data is described in Table 2.
 Implicit and Continuous Authentication of Smart Home Users 1237

Table 2. Test data description

# of normal commands # of anomalous commands

First Activity All First Activity All
AU1 34 117 151 40 36 76
AU2 27 94 121 41 31 72
AU3 22 75 97 48 33 81
AU4 43 149 192 35 34 69
AU5 43 150 193 48 373 421

4.3 Results and Discussion

The True Positive Rate (TPR) and False Positive Rate (FPR) measurements
obtained for different trained OCSVM models against different adversarial users
are given in Figs. 4 and 5, respectively. Among all the trained models, we can
see that the TPR reaches at least 95.29% while reaching up to 100%. This

Fig. 4. True positive rate measurements

Fig. 5. False positive rate measurements

1238 N. Amraoui et al.

high sensitivity rate makes sure that the system is efficiently able to detect any
anomalous control command even if it has been requested by the legitimate user.
On the other hand, the FPR reach at most 4.12% while reaching until 0%. This
low rate ensures a high level of user experience since the system rarely requests
re-authentication demands when the user shows no anomalous behavior.

5 Conclusion

In this paper, we investigated the feasibility in implicitly building normal behav-

iors of a smart home users when controlling their intelligent IoT devices. The
normal models was the basis of an authentication framework to continuously con-
firm or reject the user requested commands according to their deviation from
this baseline. This idea was successfully concretized using Anomaly Detection
techniques. In particular, a One Class Support Vector Machines (OCSVM) was
trained on normal behavioral scores which have been extracted from the user
access logs as well as IoT devices usage data. In the future, we plan to investigate
how the normal OCSVM models should be updated to cope with the change of
normal user behavior using the Incremental version of OCSVM.

References
1. Crocker, P.: Smarter authentication makes mobile experiences more secure,
user friendly. https://www.computerworld.com/article/3181710/mobile-wireless/
smarter-authentication-makes-mobile-experiences-more-secure-user-friendly.html
2. Garcia-Font, V., Garrigues, C., Rifà-Pous, H.: A comparative study of anomaly
detection techniques for smart city wireless sensor networks. Sens. J. 16, 868 (2016)
3. Haim, B., Menahem, E., Wolfsthal, Y., Meenan, C.: Visualizing insider threats: an
effective interface for security analytics. In: 22nd ACM International Conference
on Intelligent User Interfaces Companion, pp. 39–42 (2017)
4. Liao, Q., Li, H., Kang, S., Liu, C.: Application layer DDoS attack detection using
cluster with label based on sparse vector decomposition and rhythm matching.
Secur. Commun. Netw. J 8, 3111–3120 (2015)
5. Mathew, S., Petropoulos, M., Ngo, H.Q., Upadhyaya, S.: A data-centric approach
to insider attack detection in database systems. In: 13th International Workshop
on Recent Advances in Intrusion Detection, pp. 382–401, Berlin (2010)
6. Mazzawi, H., Dalal, G., Rozenblat, D., et al.: Anomaly detection in large databases
using behavioral patterning. In: 33rd International Conference on Data Engineer-
ing, San Diego, pp. 1140–1149 (2017)
7. Najafabadi, M.M., Khoshgoftaar, T.M., Calvert, C., Kemp, C.: User behavior
anomaly detection for application layer DDoS attacks. In: 18th International Con-
ference on Information Reuse and Integration, San Diego, pp. 154–161 (2017)
8. Rath, A.T., Colin, J.N.: Strengthening access control in case of compromised
accounts in smart home. In: 13th International Conference on Wireless and Mobile
Computing, Networking and Communications, pp. 1–8, Rome (2017)
9. Ruan, X., Wu, Z., Wang, H., Jajodia, S.: Profiling online social behaviors for com-
promised account detection. Trans. Inf. Forensics Secur. J 11, 176–187 (2016)
 Implicit and Continuous Authentication of Smart Home Users 1239

10. Scholkopf, B., Platt, J., Taylor, J.S., et al.: Estimating the support of a high-
dimensional distribution. Neural Comput. J 13, 1443–1471 (2001)
11. Viswanath, B., Bashir, M.A., Crovella, M., et al.: Towards detecting anomalous
user behavior in online social networks. In: 23rd USENIX Security Symposium,
San Diego, pp. 223–238 (2014)
12. Wang, C., Yang, B.: Composite Behavioral Modeling for Identity Theft Detection
in Online Social Networks (2018). arXiv preprint arXiv:1801.06825
 Supporting Internet-Based Location
for Location-Based Access Control
in Enterprise Cloud Storage Solution

Muhammad I. H. Sukmana1(B) , Kennedy A. Torkura1 , Hendrik Graupner1 ,

Ankit Chauhan2 , Feng Cheng1 , and Christoph Meinel1
1
Hasso-Plattner-Institute (HPI), University of Potsdam, Potsdam, Germany
{muhammad.sukmana,kennedy.torkura,hendrik.graupner,
feng.cheng,christoph.meinel}@hpi.de
2
GroovyTek, Centennial, CO, USA
[email protected]

Abstract. The emergence of the Internet allows the enterprise to imple-

ment telework policy in order for the employee to work and access com-
pany file anytime, anywhere. But it raises the challenge for the enterprise
to enforce physical access control on enterprise’s files to the employee
outside the enterprise network. One of the solutions for the enterprise
to enforce physical access control of its files is implementing location-
based access control (LBAC) model to allow the employee to access
a file at the pre-determined location. We propose an extension of the
LBAC model to include Internet-based location as an input for location
constraint used for the access control decision and authorization where
Internet-connected device is mapped into a physical location using the IP
geolocation method. Our approach could be used as an alternative user’s
location determination and location verification methods, especially for
mobile devices with minimum self-geolocation capability. We implement
our proposal into proof-of-concept enterprise cloud storage solution called
CloudRAID for Business (CfB) by combining Internet-based geolocation
service, delay-based measurement technique, and open source informa-
tion. Our evaluation’s result shows that our implementation is able to
grant user access control of the system once the user’s location is in the
pre-determined location.

Keywords: Location-based access control ·

Internet-based geolocation · Delay-based measurement · GPS ·
WiFi access points · Enterprise cloud storage system

1 Introduction

With the increasing availability of Internet access and mobile device usage, sev-
eral enterprises allow telework policies for its employees in order to increase
work performance and collaboration, e.g. remote access and bring your own
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1240–1253, 2020.
https://doi.org/10.1007/978-3-030-15032-7_104
 Supporting Internet-Based Location for LBAC 1241

device (BYOD). But this raises the challenges for the enterprise to enforce secure
access control in the system, such as lack of enterprise’s physical security control
or compromised communication due to insecure network [12].
One of the solutions to enforce physical security control for the enterprise is
to implement location-based access control (LBAC) model. LBAC determines
whether a user is authorized to access enterprise system or its confidential data
by checking if the user is in a pre-determined location [1,2,7]. It utilizes several
location sensing technologies to determine user’s location, such as Global Posi-
tioning System (GPS), mobile network, sensors, and WiFi access point (AP)
[1,3,15]. Unfortunately, there has not been many types of research in the LBAC
model area focusing on location verification method to ensure the location is not
manipulated [4].
In this paper, we propose an extension of the LBAC model to include
Internet-based location as an alternative location input for access control deci-
sion. We incorporate IP geolocation methods of binding Internet-connected
device to a geographic location [6] into the LBAC model into its architecture and
aspects that can be used as the alternative location determination and verifica-
tion methods. We implement our proposal into our proof-of-concept enterprise
cloud storage solution CloudRAID for Business (CfB) [13] to ensure a user can
only access a company’s confidential file in the trusted location. The evaluation
result of our implementation shows that it is able to verify a user’s submitted
location to be in the trusted location.
The structure of this paper is as follows: Sect. 2 presents several related works
in the field of location-based authentication, authorization, and access control.
Section 3 provides an overview of CfB and the challenge of physical access con-
trol enforcement in CfB. Section 4 explains our proposal of extending the LBAC
model with Internet-based location as an alternative location input for access
control decision. In Sect. 5 we implement our proposal into CfB. Section 6 eval-
uates the functionality experiment result in our implementation and discuss the
result. Finally, Sect. 7 summarizes our work and future work in this area.

2 Related Works

Baracaldo et al. [2] proposed Geo-social-RBAC, a role-based access control

(RBAC) model that is using geo-social constraints of the social dimension in a
location as part of the access control policy to ensure a user is authorized once he
is with other trusted people at a trusted location. Zhang et al. [15] proposed the
usage of smartphone for location-based authentication and authorization archi-
tecture solution using GPS location, IP address, nearby WiFi access points. Choi
et al. [3] utilized Bluetooth Low Energy beacon to challenge user in the vicinity
using one-time password and IP address filtering for convenient location-based
authentication method.
Our work is different from mentioned works above as we extend LBAC model
to include Internet-based location as an alternative location input constraint to
determine user’s location rather than relying solely on GPS or mobile network.
1242 M. I. H. Sukmana et al.

We propose supporting architecture following Internet-based geolocation meth-

ods, including delay-based measurement, that can be used for location determi-
nation and verification methods for minimum self-geolocate capability device to
ensure the user is in a secure location.

3 Problems and Background

In this Section, we explain the overview of CloudRAID for Business (CfB) and
the challenge for physical access control in the enterprise cloud storage system.

3.1 CloudRAID for Business

CloudRAID for Business (CfB) [13] is an enterprise cloud storage broker (CSB)
system based on a personal CSB system called CloudRAID [11]. It provides a
secure cloud storage service for enterprise usage by acting as a cloud storage
broker between the company and multiple Cloud Service Providers (CSPs). It
manages the company’s resources in multiple CSPs on behalf of the company,
i.e. buckets, credentials, and company data stored in the buckets. By combining
encryption and erasure methods, CfB provides availability and confidentiality
for data stored in the cloud. For more detailed explanation about CloudRAID
and CfB refer to [11,13].

3.2 Challenges for Physical Access Control Enforcement

We identify several challenges of physical access control enforcement in the CfB.

As CfB is used to securely store enterprise’s file in the cloud, it needs to
ensure file can only be accessed by its authorized user anytime anywhere. This
includes enforcing access control for the employee working remotely outside the
company’s premise or using their own BYOD device. Although CSP provides
access control to the services and resources owned by CfB, it still depends on
how CfB enforces physical access control for the employees in the system to
ensure the employees can only access their files at the pre-determined location.
A common solution to physical access control enforcement is by filtering
user’s IP address against a list of allowed or blocked IP addresses, such as the IP
address filter in the policy configuration in several services of Amazon Web Ser-
vices (AWS)1 . The main assumption is that IP address changes infrequently and
the user will have the same IP address for a long period of time. But in the reality,
the IP address assigned to a host could dynamically change for various reasons,
e.g. client is disconnected or session time limit length for dynamic IP address
assignment depending on the Internet Service Provider’s (ISP) policy [10].
LBAC could be used as an alternative solution to enforce physical access con-
trol. Location information from the GPS is mostly used by LBAC to decide if

1
https://docs.aws.amazon.com/IAM/latest/UserGuide/reference policies elements
condition operators.html.
 Supporting Internet-Based Location for LBAC 1243

user fulfills the location requirement. But its location information is “accurate”
under certain radius, e.g. 5 or 15 m, which means the user’s true location could
lie anywhere in the circle depending on the signal strength received and the sur-
rounding environment of the device itself [14]. A user can also spoof the device’s
GPS location, e.g. using a fake GPS application, which makes its information
unreliable. Verifying GPS’ location information could be a challenge since GPS
positioning is passive as the satellites could not actively determine the device’s
location [9].
Location information could also be determined based on the device’s public
IP address using the information from the DNS LOC record, WHOIS registra-
tion, and IP geolocation services, e.g. GeoIP2 and ipstack3 . But the information
provided from these sources are not very accurate locationas it can only provide
good country-level accuracy but bad city-level accuracy with the possibility of
inconsistent and outdated location information [5].
A user can also mask device’s IP address using a virtual private network
(VPN) or proxy service since the communication is encrypted and tunnelled
between client and server via VPN or proxy service that makes the user not
easily tracked. This also affects the device’s location to be masked as well since
its location depends on the IP address of the VPN server if it is mapped using
IP-Geolocation service.

4 Supporting Internet-Based Location for Location-Based

Access Control Model

We propose Internet-based location as an alternative input for location con-

straint in LBAC instead of relying only on a single location input, e.g. GPS,
mobile network, or WiFi APs. We incorporate Internet-based geolocation meth-
ods of binding Internet-connected device to a geographic location into LBAC
model [6]. Internet-based geolocation method can be used as the alternative
location determination and verification methods to ensure the user is at a cer-
tain location and does not manipulate their location. This helps to geolocate
BYOD device with minimum self-geolocation capability, e.g. most laptops do
not have GPS functionality, as the device is assumed to have an Internet con-
nection during access request.
We formalize the concept of location used in this paper. A location is a rep-
resentative of a certain region of the earth, such as country, city, or a circle. The
smallest form of location is a coordinate of latitude-longitude. In this paper, we
ignore the elevation element of the location in our proposal as we are considering
a two-dimensional plane during our calculation.
We propose four aspects that can be used for location determination as fol-
lows:

2
https://www.maxmind.com/en/geoip2-services-and-databases.
3
https://ipstack.com/.
1244 M. I. H. Sukmana et al.

• GPS: Location information is calculated from GPS satellites’ received signal

by the device. Some devices may not be able to provide its location informa-
tion as it does not have GPS functionality.
• IP Address: User’s IP address could provide basic location information of
the user with the country- and/or city-level accuracy based on the information
attached to it, such as DNS LOC record, WHOIS registration, and ASN
information. Location information could also be obtained from IP geolocation
services.
• Delay measurement result: Delay measurement result is the latency
between Internet-connected device and a collection of landmark servers, which
is a public server with a known location. Based on the delay measurement
result, the user’s location could be calculated relative to the landmark servers.
• Signal strength: Signal strength of surrounding WiFi APs or sensors could
be used to locate a user in a certain area. Several services, e.g. OpenWifi4 or
WiGLE5 , provide the mapping between WiFi APs and its locations.

The location aspects could be divided into two categories: proactive loca-
tion aspect where the device actively communicates with location entities to
determine its location (delay measurement result and signal strength) and reac-
tive location aspect where the location of the device is queried or calculated
from the device (GPS and IP address). Although the location aspects have differ-
ent location’s accuracy levels, each aspect’s location information could be verified
other aspect’s location information to ensure that it has not been manipulated
by the user, such as IP address could be masked with VPN. We consider IP
address and the delay measurement result as the primary aspects for location
determination and Signal strength and GPS as the secondary aspects due to
the availability of device’s GPS functionality and WiFi APs or sensors in the
surrounding area.
As we extend Internet-based location for LBAC model, we implement
Internet-based geolocation method for location determination and verification
purposes to ensure the user’s submitted location is not manipulated, as the
conventional LBAC system does not take into account this problem [4]. We pro-
pose an architecture to support Internet-based geolocation method in the LBAC
model as can be seen in Fig. 1 that consists of four entities following four location
determination aspects:

• Requester: The entity that sends the request to access an LBAC system.
It sends a request to the LPS in order to get the requester’s location and a
request to the LVS in order to verify requester’s location.
• Location Provider Service (LPS): A trusted third-party entity indepen-
dent from the LBAC system that provides requester’s location information.
It consists of GPS and Internet-based geolocation service, such as Google
Maps’ Geolocation6 , where Internet-based geolocation service is mandatory
4
http://www.openwlanmap.org.
5
https://wigle.net/.
6
https://developers.google.com/maps/documentation/geolocation/.
 Supporting Internet-Based Location for LBAC 1245

Fig. 1. Architecture overview of LBAC with Internet-based location support

as not all devices have the GPS capability but all devices are connected to
the Internet. Finally, it then returns requester’s location information, which
could be multiple results depending on the number of services used to locate
the requester.
• Location Verifier Service (LVS): The entity that verifies the requester’s
location to ensure the requester is in the pre-determined location. It is con-
trolled by the LBAC system as the requester sends the location verification
request with the response is then sent to LAS. LVS could consist of a set of
multiple landmark servers, registered WiFi APs, and/or sensors where the
requester sends simultaneous location verification requests to LVS’ entities.
• Location Authorization Service (LAS): The entity that grants or denies
requester’s access request to the system. It determines whether a requester
is at the secure pre-determined location based on the requester’s supplied
information and the requester’s location verification response from LVS.
The workflow of a user requesting access to LBAC system is as follows. First,
the user carrying Internet-connected device sends a request to LPS to locate
their location. Depending on the device’s capability, the requester could receive
Internet-based location information or GPS location information as well. After
the user receives its location, it then sends access location verification request to
LVS of a set of multiple landmark servers and surrounding WiFi APs or sensors,
if available, with the verification response is sent directly to LAS. Finally, LAS
decides whether the user is at a pre-determined location and sends the access
response to the user whether to allow or deny the access request.

5 Implementation
In this Section, we explain how we implemented our proposal into CfB [13]. This
Section consists of two parts: Client-side implementation to gather user’s location
information and server-side implementation in CfB main server to determine the
user’s location and verify if the user is at the pre-determined and secure location.
1246 M. I. H. Sukmana et al.

5.1 Client-Side Implementation

If a user wants to upload or share a file, they have the option to include file access
restriction to ensure file can only be accessed by an authorized user, including
location restriction where a file can only be accessed at pre-determined locations,
e.g. home or office. File access restriction is then stored on the CfB main server
and used to determine whether a user is authorized to access the file.

Fig. 2. Overview of how CfB’s client application collects location information before
requesting file access

When a user wants to access a file, the client application first sends a file
access request to CfB main server. If the file has a list of access restrictions,
including location restriction, CfB main server then requests the user’s support-
ing information and current location to the client application while sending the
list of landmark servers that need to be contacted, otherwise the access to the
file is given to the user.
Client application first determines user’s location by querying it from trusted
third-party Internet-based geolocation service, e.g. Google Maps’ Geolocation,
based on the user’s IP address and surrounding available WiFi APs with its
signal strength. It also determines the user’s GPS location, depending on the
availability of the device’s GPS functionality. This is done in order for the CfB
main server to have the reference of the user’s location since it is dealing with
the imprecise location and the possibility of fake user’s location.
It then sends simultaneous location verification requests to the list of land-
mark servers given by CfB main service. It sends ping requests to the servers,
which generates the round-trip time (RTT) between a user and the servers, and
traceroute requests to identify the route and transit delays of packets from a
user to the servers.
Finally, the client application then sends the file access response to CfB main
service with the client’s location and its supporting location information (RTT
and traceroute results and WiFi access points information) as it can be seen in
 Supporting Internet-Based Location for LBAC 1247

Fig. 2. Based on this information, CfB main service decides whether submitted
user’s current location fulfils the file’s location restriction.

Fig. 3. Overview of CfB main server’s Cloud File service architecture for location-based
access control decision with Internet-based location

5.2 Server-Side Implementation

This Subsection explains our proposal’s implementation into CfB main server’s
Cloud File service [13]. It consists of three parts: Delay-based measurement for
user location estimation; open source intelligence to check for the user’s spoofed
IP address and location; and the location restriction decision algorithm to decide
if a user can access the files based on their location (Fig. 3).

5.2.1 Delay-Based Measurement

CfB needs to verify the user’s supplied location in the file access request by
checking if the user is at the aforementioned location and fulfils the file’s location
restriction. One of the Internet-based geolocation methods to estimate the user’s
location is delay-based measurement method where it correlates the Internet
delays and geographical distance between the Internet-connected device and a
set of landmark servers with known locations.
We chose Constraint-Based Geolocation (CBG) proposed by Gueye
et al. [6] as our delay-based geolocation measurement scheme since it provides
a simpler approach to estimate a user’s location with fewer resources needed
compared with other delay-based measurement methods.
We used four servers located in Potsdam, Karlsruhe, Nuremberg, and Frank-
furt am Main in Germany as our landmark servers as can be seen in Table 1.
Since the information about the exact physical location of Frankfurt server is
kept hidden by the CSP, we chose a random coordinate in the city as its repre-
sentative location.
1248 M. I. H. Sukmana et al.

Table 1. Location of landmark servers

Landmark server GPS coordinate

(Latitude, Longitude)
Potsdam 52.394033, 13.133462
Karlsruhe 48.998733, 8.378024
Frankfurt am Main 50.1198925, 8.7351763
Nuremberg 50.1198925, 8.7351763

We decided for the client to ping the landmark servers instead of the servers
pinging the client as it is originally proposed in the CBG approach [6] in order
to get delay measurement result. The reason is after the initial testing landmark
servers were not able to ping several client’s IP addresses as ICMP request is
blocked by some of the user’s ISPs, while TCP request requires port scanning
for pinging the open port of the IP address with the possibility of the null result.

5.2.2 Open Source Intelligence

In order to keep extensive knowledge of the location and check for the spoofed
IP address with VPN/Proxy service, we chose to use several services available
as open source intelligent for CfB main service. We use GeoLite2 database7 for
IP Geolocation service to geolocate user’s IP address, Google’s Geocoding8
as geocoding service where it converts GPS coordinates into human readable
address, and VPN Blocker9 for VPN/Proxy detection service.

5.2.3 Location Restriction Decision

We developed a location restriction decision algorithm for CfB main service
based on the techniques explained in Sects. 5.2.1 and 5.2.2. Our algorithm con-
sists of three major steps: (1) Verify if the user’s claimed location is correct and
not spoofed; (2) Calculate the possible area of the user’s location; and (3) Verify
if the user is at the pre-determined location. Based on the algorithm’s result, the
CfB main server determines whether to allow or deny a user’s file access request.
First, CfB verifies the user’s location for the city- and/or country-level accu-
racy and checks whether the user’s IP address and/or location is not spoofed. It
compares the possible city and country of the user’s location based on the user’s
IP address and claimed location, which could be GPS or Internet-based location,
using geocoding service and IP geolocation service. If both results in different
locations, then most likely user spoofs his location, therefore, CfB stops the loca-
tion verification process and denies the user’s request, else it will continue the
process.

7
https://dev.maxmind.com/geoip/geoip2/geolite2/.
8
https://developers.google.com/maps/documentation/geocoding/.
9
https://vpnblocker.net/.
 Supporting Internet-Based Location for LBAC 1249

For each hop in the user’s traceroute result to landmark servers, it checks
whether the hop is a possible VPN or proxy server as most likely the connection
is tunneled through the server that will appear the connection originates from the
server. If all hops are not detected as VPN server, then the location verification
process will continue. Else if at least one of the hops is detected as a VPN server,
then the process is stopped and the user’s file access request is denied.

Algorithm 1. User’s location verification to be in the trusted location

Data: userP ings User’s ping result to landmark servers, landmarkList List of landmark
servers and its information, landmarkP ings Landmark servers’ ping results between
each other, landmarkDists Landmark servers’ distances between each other,
userLoc User’s claimed location, trustedLocation Trusted location’s coordinate, Cτ
List of K landmark circles predicting user’s location
Result: requestAllowed Decision result whether to allow or deny user’s file request
1 foreach landmark ∈ landmarkList do
2 ping = getP ing(landmark, landmarkP ings);
3 dist = getDist(landmark, landmarkDists);
4 bestline = getBestline(ping, dist);
5 uP ing = getP ing(landmark, userP ings);
6 circle = getCircle(uP ing, bestline, landmark);
7 Cτ .insert(circle);
8 end

K
9 userCalcLoc = Ciτ ;
i
10 if intersecting(userCalcLoc, userLoc) then
11 userV erLoc = userCalcLoc ∩ userLoc;
12 if intersecting(userV erLoc, trustedLoc) then
13 requestAllowed = true;
14 else
15 requestAllowed = false;
16 end
17 else
18 requestAllowed = false;
19 end

CfB then calculates user’s possible location based on the user’s delay mea-
surement results of landmark servers following [6] and verifies whether the user’s
location fulfils the file’s location restriction as can be seen in Algorithm 1. First,
it calculates the bestline equation for each landmark server using the ping result
and distance between a landmark server to other servers. It then calculates the
user’s possible location circle for each landmark server using the server’s bestline
equation and the user’s ping result to the server.
The user’s calculated location could be determined by finding the intersection
region between the list of the user’s possible location circles. CfB then checks
whether the user’s calculated location intersects with user’s claimed location.
If both locations intersect, then the intersection region is considered as user’s
verified location, else user’s claimed location is manipulated and the file access
request is denied. Finally, CfB checks if pre-determined location intersects with
1250 M. I. H. Sukmana et al.

user’s verified location. If it is, then the user’s file access request is granted, else
it will be denied. In theory, the running time for our proposal is O(n3 ) where
n is the number of landmark servers as for each landmark server requires to
ping other servers in order to calculate its bestline and to be pinged by user to
calculate their possible location.

6 Evaluation
We evaluate our proposal’s implementation in CfB by running an experiment of
a user accessing files with location restriction. For our experiment, a user can
only access the file once they are at a trusted coordinate (52.393518, 13.132989)
with the experiment area is limited to Germany only. We assume the user is
using a device with no GPS functionality and connecting to the Internet with
WiFi only.
Since [6] does not specify many details for the delay measurement result and
geographical distance, we tried all possible combinations with different parame-
ters during our experiment. We used 4 parameters of ping statistic result of maxi-
mum, minimum, average, and median and two approaches to calculate geograph-
ical distance between landmark servers: using great circle haversine formula10
to calculate the distance of two latitude-longitude coordinates or transforming
latitude-longitude coordinate into Cartesian coordinate (x, y) using Elliptical
Mercator projection11 and using Pythagoras theorem to calculate the distances
between two coordinates.
We run 11 trials for each type of ping statistic parameter as the two types
of geographical distance parameter can be calculated at the same time. Each
landmark server and the client sent 5 TCP ping requests for its delay measure-
ment result. We then calculated landmark server’s self-calibrated bestline and
the user’s possible location circles using the combination of ping statistic param-
eter and geographical distance parameter, e.g. latitude-longitude coordinate and
maximum ping, and determined user’s possible location and whether the user
is at the trusted coordinate. Each trial we run is considered success/true if the
trusted location is located inside or intersects all landmark server’s circles and
user’s location circle, otherwise it is considered failed/false. We can then calcu-
late the overall result for each combination of ping statistic and geographical
distance parameters from 11 trials using the majority rule. Each trial can be
visualized into a map as can be seen in Fig. 4.
The overall result of our experiment can be seen in Table 2. Maximum and
minimum ping experiments are successful with the rate of 54.55%, while average
ping and median ping experiments are failed with the rate of 36.36%, and 27.23%,
respectively. The experiment for minimum, median, and average ping results
require more running time to calculate landmark server’s bestline compared to
maximum ping result due to the landmark server’s calculated bestline does not
fulfil the requirement of bestline as mentioned in [6] that required us to run the
10
https://rosettacode.org/wiki/Haversine formula.
11
https://wiki.openstreetmap.org/wiki/Mercator.
 Supporting Internet-Based Location for LBAC 1251

Fig. 4. Example of landmark server’s circles map visualization. The pin marks the
trusted coordinate and the circles visualize different user’s locations depending on the
colours: user’s location from Google Maps (black), Potsdam server (red), Nuremberg
server (green), Karlsruhe server (blue), Frankfurt server (yellow)

Table 2. Overall result experiment of our proof-of-concept using combination of ping

statistic and geographical distance parameters

Coordinate\Ping Maximum Minimum Average Median

Latitude Longitude True True False False
Cartesian True True False False

trial on the loop until it found the bestline. Maximum ping result experiment
requires 1–2 min for each trial while other experiments could be running for 1 h.
Geographical distance methods using latitude-longitude or Cartesian coordinate
do not affect the experiment’s result since both calculate the same fixed distance.
Based on these results, we conclude that our implementation following our
proposal is able to determine whether a user is at a trusted location by com-
paring user’s submitted location against user’s possible location calculated using
Internet-based geolocation methods. We achieve the best result and performance
using maximum ping result statistic of landmark servers and clients to calculate
the landmark server’s bestline and user’s possible location to verify if a user is
at the trusted location.
Our proposal relies on the multiple landmark servers as Location Verifier
Service in order to calculate user’s possible location. In this paper, we use 4
landmark servers in our implementation where 3 of our landmark servers are
located tend to be in the southern part of Germany due to available CSPs we can
use to set up the servers. This affects our calculation of landmark server’s bestline
to require more time as landmark server’s line calculation does not fulfil bestline’s
criteria, which makes it to be in the loop until it finds its bestline equation. It also
affects user’s possible location calculation to be not very accurate as it generates
1252 M. I. H. Sukmana et al.

big region of user’s possible location as can be seen in Fig. 4 that could create
false positive decision as user is not at the pre-determined location to access
the file. Other Internet-based geolocation methods could be combined with our
proposal that could give better location accuracy and user’s privacy to verify
user’s location, such as [8].

7 Conclusion and Future Works

In this paper, we propose location-based access control (LBAC) model exten-
sion to include Internet-based location as an alternative location input for access
control decision. We introduce the architecture and the components to support
Internet-based geolocation method for alternative location determination and
location verification methods. We implement our proposal into enterprise cloud
storage solution called CloudRAID for Business to ensure the user can only
access the file when they are in the trusted location. Our evaluation result shows
that our implementation is able to authorize a user’s file access request by calcu-
lating the user’s possible location and comparing it against the user’s submitted
location and the trusted location.
We would like to investigate the effectiveness of other Internet-based geoloca-
tion methods and components to be used in our proposal to determine and verify
the user’s location, such as sensors for indoor geolocation or privacy preserving
geolocation method. Also, we would like to integrate our extended LBAC with
traditional access control models that could be used for enterprise usage, e.g.
role-based access control or attribute-based access control.

Acknowledgments. We would like to thank Bundesdruckerei GmbH for the support

of this paper. We also would like to thank Lasse Jahn, Marco Schaarschmidt, Kerstin
Andree, and Niklas Dornick for their help developing the prototype of our approach.

References
1. Ardagna, C.A., Cremonini, M., Damiani, E., di Vimercati, S.D.C., Samarati, P.:
Supporting location-based conditions in access control policies. In: Proceedings
of the 2006 ACM Symposium on Information, Computer and Communications
Security, pp. 212–222. ACM (2006)
2. Baracaldo, N., Palanisamy, B., Joshi, J.: Geo-social-RBAC: a location-based
socially aware access control framework. In: International Conference on Network
and System Security, pp. 501–509. Springer (2014)
3. Choi, M., Lee, J., Kim, S., Jeong, Y.S., Park, J.H.: Location based authentica-
tion scheme using ble for high performance digital content management system.
Neurocomputing 209, 25–38 (2016)
4. Decker, M.: Requirements for a location-based access control model. In: Proceed-
ings of the 6th International Conference on Advances in Mobile Computing and
Multimedia, pp. 346–349. ACM (2008)
5. Gharaibeh, M., Shah, A., Huffaker, B., Zhang, H., Ensafi, R., Papadopoulos, C.:
A look at router geolocation in public and commercial databases. In: Proceedings
of the 2017 Internet Measurement Conference, pp. 463–469. ACM (2017)
 Supporting Internet-Based Location for LBAC 1253

6. Gueye, B., Ziviani, A., Crovella, M., Fdida, S.: Constraint-based geolocation of
internet hosts. IEEE/ACM Trans. Netw. (TON) 14(6), 1219–1232 (2006)
7. Kirkpatrick, M.S., Ghinita, G., Bertino, E.: Privacy-preserving enforcement of spa-
tially aware rbac. IEEE Trans. Dependable Secure Comput. 9(5), 627–640 (2012)
8. Konstantinidis, A., Chatzimilioudis, G., Zeinalipour-Yazti, D., Mpeis, P., Pelekis,
N., Theodoridis, Y.: Privacy-preserving indoor localization on smartphones. IEEE
Trans. Knowl. Data Eng. 27(11), 3042–3055 (2015)
9. Lenders, V., Koukoumidis, E., Zhang, P., Martonosi, M.: Location-based trust for
mobile user-generated content: applications, challenges and implementations. In:
Proceedings of the 9th Workshop on Mobile Computing Systems and Applications,
pp. 60–64. ACM (2008)
10. Padmanabhan, R., Dhamdhere, A., Aben, E., Spring, N., et al.: Reasons dynamic
addresses change. In: Proceedings of the 2016 Internet Measurement Conference,
pp. 183–198. ACM (2016)
11. Schnjakin, M., Korsch, D., Schoenberg, M., Meinel, C.: Implementation of a secure
and reliable storage above the untrusted clouds. In: 2013 8th International Con-
ference on Computer Science and Education (ICCSE), pp. 347–353. IEEE (2013)
12. Souppaya, M., Scarfone, K.: Guide to enterprise telework, remote access, and bring
your own device (BYOD) security. NIST Spec. Publ. 800, 46 (2016)
13. Sukmana, M.I., Torkura, K.A., Meinel, C., Graupner, H.: Redesign cloudraid for
flexible and secure enterprise file sharing over public cloud storage. In: Proceedings
of the 10th International Conference on Security of Information and Networks, pp.
3–10. ACM (2017)
14. Zandbergen, P.A., Barbeau, S.J.: Positional accuracy of assisted GPS data from
high-sensitivity gps-enabled mobile phones. J. Navig. 64(3), 381–399 (2011)
15. Zhang, F., Kondoro, A., Muftic, S.: Location-based authentication and authoriza-
tion using smart phones. In: 2012 IEEE 11th International Conference on Trust,
Security and Privacy in Computing and Communications, pp. 1285–1292. IEEE
(2012)
 Exploiting Multiple Paths in Multi-hop
Co-operative Ad-Hoc Networks
for Providing Security without Trusted
Third Party

T. Dinesh Ram Kumar1 , Ashutosh Bhatia1(B) , and R. C. Hansdah2

1
Department of Computer Science and Information Systems,
Birla Institute of Technology and Science, Pilani, India
{f2014302,ashutosh.bhatia}@pilani.bits-pilani.ac.in
2
Department of Computer Science and Automation,
Indian Institute of Science, Bangalore, India
[email protected]

Abstract. Ensuring secure communication in large scale wireless net-

works is a challenging issue, especially when the nodes participating in
the network itself cannot be trusted fully. This situation is very much
pertinent in the case of multi-hop wireless cooperative networks, where
the nodes co-operate with each other to support multi-hop communica-
tion in the network. In such networks, assuming the presence of a trusted
third party to provide key exchange and authentication services in the
network is not viable. Additionally, the classical Diffie-Hellman based
key establishment mechanism as such cannot be used because of Man-
In-The-Middle and impersonation attacks, as the nodes participating in
a multi-hop path from source to destination cannot be trusted. In this
paper, we investigate the use of the multi-paths and secret sharing in
multi-hop cooperative adhoc networks to provide practical secure com-
munication between the nodes in the network. We propose a couple of
routing algorithms that give multiple-paths to send different segments of
information required to be communicated securely, under various behav-
ioral models of the participating nodes. The simulation results show that
the majority of times, we are able to find the multiple paths in the net-
work through which the different segments of a piece of information can
be sent to achieve desirable security.

Keywords: Cooperative adhoc networks · Secret sharing ·

Diffie Hellmann key establishment · Secure routing

1 Introduction
Typically, cooperative networks involve multi-hop wireless communication and
are Adhoc in nature. Providing secure communication in such Mobile Adhoc
and Cooperative NETworks (MACNETs) requires the establishment of a shared
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1254–1265, 2020.
https://doi.org/10.1007/978-3-030-15032-7_105
 Exploiting Multiple Paths in MACNETs for Providing Security 1255

secret between every communicating nodes in the network. In general, the mech-
anisms to establish a shared secret (key) between two parties are either public key
(RSA) based or Diffie Hellman based. The public key based mechanisms involve
a trusted third party (certification authority) to verify the identity of nodes by
means of digital signatures. Consider networks like a peer-to-peer MACNET,
where networks are formed on the fly and peers in the network cooperate with
each other to facilitate communication between parties that are separated by
more than one hop. In such networks, there is no clear party trusted by all
nodes, and therefore, the public key based key establishment mechanism are
not suitable for such MACNETs. Additionally, using the certificate issued by
trusted third party cannot be used to authenticate each other as participating
nodes may not want to register themselves with any other trusted third party in
order to protect their anonymity. Another option is to use Diffie Hellman based
key exchange mechanism. But, as in this case every body except the source and
the destination can be a potential adversary, it is prone to Man-In-The-Middle
and impersonation attacks, and also, there is no party to verify the authenticity
of nodes involved in key establishment.
Let us try to understand the issue that we are trying to address in this paper,
with the help of a specific type of MACNET, known as Vehicular Adhoc Net-
works (VANET). In VANET, the vehicles (nodes) cooperate with each other to
disseminate various useful information related to the traffic which is then utilized
among themselves to improve the safety and comfort of the drivers. The vehicles
in VANET, are typically identified by their position (physical or logical) in the
network and require to establish shared secret between them to communicate
securely as well as authenticate each other. It would be a serious privacy issue,
if in any sense, the identification information of vehicles is associated with any
kind of Identity of its owner, driver or passengers. Apart from their positions in
the network, nodes do not know any other information about their counter parts
with whom they want to have secure communication, and therefore, wanted to
establish a shared secret or key.
In this paper, we propose a mechanism for secure communication between
two parties in MACNETs, using the concept of “Secret Sharing” [1] that involves
multiple paths. “Secret Sharing” is a general technique, where a secret is bro-
ken down into multiple pieces, in such a way that individual piece or group of
pieces below a defined threshold convey no information about the secret. The
core idea behind the proposed method is to break the message into pieces and
transmit them along multiple paths towards the destination. Since, for each pri-
vate message, we need to break the message into multiple shares and transmit
them resulting in lots of effort at source and destination. It is better to use this
mechanism to establish a shared key, then to send message itself. We use secret
sharing to break down the message into multiple shares and route them along
multiple paths. Choice of paths and shares is used to provide required confiden-
tiality. Moreover it provides integrity also, that is any modification of share can
be identified at destination. Other than the shared key, the proposed mecha-
nism can also be used to communicate either short messages or the public keys
1256 T. Dinesh Ram Kumar et al.

involved in DH based key exchange. For effective choice of paths and number
of shares to break into, we model the behaviour of the adversarial nodes and
describe a routing algorithm for maximizing effectiveness. By maximizing effec-
tiveness, we mean that the proposed algorithm try to minimize the likelihood
of adversarial nodes from gathering required number of shares to decode the
message while ensuring the intended destination gets the message. We develop
a probabilistic model of nodes and try to achieve a desired level of security of
the communication.

2 Related Work

The research works that can be considered as closely related to the work proposed
in this paper, are the works on enabling secure communication in Wireless Sensor
Networks. The sensor nodes that are directly in the range of each other can
establish a common key using Diffie Hellman based key exchange mechanism and
subsequently use it to communicate with each other. But, the nodes that are not
connected directly, cannot use the similar scheme. Initial proposals involved the
formation of a secure path, between the two nodes that are not directly reachable
to each other, using the sequence of nodes that are directly connected and all the
nodes in the path have already established a share key with the next node in the
sequence. However, in this case, messages are revealed to the intermediate nodes
along the path. Compromise of any node along the path would compromise the
end-to-end communication.
Due to energy constraints of Wireless Sensor Networks, many Random Key
Distribution Schemes have also been proposed to enable secure communication
among all the nodes in the network [2–5]. The basic scheme works as follows.
Before sensor nodes are deployed, an initialization phase is performed. In the
initialization phase, the basic scheme picks a random pool (set) of keys S out
of the total possible key space. For each node, m keys are randomly selected
from the key pool S and stored into the node’s memory. This set of m keys is
called the node’s key ring. The number of keys in the key pool, |S|, is chosen
such that two random subsets of size m in S will share at least one key with
some probability p. In case of MACNETs, the pre-distribution of key cannot be
assumed.
The core concept of using multiple paths to establish a shared secret has
been previously explored in [6] with a specific purpose of establishing end-to-
end key in wireless sensor networks (WSNs). The authors in [6] propose an
end-to-end pair-wise key establishment mechanism using multiple node-disjoint
paths. Assuming that node A needs to set up a pair-wise key with another node
B. Firstly, node A, finds N node-disjoint paths to B. Secondly, node A selects
a key K and divides it into N non-overlapping segments. Then node A sends
each segment through the separate node disjoint path to node B. When node
B receives all N segments of the key K, it can reproduce the key K and use it
to secure communication with node A. To decide the number of node-disjoint
paths that would be sufficient to attain a high level of security while keeping
 Exploiting Multiple Paths in MACNETs for Providing Security 1257

the transmission overhead as low as possible the authors considered the number
of sensor nodes that an attacker can compromise as an input parameter and
calculated the probability of the path-key K is revealed if a node-disjoint path
set of size N is used.
The authors of [7] argued that the scheme of breaking the message into N
segments and then sending each segment through a node disjoint path is vul-
nerable to the scenario when compromised nodes in a node-disjoint path do not
forward the share passing through them. This scenario is very much applicable
in cooperative networks, where the nodes may exhibit selfish behavior by not
forwarding others packets. To solve this problem, the authors in [7] used shamir’s
secret key sharing scheme [1]. In this scheme, the only k out of n segments are
sufficient to reconstruct the secret share. Shamir’s secret key sharing scheme
together with the multi-path scheme is computationally inefficient and requires
more bandwidth to share part-keys through different paths between source and
destination.
In Wireless Sensor Networks, due to limitations of the nodes, public key or
Diffie Hellman based schemes could not be used and thus secret key was itself
sent. There are few differences in approach of modeling the adversaries in WSN
and the way we have modeled the behaviour of nodes in this paper. In WSN,
adversaries are external to the network who can compromise the nodes that are
trusted by other nodes. In our case, the nodes themselves are not trustable as
they are curious. They can collaborate interchange messages with each other to
obtain the pieces of information. Nodes in our case are not energy constraint and
we can use Diffie Hellman based schemes so that even if the message is discovered
it provides forward secrecy. Since Diffie Hellman without PKI for authentication
in prone to man in the middle attack, we need to ensure that likelihood of mes-
sage being discovered while exchange during key establishment is minimized so
as to avoid MITM attacks. Moreover, PKI based authentication needs registra-
tion with trusted third party. Thus any breach in the trusted third party may
reveal information about nodes themselves which may be security breach where
anonymity is important. We model the malicious behavior of nodes in a coop-
erative network as curious and collaborative (discussed in section IV), where
each node can have a different degree of curiosity and the level of collabora-
tion with other nodes. This model allows the source node to send the segments
through multiple-paths which are not necessarily node-disjoint and still achieve
a reasonable security.

3 Assumptions

We consider a simple MACNET, where nodes cooperate to communicate with

each other. We assume that the path to be taken by a message is specified by the
source node i.e, the source node knows the id of all intermediate nodes belong-
ing to the path. While sending a message, the source nodes want some level
(defined later) of secrecy in communication. We assume that no adversary node
in the network modifies the message, and faithfully forwards them towards the
1258 T. Dinesh Ram Kumar et al.

destination as dictated by the source. But their innate curiosity urges them to
eavesdrop the communication and also these adversaries are willing to collabo-
rate to decode the message. Moreover, no secret key is involved in communication
among the end hosts and there is no pre-shared secret. Thus to ensure desired
level of secrecy, that is to lower the likeliness of eavesdropper from discover-
ing (decoding) the message, source breaks the message into multiple shares and
routes them along multiple paths.
Nodes model the behaviour of adversaries to decide suitable choice of paths
and suitable number of shares to achieve desired level of security. Also, we con-
sider that threshold in “secret sharing” is set to maximum that is to say all
shares are needed to recover the original message. To nullify the possibility of
eavesdropping due to overhearing, we assume that the communication with one-
hop neighbors is secure i.e. all nodes in the network share pairwise secret keys
with their neighboring nodes.
In a nutshell under this model, goal of the source node is to send a message
to the intended destination with desired level of secrecy (privacy) and goal of the
adversaries is to eavesdrop on their communication, i.e, to decode the message
without disrupting their communication (Passive Adversaries w.r.t. the source
but work together to achieve their respective goals).

4 Behavioural Modelling of Nodes

Adversarial nodes are modeled using two types of behaviour: individual malicious
intent and collective collaborative intent.
Individual Malicious Intent: Each node is inherently curious, and want to
eavesdrop on the communication and decode the message to satisfy their own
curiosity. But curiosity levels among nodes can vary. That is, to say, some nodes
are more curious than others. Curiosity can be seen as a measure of effort that an
adversarial node is willing to put to decode the secret being communicated via
them. Some nodes are more determined than others, and are willing to put more
effort to decode the message from the shares. Thus to counter this behaviour of
adversaries, source increases the complexity of decoding the message by increas-
ing the number of secret shares of message and this also increases the amount
of eavesdropping required.
Collective Collaborative Intent: Collaborative behaviour of nodes is that
nodes are willing to co-operate with each other to help each other to break
the secret by conveying their share of information (communication) they eaves-
dropped to other nodes. We consider that collaboration intent is mutual between
parties. Some nodes are more collaborative with few nodes than others. Curious
and collaborative behaviour are independent of each other. Some non-curious
nodes may be more collaborative with other nodes. That is, they themselves are
not interested in decoding their message but however are willing to give their
share of information to other nodes. While some nodes tend to be highly curious
but not collaborative, i.e., they want to decode the secret on their own.
 Exploiting Multiple Paths in MACNETs for Providing Security 1259

To explain where curiosity and collaboration come into play consider this
scenario, source has modeled the behaviour of nodes and decided the number of
shares and routed the shares and in-between nodes have forwarded the shares
as intended by the source. Now adversaries begin their malicious activity. Some
nodes have obtained some of the shares of the message as the source forwarded
the shares through them. However, they need to gather the remaining shares
from others to obtain all shares to decode the secret. Now comes the gathering
phase. Nodes may convey their shares of message to some nodes and may not
convey to other nodes which depends on their level of collaboration between
them. Some nodes may obtain a copy of all the shares either as a result of
collaboration with appropriate nodes or they could have obtained as a result of
routing by source. Some nodes may be curious and put the effort to decode the
messages while some may not. Source routes the messages so that destination
gets all the shares of the message. We assume the destination node (end point
of communication) is willing to put the effort to decode the message and also
assume that destination does not convey the information of the shares or the
message to other nodes. So goal of the source is to make sure that no node apart
from destination (all the other nodes are considered adversaries) decodes the
secret. In probabilistic model, the goal of the source is to keep the likeness that
adversaries break the message below a desired threshold. To achieve this, source
tries to ensure that no curious nodes gathers all information by deciding the
number of shares and choosing paths appropriately. But the success of routing
depends on the effectiveness of behavioral model. Source dynamically models
the behaviour of adversarial nodes. How source models the behaviour of other
nodes is not covered in this work. However, we describe two types of modeling
of adversaries: a deterministic model and a probabilistic one.

4.1 Deterministic Model

Before deciding the number of shares and choice of routes for the shares, the
source needs to model the behaviour of intermediate adversarial nodes and has
to discover paths towards intended destination. Deterministic model involves the
source node modeling the behaviour of adversarial nodes discretely (concretely).
To explain our model we consider one behaviour at a time.
Curiosity: In this case, source considers only curious behaviour of nodes and
models the intermediate adversarial nodes as either curious or non-curious.
Under this model, source tries to find a path from source to destination involv-
ing only non-curious nodes. If such a path is found then it forwards the message
along the path and can be sure that no curious node can get the message. How-
ever if such a path cannot be found, it has to break the message into multiple
shares. Node now constructs an induced network consisting of only curious nodes
but maintaining the paths relationships. As non curious nodes cause no prob-
lem. This is also the case when node considers all nodes as curious. Now node
has to decide the choice of paths and number of shares to break into. A simple
approach is find paths such that no node lies in all the chosen paths towards the
1260 T. Dinesh Ram Kumar et al.

destination in the induced network, break the message into number of shares
equal to number of paths and transmit one along each of the path. This ensures
that no node gets all the shares. In this case all we need is only two shares and
two node disjoint paths (exception of source and destination). That is say that
we need to find two disjoint paths for transmitting the shares. If we cannot find
such paths, we need to discover more paths. But, we observed that if there are
no two paths which are node- disjoint then there exists some nodes which lie
in all the discovered paths. In such a case, we need to discover more paths till
we find paths such that those nodes don’t appear and then chose two disjoint
paths and route the two shares of the message through them. (Note we consider
disjoincy in induced network. That is non-curious node may get all the shares).
Collaborative Behaviour: Now, we consider that source also considers col-
laborative behaviour of nodes. That is source considers (models) a pair of nodes
as collaborating or not. Such a modeling results in collaborating groups. A col-
laborating group is defined a set of nodes such that for any two nodes either
they directly collaborate or there exists a sequence of nodes through which they
indirectly collaborate. That is to say that any node in the group get a share
then all the other members are also notified of the share. Note that resulting
collaborative groups are disjoint. Under this model, their can be collaborative
groups, where no member is curious. That is collaborative group exhibits no
curious behaviour. If some member in collaborative group is curious, then if
any member of the group is involved in forwarding then the curious member
also obtains the share. Thus this groups are labeled as curious. Here as we con-
struct an induced network of curious nodes, but we cannot remove members
of curious collaborative groups, thus we label all members of curious collabora-
tive groups as curious whether they were inherently curious or not. And then
construct an induced network of curious nodes. Now node discovers more and
more disjoint paths till no member of any (curious) collaborative group lies in
all disjoint paths. That is to say “node-disjoint paths such that they are also
group-disjoint paths”.
We need disjoint paths to ensure no nodes (in induced network) gets all the
shares either they themselves curious or belong to a curious collaborating group.
We also need to ensure that they are group disjoint to ensure no member of
any curious group gets all shares by gathering. Now to determine the number of
shares and set of all paths for routing, model each discovered disjoint path as a
set with each element representing a curious collaborative group to which nodes
in the path belong to. This results in a set of sets. Aim is to find the minimal
number of paths such that the intersection of sets corresponding to each path
when represented as the set of distinct collaborative groups involved in the path,
is a null set. This results in a set of chosen paths and number of shares is equal
to number of paths. The trivial case is each adversary node belongs to its own
collaborative group.
Deterministic model may not always yield a solution that is choice of num-
ber of shares and paths to route the message towards the destination. This
may happen when source is unable to find paths that are node-disjoint if node
 Exploiting Multiple Paths in MACNETs for Providing Security 1261

considers only curiosity and group-disjoint if node considers curiosity and collab-
oration together. This is because of deterministic modeling of node behaviour.
Deterministic solution results in a set of paths. It does not consider reuse of
paths. It does not consider the fact that nodes exhibit varying levels of curiosity.
Collaboration may also vary between nodes due to distrust between them in a
cooperative network. Thus, we go for a probabilistic model, where curiosity and
collaborations are probabilistic measures. Idea behind probabilistic model is the
fact that as the number of shares increases for a given message, the amount of
effort required to decode the message increases, that is likelihood of an adversary
decoding the message decreases, or curiosity decreases with number of shares.
In other words, increase in number of shares results in only determined adver-
saries decoding the message. Also, the amount of effort required to gather from
collaborators also increases with increase in number of shares of the message.

4.2 Probabilistic Model

We consider that source node models the behaviour of adversarial nodes prob-
abilistically in this model. For an adversarial node to break a message, it must
first gather all the required number of shares of the message, then combine them
to decode the message. This process can be broken into phases: gathering and
decoding. Gathering phase involves either cooperation with other nodes or it
already has that share. While decoding phase depends on the curiosity of the
node and the effort it is willing to put. We now re-define curiosity and collabo-
ration in this model to account for the probabilistic model.

Definition 1. Curiosity(A, n) indicates the probability that node ‘A’ would try
to decode/breaks the message given all the n shares of a message.

Curiosity is used to model the effort the adversarial node is willing to put to
decode the message. We know that more effort is needed to decode the message
when there is more number of shares for a message due to increase in complexity
of decoding algorithm. Thus Curiosity(A, n) is a decreasing function with respect
to ‘n’.

Definition 2. Collaboration(A, B) indicates the probability that node ‘A’

receives a share of message given that node ‘B’ has that share (as node ‘B’
lies along the path chosen by the source).

Collaboration models the likelihood that a node not along the path that the
source took to route the share, is able to gather information about the share. We
consider that this likelihood is independent across shares, that is independent
of previous collaboration. We also consider likelihood of collaboration is mutual,
that is Collaboration(A, B) equals Collaboration(B, A).
Because of the probabilistic behavioral model, we need probabilistic measures
for performance evaluation or comparison of various choices of paths and num-
ber of shares which measures likelihood of adversaries succeeding (in breaking
the message). Decoding phase depends only on curiosity of individual node and
1262 T. Dinesh Ram Kumar et al.

number of shares. Gathering phase depends on the choice of paths and number
of shares. This is because, more cooperation is needed to gather all shares. Thus
the increase in number of shares, both curiosity and collaboration decreases. But
note that end nodes also must spend effort to decode and encode the message.
Since all messages do not need same levels of privacy, we use these performance
measures as a measure of privacy level. Thus source specifies a minimum/ max-
imum threshold (depending on type of measure) on these performance measure
for each message. Aim of routing algorithm must be to meet the specified privacy
level.
Before we define the performance measures, we first look at gathering phase.
To break the message, it must first gather all shares. Either it lies along the path
or it must collaborate with other nodes to obtain that share.
Let PA be the probability of node having all the shares. To calculate PA , we
define PAi as the probability that an adversary node A obtains the ith share of
the message. PAi = 1, if node A lies along the path. Assuming that probability of
obtaining a particular share is independent of the probability
n of obtaining other
shares, the value of PA can be calculated as, PA = i=1 PAi .
Let S be the set of all nodes lying along the path taken by the ith copy
of the share with cardinality m. The value of PAi is the probability that the
node
m A collaborate with at least one node in the set S. Therefore, PAi = 1 −
th
j=1 (1 − Collaboration(A, Sj )), where Sj is the j element of the set B. We
n  
m

get the value of PA as, PA = i=1 1 − j=1 (1 − Collaboration(A, Sj )) .
Now once an adversary gathers all shares, it must now decode it to break it.
Let QnA be the probability that node A would be able to decode the message.
The value of QnA can be calculated as, QnA = Curiosity(A, n) ∗ PA .
We now define the probabilistic performance evaluation measures to compare
the various choices of paths and number of shares under the probabilistic model
of adversaries behaviour. Let M1 be the probability that no node in the network
is able to break the message. The value of M1 can be calculated as, M 1 =
 n
A∈N (1 − QA ), where N is the set of all nodes in the network. Similarly, let M2
be the maximum probability that some node breaks the message. The value of
M2 is, M 2 = max(QnA ), ∀A ∈ N .
Source specifies a desired privacy level using one of these measures to the
routing algorithm along with the behavioral model of adversarial nodes and
its discovered network, routing algorithm returns a list of paths and number of
shares required. Here paths may repeat, that is two or more shares of the message
can pass through the same path unlike in Deterministic routing algorithms. The
simplest algorithm would be to choose a path, increase the number of shares
such that required performance measure is met. This may yield a solution, but
increases the effort that source and destination must spend when alternative
choice could exist with lesser number of shares. This solution is only based on
idea of increasing effort needed to break, but it would be better if nodes are also
involved in gathering.
To better understand the probabilistic model, we start by considering
only probabilistic curiosity and as extension of Deterministic model. The
 Exploiting Multiple Paths in MACNETs for Providing Security 1263

Deterministic model yields a solution whenever disjoint paths exist. Else, some
nodes exist along all the paths. Thus, we find two paths such that they have only
these nodes in common and these are our choice of paths. Since no other node
apart from them can break (as no collaboration considered), we choose a large
enough number of shares such that the performance measure is met. We then
route the shares alternatively through these paths. In this model, only those
common nodes can gather all shares, we increase the effort needed for them to
break the secret while no other node can gather all the shares.

5 Results
To perform the experiment, we model the multi-hop cooperative network as
graphs, with communicating entities as nodes or vertices and their neighbor-
hood relationships as the edges. Graphs are generated using random points in
N-Dimensional space. Adjacency is established based on a maximum neighbor-
hood threshold (proximity) parameter α and the resultant graph is checked for
connectivity. If the resultant graph is not connected then generate the graph
again. In case we do not get the connected graph after trying for a multiple
number of times, then increase the value of α. Nodes are labeled curious or not
based on a curiosity level threshold β and random values chosen from uniform
threshold. For collaboration, nodes are randomly assigned to a predefined num-
ber of collaboration groups γ. Then nodes belonging to a group are collaborating.
Higher value of γ leads to lesser degree of collaboration in the network. As the
neighbor relationship and the curiosity of the nodes are defined randomly, we get
a different graph every time we perform the simulation while keeping the values
of all experimental parameters such as number of nodes in the network, prox-
imity parameter α, curiosity level threshold β, and the number of collaboration
groups γ, same.

(a) (b)

Fig. 1. A multi-hop cooperative network Fig. 2. Multipath solutions only when

and the collaboration among the nodes curiosity is considered

Figure 1(a) shows the network that we simulated for deterministic model.
Nodes 4, 5 and 6 form a collaboration group and nodes 3 and 8 form another
1264 T. Dinesh Ram Kumar et al.

Fig. 3. Multipath solutions when Fig. 4. Multipath solutions when prob-

both curiosity and collaboration is abilistic curiosity and collaboration is
considered considered

collaboration group. Nodes 1, 2, 3, 4, 5, 6 and 8 are curious nodes while 7 is not

(see Fig. 1(b)). If collaborations were ignored two solutions were found as seen
in Fig. 2. One of them involves directly sending the message through the path
0 → 7 → 9 as node 7 is not curious (Fig. 2(b)). Here 0 (zero) is the start and 9
is the end node. Other solution (Fig. 2(a)) involves two shares passing through
disjoint paths 0 → 8 → 5 → 1 → 9 and 0 → 2 → 4 → 9. Now considering
collaboration groups as shown in Fig. 3, first solution still works as node 7 does
not collaborate with any other node. However the latter solution does not work
as nodes 4 and 5 can gain all the shares. Thus we need to ensure that these
nodes don’t get all the required shares. Thus choosing the an additional path
0 → 2 → 1 → 9 ensures that no adversarial nodes gain all the three shares even
after collaboration. That is nodes 2 and 1 learn two shares as they are along the
paths, nodes 4, 5 and 6 learn information about two shares by collaboration.
Figure 4, shows a simulation of probabilistic model. Node 0 and node 9 are
labeled as start and end indicating source and destination. Sizes of nodes indicate
the curiosity levels of nodes. For example node 7 is less curious than node 1.
Links between nodes indicate the level of collaboration of nodes (Fig. 4(b)). For
example if a node 2 receives a share, node 8 is mostly to get the share than node
1. Current Solution involves 3 shares. Two of them passing through the path
0 → 3 → 9 and the remaining through 0 → 1 → 9. Note that even though node
7 is in neighborhood of 0 and 0 → 7 → 9 is a candidate path, however node 4
is likely to gain the share if node 7 is chosen which is highly curious. Node 1 is
chosen even though it is highly curious because choosing it does not convey much
information to other nodes. Node 1 and node 3 do not share information with
each other. Thus disjoint paths are chosen. More shares pass through node 3 than
node 1 as node 3 is less curious than node 1. Finally, Fig. 5 shows the performance
of proposed solution in terms of number of shared required with respect to the
curiosity and collaboration level of node participating in the network.
 Exploiting Multiple Paths in MACNETs for Providing Security 1265

Fig. 5. Shares required with change in collaboration and curiosity levels

6 Conclusions
We proposed a simple model to capture the behaviour of adversarial nodes in
ad-hoc cooperative networks using the concepts of curiosity and collaboration.
Then, we proposed a practical solution for key establishment using Diffie Hell-
man under the proposed adversarial model, when there are no trusted third
parties, by exploiting the multipath nature of multi-hop networks and using the
concept of secret sharing. We avoid MITM attacks by minimizing the likelihood
of adversaries from getting the message being transmitted as per security needs.
However we did not propose how to obtain the curiosity and collaboration lev-
els of the nodes. Finally, we have considered networks where adversaries do not
modify the messages.
However whenever nodes modify or drop messages, we need a mechanism to
detect these modifications or drops.

References
1. Shamir, A.: How to share a secret. Commun. ACM 22(11), 612–613 (1979). https://
doi.org/10.1145/359168.359176
2. Chan, H., Perrig, A., Song, D.: Random key predistribution schemes for sensor
networks. In: 2003 Symposium on Security and Privacy, pp. 197–213, May 2003
3. Du, W., Deng, J., Han, Y.S., Chen, S., Varshney, P.K.: A key management scheme
for wireless sensor networks using deployment knowledge. In: IEEE INFOCOM
2004, vol. 1, p. 597, March 2004
4. Du, W., Deng, J., Han, Y.S., Varshney, P.K., Katz, J., Khalili, A.: A pairwise key
predistribution scheme for wireless sensor networks. ACM Trans. Inf. Syst. Secur.
8(2), 228–258 (2005). https://doi.org/10.1145/1065545.1065548
5. Eschenauer, L., Gligor, V.D.: A key-management scheme for distributed sensor net-
works. In: Proceedings of the 9th ACM Conference on Computer and Communica-
tions Security, pp. 41–47. ACM (2002)
6. Ling, H., Znati, T.: End-to-end pairwise key establishment using multi-path in wire-
less sensor network. In: IEEE Global Telecommunications Conference on GLOBE-
COM 2005, vol. 3, p. 5, November 2005
7. Yin, L., Qiao-Yan, W., Zheng-Ping, J., Meng, S.: Secure pairwise key establishment
for key predistribution in wireless sensor networks. In: 2012 International Conference
on Computer Science and Service System (CSSS), pp. 822–825. IEEE (2012)
 Deniable Secret Handshake
Protocol - Revisited

Somnath Panja1(B) , Sabyasachi Dutta2 , and Kouichi Sakurai2

1
Applied Statistics Unit, Indian Statistical Institute, Kolkata, India
[email protected]
2
Faculty of Information Science and Electrical Engineering, Kyushu University,
Fukuoka, Japan
[email protected], [email protected]

Abstract. The notion of deniability ensures that the transcript gener-

ated in an interactive protocol does not yield any evidence of the interac-
tion. In the context of key-exchange protocols for secure message trans-
mission, the notion of deniability is well-explored. On the other hand, a
secret handshake protocol enables a group of authorized users to establish
a shared secret key and authenticate each other. Recently, a framework
for deniable secret handshake is proposed by Tian et al. in ISPEC 2018.
We analyze the protocol, show three flaws and give solutions to prevent
them.

Keywords: Secret handshake · Key exchange · Deniability ·

Public random oracle

1 Introduction
Privacy concerns in modern day electronic communications triggers the need of
examining familiar security services such as authentication and key agreement.
The paramount importance of protecting user’s security and privacy, especially
when two users want to communicate between themselves in a hostile network
condition, initiated the study of secret handshake protocols. Consider the situa-
tion when two parties A and B want to identify each other as member of a secret
agency and then communicate. However, the situation is really hostile and they
do not know each other. Consequently, A wants to make sure that if B is not
a member of the group then he will learn nothing about A s identity once the
protocol is run. If only B is a member of the group then he can identify A as a
member of the group, otherwise not. In a secret handshake protocol there is a
central authority (CA) who creates a group of authorized users and is responsible
for generating keys, certificates etc. for the users.

1.1 Motivation and Our Contribution

The concept of deniability is defined following the simulation based paradigm.
An adversary tries to convince a third party (judge) by producing a proof of a
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1266–1278, 2020.
https://doi.org/10.1007/978-3-030-15032-7_106
 Deniable Secret Handshake Protocol - Revisited 1267

conversation held between two authorized users of the same organization. An

secret handshake (SH) protocol is said to be deniable if a simulator can generate
the same view of the protocol transcript which is indistinguishable from the real
view. Mainly, two types of deniability exist in the literature - full deniability and
strong deniability. In a fully deniable SH protocol, an adversary’s view should be
indistinguishable from the view of a simulator when two honest parties faithfully
perform a SH protocol. We note that the simulator is able to produce the view
without the honest users’ certificates and CA’s master secret key. However, the
notion of strong deniability takes care of the scenario when one of the users is
malicious and “acts” as the aforementioned adversary to trap an honest user
with whom he is communicating. The definition of strong deniability requires
that the simulator must be fed with the same inputs as the malicious party/
adversary which includes secret certificates and randomness of the adversary.
Now if the simulator can produce an indistinguishable view from the real view
then the protocol is said to be strongly deniable. For more details, we refer to
the paper of [18].
In this paper, we consider the situation when a user behaves semi-honestly
and participates in an SH protocol with an honest user. A semi-honest user is not
completely honest but follows the protocol as an honest user. However, he may
store the randomness that he uses in the protocol and may try to gather more
information from the run of the protocol than an honest user would. We first show
that in presence of a semi-honest user, the protocol of Tian et al. [18] is not fully
deniable. More specifically, if the responder of a SH protocol is semi-honest and
maintains receipt(s) then a simulator is unable to generate a transcript which is
indistinguishable from the transcript generated during the real execution of the
protocol- resulting in the loss of full deniability for the initiator. We then propose
a possible countermeasure to fix the issue. We furthermore present two attacks-
viz. “man-in-the-middle” (MITM) attack and “cutting-last-message-attack” to
the protocol. We conclude the paper with solutions to resist these two types of
attacks.

1.2 Related Work

The concept of secret handshake was first introduced by Balfanz et al. [1]. The
security of the scheme was based on the hardness of bilinear Diffie-Hellman
problem in random oracle model. The scheme was constructed from a pairing-
based key agreement scheme. Castelluccia et al. [4] gave a more efficient scheme
under the computational Diffie-Hellman assumption in the random oracle model.
Tsudik-Xu [20] proposed a protocol where in a single run any number of members
can authenticate themselves to other members in the group. Yamashita-Tanaka
[23] gave a scheme with multiple groups that can accommodate change of mem-
berships. That is, if a member is added to a new group or is deleted from one,
it is not necessary to change his other memberships. This work was forwarded
by Kawai et al. [12] to achieve a monotonicity condition. Apart from anonymity
(which is inherent) of the participant, few other properties e.g. affiliation-hiding
1268 S. Panja et al.

[7], unlinkability [9], and user untraceability [13] are also explored in the litera-
ture of secret handshake protocol. Tian et al. [19] proposed a solution to achieve
full-unilinkability based on a k-size one-time certificate set. Untraceble secret
handshake allows authorized users (participants) to remain untraceable if the
issuing authority is not trusted. To cope against untrusted CA, Manulis et al.
proposed two solutions [13,14].
Deniability in the context of authentication was formally introduced by
Dwork et al. [5]. Raimondo et al. [16] explored the notion of deniability for secure
key-exchange protocols and put forward the notion of strong deniability and
partial deniability. Jiang-Naini [10] gave an efficient key exchange protocol that
achieves full deniability in the public random oracle model. Public random oracle
model was defined by Pass [15], which is a weaker assumption than the random
oracle. Unger-Goldberg [21] studied deniable authenticated key exchanges in the
generalized universal composability framework. They gave two protocols offering
forward secrecy and both full and strong deniability. Later, they extended their
study [22] to propose three strongly deniable key exchange protocols- deniable
authenticated key exchange with zero-knowledge, zero-knowledge Diffie-Hellman
and extended zero-knowledge Diffie-Hellman. Yao-Zhao [24] proposed a provably
secure internet key exchange protocol which provides simultaneous strong deni-
ability for both the participants. Their construction achieves deniability in the
restricted random oracle model (defined in [25]). Recently, Tian et al. proposed
a framework for deniable secret handshake protocol (DSH) [18]. They have given
a generic construction of a DSH protocol from any forward-secure secret hand-
shake protocol.

2 Preliminaries
Online and Offline Judge: When we discuss deniability, we must do so with
respect to a type judge (or distinguisher). Two primary types of judges have
been discussed in the secure messaging literature such as offline judges, and
online judges.
An offline judge examines the transcripts of a protocol execution that
occurred in the past and decides whether or not the parties mentioned in the
transcript were actually involved in the conversation. An online judge interacts
with a protocol participant, referred to as the informant, while the protocol
conversation is occurring.

2.1 Security Model

In this section, we introduce the security model due to [18]. The linkable
affiliation-hiding (LAH) and untraceability properties of their model follow
directly from [8,13] respectively.
States: Let us define a user set U of n users. The i-th session established by a
user U is denoted by ΠUi . An oracle ΠUi may be considered as used or unused.
An oracle ΠUi is considered as unused if it has never been initialized. When
 Deniable Secret Handshake Protocol - Revisited 1269

an oracle ΠUi is initialized, it becomes part of a group. After initialization, the

oracle is marked as used. The internal states state iU are stored by the oracle. The
oracle ΠUi accepts and terminates the protocol as soon as the session key KUi is
computed. The oracle stops sending or receiving messages after terminating the
protocol. If the protocol execution fails, the oracle terminates without having
accepted.
Partnering: We denote the pseudonym of all users recognized by the i-th session
ΠUi as pidiU and the corresponding session identifier as sidiU . Two instance oracles
ΠUi and ΠUj  are partners if and only if pidiU = pidjU  and sidiU = sidjU  .

2.2 System Model

A deniable secret handshake scheme is a collection of the following algorithms.
Setup: On input of a security parameter κ, the algorithm outputs public param-
eters params.
KeyGen: This algorithm is executed by the central authority (CA). The algo-
rithm, on input params, outputs the group public key mpk, corresponding secret
key msk and an empty pseudonym revocation list L .
Add: This algorithm is an interactive algorithm between the CA and a user.
The algorithm, on input msk and a user U ∈ U , generates a public pseudonym
pk and the corresponding secret certificate cert for the user U . The CA updates
the group pseudonym list by adding public pseudonym pk. The user will be
registered after executing this algorithm.
Revoke: This algorithm is executed by CA. On input pk from a user, the algo-
rithm updates the group pseudonym revocation list L by adding the public
pseudonym pk of the user.
Handshake: This is an interactive algorithm executed by some set of partic-
ipants Δ = {U1 , U2 , . . . , Un } ⊆ U . Each user Ui runs the session Πiτ of the
protocol on some inputs: (pkiτ , certτi ), mpkiτ and L , where (pkiτ , certτi ) is the
public pseudonym/secret certificate pair of user Ui , mpkiτ is the group public
key in Ui ’s view and L is the group pseudonym revocation list. The algorithm
outputs a session key SK if and only if her counter part users are registered and
non-revoked users, otherwise the algorithm rejects.

2.3 Session Key Security

Now we describe the adversarial behavior. An adversary is allowed to fully control
the communication network. The adversary can inject, modify, or delete messages
at will. He can also launch man-in-the-middle attack. He can corrupt some users
and obtain their secret keys, internal states and session keys. A session ΠUi is
secure if an adversary can not obtain the established session keys unless it is
compromised trivially through party corruption. We consider the same session
key security model and security definition as described in Tian et al. [18].
1270 S. Panja et al.

Generic Concurrent Knowledge Extraction Assumption (GCKEA): We fol-

low the same assumption used in Tian et al. This is a generalized version of
Concurrent KEA [24] and Knowledge of Pairing Pre-Image Assumption (KPA)
[17]. Due to the page limitation, we refer to [18] for details about this assumption.

3 Revisiting DSH Protocol of Tian et al. [18]

3.1 The Protocol
Tian et.al. have introduced the notion of deniable secret handshake framework
DSH [18]. For the sake of completeness, we describe the protocol almost verba-
tim. Their proposed generic framework consists of the following building blocks.
1. A forward-secure secret handshake protocol SH = (Setup, KeyGen, Add,
Revoke, Handshake),
2. A blind digital signature scheme BS = (KeyGen, Signer and User, Verify),
3. A public key based key derivation function PKDF,
4. A proof of knowledge PoK and
5. A collision-resistant hash function H.
The DSH protocol given in [18] is as follows.
Setup: Given a security parameter λ as input, the Setup algorithm outputs
public parameters: params ← SH.Setup.
KeyGen: The CA runs the SH.KeyGen algorithm to obtain the group pub-
lic/secret key pair (mpk, msk) and an empty pseudonym revocation list L .
Add: The CA and user  run the BS.Signer and User(msk) interactive algorithm
to obtain a pseudonym/certificate pair (pka , certa ) of user Â. User  takes pka
as public pseudonym.
Revoke: The group CA runs the SH.Revoke(pka ) algorithm to update the group
pseudonym revocation list L . Note that public pseudonym pka is added to
revocation list L .
Handshake:
• User  runs the SH.Handshake.Ephemeral algorithm to obtain ephemeral
secret/public key pair (eska , epka ) and sends (epka , pka ) to user B̂;
• After receiving (epka , pka ) from user Â, user B̂ performs the following steps:
1. Run the SH.Handshake.Ephemeral algorithm to obtain ephemeral secret
and public key pair (eskb , epkb );
2. Compute the proof of knowledge PoK{(eskb ) : H(P KDF (epkb , epka ))};
3. Send (epkb , pkb , P oK(eskb )) to user Â.
• Upon receiving (epkb , pkb , P oK(eskb )) from user B̂, user  computes the
proof of knowledge (i.e., non-malleable zero-knowledge)
PoK{(eska , certa ) : H(P KDF (epka , epkb )||P KDF (pka , epkb ))} and sends it
to user B̂. Meanwhile, Â computes the final session key SKa = H(Ka ||sid),
where Ka = SH.Handshake.KDF (eska , epkb , certa , mpk, L, init) and the
session identifier is sid = (epka ||epkb ).
 Deniable Secret Handshake Protocol - Revisited 1271

• Upon receiving PoK(eska , certa ) from user Â, user B̂ computes the
proof of knowledge PoK{(eskb , certb ) : H(P KDF (epkb , epka )||P KDF (pkb ,
epka ))} and sends it to user Â. Meanwhile, B̂ computes the final session
key SKb = H(Kb ||sid), where Kb = SH.Handshake.KDF (eskb , epka , certb ,
mpk, L, resp). We notice that Ka = Kb holds due to the correctness of
SH.Handshake algorithm.

3.2 Analysis of the Protocol

According to the protocol, at the beginning of the handshake session, the user Â
sends her public pseudonym pka to the user B̂. In the next step, upon receiving
pka from the Â, a semi-honest user B̂ can generate her ephemeral secret and
public key pair (eskb , epkb ) and can keep a receipt of its randomness and keys.
User B̂ can show this receipt to a judge to prove their conversation. As a result of
this attack, a simulator can not generate the transcript that is indistinguishable
from the transcript during real protocol execution. The next subsection describes
an example of such flaw. We note that such a flaw shares a similar idea as given
in [6].
Receipt Flaw: Suppose the user  is an honest user and B̂ is a semi-honest
user. They execute the deniable secret handshake protocol and communicate
with each other following the protocol specification. Meanwhile, user B̂ keeps
the receipt of its randomness whenever he receives a request from the initiator
user  with public key pka . Later if only the receipt is obtained by the police
then the full deniability will be lost for both the users. This is a flaw in the
system in the sense that it is easy for user B̂ to keep a receipt depending on the
public key in the first message.
To accomplish this task, upon receiving (epka , pka ) from user Â, user B̂
chooses a value uniformly at random from Z∗q for generating ephemeral pub-
lic key/secret key pair and keeps a receipt of the randomness to prove how
she generated the public key/secret key pair. Since user B̂ keeps a receipt, the
exact ephemeral public key/secret key of user B̂ can be generated later from
that receipt. However, whenever B̂ receives a pseudonym different from pka , she
chooses a value uniformly at random from Z∗q for generating her ephemeral public
key/secret key pair and may not keep a receipt. In future, police approaches to
judge with B̂’s receipt to prove the conversation with user  whose pseudonym
is pka .
In the above scenario: (1) a simulator S can not simulate the tran-
script of the ephemeral public/secret keys generated by user B̂ since user
B̂ maintains a receipt of randomness and the exact ephemeral public/secret
key pair of B̂ can be generated from that receipt. This implies that the
simulator will no be able to simulate the second message of the hand-
shake protocol (epkb , pkb , P oK(eskb )). (2) A simulator S can not generate the
proof of knowledge (i.e., non-malleable zero-knowledge) PoK{(eska , certa ) :
H(P KDF (epka , epkb )||P KDF (pka , epkb ))} in the third message without hav-
ing the secret certificate, certa , of user  and the secret key of the semi-honest
1272 S. Panja et al.

user B̂. As user B̂ keeps the receipt of randomness, the exact real transcript of
this third message can be generated from that receipt. Thus, user  can not deny
of having handshake session with user B̂ i.e. breaking full deniability property
of the protocol.
Man-in-the-Middle (MITM) Attack: In this attack, we consider two hon-
est users  and B̂ trying to execute the deniable secret handshake protocol
[18]. However, upon observing the public pseudonym pka , a man-in-the-middle
adversary (MITM) M̂ tries to interfere in their communication. We now present
this attack on DSH, which is depicted in Fig. 1. Since the PKDF is assumed
to be a public key bases key derivation function, let us, for example, assume
PKDF{(epka , epkb )} = g xy if epka = g x and epkb = g y .

Fig. 1. Full deniability loss of user  under man-in-the-middle (MITM) adversary

attack.

Here, user  and user B̂ are honest users. In this scenario, the MITM adver-
sary M keeps a receipt of its random variable r and presents it to an offline or
online judge. The MITM adversary M communicates with the honest user  in
the name of user B̂ as shown on the left session of the Fig. 1. The right session of
the Fig. 1 depicts the communication between the MITM initiator M with the
honest responder B̂ in the name of user Â.
Upon receiving the first round message (epka , pka ) (for example, (g x , g a ))
from Â, the adversary M chooses a number uniformly at random from Z∗q
and sends ((epka )r , pka ) (for example, (g rx , g a )) to user B̂ in the name
of Â. The adversary keeps a receipt of its randomness r. Upon receiving
the second round message (epkb , pkb , P oK{eskb : H(P KDF (epkb , (epka )r ))})
(for example, (g y , g b , H(g rxy ))) from honest user B̂, the adversary modi-
fies the message to ((epkb )r , pkb , P oK{eskb : H(P KDF (epkb , epka ))}) (for
example, (g ry , g b , H(g rxy ))) and sends it to user  in the name of user B̂.
In the third round, the adversary receives the message P oK{(eska , pka ) :
H(P KDF (epka , epkb )||P KDF (pka , epkb ))} (for example, H(g rxy ||g ray )) from
honest user  and aborts the conversation with both user  and user B̂. Note
that in these conversations, the adversary M does not need to do any oracle
query. After these conversations, the adversary M̂ approaches to judge with her
 Deniable Secret Handshake Protocol - Revisited 1273

receipt to prove that the user  whose pseudonym is pka is involved in the
conversation.
In the above scenario, in case of honest user  and B̂, a simulator S
can not generate the proof of knowledge (i.e., non-malleable zero-knowledge)
PoK{(eska , certa ) : H(P KDF (epka , epkb )||P KDF (pka , epkb ))} in the third
message without having the secret certificate, certa , of user  and the random
number r generated by the adversary M . Since the adversary does not make
any oracle query, the simulator can not extract the exponent r chosen by the
adversary M . Therefore, The user  losses deniability in this case.
Cutting-Last-Message Attack: Since the PKDF is assumed to be a public
key based key derivation function, we assume PKDF{(epka , epkb )} = g xy if
epka = g x and epkb = g y .
This attack works as follows. A man-in-the-middle adversary M interacts
with the uncorrupted user  in the name of user M = B̂ in a session (referred to
as first session), while concurrently communicating with the uncorrupted user
B̂ in the name of user  in another session (referred to as second session).
The adversary M relays the messages between user  and user B̂ in these two
sessions, but aborts the first session after receiving last message from the user
B̂ in the second session.
Such an attack results in authentication failure as follows: the user B̂ is
perfectly fooled to believe that the user B̂ has taken a part in conversation with
user  in the second session and shared a session key with that user. However,
user  thinks that it only took part in an aborted session with user M in the
first session. The Fig. 2 depicts this attack.

Fig. 2. “cutting-last-message” attack.

Such an attack can be ruled out by adding pkb in the proof-of-knowledge

in the second message, such as PoK{(epkb ) : H(pkb ||P KDF (epkb , epka ))}
or by including pkb in the PKDF function. Specifically, after receiving the
second message (epkb , pkb , P oK{eskb : H(pkb ||P KDF (epkb , epka ))}) from
user B̂, the adversary can not send correctly (epkm , pkm , P oK{eskb :
1274 S. Panja et al.

H(pkm ||P KDF (epkb , epka ))}), where epkm is the ephemeral public key of adver-
sary M , pkm is the public pseudonym of adversary M = B̂.

3.3 Possible Countermeasures

Solution: Using Public Random Oracle Model (pRO)

(Public) Random Oracle: Random oracle [2] H : {0, 1}∗ → {0, 1}k is a random
function such that it has following two properties: (i) it is a function such that
the same input gives the same output, and (ii) it is completely random in the
sense that for any input x, H(x) is uniformly distributed over {0, 1}k . A random
oracle (RO) can be described as follows.
Let us consider a set L ⊂ {0, 1}∗ ×{0, 1}k . Initially, L = φ. For an input x, the
value of H(x) is computed as follows. First, check whether there exists y ∈ {0, 1}k
such that (x, y) ∈ L. If there is no such y, choose a random y ← {0, 1}k and
add the tuple (x, y) into the set L. In any case, return y as the computed value
H(x).
Our protocol in this section is proven to be deniably secure in the public ran-
dom oracle (pRO) model. In this model, the random oracle is a public random
function that is accessible by the adversary and the simulator by submitting
input and receiving output. The simulator can see the input and output pairs of
all random oracle queries. This type of oracle is introduced in [15] for proving
deniable zero knowledge. Note that deniability means the simulator’s code can
also be executed by the adversary himself and thus, he can simulate the tran-
script in adversary’s view without actually interacting with honest parties. In
this case, the adversary is the only entity interacting with the public random
oracle (The adversary can forward the simulator’s query to the public random
oracle) and he can feed to the simulator with the oracle input/output. Thus,
the simulation under the public random oracle model can be replayed by the
adversary. However, in the traditional random oracle model [2], the simulator
maintains the random oracle evaluation. Now if the simulator’s code is executed
by the adversary, then the random oracle evaluation is maintained by the adver-
sary since the simulator is his subroutine. However, in the real protocol, the
adversary can only submit input to the random oracle and receive the output.
Therefore, in the traditional random oracle model, the simulator’s code can not
be played by an adversary and thus, the simulation is not guaranteed to be
deniable.
The Protocol: Let H : {0, 1}∗ → {0, 1}k be a hash function. We have described
the modified procedure below under the pRO model described above. The pro-
tocol uses following components: (1) a forward secure secret handshake protocol,
(2) a blind signature algorithm BS=(KeyGen, Signer and User, Verify); (3) a
public key derivation function PKDF, a proof of knowledge PoK and a collision-
resistant hash function H. For simplicity, in this section, we have described the
protocol in the two-party setting using user  and user B̂. It can be extended
to multi-party setting using BD protocol [3].
 Deniable Secret Handshake Protocol - Revisited 1275

– Setup: On input of a security parameter κ, this algorithm outputs public

parameters params ← SH.Setup.
– KeyGen: This algorithm is executed by the group CA. The algorithm, on
input params, outputs the group public key mpk, corresponding secret key
msk and an empty pseudonym revocation list L .
– Add: This algorithm is an interactive algorithm between the CA and a user.
The algorithm, on input msk and a user Â, executes the BS.Signer and
User(msk) interactive algorithm ([11]) to generate a public pseudonym pka
and the corresponding secret certificate certa for the user Â. User  takes
pka as public pseudonym.
– Revoke: This algorithm is executed by CA. On input pka from a user Â,
the algorithm executes the SH.Revoke(pka ) algorithm to update the group
pseudonym revocation list L by adding the public pseudonym pka of the user
Â.
– Handshake:

• User  executes the SH.Handshake.Ephemeral algorithm that outputs the

ephemeral secret/public key pair (eska , epka ) for user Â. User  sends epka
to user B̂.
• Upon receiving epka from user Â, user B̂ performs the following steps.
1. It executes the SH.Handshake.Ephemeral algorithm to get ephemeral
secret and public key pair (eskb , epkb );
2. It computes the proof of knowledge PoK{(eskb ) : H(pkb ||P KDF (epkb ,
epka ))};
3. User B̂ sends (epkb , pkb , P oK(eskb )) to user Â.
• Upon receiving (epkb , pkb , P oK(eskb )) from user B̂, user  evaluates the proof
of knowledge (i.e., non-malleable zero-knowledge)
PoK{(eska , certa ) : H(P KDF (epka , epkb )||P KDF (pka , epkb ))} and sends
it along with public pseudonym pka to user B̂. Â calculates the final ses-
sion key SKa = H(Ka ||sid), where Ka = SH.Handshake.KDF (eska , epkb ,
certa , mpk, L, init) and the session identifier sid = (epka ||epkb ).
• Upon receiving PoK(eska , certa ) and pka from user Â, user B̂ com-
putes the proof of knowledge PoK{(eskb , certb ) : H(P KDF (epkb , epka )||
P KDF (pkb , epka ))} and sends it to user Â. Meanwhile, B̂ calculates
the final session key SKb = H(Kb ||sid), where Kb = SH.Handshake.
KDF (eskb , epka , certb , mpk, L, resp) and the session identifier sid = (epka ||
epkb ). The correctness of the protocol i.e. the equation Ka = Kb holds due
to the correctness of the underlying SH.Handshake algorithm.

Theorem 1. The proposed generic framework achieves session key security in

the public random oracle model provided the underlying secret handshake protocol
SH is session key secure.

Proof. The proof follows from essentially a similar argument of Theorem 1 in

[18]. The detailed proof is deferred to the full version of the paper. 

1276 S. Panja et al.

Discussions on Deniability: We outline the main idea of the proof of deni-

ability. To prove the deniability, a simulator needs to simulate the answer of
send, session key reveal, test and ephemeral secret key reveal, long-term secret
key reveal and group secret key reveal queries such that the adversary’s view in
the simulated transcript is indistinguishable from the real transcript, while the
simulator should not use any uncorrupted secret keys. Due to the page limita-
tion, we refer to the session key security section of [18] for description about these
queries. The difficult part is to answer the send query. Consider a send query of
the form (Ui , Gj , s, m) to simulate a message m. Let us assume m = H(g s ||K);
where K is publicly known, s is a secret, g is an publicly known element of the
considered group and ‘||’ is the concatenate operation. Therefore, (g s ||K) must
have been queried to the oracle H to be consistent with the input (g s ||K) and
output m, that can be verified by the simulator since he sees all the input/output
for all H oracle queries. Otherwise, the message m is inconsistent with probabil-
ity 1 − negligible(κ) if (g s ||K) is not queried to the oracle, where negligible(κ)
is a negligible function in security parameter κ (i.e. for any positive polynomial
p(n), there exists integer n0 > 0 such that negligible(n) > (1/p(n))), ∀n > n0 .).
Thus, if (g s |K) is queried to the oracle, the simulator can extract s by the generic
concurrent knowledge extraction assumption (GCKEA). This assumption is also
used in [18] (a similar assumption is used in [24]). Thus, the simulator can answer
adversary’s send queries. Similarly, the simulator can answer other send queries.

Theorem 2. Let H be a public random oracle. Then the secret handshake pro-
tocol described above is deniable.

The proof is similar to the proof described in [24] and is deferred to the full
version of the paper.

Note 1 (Unlinkability). This solution does not satisfy the unlinkability property
since the public pseudonym is being sent by each user in every session. Thus,
these sessions are linkable.

4 Conclusion

We have discussed some possible attacks to the fully deniable secret handshake
protocol of Tian et al. [18]. We mention that there may be a subjective issue
about the model of deniability. However, this paper opens up possibilities to
scrutinize definitions for deniability in the context of secret handshake.

Acknowledgements. The first author is financially supported by Indian Statistical

Institute, Kolkata, India under a research fellowship program. The work presented in
this paper was carried out while the first author visited Kyushu University, Japan. The
second author is financially supported by the National Institute of Information and
Communications Technology (NICT), Japan under an International Exchange Pro-
gram. The third author is partially supported by JSPS Grant-in-Aid for Scientific
Research KAKENHI (C) JP18K11297.
 Deniable Secret Handshake Protocol - Revisited 1277

References
1. Balfanz, D., Durfee, G., Shankar, N., Smetters, D.K., Staddon, J., Wong, H.-C.:
Secret handshakes from pairing-based key agreements. In: IEEE S&P 2003, pp.
180–196 (2003)
2. Bellare, M., Rogaway, P.: Random oracles are practical: a paradigm for designing
efficient protocols. In: ACM CCS 1993, pp. 62–73 (1993)
3. Burmester, M., Desmedt, Y.G.: Efficient and secure conference-key distribution.
In: Lomas, M. (ed.) Security Protocols 1996. LNCS, vol. 1189, pp. 119–129 (1997)
4. Castelluccia, C., Jarecki, S., Tsudik, G.: Secret handshakes from CA-oblivious
encryption. In: ASIACRYPT 2004. LNCS, vol. 3329, pp. 293–307. Springer, Hei-
delberg (2004)
5. Dwork, C., Naor, M., Sahai, A.: Concurrent zero-knowledge. J. ACM 51(6), 851–
898 (2004)
6. Goldreich, O., Krawczyk, H.: On the composition of zero-knowledge proof systems.
SIAM J. Comput. 25(1), 169–192 (1996)
7. Jarecki, S., Kim, J., Tsudik, G.: Group secret handshakes or affiliation-hiding
authenticated group key agreement. In: CT-RSA 2007. LNCS, vol. 4377, pp. 287–
308 (2006)
8. Jarecki, S., Kim, J., Tsudik, G.: Beyond secret handshakes: affiliation-hiding
authenticated key exchange. In: CT-RSA 2008. LNCS, vol. 4964, pp. 352–369
(2008)
9. Jarecki, S., Liu, X.: Private mutual authentication and conditional oblivious trans-
fer. In: CRYPTO 2009. LNCS, vol. 5677, pp. 90–107 (2009)
10. Jiang, S., Safavi-Naini, R.: An efficient deniable key exchange protocol (extended
abstract). In: FC 2008. LNCS, vol. 5143, pp. 47–52 (2008)
11. Juels, A., Luby, M., Ostrovsky, R.: Security of blind digital signatures. In: Kaliski,
B.S. (ed.) CRYPTO 1997. LNCS, vol. 1294, pp. 150–164. Springer, Heidelberg
(1997)
12. Kawai, Y., Tanno, S., Kondo, T., Yoneyama, K., Ohta, K., Kunihiro, N.: Extension
of secret handshake protocols with multiple groups in monotone condition. IEICE
Trans. 93–A(6), 1122–1131 (2010)
13. Manulis, M., Poettering, B., Tsudik, G.: Affiliation-hiding key exchange with
untrusted group authorities. In: ACNS 2010, pp. 402–419 (2010)
14. Manulis, M., Poettering, B., Tsudik, G.: Taming big brother ambitions: more pri-
vacy for secret handshakes. In: Privacy Enhancing Technologies 2010, pp. 149–165
(2010)
15. Pass, R.: On deniability in the common reference string and random oracle model.
In: CRYPTO 2003, pp. 316–337 (2003)
16. Di Raimondo, M., Gennaro, R., Krawczyk, H.: Deniable authentication and key
exchange. In: ACM CCS 2006, pp. 400–409 (2006)
17. Schäge, S.: TOPAS: 2-pass key exchange with full perfect forward secrecy and
optimal communication complexity. In: CCS, pp. 1224–1235. ACM (2015)
18. Tian, Y., Li, Y., Zhang, Y., Li, N., Yang, G., Yu, Y.: DSH: deniable secret hand-
shake framework. In: ISPEC 2018, pp. 341–353 (2018)
19. Tian, Y., Zhang, S., Yang, G., Mu, Y., Yu, Y.: Privacy-preserving k-time authen-
ticated secret handshakes. In: ACISP (2), pp. 281–300 (2017)
20. Tsudik, G., Xu, S.: A flexible framework for secret handshakes. In: Privacy Enhanc-
ing Technologies 2006, pp. 295–315 (2006)
1278 S. Panja et al.

21. Unger, N., Goldberg, I.: Deniable key exchanges for secure messaging. In: ACM
CCS 2015, pp. 1211–1223 (2015)
22. Unger, N., Goldberg, I.: Improved strongly deniable authenticated key exchanges
for secure messaging. PoPETs 2018(1), 21–66 (2018)
23. Yamashita, N., Tanaka, K.: Secret handshake with multiple groups. In: WISA 2006,
pp. 339–348 (2006)
24. Yao, A.C.-C., Zhao, Y.: Privacy-preserving authenticated key-exchange over inter-
net. IEEE TIFS 9(1), 125–140 (2014)
25. Yung, M., Zhao, Y.: Interactive zero-knowledge with restricted random oracles. In:
TCC 2006. LNCS, vol. 3876, pp. 21–40 (2006)
 Detection of Algorithmically Generated
Domain Names in Botnets

Deepak Kumar Vishvakarma1 , Ashutosh Bhatia2(B) , and Zdenek Riha3

1
Center for Artificial Intelligence and Robotics, CAIR, Bangalore, India
[email protected]
2
Department of Computer Science, BITS, Pilani, India
[email protected]
3
Faculty of Informatics, Masaryk University, Brno, Czech Republic
[email protected]

Abstract. Botnets pose a major threat to the information security of

organizations and individuals. The bots (malware infected hosts) receive
commands and updates from the Command and Control (C&C) servers,
and hence, contacting and communicating with these servers is an essen-
tial requirement of bots. However, once a malware is identified in the
infected host, it is easy to find its C&C server and block it, if the domain
names of the servers are hard-coded in the malware. To counter such
detection, many malwares families use probabilistic algorithms known as
domain generation algorithms (DGAs) to generate domain names for the
C&C servers. This makes it difficult to track down the C&C servers of
the Botnet even after the malware is identified. In this paper, we propose
a probabilistic approach for the identification of domain names which are
likely to be generated by a malware using DGA. The proposed solution is
based on the hypothesis that human generated domain names are usually
inspired by the words from a particular language (say English), whereas
DGA generated domain names should contain random sub-strings in it.
Results show that the percentage of false negatives in the detection of
DGA generated domain names using the proposed method is less than
29% across 30 DGA families considered by us in our experimentation.

Keywords: Domain name system · Domain generations algorithms ·

Botnets · Command and control servers

1 Introduction
In modern era, the Internet has become an essential part of our lives, just like any
other unity such as water or electricity. However, the use of Internet opens up a
lot of doors for security threats. It makes computers prone to be infected by the
malicious softwares in form of Internet worm, computer virus, trojan horse, key-
logger, sniffer, etc. These malicious softwares pose a major threat to the infected
systems. Further, these infected hosts can form a network known as Botnet,
where each compromised host or “Bot” can be controlled remotely by a command
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1279–1290, 2020.
https://doi.org/10.1007/978-3-030-15032-7_107
1280 D. K. Vishvakarma et al.

and control (C&C) server. Botnet pose a significant threat for cyber-security
as they provide a distributed platform for cyber-crimes. In general, these Bots
are software programs that run on host computer, without host’s knowledge,
allowing C&C servers to control host actions remotely. There are various ways
a Botnet can be utilized to perform malicious operations. A prominent use of
Botnet is to launch distributed denial of service (DDOS) attack, where multiple
Bots present in a Botnet can start flooding the bandwidth or other resources of
the target system, after receiving the command from C&C servers.
The malware population has been increasing exponentially. As per the statis-
tics published in the Panda security report [3], there were 304 million unique
malware samples in year 2015. The diversity and number of types of malware
and mechanisms of communication between infected hosts and the C&C servers
undermine the effectiveness of classical signature-based detection techniques.
This makes detecting presence of attack, infected hosts and the C&C server a
challenging task. Often signature, and heuristic based detection and analysis
techniques generate a large amount of false positives, and further, falls short of
detecting unknown malware families [4].
To get the updated malware or commands from the servers, a “Bot” contacts
the C&C servers using their domain names. If the domain names of C&C servers
are hard coded in the malware itself, then the C&C servers can be tracked down
easily, once the malware in the host is identified. To counter this detection,
a number of malware families use domain generation algorithms (DGAs) to
dynamically generate domain names of C&C servers instead of storing the list
in the malware executable. A malware may generate a number of domains every
day and try to connect all of them until an active C&C server is found. This
way, a malware eventually finds the C&C server and get the updates or receive
commands. This techniques of generating the domain names algorithmically by
the malwares instead of hard coding the names in the malware itself makes
difficult for the law enforcement agencies (LEA) to track down and neutralize the
C&C servers. Thus, for a malware having DGA the traditional ways of analyzing
malware do not help much in revealing the C&C servers. This technique for
keeping a malicious Botnet in operation by constantly changing the domain
name of the Botnet owner’s C&C server is also called Domain Fluxing.
One way to identify the infected hosts, the family of DGA used by the mal-
ware and to know the domain name of C&C server, is to look for the traces
in the host system and in the network traffic, left by the “Bot”, when it was
trying to reach the C&C server. The features like the period of DNS queries
performed by the host and the failure rate of DNS queries could help to identify
the domain names generated by the DGA. However, a smart DGA can limit the
number of domain names tried for C&C server in a day and hence normalizing
the DNS query period and failure rates. It may also include dictionary words
while creating domain names to make it appear human generated domain names.
In this work, we try to find out the thin line separating the human created
domain names from DGA generated domain names. In particular, we propose a
mechanism to address an important research problem of identifying C&C servers
 Detection of Algorithmically Generated Domain Names in Botnets 1281

in Botnets for the scenario when the infected hosts make use of DGA to reach
out their C&C servers, instead of making an static entry of domain names for
their C&C servers.
Our idea to identify whether a domain name has been generated by a human
or a computer program (DGA algorithm) is based on the amount of information
i.e., entropy carried by the domain names generated by these two different types
of mechanisms (human and DGA). The entropy is a measure of the amount of
information carried by the subject and it is extensively used to measure the ran-
domness of a bit string in Cryptography. In this work, we use a similar approach
to measure the likelihood of a domain name being human created or generated by
a DGA family. We learn the probability distribution of characters and symbols
used in 1 million top domain names [11] and use it to compute the likelihood of
a subject domain name. If likelihood of domain name is above a threshold then
the domain name considered human generated otherwise it is marked as gener-
ated algorithmically by DGA equipped malware. To get the human generated
domain names, we have used publicly available resources of domain names [11].

2 Background
Based on the topology of the network and distribution of C&C servers, Botnets
can be classified as follows:
1. Centralized Topology: A small number of HTTP or IRC (Internet Relay
Chat) servers working as C&C servers.
2. Decentralized Topology: Assumes a peer to peer network, where every
Bot can be a C&C server.
3. Locomotive Topology: The location of C&C server keep changing over
time.
Usually, Centralized and Locomotive Botnets address the C&C servers using
DNS. Centralized topology of Botnet uses a small number of C&C servers. The
domain names of these servers are usually hard-coded in the malware executable
and could be easily discovered by analyzing the malware executable. This can
lead to quick neutralization of the C&C servers of the Botnet. The Bots (infected
hosts) may still exist but in the absence of the C&C servers they may not be
harmful.
The use of domain fluxing increases the resilience of Botnet against their
take-downs by increasing the flexibility of the C&C server addressing layer. The
domain fluxing Botnets may use deterministic DGA using some predefined tem-
plate or it may use a pseudo-random domain names. Further, the DGA may be
time dependent or independent. Therefore, the set of all DGAs can be classified
into four families (classes) [4].
1. Time Independent DGA (TID-DGA): The DGA of this type, use a
static seed for pseudo-random algorithm and generates the same set of domain
names every time they are executed. Kraken is a well known Botnet that used
TID-DGA [15].
1282 D. K. Vishvakarma et al.

2. Time Dependent and Deterministic DGA (TDD-DGA): This class of

DGAs, use a deterministic algorithm and seed hence domain names can be
easily computed for a given seed. However the seed is changed based on time.
A famous Botnet that used this type of DGA is Conficker.c worm [16].
3. Non-deterministic and Time Dependent DGA (TDN-DGA): The
algorithm used in this type of DGA is time dependent and non deterministic.
The anticipation of seed is infeasible and hence precomputation of domain
names [1]. An example Botnet which uses this type of DGA is Torpig Botnet.
It uses the popular trending topics of the social networking service, Twitter
as a seed to generate different set of domain names on subsequent every time
the algorithm is executed [2].
4. Non-deterministic and Time Independent DGA (TIN-DGA): In this
type of DGAs, the algorithm used for domain name generation is time inde-
pendent and non deterministic. Since algorithm is non deterministic and time
independent, guessing or precomputing domain names is infeasible. At the
same time, due to the randomness of generated domain names, the probabil-
ity of generating a correct domain name decreases drastically [2].

Table 1, illustrates the taxonomy of DGA families based on the algorithm

type.

Table 1. Taxonomy of DGA family based on algorithm type [2]

DGA type Time dependence Deterministic Example

TID-DGA No Yes Kraken
TDD-DGA Yes Yes Conficker, Torpig
TDN-DGA Yes No Newer Conflicker (D)
TIN-DGA No No Not seen

3 Related Work
The works in [7–9] analyzed the DNS traffic of the network to detect the DGA
generated domain names, assuming that the presence of domain fluxing (fre-
quent change in the address of C&C servers) indicates the existence of Bots
(infected hosts) in the network. This technique allows the detection of Bots in
the network and domain names of C&C servers. However, the technique men-
tioned fails in anticipating future domain names generated by the DGA. In [5],
the authors talk about an approach for extracting algorithm used by DGA from
a malware executable using dynamic taint analysis. The paper, also proposes to
perform static analysis techniques to over come the limitation of single execution
path coverage of dynamic analysis. In paper [6], authors mention a combination
of dynamic and static analysis techniques for extracting binary functions and
claim to extract complete DGA used by malware with proposed technique. In
 Detection of Algorithmically Generated Domain Names in Botnets 1283

an another work, Barabosch, Leder et. al. [2] propose a technique for automatic
extraction of domain name generation algorithm from the binary of malware
executables.
The work proposed in [12], talks about reputation based system to detect
DGA based Botnets. In this work, the algorithm assigns high negative reputa-
tion score to each host that is involved in suspicious Bot activities. It maintains a
suspicious group activity matrix and adds a hosts into that, which then algorith-
mically generates a large set of suspicious domain names. Similarly, it maintains
a failure matrix for hosts with a high number of failed DNS queries. Finally, it
calculates the negative reputation score of each host using the two matrices and
detects hosts with high negative reputation scores as bot-infected. Authors claim
that it can successfully detect DGA-based Botnets with a high detection rate
and a low false alarm rate while providing real-time monitoring in large-scale
networks.
In [13], authors propose a methodology to detect domain fluxes in DNS
traffic by looking for patterns inherent to domain names that are generated
algorithmically, in contrast to those generated by humans. They analyze the
distribution of alphanumeric characters as well as bigrams in all domains that
are mapped to the same set of IP addresses and compare the performance of
several distance metrics, including K-L distance, Edit distance, and Jaccard
measure. Authors claim that they were able to detect unknown Botnet exhibiting
advanced domain-name generation techniques from DNS records.

4 Identifying DGA Generated Domain Names

The problem that we have addressed in this work is to find out whether a given
domain name is algorithmically generated by a DGA or not. The subject domain
name may be captured from a DNS query in the network. The effectiveness of
our solution depends on the sample database of benign domain names created
to be accessed by humans.
Before having a detailed discussing on the solution of the aforementioned
problem that we propose, we first give the basic approach taken by us for iden-
tifying domain names generated using DGA and the concept of measuring ran-
domness present in given set of domain names. We assume that the purpose
of domain names is to avoid remembering numeric IP address and hence the
legitimate domain names should be easy to remember.

4.1 Basic Approach

We begin with the hypothesis that due to human memory limitations, domain
names should be predictable and memorable. The distribution of symbols in
the domain names follows the distribution of characters in the users’ language
(usually English). Domain names generated by malware using DGA do not have
such restrictions. Hence, randomness of human generated domain names should
be less than that of DGA generated domain names which can be used as a
1284 D. K. Vishvakarma et al.

(a) Domain Names Length Dist. (b) DGA: Domain Names Length Dist.

Fig. 1. Comparison of length distribution

(a) Numerics distribution (b) Alphabets distribution

Fig. 2. A comparison of symbol frequency distribution between top domain names,

English text and DGA generated domain names

measure to identify DGA generated domain names. Further, for DGA which
generate meaningful domain names, such as combination of dictionary words
and numerics are bound to generate the same pattern for all domain names
generated by them and hence the domain names generated by such DGA would
approximately have the same entropy values.
In our approach, we try to learn the distribution of characters in a collection
of popular domain names and use it to compute the randomness of a test domain
names. A subject domain name from test domain names having entropy more
than δ from the a threshold value can be considered as a suspected domain name
(i.e generated by DGA) and can be subjected to further investigations.
 Detection of Algorithmically Generated Domain Names in Botnets 1285

4.2 Measuring Randomness of Domain Names

Human generated domain names have limitations in terms of randomness and

generally follow some memorable pattern [10]. These requirements are similar to
the one required for user passwords. Password needs to be memorable by user
and this requirement brings the entropy (randomness) of the password down.
According to an article [10] published by National Institute of Standards and
Technologies (NIST), a human created password of 8 characters has random-
ness of 18 to 30 bits even with certain composition rules and dictionary words
restrictions.

Kolmogorov Complexity. Kolmogorov complexity(K-complexity) of a char-

acter string is a measure of the computational resources needed to specify the
object. In paper [14], author defines K-complexity as the length of the short-
est Turing Machine that outputs it and then halts. It can be considered as a
measure of randomness of strings.
K-complexity of a human-generated string is limited and a naive approach,
yet it is a hard problem to compute K-complexity of an arbitrary string.
Entropy (H ). Measuring entropy is a classical approach to compute the ran-
domness of the string. To work with the entropy, we need to know the probability
distribution of characters. Generally, the distribution is learned from a given data
set. In this work, we limit our study to the domain names in English language.
We use entropy of DGA generated domain names to classify them to one of DGA
families.

Likelihood. Likelihood is a measure to estimate the odds in favour of any event

to occur. As above to compute the likelihood of a string to appear in text, we
need to learn the distribution of characters from a given data set. In this work,
we use likelihood to identify DGA generated domain names.

4.3 Detailed Description and Algorithms

We used 1 million top domain names list available at [11] in the Internet and
studied the probability distribution of English alphabets in the listed domain
names. Then, we analyzed the distribution of length of the domain names and
frequency distribution of characters from top 1 million domain names collection.
Figure 1a, shows the plot of the length distribution in top 1 million domain
names.
We observe that the probability distribution of alphabets in the domain
names has similarity with the distribution of alphabets in the English text which
supports our hypothesis discussed in Sect. 4.1. Hence human created domain
names are likely to follow similar distribution and we can use the distribution
learned from the sample set for analysis of domain names.
1286 D. K. Vishvakarma et al.

After analysing human generate domain names, we analyzed 840836 domain

names generated by different DGA families available at [11], and studied the
length distribution and frequency of characters in generated domain names.
Figure 1b, shows the plot of the length distribution in the domain names gen-
erated by various DGA families. We observe that the distribution in sample set
is bell curved and centered near domain names with length 8 (Fig. 1a). Whereas
DGA generated domain names are skewed and majority of them are of length
19 (Fig. 1b).
Figure 2 shows the comparison in terms of frequency distribution of symbols
among top domain names, DGA generated domain names and letters in English
text.
In Fig. 2, we plotted the distribution for numerics and alphabets separately
for clarity, symbols which are not shown in the plot have zero frequency. The
DGA distribution curve differ from other curves but the difference is not very
drastic. This is because we have merged the domain names from different DGA
families in this figure. Later we will discuss the distribution of individual DGA
family in comparison with our sample set (top 1 million domain names) and will
be able to observe the difference in DGA families and standard domain names
distribution.
Algorithm 1, describes the process of identifying a domain name if it is gen-
erated automatically by a malware using known or unknown DGA. Once the a
domain name is identified to be DGA generated it can be assigned to one of the
known DGA families or to a class of unknown DGA family.
Algorithm 2 discusses the approach of domain name classification. We com-
pute the entropy of the domain name and compare it with min and max entropies
of the known DGA families and associate it with one or more of the known
families.

Algorithm 1. Identify DGA generated domain name

Data: sample data, test data
Result: List of DGA generated domain names;
1
2 Step1: Compute 1-gram probability distributions (PD) of symbols from sample
data of domain names;
3 Step2: Find out likelihood threshold, θ, for a domain name to be marked as
human created;
/* we have set θ = 0.00001 in our experiment. */
4 Step3: For all domain names from the test data, compute the likelihood of the
domain name;
5 if (likelihood < θ) then
6 mark the domain name to be DGA generated
 Detection of Algorithmically Generated Domain Names in Botnets 1287

Algorithm 2. Identify DGA family

Data: test domain name, DGA generated domain data set, sample domain
names data set(human created)
Result: List of DGA families the domain name belongs to or NULL in case the
domain name family could not be identified
1 Step1: Compute probability distributions (PD) of symbols from sample data of
domain names
2 Step2: Find out min and max entropies for all DGA families from DGA data
set.
3 Step3: Compute the entropy of the test domain name and compare with min
and max entropies of DGA families.
4 Step4: Assign one or more DGA families based on the previous step.
5 Step5: If the entropy of test domain name does not fall in range of entropies of
any of DGA family then mark it as unknown DGA family.

5 DGA Identification Results

In this section, we discuss the results of the experiments performed with training
data set and learning data set. We learn the probability distribution of the
symbols used in the 1 million top domain names [11] and uses that to compute
the likelihood of every domain name of the test data set (domain names belonging
to various DGA families).

5.1 DGA Likelihood Statistics

Figure 3, shows the summary of the minimum and maximum likelihood range of
a DGA family along with their set size.
We observe that the Minimum and Maximum values of likelihood of DGA
generated domain names varies from zero to 0.001064 for DGA families. And for
the Top200domainNames (test set created from normal domain names) it varies
from 0 to 0.07330. In the process of computing the likelihood of every domain
names of all DGA families from test data, we store it for further analysis to
identify the DGA generated domain names in test data set.

5.2 Likelihood Threshold Value

Experimentally, we fix our threshold value θ to 0.00001. We use θ as a reference
point to mark a domain name as DGA generated or human created. In the
Sect. 5.1 we compute the likelihood of every domain name of all DGA families
available in the test data and we also computed the likelihood of top200domains
data set which is human created domain names set to measure the false positive
errors of our approach. Now we compare the likelihood values for domain names
for each of the class from test data set with θ and all the domain names with
likelihood value lesser than the θ are marked as DGA generated domain names
Sect. 5.3.
1288 D. K. Vishvakarma et al.

Fig. 3. Comparison of Domain Names Fig. 4. Result Of DGA identification

based on likelihood based on likelihood

5.3 Results Statistics

In this section, we discuss the result of identification of DGA generated domain

names based on the likelihood of domain names and the likelihood threshold
θ from Sect. 5.2. We computed the likelihood of every domain name of DGA
families and compared it with the threshold θ. The result of comparison is shown
in the Fig. 4.
We observe that the maximum likelihood of domain names generated by DGA
families is 0.001064, refer Fig. 3. We use likelihood threshold value, θ = 0.00001.
A DGA family may have a portion of domain names with likelihood more than
θ, and hence the DGA identification Algorithm 6 fails to mark them as DGA gen-
erated domain names. This results in error in terms of false negative. Similarly,
for the test set top200domainNames, there is a subset of domain names with like-
lihood value less than θ and hence they are marked as DGA generated domain
names. This generates error in terms of false positives. We have quantified the
errors generated while identifying the DGA families and shown in the Fig. 4.
 Detection of Algorithmically Generated Domain Names in Botnets 1289

From Fig. 4, we observe that the error for the test DGA domain names varies
from zero (sisron family) to 28.85% (matsnu family) for different DGA families.
The reason for high error rates are similarity in the distribution of characters in
the DGA family domain names with the distribution of characters in the normal
domain names. The similarity results in high likelihood value and hence increases
the false positives.

6 Conclusion and Future Work

In this work, we tried to identify a domain name generated by a known or
unknown DGA family. Working on hypothesis, section that human created
domain names need to be memorable and hence usually inspired by words in the
language, whereas DGA generated domain names would have some random or
algorithmic generated strings in it. The randomly or algorithmically generated
domain names, have high entropy compared to the human generated domain
names. We observed that the error rates vary from 0% to 28.85% in identifying
the DGA generated domain names. A DGA may generate a domain name which
is similar to human created domain name and thus failing our hypothesis. But
in a large scale deployment of DGA families across various malware families the
generated domain name is still likely to have some component which is different
form normal human behaviour of created names.
Identifying and analyzing DGA generated domain names and the algorithm
used is an challenging research field. The work can be extended to:
1. Identify DGA family: To identify DGA family based on their entropy range
of domain names. Once a domain name is marked to be suspicious then
identifying family name may help a lot it analyzing the malware and in finding
the variants of same DGA family.
2. Compute Likelihood threshold θ: In this work we compute the θ experimen-
tally. For efficiency and accurate identification θ needs to be optimized.
3. Error Rates: The present technique gives hight false positives and needs to
be improved. Machine leaning techniques may give better results.

References
1. Stone-Gross, B., Cova, M., Cavallaro, L., Gilbert, B., Szydlowski, M., Kemmerer,
R., Kruegel, C., Vigna, G.: Your botnet is my botnet: analysis of a botnet takeover.
In: Security, pp. 635–647 (2009)
2. Barabosch, T., Wichmann, A., Leder, F., Gerhards-Padilla, E.: Automatic extrac-
tion of domain name generation algorithms from current malware
3. Panda Security: Pandalabs annual Report - 2015 summary (2016)
4. Yin, H., Song, D.: Panorama: capturing System-wise information flow for malware
detection and analysis. In: CCS 2007, Alexandra, Virginia, USA, 29 November–2
November 2007 (2007)
5. Kolbitsch, C., Holz, T., Kruegel, C., Kirda, E.: Inspector gadget: automated extrac-
tion of proprietary gadgets from malware binaries. In: Security and Privacy, pp.
29–44 (2010)
1290 D. K. Vishvakarma et al.

6. Caballero, J., Johnson, N.M., Mccamant, S., Song, D.: Binary code extraction and
interface identification for security applications. Electr. Eng. (2009)
7. Yadav, S., Reddy, A., Reddy, A.: Detecting algorithmically generated malicious
domain names. In: IMC 2010 (2010)
8. Bilge, L., Kirda, E., Kruegel, C., Balduzzi, M., Antipolis, S.: Exposure: finding
malicious domains using passive DNS analysis. In: 18th Annual Network and Dis-
tributed System Security Symposium, pp. 1–17 (2011)
9. Antonakakis, M., Perdisci, R., Dagon, D., Lee, W., Feamster, N.: Building a
dynamic reputation system for DNS. In: Proceedings of the 19th Conference on
Security, USENIX Security 2010, p. 18 (2010)
10. Burr, W., Dodson, D., Polk, W.: Electronic authentication guideline. NIST Special
publication 800-63 (2004)
11. http://osint.banbenekconsulting.com
12. Sharifnya, R., Abadi, M.: A novel reputation system to detect DGA-based botnets.
In: Proceedings 2013 ANR (2013)
13. Yadav, S., Reddy, A.K.K., Reddy, A.L.N., Ranjan, S.: Detecting algorithmically
generated domain-flux attacks with DNS traffic analysis. IEEE/ACM Trans. Netw.
20, 1663–1677 (2012)
14. Li, R., Vitanyi, P.: An Introduction to Kolmogorov Complexity and Its Applica-
tions. Springer, Heidelberg (1997)
15. Royal, P.: On the Kraken and Bobax botnets (2008). www.damballa.com/
downloads/press/Kraken Response.pdf. Accessed 06 Aug 2012
16. Leder, F., Werner, T.: Know your enemy: containing conficker. The Honeynet
Project (2009)
 How Important Are Logs of Ordinary
Operations? Empirical Investigation
of Anomaly Detection

Akinori Muramatsu1(B) and Masayoshi Aritsugi2

1
Computer Science and Electrical Engineering,
Graduate School of Science and Technology, Kumamoto University,
2-39-1 Kurokami, Chuo-ku, Kumamoto 860-8555, Japan
[email protected]
2
Big Data Science and Technology, Faculty of Advanced Science and Technology,
Kumamoto University, 2-39-1 Kurokami, Chuo-ku, Kumamoto 860-8555, Japan
[email protected]

Abstract. Anomaly detection is supposed to improve safety of com-

puters connected to the Internet. Cyberattackers would thus try to cheat
anomaly detection systems. In this paper, we focus on feasibility of cheat-
ing anomaly detection. We investigate anomaly situations which could
not be detected based on a detection technique and attempt to generate
such situations with using ordinary operations. We evaluate our attempt
empirically for demonstrating that logs of ordinary operations are signif-
icant information which should not be leaked.

1 Introduction

Many sophisticated and insistent cyberattacks have been observed in various

organizations. One of the recent ones is an advanced persistent threat (APT),
which is a set of stealthy and continuous computer network exploitation pro-
cesses. An example of this is operation aurora [1,9], where a user accessed a
malicious website and then malware was downloaded through the access. The
malware generated a backdoor at the organization, and then everything on it
could be accessed through it.
Anomaly detection has been studied for preventing cyberattacks [3,7,11]. In
the detection, a model is constructed with normal (and anomaly) data and the
difference between the current situation and the model in metric distances or
similarities determines if the situation is normal or not. [3] and [11] proposed to
introduce it in enterprise security systems and system of systems, respectively. [7]
developed an anomaly detection approach for host-based APT detection.
Note that cyberattackers may tend to devise their ways of attacks, as written
in [2,6,10]. However, to our knowledge, every study of anomaly detection does
not adequately take account of that in their work. Wagner and Soto [10] focused
on system call traces and discussed that evasion attacks with using system calls,

c Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1291–1302, 2020.
https://doi.org/10.1007/978-3-030-15032-7_108
1292 A. Muramatsu and M. Aritsugi

which are accepted by an intrusion detection system, could avoid to be detected

by the system.
In this paper, we empirically investigate to generate anomaly data, which
could not be detected, with using logs of ordinary processes and demonstrate
how the logs are important. We borrow StreamSpot [7], which is a state-
of-the-art anomaly detection approach, and evaluate our anomaly data on it
experimentally.
The rest of this paper is organized as follows. Section 2 mentions related
work. Section 3 observes the similarity calculation in the detection of StreamSpot
and devises a way of generating sophisticated attacks not to be detected, and
then reports results of experiments where generated attacks were examined. We
conclude this paper in Sect. 4.

2 Related Work
In this section we briefly introduce StreamSpot [7] and mimickry attacks [10].

2.1 StreamSpot

Manzoor et al. [7] proposed a clustering based anomaly detection approach,

StreamSpot, where streams to be monitored for determining if they are anomaly
or not, are expressed in directed graphs. A node in a graph stands for a type,
such as socket, file, and memory, and an edge represents a system call, such as
read and fork. Given a directed graph of nodes and edges, which have types and
change dynamically, the approach generates a set of strings, each of which is
denoted as s(v, k) and is constructed via a k-hop breadth-first traversal starting
from node v. A set of strings in the graphs is denoted as S.
The strings are then used for vectorizing each graph. Let L be the size of
their sketches, hashes hl (l = 1, . . . , L) are considered. A hash hl takes a string
as its argument and returns +1 or −1. A projection vector of graph G, yG , and
its sketch vector, xG , are as follows:

yG (l) = hl (si ) (1)
i=1,...,|S|


+1 (yG (l) ≥ 0)
xG (l) = sign(yG (l)) = (2)
−1 (yG (l) < 0)
Given two graphs G1 and G2 , the similarity between them can be calcu-
lated with the cosine distance of their sketch vectors xG1 and xG2 . StreamSpot
constructs a classifier with data in normal situations. Then, given a data, the
classifier determines if it is anomaly based on the cosine distance between the
data and those in the normal situations.
 How Important Are Logs of Ordinary Operations? 1293

2.2 Mimicry Attacks

Wagner and Soto [10] considered host-based anomaly detection systems and
proposed mimicry attacks which could be realized by six simple ideas, two of
which were “Insert no-ops” and “Generate equivalent attacks.” We borrow these
ideas and, in the following, we attempt to generate sequences of operations for
anomaly processes with adding some operations appearing in ordinary processes.
Particularly, we focus on this attempt in terms of variations of the addition and
examine if such evasion could allow us to make attacks be undetected.

3 Devised Attacks with Ordinary Operations Under

StreamSpot
3.1 Similarity Calculation in StreamSpot
We observe the similarity calculation for anomaly detection in StreamSpot. As
described in the previous section, strings are vectorized into sketch vectors and
then the sketch vectors are used for calculating the similarity for the detection.
Here we consider cases where two strings with different numbers of characters
look similar under the calculation.
Let graph G1 consist of strings s1 × 1, s2 × 2, and s3 × 2, its projection vector
yG1 is
yG1 (l) = hl (s1 ) + 2 × hl (s2 ) + 2 × hl (s3 ) (3)
and thus its sketch vector xG1 is as follows:

xG1 (l) = sign(yG1 (l)). (4)

Similarly, let graph G2 consist of strings s1 × 3, s2 × 6, and s3 × 6, its projection

vector yG2 is as follows:

yG2 (l) = 3 × hl (s1 ) + 6 × hl (s2 ) + 6 × hl (s3 ) (5)

= 3 × yG1 (l) (6)

We thus have xG2 = xG1 . This means that in the anomaly detection systems
the two graphs are classified into the same class. We could therefore say that
given two graphs having strings with similar ratios their sketch vectors tend to
be similar.
We performed an experiment for verifying this conjecture. We used BOS
dataset in MWS2017 Datasets [5] of MWS Datasets [4,8]. BOS dataset was con-
structed by observing behaviors of intruders to Windows machines and consists
of such data as files intruders accessed and/or executed. A synthetic example in
Fig. 1 shows the format of the dataset.
An event includes attributes of process ID, its parent process ID, name of
executed file, name of executed file of its parent process, its command line data,
and command line data of its parent process. We constructed graphs whose
nodes, their IDs, and edges stood for name of executed file, process ID, and
1294 A. Muramatsu and M. Aritsugi

Fig. 1. Log example

Table 1. Process log example

UtcTime ProcessId Image ParentProcessId ParentImage ParentCommandLine

EventData 1 200 B 100 A x
EventData 2 300 C 200 B y

Fig. 2. A graph of Table 1

 How Important Are Logs of Ordinary Operations? 1295

command line data of its parent process, respectively. For example, given data
shown in Table 1, we have the graph shown in Fig. 2. The number of the original
graphs in the dataset was 27.
We vectorized the ratio of strings in each of the graphs as r. The size of r
was set as the number of distinct strings in the graph. Given a graph G, each bit
of vector rG stored the ratio of each string. We calculated the cosine distances
between any two vectors of ratios and between their sketch vectors.

Fig. 3. Cosine distances between r and x

Figure 3 shows the relationship between the two distances. We could say if
two graphs have strings with similar ratios then they could look similar under
a system of sketch vector distances. We investigate to generate anomaly data
based on this conjecture in the following.

3.2 Undetected Anomaly Data Generation

In this paper, we attempt to generate undetected anomaly data by adding oper-
ations appearing in ordinary operation logs to anomaly data. The process steps
are as follows:
1. We find an Event and extract Image, ParentImage, ParentCommandLine
from its EventData with their orders in the ordinary operation logs.
2. We add them at any point of anomaly data with appropriate ProcessId and
ParentProcessId.
3. We iterate step 2 in t times, where t is set in advance.
Note that the above process was developed on treating data of [5] in this study,
and the process could be extended and applied in cyberattack situations without
loss of generality.
1296 A. Muramatsu and M. Aritsugi

For example, given graph G3 which consists of string s4 in 5 times, its pro-
jection vector yG3 is calculated as
yG3 (l) = 5 × hl (s4 ) (7)
and thus G3 does not look as similar to graphs G1 nor G2 . However, let us
consider adding some strings in those graphs to G3 . For example, let us add
s1 × 10, s2 × 20, and s3 × 20 to G3 , then we have
yG3 (l) = 10 × hl (s1 ) + 20 × hl (s2 ) + 20 × hl (s3 ) + 5 × hl (s4 ) (8)
1
= 10 × yG1 (l) + × hl (s4 ) (9)
2
 10 × yG1 (l). (10)

2 ×hl (s4 )
in Eq. (9) becomes +0.5 or −0.5, and thus it has limited influence
1
Here
on the calculation of its sketch vector and we could finally get Eq. (10) meaning
that we could make G3 look similar to graphs G1 and G2 .

3.3 Experiments
We did experiments to evaluate how well our generated data could be undetected
by StreamSpot [7]. We describe the method and report its results.

3.3.1 Method
We used BOS dataset in MWS2017 Datasets [5]. There were 17 graphs for normal
situations and 10 graphs for anomaly ones. In a preliminary experiment on these
original graphs with StreamSpot, 5 out of the 10 were detected as anomaly. Note
that the results are not so good as those shown in [7]. This would be mainly
because of the small amount of data we could use. We apply our data generation
method on the 5 detected anomaly graphs in the following.
The added strings were extracted from the 17 normal graphs, namely, Win-
dows Defender, C# Compiler, wsqmcons.exe, and some batch processes. We
assumed the following four cases:
1. Only one operation, Windows Defender, is known as an ordinary process;
2. Two operations, namely, Windows Defender and C# Compiler, are known as
ordinary processes;
3. All the four operations of ordinary operations are known; and
4. All the four operations of ordinary operations with their ratios of occurrences
are known.
We did a 4-fold cross validation and in the following we report accuracy, AUC
(area under the ROC curve), and AP (area under the precision-recall curve).
They were calculated every addition of 100 edges in test data.
There are several parameters in StreamSpot [7]. We set the sketch size L =
1000, the hop count k = 1, the chunk length C = 50, the maximum number of
stored edges was bigger than the total number of edges in test data, and the
number of tested graphs at a time B was the number of those in test data.
 How Important Are Logs of Ordinary Operations? 1297

3.3.2 Results
We report results of one of the 4-fold cross validation because tendencies shown
in the rest three were the same to the results and due to the page limitation.
Figure 4 shows results with the original dataset. Figures 5, 6, 7, and 8 show
results in case 1 with 40 Windows Defenders, case 2 with 20 Windows Defenders
and 20 C# Compilers, case 3 with 10 each of the four operations, and case 4
with 11 Windows Defenders, 6 C# Compilers, 12 wsqmcons.exe, and 11 batch
processes, respectively.
Figures 4 and 5 show similar results, but Figs. 6 and 7 show that all metrics
of accuracy, AUC and AP decreased as the amount of addition increased. On
the other hand, Figs. 7 and 8 look similar, but in precisely AUC and AP shown

Fig. 4. Results with the original dataset

Fig. 5. Results in case 1 with 40 Windows Defenders

1298 A. Muramatsu and M. Aritsugi

Fig. 6. Results in case 2 with 20 Windows Defenders and 20 C# Compilers

Fig. 7. Results in case 3 with 10 each of the four operations

in Fig. 8 were slightly worse than those in Fig. 7. This indicates that added
operations with normal ratios may help us to make data be more undetected.
Figures 9, 10, and 11 show results with varying the total numbers of added
operations as 200, 400, and 800, respectively, while keeping their ratios.
In Figs. 9, 10, and 11, we can see that accuracy, AUC and AP decreased as
the number increased. In fact, 3 out of the 5 were determined as normal in the
case of Fig. 9, while 4 and 5 in the cases of Figs. 10 and 11 were determined as
normal, respectively.
 How Important Are Logs of Ordinary Operations? 1299

Fig. 8. Results in case 4 with 11 Windows Defenders, 6 C# Compilers, 12 wsqm-

cons.exe, and 11 batch processes

Fig. 9. Results in case 4 with 200 in total operations with the same ratio of Fig. 8

3.3.3 Discussions
According to the results, we can say that it is not so hard to cheat the
anomaly detection if we could have some knowledge about ordinary operations.
In StreamSpot, a sketching technique is exploited in the detection in order to
make it possible to calculate distances with small amount of memory in real
time. The results also indicate that the more the leakage of ordinary operations
the worse the organization’s security.
Note that we could not say our undetected anomaly data generation can
work well in every anomaly detection systems. We, however, can say that the
1300 A. Muramatsu and M. Aritsugi

Fig. 10. Results in case 4 with 400 in total operations with the same ratio of Fig. 8

Fig. 11. Results in case 4 with 800 in total operations with the same ratio of Fig. 8

probability to cheat anomaly detection must increase as the leakage of ordinary

operations increases.
It should be also noted that such large number of additions as 800 can be
reasonable. A way to detect such evasion is to take account of processing time of
each operation in constructing graphs. However, processing time is influenced by
many factors and the way must be hard to perform in real time, and thus may
lead us to have a large number of detection faults. Thus what a cyberattacker
needs to do is to insert ordinary operations in which almost nothing is processed
actually.
 How Important Are Logs of Ordinary Operations? 1301

4 Conclusion
In this paper, we focused on a state-of-the-art anomaly detection approach
and investigated to devise attacks with ordinary operations in order not to be
detected by the approach. We found that data consisting of processes in simi-
lar ratios tend to be treated as similar in the approach, and considered how to
cheat the detection system by adding ordinary operations into anomaly data.
We empirically evaluated our proposal with using a real dataset constructed by
observing behaviors of intruders to Windows machines. In the experiments, we
varied the level of attackers’ knowledge about ordinary operations to be added.
Although our idea was simple, the experimental results indicated that our idea
could work in real situations and that the more the leakage of ordinary operations
the worse the detection performance.
In the future, we intend to examine various anomaly detection systems and
examine if our findings are true for the systems. Also, we would like to investigate
how worse as the leakage of ordinary operations increases.

References
1. Alperovitch, D.: Revealed: operation shady RAT. Technical report, McAfee
(2011). https://web.archive.org/web/20110804083836/ http://www.mcafee.com/
us/resources/white-papers/wp-operation-shady-rat.pdf
2. Dang, H., Huang, Y., Chang, E.C.: Evading classifiers by morphing in the dark.
In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Commu-
nications Security, CCS 2017, pp. 119–133. ACM, New York (2017). https://doi.
org/10.1145/3133956.3133978
3. Dong, B., Chen, Z., Wang, H.W., Tang, L.A., Zhang, K., Lin, Y., Li, Z., Chen, H.:
Efficient discovery of abnormal event sequences in enterprise security systems. In:
Proceedings of the 2017 ACM Conference on Information and Knowledge Man-
agement, CIKM 2017, pp. 707–715. ACM, New York (2017). https://doi.org/10.
1145/3132847.3132854
4. Hatada, M., Akiyama, M., Matsuki, T., Kasama, T.: Empowering anti-malware
research in Japan by sharing the MWS datasets. J. Inf. Process. 23(5), 579–588
(2015). https://doi.org/10.2197/ipsjjip.23.579
5. IPSJ and MWS Organizing Committee: Anti malware engineering workshop
(MWS) datasets 2017 (2017). http://www.iwsec.org/mws/
6. Liang, B., Su, M., You, W., Shi, W., Yang, G.: Cracking classifiers for evasion:
a case study on the Google’s phishing pages filter. In: Proceedings of the 25th
International Conference on World Wide Web, WWW 2016, pp. 345–356. Interna-
tional World Wide Web Conferences Steering Committee, Republic and Canton of
Geneva, Switzerland (2016). https://doi.org/10.1145/2872427.2883060
7. Manzoor, E., Milajerdi, S.M., Akoglu, L.: Fast memory-efficient anomaly detection
in streaming heterogeneous graphs. In: Proceedings of the 22nd ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining, KDD 2016,
pp. 1035–1044. ACM, New York (2016). https://doi.org/10.1145/2939672.2939783
8. Takata, Y., Terada, M., Matsuki, T., Kasama, T., Araki, S., Hatada, M.: Datasets
for anti-malware research-MWS datasets 2018-(in Japanese). SIG Technical reports
2018-CSEC-82 38, IPSJ (2018). http://id.nii.ac.jp/1001/00190441/
1302 A. Muramatsu and M. Aritsugi

9. Varma, R.: McAfee Labs: combating aurora. Technical report, McAfee Labs
(2010). https://paper.seebug.org/papers/APT/APT CyberCriminal Campagin/
2010/Combating%20Threats%20-%20Operation%20Aurora.pdf
10. Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems.
In: Proceedings of the 9th ACM Conference on Computer and Communications
Security, CCS 2002, pp. 255–264. ACM, New York (2002). https://doi.org/10.1145/
586110.586145
11. Zoppi, T., Ceccarelli, A., Bondavalli, A.: Exploring anomaly detection in systems
of systems. In: Proceedings of the Symposium on Applied Computing, SAC 2017,
pp. 1139–1146. ACM, New York (2017). https://doi.org/10.1145/3019612.3019765
 Access Control Based Dynamic Path
Establishment for Securing Flows
from the User Devices with Different
Security Clearance

Uday Tupakula(B) , Vijay Varadharajan, and Kallol Karmakar

Advanced Cyber Security Engineering Research Centre,

The University of Newcastle, Newcastle 2308, Australia
{uday.tupakula,vijay.varadharajan,
kallolkrishna.karmakar}@newcastle.edu.au

Abstract. In this work, we proposed Software Defined Networking

(SDN) based access control techniques for preventing unauthorised access
to traffic flows in secure networks. We have developed an Access Con-
trol Application (ACA) for the SDN Controller for differentiating the
flow requests from the user/devices that are classified at different secu-
rity levels and configuring the routes with physical or virtual separation
between the flows. This separation of flows makes it difficult for the mali-
cious users with lower security clearance to access the flows that belong
to the users with higher security clearance. Hence, our work significantly
minimises the attack surface in secure environments. We will also dis-
cuss the prototype implementation of our model and some performance
characteristics.

Keywords: Software defined networks · Services security ·

Access control · Flow security

1 Introduction
Secure environments such as defence have users with clearance at various secu-
rity levels such as Unclassified, Classified, Secret and Top Secret. The users
have different level of access to the information and/or resources depending on
their security clearance. However, such varying level of access is only provisioned
on the servers. Often, the users access the information through the network and
there is no separation of the flows from the users with different security clearance.
Hence, users that are classified at lower privileges can still have unauthorised
access to the information with higher security clearance by performing network
attacks and accessing the flows that are related to the user devices with higher
security clearance. In this paper we propose SDN based techniques for securing
the flows between end points for users with different security clearance. Our
model makes use of Access Control Application for maintaining the information
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1303–1315, 2020.
https://doi.org/10.1007/978-3-030-15032-7_109
1304 U. Tupakula et al.

related to the users and global network information available at the SDN Con-
troller to dynamically determine the available routes and establishing physically
or virtually separate paths for the flows originating from the user devices with
different security clearance.
The paper is organised as follows: Sect. 2 considers a simple network sce-
nario and presents the attacker model. In Sect. 3 we propose an Access Control
Application for SDN Controllers which can be used for dynamically establishing
separate communication path for the flows from the user devices with different
security clearance. In Sect. 4 we present the implementation and performance
analysis of our model. In Sect. 5, we consider some of the related work and
Sect. 6 makes some concluding remarks.

2 Attacker Model
In this section, we consider a simple network scenario and present the attacker
model that will be addressed with our model.
We consider a secure environment such as defence networks with hetero-
geneous devices such as client machines, server machines and wireless mobile
devices. The client and server machines are implemented on physical hosts or
as virtual machines (VMs) that are running on a hypervisor or virtual machine
monitor (VMM) [3] based system. The users have clearance at different security
levels such as Unclassified (UC), Classified (C), Secret (S) and Top Secret (TS).
Similarly, the servers that are hosting security services has information (such as
files) that are categorised at different security levels and can only be accessed
by the users that are cleared at the required security level.

Fig. 1. Attacker model

In the current networks, the access control is only performed at the server
end where the users access the resources. For example, if the user cleared at
Secret level is trying to access the files that belong to Top Secret level, then the
access control module in the security server prevents such unauthorised access
to the resources. However, all the user flows with varying security clearance
are sharing the same network. As shown in the Fig. 1, when the users try to
access the resources at the security server, the routes are established based on
different parameters such as shortest path, number of parallel paths, load on the
path but not on the security clearance of the user devices. Even in the case of
SDN network, although the routes are established by the SDN controller, the
 Access Control Based Dynamic Path Establishment 1305

route establishment is based on these parameters. None of the SDN Controllers

makes use of the security clearance level of the user as a metric to establish
secure paths between the end points. Hence, the malicious users can make use
of network based attacks to have unauthorised access to the flows that belong to
users with higher security clearance. It is not an easy task to detect or prevent
such attacks in the current networks. Furthermore, the users can be accessing
the information from the fixed devices such as desktops or using their mobile
devices. Usage of mobile devices for accessing the secure resources makes it more
challenging to deal with such attacks. Hence, there is need for end to end security
techniques to deal with such attacks. In this paper, we propose techniques for
securing communication between end points using SDN. The main contributions
of this work are as follows:
• Access control based dynamic path establishment between the end
points: This is important since the users can be accessing the security ser-
vices from their desktop machines or mobile devices. The desktops have a
fixed location and relocation of the desktop machines can be considered as
a rare event. Hence if the users are accessing the security services using the
desktop machines, then it is reasonable to establish static separate routes
from the location of the desktop to the security servers depending on the
security clearance of the user. Currently, there is increasing trend of using
the mobile devices for accessing the services using Wi-Fi networks since they
provide the flexibility for the users to access the services from any location
in the organisation. The Access Points that are located with the range of
the users device enable the user to access the services in the network. Hence
there is need to establish dynamic secure path from the Access Point (that is
connected to users mobile device) to the servers with security services.
• Provisioning physical or virtual separation between the flows:
Encryption can be used to provide some level of security when multiple flows
are making use of the same network for accessing the security services. How-
ever the current attacks are very sophisticated. If the attackers have access
to the flows, they are able to extract the fine granular information [20] from
the flows even if the traffic is encrypted [11]. Furthermore, attacks such as
heart-bleed [18] can expose the secure key used for encrypting the messages.
Hence preventing access to the flows is a fundamental security requirement
for protecting the flows with different security clearance.
• Prototype implementation of the model: We demonstrate the effec-
tiveness of our model with the implementation of the ACA for ONOS SDN
Controller and mininet simulations.

3 Our Approach
In this section, we make some assumptions and propose our model to isolate the
flows from the user devices with different security clearance.
We assume that there is one logical SDN Controller per domain. We also
assume that the Controller and the virtual and physical switches in the data
1306 U. Tupakula et al.

plane are trusted. For example, TPM attestation [21] can be used for validating
the state of the physical and virtual switches.
We assume that TLS/SSL is used for securing the communication between
the Controller and the switches. Currently, TLS/SSL is supported on all the
SDN switches.
We assume that the servers hosting the security services are trusted and
secure. If these services are vulnerable, then the attackers can have unrestricted
access to all the information stored in the servers. Host based security tools can
be used for securing the security services from the attackers.
We use flow to represent the communication between the end hosts. Flow is
defined as sequence of packets, and each packet is identified by Source IP, Source
Port, Destination IP, Destination Port and IP protocol. When the hosts initiate
a packet flow, the switches receive the traffic and forward it to the Controller.
The Controller makes the decision on how to deal with the new flow based on the
policies specified in the Controller and it establishes the route for communication
between the specific hosts.
We have developed an ACA that can be hosted on the NBI of the SDN Con-
troller. The ACA is used for maintaining the details of the users. This includes
information such as security clearance level of the users and all the devices used
by the users. For example, the users register their mobile devices with the secu-
rity administrator prior to connecting them in the secured network. The ACA
also makes use of the global network information available in the SDN Con-
troller to analyse the network topology and establishing secure paths for the
flows originating from the users with different clearance. When the users make
use of their device for accessing some information on the security server, this
will initiate new flow from the user’s device to the security server. The switch
that is connected to the users device will receive the flow request and generate
a Packet in message to the controller which will be subsequently forwarded to
the ACA.
The work flow for the ACA is shown in Fig. 2. The ACA analyses the
Packet in message to determine the user security level and extract the network
topology information from the SDN Controller. Now ACA uses the security level
of the user as primary metric and other parameters such as shortest path, num-
ber of parallel path for determining the routes and establish a secure path to the
server. If there are multiple paths to the destination, then the ACA chooses a
physically separate path for the users with different clearance. However, if there
are limited paths to the destination, then the ACA distributes the flows among
the available paths and makes use of virtual lans in the switches to separate
flows from the users with different security clearance. As shown in the Fig. 3, the
ACA makes use of virtual lans in the OFS1 and OFS4 for separating the flows
with different security clearance. Since there are two parallel paths between the
switches OFS1 and OFS4, the ACA chooses OFS2 to route the flows from Top
Secret and Secret level clearance users to access the information on the security
server. Furthermore, virtual lans are used to separate the flows from TS and S
users. Similarly ACA chooses OFS3 to route the flows from the users devices that
 Access Control Based Dynamic Path Establishment 1307

Get Packet_IN

ACA: Extract User

InformaƟon & Security
Level

No Match ACA: Search Policy Match

Mapping for specific
User

ACA: Use topology repo,

Normal Route for routes

ACA: Allocate separate

ACA: End operaƟon VLANs to Users based on
the security level

ACA: End operaƟon

Fig. 2. ACA work flow

Fig. 3. Secure path establishment between end points

are cleared at classified and Unclassified level with virtual separation between
the flows for classified and unclassified users. Hence our model is able to estab-
lish secure paths with physical or virtual separation between the flows based on
availability of the network infrastructure.

4 Implementation

In this Section, we present the implementation and performance analysis of our

model.
We have developed an ACA for ONOS SDN Controller [1] and validated our
model using Mininet simulations. ONOS is an open-source SDN controller that
1308 U. Tupakula et al.

Fig. 4. Mininet network view

is designed for performance, high availability, scale-out and well-defined north-

bound and southbound abstractions and interfaces. It is a multi-module project
written in Java and uses OSGi for functionality management. The individual
features are loaded using the OSGi runtime called Karaf. ONOS was designed
to meet the needs of operators wishing to build carrier-grade solutions by offering
the flexibility to create and deploy new dynamic network services with simpli-
fied programmatic interfaces. Several modules in ONOS such as switch man-
ager, module management, link discovery and REST APIs are adapted from the
Floodlight Controller. The data model in ONOS is implemented using Titan
graph database, Cassandra key value store and the Blueprints graph API to
expose network state to applications. Our ACA application uses ONOS appli-
cation development interface. In particular we made use of ONOS application
features such as device, route selection, update, maintaining sessions for creating
& updating our topology repository. We have created specific mininet scripts to
create the topology. The role based policies are stored in a JSON repository.
For user and device details, we have created a dummy user to device relational
database. To update, modify and read the stored JSON policies, we have used
JAVA parser module. We have made additional modifications to mininet script
to enforce the flow isolation based on Controller response.

4.1 Network Setup

We have validated our model using mininet [13,19] simulations for varying num-
ber of hosts and switches. For easy presentation, we present the operation of
model using simple network topology as shown in Fig. 3. In this scenario, we
have created four individual Linux virtual machines with TS, S, C and UC user.
Mininet hosts connect to these virtual machines. The IP mapping for users is:
(10.0.0.7 → T S, 10.0.0.5 → S, 10.0.0.9 → C and 10.0.0.6 → U C). The Mininet
host 10.0.0.8 connects to a real physical security server. We have programmed the
S1–S4 OpenFlow switches for various logical VLAN paths (as shown in Fig. 4).
 Access Control Based Dynamic Path Establishment 1309

Our ACA creates and assigns flows from various host to these VLANs. Since
they are dependant on our flow policies described in the ACA, we refer to these
VLAN paths as logical or programmable.

4.2 Policy Syntax

Figure 5 shows the ACA repository schema. Here, each User contains multiple
devices. Each device is associated with different Policy Expression with actions.
The action consists of specific VLANs and their security level. We will describe
a sample policy expression in this section.

USER DEVICE POLICY_EXPRESSION

PK UserID PK DeviceID PK PEID

UserName DeviceName hostIP

UserStatus DeviceType hostMAC

DeviceMac desnaonIP

desnaonMAC

Acon flowID

PK VlanID AconID

SecurityLVL

Fig. 5. ACA repository schema

Our ACA uses JSON -based policies to transfer the user flows through differ-
ent VLANs. The policies can be applied based on flows, users, devices or services.
In this subsection, we will describe a sample policy. In the following scenario, we
will use this policy to defend a MiTM (Man-in-The-Middle) attack.
Listing 1.1. A Sample Policy
{
” id ” : ”02” ,
” hostip ” : ”10.0.0.7” ,
” hostmac ” : ” 0 0 : 0 9 : 0 0 :AA” ,
” destip ” : ”10.0.0.8” ,
” dstmac ” : ” 0 0 : 0 9 : 0 0 :BB” ,
” f l o w i d ” : ”78 b34x ” ,
” actions ” : [
” securitylvl ” : ” top secret ” ,
” v l a n i d ” : ”VLAN500” ,
],
},
}
1310 U. Tupakula et al.

4.3 Scenario
In this section, we present a scenario which is more relevant to an insider attack.
Here, an adversary being a company employee intercepts the traffic from other
high ranked employees. We have used an SDN network to simulate the scenario.
First, we show how an adversary is able to intercept flows that belong to the
user devices with higher security clearance. Then we show how our ACA creates
VLANs and secures the user flows with higher security clearance.

Fig. 6. MiTM attack (a) Successful approach and (b) Our approach

Figure 3 shows the network setup. In this case the user that is cleared at TS
level is using his device with 10.0.0.7 IP address to access the services on the
server with IP 10.0.0.8. Figure 6a shows the case of successful attack where the
adversary who is cleared at UC level uses Kali Linux with IP 10.0.0.6 to launch
MiTM attack and access the flows that belong to the user with TS clearance.
Now, we activate our ACA. Our application enforces host specific flow policies
associated with the VLAN. We introduce a policy shown in listing 1.1. According
to the policy, if a user is accessing the server 10.0.0.8 from IP 10.0.0.7, his flows
will be isolated through V LAN 500 in all the OpenFlow switches in the SDN
network. Our application continuously checks the flow of the communicating
hosts, and it does not allow the adversary to launch MiTM attacks. Figure 6b
shows the alerts when the ACA detects the attacks from the adversary.

4.4 Results

In this section, we present the performances of the SDN network, while running
our ACA application and without running it.
First, we measure the path setup time for the SDN network. The overall
time taken by an SDN controller to set up an end to end path connection is
referred to as path setup time. We varied the number of OpenFlow switches
from 50–200 to measure the difference in path setup time between the end hosts.
The hosts are connected to head and tail switches in the topology. In Fig. 7a, we
present a comparison of path setup time while running our ACA application and
without running ACA application. Figure 7a shows that with increasing number
 Access Control Based Dynamic Path Establishment 1311

Path Setup Time

With ACA
1000
Without ACA
821.08
Path Setup Time (ms)

800 790.86

600 606.89 580.78

400
355.886 340.66

200 178.04 170.8

0
50 OFS 100 OFS 150 OFS 200 OFS
Number of Switches
(a)

Flow Setup Latency

Packet_in Rate (Packets/s)

2000 55.36
58.88

1500 40.19
42.08

1000 25.6
27.56

500 12.1
13.6
Without ACA With ACA
0 10 20 30 40 50 60
Latency (ms)
(b)

Fig. 7. (a) Path setup time and (b) Flow setup latency

of OpenFlow switches in the SDN network the path setup time increases linearly
in both the cases. However the path setup time with ACA is more compared to
the path setup time without ACA. With 50 OpenFlow switches, path setup
time without ACA is 170.8 ms and it increases to 790.86 ms with 200 OpenFlow
switches. With ACA application running on the Controller increases the path
setup time by 3–5%. In this case, with 50 OpenFlow switches, the path setup
time is 178.04 ms and it increases to 821.08 ms with 200 OpenFlow switches.
1312 U. Tupakula et al.

The amount of delay the Controller incur to send the flow entry to the Open-
Flow switches is termed as flow setup latency. We have generated a varying
number of packet in request(from 500 to 2000 packets/s) and measured the time
taken by controller to send flow mod messages to the switches. We conducted
the same test cases while running ACA application and without running it.
Figure 7b shows flow setup latency comparison for both the cases. Without
running the ACA application on the SDN controller, flow setup latency increases
steadily from 12.1 ms to 55.36 ms with the increase in packet in rate from 500
to 2000 packets/s respectively. The Flow setup latency rate increases by 2–4%,
with the activation of ACA application. For 500 packets/s flow setup latency is
13.6 ms and with 2000 packets/s it increases to 58.88 ms.
ACA application checks the all the policy rules installed in the repository.
Hence, our ACA application incurs a delay in the flow processing time. Both the
cases explained above reflects this time overhead.

5 Related Work

Although there are significant benefits with SDN, security in SDN is still in
infancy. There is ongoing work [7,10,12,14,17] for analysing the security issues
with the SDN technology. In this section, we briefly discuss important related
works that are relevant to our work.
An access control is a method of enforcing rules on the usage of resources
by authorised entities. A trusted component known as access control module or
reference monitor is used to mediate the all the interactions between the sub-
jects and objects according to the policies specified by the administrator. Access
Matrix is one of the well known access control models where subjects and objects
are represented as rows and columns of a matrix and each cell species the access
rights granted to the corresponding subject on the corresponding object. How-
ever, this approach can become very complex with increasing number of subjects
and objects. Hence several access control models such as Discretionary Access
Control (DAC), Mandatory Access Control (MAC), Role-Based Access Control
(RBAC) and Attribute Based Access Control (ABAC) have been proposed and
developed in the past. Here, we provide a basic description of each access control
mechanism.
Typically, discretionary access control (DAC) models leave the specification
of access control policies to individual users, where the owner of the object
specifies which subjects can have which access rights to the object. A mechanism
implementing a DAC policy must be able to verify that a given subject has the
right to access the object. DAC policies are typically implemented using Access
Control Lists (ACLs).
In mandatory access control (MAC) models the standard approach is to
have the access defined by a system administrator, and employ attributes such
as classifications and clearances. Typically, The MAC model is based on security
labels with various security levels. Subjects have a security clearance attribute,
and data objects a security classification or sensitivity level. The clearance and
 Access Control Based Dynamic Path Establishment 1313

classification data are stored in the security labels, which are assigned to the
specific subjects and objects. Typical multi-level security policies include Bell
LaPadula secrecy model [4] and Biba integrity models. Other variants of access
control policy models that using security labels include Type enforcement [2] and
Chinese Wall [6]. Security Enhanced Linux (SELinux) [15] can be considered as
a good example of the MAC-based commercial system, where type enforcement
is implemented based on the labels of the subjects and objects.
In RBAC models [8,16], the attributes used in access control are the roles
associated with the principals and the permissions associated with the roles.
Typically roles are intended to reflect the real world job functions within an
organisation. The permissions that are attached to the roles reflect the actual
operations that may be carried out by members of the role. The policies that
need to be expressed in an RBAC system can therefore be seen to have the
potential to be both extensive and intricate. Another important characteristic
of RBAC is that of role inheritance. Role hierarchies allow roles to be inherited
from other roles. Then there is the ability to specify role constraints such as
constraints on users taking specific roles (for example, users may not be allowed
to take on one role if the user has another specific role).
With the increasing migration of the services to the cloud, there have been
increasing security concerns for secure storage of data in the cloud. Hence, there
is considerable research interest for developing data access control schemes to
protect the stored data in the cloud computing. In an ABE system, the access
permission to a resource is associated with a set of attributes and a security policy
based on these attributes. Only the users who hold the keys for the attributes
are able to decrypt the content. This feature allows ABE schemes to be used in
protecting the privacy of resources in a cloud storage system which uses ABAC
model to control access privileges. The first ABE scheme was proposed in [9],
in which cipher-texts are labelled with sets of attributes and private keys are
associated with access structures that control which cipher-texts a user is able
to decrypt. Hence, this scheme is also referred to as key-policy ABE or KP-ABE.
In [5], another form of ABE scheme was introduced. This scheme works in the
reverse manner where the user keys are associated with sets of attributes, and
the cipher-texts are associated with the policies. Hence, it is referred to as the
cipher-text-policy ABE or CP-ABE scheme. Zhou et al. in [22] proposed a role-
based encryption (RBE) scheme that integrates the cryptographic techniques
with RBAC. This scheme allows RBAC policies to be enforced for the encrypted
data stored in public clouds. They also presented the implementation of a RBE-
based hybrid cloud storage architecture that allows an organization to store data
securely in a public cloud, while maintaining the sensitive information related
to the organization’s structure in a private cloud.
As already mentioned, these access control models mainly enforce security
policies at the server end. The attacker can obtain sensitive information by
having unauthorised access to the traffic flows related to the users with higher
privileges. Hence there is need for access control techniques at the server end and
1314 U. Tupakula et al.

also at the network level. Our model provides a physical or virtual separation
between the flows with different security level.

6 Conclusion

In this paper we proposed techniques for establishing secure path for communi-
cation between the end nodes based on the security clearance of the users. Our
model makes use of ACA that can be hosted in the NBI of the SDN Controller
to secure the flows from the user devices with different security clearance. The
ACA maintains the details of all the user devices and make use of global network
knowledge available at the Controller to differentiate the flows originating from
the user devices with different security clearance and establish a physical or vir-
tual separate path to the flows. We have implemented our model with ONOS
SDN Controller and presented some performance results.

References
1. Berde, P., et al.: ONOS: towards an open, distributed SDN OS. In: Proceedings of
the 3rd Workshop on Hot Topics in Software Defined Networking, pp. 1–6. ACM
(2014)
2. Badger, L., Sterne, D.F., Sherman, D.L., Walker, K.M., Haghighat, S.A.: Prac-
tical domain and type enforcement for UNIX. In: 1995 Proceedings of the IEEE
Symposium on Security and Privacy, pp. 66–77 (1995)
3. Barham, P., et al.: Xen and the art of virtualization. In: Proceedings of the Nine-
teenth ACM Symposium on Operating Systems Principles, SOSP 2003, pp. 164–
177. ACM Press, New York (2003)
4. Bell, D.E., LaPadula, L.J.: Secure computer system: unified exposition and multics
interpretation (1976)
5. Bethencourt, J., Sahai, A., Waters, B.: Ciphertext-policy attribute-based encryp-
tion. In: 2007 IEEE Symposium on Security and Privacy, SP 2007, pp. 321–334.
IEEE (2007)
6. Brewer, D.F., Nash, M.J.: The Chinese Wall security policy. In: IEEE Symposium
on Security and Privacy, pp. 206–214 (1989)
7. Dacier, M.C., et al.: Security challenges and opportunities of software-defined net-
working. IEEE Secur. Priv. 15(2), 96–100 (2017). https://doi.org/10.1109/MSP.
2017.46
8. Ferraiolo, D., Kuhn, R.: Role-based access control. In: 15th National Computer
Security Conference, pp. 554–563. US National Institute of Standards and Tech-
nology (1992)
9. Goyal, V., Pandey, O., Sahai, A., Waters, B.: Attribute-based encryption for fine-
grained access control of encrypted data. In: Proceedings of the 13th ACM Con-
ference on Computer and Communications Security, pp. 89–98. ACM (2006)
10. Hong, S., et al.: Poisoning network visibility in software-defined networks: new
attacks and countermeasures. In: NDSS (2015)
11. Jansen, R., Juarez, M., Gálvez, R., Elahi, T., Diaz, C.: Inside Job: applying traffic
analysis to measure tor from within. In: Network and Distributed System Security
Symposium. IEEE Internet Society (2018)
 Access Control Based Dynamic Path Establishment 1315

12. Kreutz, D., Ramos, F., Verissimo, P.: Towards secure and dependable software-
defined networks. In: Proceedings of the 2nd ACM SIGCOMM Workshop on Hot
Topics in Software Defined Networking, pp. 55–60. ACM (2013)
13. Lantz, B., Heller, B., McKeown, N.: A network in a laptop: rapid prototyping for
software-defined networks. In: Proceedings of the 9th ACM SIGCOMM Workshop
on Hot Topics in Networks, p. 19. ACM (2010)
14. Lee, S., et al.: DELTA: a security assessment framework for software-defined net-
works. In: Proceedings of NDSS, vol. 17 (2017)
15. NSA: SELinux Related Work (2012). http://www.nsa.gov/research/selinux/
related.shtml. Accessed July 2012
16. Sandhu, R.S., Coyne, E.J., Feinstein, H.L., Youman, C.E.: Role-based access con-
trol: a multi-dimensional view. In: 1994 Proceedings of the 10th Annual Computer
Security Applications Conference, pp. 54–62 (1994)
17. Schehlmann, L., Abt, S., Baier, H.: Blessing or curse? Revisiting security aspects
of software-defined networking. In: 10th International Conference on Network and
Service Management, pp. 382–387. IEEE (2014)
18. Schneier, B.: Heartbleed. Schneier On Security, Blog (2014)
19. Mininet Team: Mininet: an instant virtual network on your laptop (or other PC).
Google scholar (2012)
20. Tomšŭ, R., Marchal, S., Asokan, N.: Profiling users by modeling web transactions.
In: 2017 IEEE 37th International Conference on Distributed Computing Systems
(ICDCS), pp. 2399–2404. IEEE (2017)
21. Trusted Computing Group: Incorporated: TPM main, part 1 design principles.
Technical report, revision 116, Trusted Computing Group, Inc. (2011)
22. Zhou, L., Varadharajan, V., Hitchens, M.: Achieving secure role-based access con-
trol on encrypted data in cloud storage. IEEE Trans. Inf. Forensics Secur. 8(12),
1947–1960 (2013)
 Interactive Aggregate Message Authentication
Scheme with Detecting Functionality

Shingo Sato1(B) and Junji Shikata2

1
Graduate School of Environment and Information Sciences,
Yokohama National University, Yokohama, Japan
[email protected]
2 Graduate School of Environment and Information Sciences,

Institute of Advanced Sciences, Yokohama National University, Yokohama, Japan

[email protected]

Abstract. In this paper, we newly propose a formal model and a construction

methodology of interactive aggregate MAC schemes with detecting functionality
(IAMD). The IAMD is an interactive aggregate MAC protocol which can iden-
tify invalid messages with a small amount of tag-size. Several aggregate MAC
schemes that can specify invalid messages have been proposed so far by using
non-adaptive group testing in the prior work. Instead, we utilize adaptive group
testing to construct IAMD scheme and the resulting IAMD scheme can iden-
tify invalid messages with a small amount of tag-size compared to the previ-
ous schemes. In this paper, we propose a generic construction of IAMD starting
from any adaptive group testing protocol and any aggregate MAC scheme, and
we apply several concrete constructions of adaptive group testing protocols and
aggregate MAC schemes. In addition, we compare and analyze those IAMD con-
structions in terms of efficiency and security.

1 Introduction

The message authentication code (MAC) is one of the most fundamental cryptographic
primitives. Furthermore, Katz and Lindell [7] proposed the aggregate MAC that can
compress multiple MAC tags on multiple messages generated by different signers into
a single aggregate tag. The advantage of the aggregate MAC lies in that the size of an
aggregate tag is much smaller than total sizes of MAC tags, and hence it will be useful
in applications in mobile networks or IoT (Internet of Things) networks where many
devices sending messages are connected. The model and security of aggregate MACs
were introduced by Katz and Lindell [7], and they proposed the simple construction
satisfying the security by using exclusive-or of MAC tags.
Group testing (e.g., see the book [2] for the survey) is a method to specify some
special items called defectives among many whole items with a small number of tests
than the trivial individual testing for each item. Suppose that there are totally n items of
which there are d defectives in the following. The first paper about group testing is pub-
lished by Dorfman [1]. The group testing techniques are classified into two types: the
first type means the testing techniques by non-adaptive strategies, called non-adaptive
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1316–1328, 2020.
https://doi.org/10.1007/978-3-030-15032-7_110
 Interactive Aggregate Message Authentication Scheme 1317

group testing [3, 10, 11], and the second type means the techniques by adaptive strate-
gies, called adaptive group testing (or called sequential group testing) [1, 3, 6, 8]. In
non-adaptive group testing, we have to know d beforehand and we have to select all the
subsets of n items to be tested without knowing the results of other tests. Finally, we
try to specify all the d defectives from the results of the tests1 . On the other hand, in
adaptive group testing, we can do tests several times such that we can select a subset of
items to be tested after observing the result of the previous test. In particular, a compet-
itive group testing (e.g., see [2] for the survey) is an adaptive group testing which does
not need to know d (i.e., the number of defectives) beforehand, and this type of testing
is useful in a real application when it is not easy to estimate d beforehand.
It is interesting to consider applications of group testing techniques in cryptogra-
phy, in particular for lightweight authentication to detect invalid messages to make the
existing networks using MACs more efficient than the present state of affairs. Actually,
aggregate MACs can be used in the applications to make the networks using MACs
more efficient. However, it is impossible to identify invalid messages once the mul-
tiple messages are regarded as invalid using an aggregate-tag in aggregate MACs in
general. Therefore, several aggregate MAC schemes that can specify invalid messages
have been proposed by using group testing in the prior work [4, 5, 9]. It should be noted
that all these schemes are proposed based on non-adaptive group testing, in particular by
the use of d-disjunct matrices (or d-separable matrices). However, no aggregate MAC
scheme based on adaptive group testing to detect invalid messages has been proposed
so far. Since adaptive group testing is expected to be more efficient than non-adaptive
group testing in terms of the number of tests in general, we consider the application of
adaptive group testing in aggregate MACs so that the resulting aggregate MAC scheme
can identify invalid messages with a small amount of tag-size than those based on non-
adaptive group testing in this paper.
The contribution of this paper is as follows.
• We provide a formal model of adaptive group testing (AGT) protocols and define
the notion of detection-completeness in Sect. 3. This model and the notion is impor-
tant, since we want to show that any AGT protocol can be combined with any aggre-
gate MAC scheme to construct an interactive aggregate MAC scheme with detecting
functionality.
• We propose a formal model of interactive aggregate MAC schemes with detecting
functionality (IAMD), and we formalize a security definition of IAMD in Sect. 4.
Specifically, we define security definitions of unforgeability and identifiability for
IAMD. In addition, the notion of identifiability consists of completeness and sound-
ness: Completeness is the security against adversaries who try to make IAMD clas-
sify valid message/tag pairs as invalid ones; Soundness is the security against adver-
saries who try to make IAMD classify invalid message/tag pairs as valid ones. We
also define a notion of weak-soundness which is a relaxed version of soundness.
Furthermore, we propose a generic construction of IAMD starting from any adaptive
group testing protocol and an aggregate MAC scheme in a simple way in Sect. 4, and
we prove that the resulting IAMD meets completeness and weak-soundness if the
1 This type of testing is typically designed by providing a d-disjunct matrix or a d-separable
matrix (e.g., see [2] for the survey of non-adaptive group testing).
1318 S. Sato and J. Shikata

underlying AGT meets detection-completeness and aggregate MAC meets unforge-

ability.
• We provide concrete constructions of IAMD by specifying an AGT protocol and an
aggregate MAC scheme in Sect. 5. Specifically, we consider XOR-based aggregate
MAC scheme (i.e., Katz-Lindell construction [7]) and Hash-based aggregate MAC
scheme as aggregate MAC schemes. On the other hand, as AGT protocols, we con-
sider the binary search, the rake-and-winnow algorithm [3], Li’s multi-stage algo-
rithm [8], and the digging algorithm (see Sect. 4.6 of [2]). In particular, the binary
search and the digging algorithm are competitive group testing, and the resulting
IAMD can detect invalid messages without knowing the number of those ones (i.e.,
the number d of defectives). We summarize comparison of those resulting IAMD
constructions in Table 2 in terms of security and efficiency in Sect. 5.

2 Preliminaries
In this paper, we use the following notations: For a positive integer n, let [n] :=
{1, . . . , n}. For any values x1 , . . . , xn , let {xi }i∈[n] := {x1 , . . . , xn }. Let 0/ be an empty
set. If we write negligible ε in n (or denoted by ε (n) = negl(n)), it means a function
ε : N → [0, 1] where ε (n) < 1/g(n) for any polynomial g and sufficiently large n. Prob-
abilistic polynomial-time is abbreviated as PPT.
Message Authentication Code (MAC). A MAC function is defined as a keyed function
F : K × {0, 1}∗ → T, where K is a key space and T is a tag space.
As security of MAC functions, we define the security notion of unforgeability as fol-
lows: A MAC function F meets unforgeability if Advuf MAC,A (λ ) < negl(λ ) holds for any
PPT adversary A against F. Advuf (λ ) is the advantage of A and defined as follows:
 MAC,A  
 U
∗ ∗ ∗  K ← K; QTag ← 0; /
AdvF,A (λ ) := Pr τ = FK (m ) ∧ m ∈
uf / QTag  ∗ ∗ .
 (m , τ ) ← AOTag (1λ )
Here, when the tagging oracle OTag takes a message m ∈ {0, 1}∗ as a query, it out-
puts τ = FK (m) ∈ T and sets QTag ← QTag ∪ {m}.
Aggregate MAC (AMAC). An aggregate message authentication code (AMAC) con-
sists of a four-tuple of polynomial-time algorithms (Gen, Tag, Agg, Ver) as follows: n
is the number of senders, K is a key space, T is a tag space, and Tagg is an aggregate-
tag space: Gen is a randomized algorithm which takes a security parameter λ as input
and outputs a secret key K ∈ K. Tag is a deterministic algorithm which takes a secret
key K, and an ID id and a message m ∈ {0, 1}∗ as input and outputs a tag τ ∈ T. Agg
is a deterministic algorithm which takes ID/tag pairs M := {(idi , τi )}i∈[n] as input and
outputs an aggregate tag T ∈ Tagg . Ver is a deterministic algorithm which takes secret
keys {Kidi }i∈[n] , ID/message pairs M and an aggregate tag T as input and outputs 1
(acceptance) or 0 (rejection).
As the correctness of AMACs, we require that an AMAC scheme meets the fol-
lowing condition: For all i ∈ [n], all Kidi ← Gen(1λ ), and all mi ∈ {0, 1}∗ , it holds that
1 ← Ver({Kidi }i∈[n] , M, T ), where M := {(idi , mi )}i∈[n] , T ← Agg({(idi , τi )}i∈[n] ) and
τi ← Tag(Kidi , mi ) for all i ∈ [n].
 Interactive Aggregate Message Authentication Scheme 1319

We define the unforgeability of AMACs as follows: An AMAC meets unforge-

AMAC,A (λ ) < negl(λ ) holds for any PPT adversary A against AMAC =
ability if Advuf
AMAC,A (λ ) is the advantage of A and defined as follows:
(Gen, Tag, Agg, Ver). Advuf

⎡  ⎤
1 ← Ver({Kidi }i∈[n] , M, T )∧  ∀i ∈ [n], Kidi ← Gen(1λ );
AMAC,A (λ ) := Pr
Advuf / QCorr and  QTag ← 0;
⎣ ∃(idi , mi ), idi ∈ / QCorr ← 0; / ⎦.
(idi , mi ) ∈
/ QTag  (M, T ) ← A O Tag ,OCorr (1λ )

Here, OTag and OCorr are oracles called tagging oracle and corrupt oracle, respec-
tively. Given a pair of an ID and a message (idi , mi ) as a query, the tagging oracle
OTag returns τ ← Tag(Kidi , mi ), and sets QTag ← QTag ∪ {(idi , mi )}. Given an ID idi
as a query, the corrupt oracle OCorr returns the corresponding secret key Kidi , and sets
QCorr ← QCorr ∪ {idi }.

3 Adaptive Group Testing

In this section, we define a model of adaptive group testing (AGT) protocols. The pur-
pose of AGT protocols is to detect specific items called defectives among many items.
The applications of AGT protocols include screening blood samples for detecting a dis-
ease, detecting clones which have a particular DNA sequence. The advantage of AGT
protocols lies in the capability of detecting defectives more efficiently than individ-
ual testing or non-adaptive group testing in terms of the number of tests required in
protocols.
An item denotes a pair (id, data) consisting of an ID and a value, a defective means
a particular item which an AGT protocol tries to detect, and a good item means an
item which is not a defective. An answer consists of (id, ans) which will be used to
decide whether an target item is a defective or not. Let I be a set of n items including
d defectives. The purpose of AGT protocols is to detect such d defectives from n items
by running two Turing machines called Query and Detect.
In order to give a definition of good items and defectives more strictly, we define a
relation R which takes two values data and ans as input and outputs
or ⊥ in polyno-
mial time, namely R(data, ans) =
(resp., R(data, ans) = ⊥) if given values data and
ans meet the relation R (resp., does not meet R). For example, R(data, ans) =
if we
have |v − v | ≤ ε for data = v and ans = (v, ε ), where v, v , ε ∈ R. Another example is
given by using a MAC function F as follows: R(data, ans) =
if we have τ = FK (μ )
for data = τ ∈ T and ans = (K, μ ) ∈ K × M. Let I be a set of n items, and let A be a
set of answers. Then, defectives (or good items) are defined by using a relation R as
follows: An item (idi , datai ) ∈ I is defined as a defective (resp., an good item) by using
an answer (idi , ansi ) ∈ A if R(datai , ansi ) = ⊥ (resp., R(datai , ansi ) =
).
Formally, an AGT protocol consists of polynomial-time algorithms (Compress,
Check, Group, GT) as follows, where GT is interactive between Query and Detect.
• A compression algorithm Compress is a deterministic algorithm which takes possi-
bly multiple items {(idi , datai )}i∈G for a group G ⊂ I as input, and it outputs a value
called a test.
1320 S. Sato and J. Shikata

• A checking algorithm Check is a deterministic algorithm which takes answers

{(idi , ansi )}i∈G for a group G ⊂ I and a test test as input, and it checks whether test
contains a value from a defective or not. It outputs a valid symbol 1 (acceptance) if
test is generated from good items, and outputs an invalid symbol 0 (rejection) if test
is generated from items of which there exists at least one defective.
• A grouping algorithm Group is a deterministic algorithm which takes a set of defec-
tive candidates and a family of subsets of I as input, and it outputs a family of subsets
of I. We denote it by I( j) ← Group(D( j−1) , I( j−1) ) at the j-th stage.
• A group testing protocol GT takes two sets, I = {(idi , datai )}i∈[n] and A =
{(idi , ansi )}i∈[n] . Then, it runs the following Turing machines.
– Query takes I as input and outputs a string str ∈ {complete, not-complete}.
– Detect takes A as input and outputs IDs of defectives D = {idi | R(datai , ansi ) =
⊥ for i ∈ [n]}.
Then, GT outputs (str, D).

Let N be the number of stages in an AGT protocol. An N-stage group testing pro-
(0)
tocol GT(I, A) = Query(I), Detect(A) is expressed as follows: Let j := 1, I1 := I,
(0)
and A1 := A. At the j-th stage, GT(I( j) , A( j) ) = Query(I( j) ), Detect(A( j) ) works as
follows:

Query(I( j) ): If j = 1, it starts from Step 2.

• Step 1: Receive D( j−1) and str ∈ {complete, not-complete} from Detect.
– If str = complete, output str and halt.
– If str = not-complete, go to Step 2.

( j)
• Step 2: I( j) ← Group(D( j−1) , I( j−1) ), where I( j) = Ii ( j)
and t ( j) is the
i∈[t ]
number of tests at the j-th stage. 
( j) ( j) ( j)
• Step 3: test( j) ← testi | ∀i ∈ [t ( j) ], testi ← Compress(Ii ) .
• Step 4: Send test( j) to Detect.
Detect(A( j) ): If j = 1, it sets D(0) = I and starts from Step 1.
• Step 1: Receive test( j) from Query.

( j)
• Step 2: A( j) ← Group(D( j−1) , A( j−1) ), where A( j) = Ai .
i∈[t ( j) ]
• Step 3: Update the list D( j−1)
as follows.
( j) ( j)
1. bi ← Check(Ai , testi ) for all i ∈ [t ( j) ].
( j) t ( j)
2. ∀i ∈ [t ( j) ], Gi ← {id | bi = 1 ∧ (id, ans) ∈ Ai }, and D( j) ← D( j−1) \ i=1 Gi .
• Step 4: Do the following:
– If j = N, output D( j) and str := complete, and halt.
– If j < N, let str := not-complete.
Send (str, D( j) ) to Query.

We define two notions, correctness and detection-completeness, for an AGT proto-

col AGT = (Compress, Check, Group, GT) as follows.

Definition 1 (Correctness and Detection-Completeness). An AGT meets correct-

ness, if it meets the following conditions:
 Interactive Aggregate Message Authentication Scheme 1321

• For any sets I, A such that R(datai , ansi ) =

for all i ∈ [n], we have 1 ←
Check(A, test), where test ← Compress(I).
• For any sets I, A such that R(datai , ansi ) =
for all i ∈ [n], we have str = complete
and D = 0,
/ where (str, D) ← Query(I), Detect(A) if Query and Detect follow the
group testing protocol GT.

An AGT meets detection-completeness if for any two sets I, A, it holds that str =
complete and D = {idi | i ∈ [n] s.t. R(datai , ansi ) = ⊥} with overwhelming probabil-
ity 1 − negl(λ ), where (str, D) ← Query(I), Detect(A) if Query and Detect follow
the group testing protocol GT.

We can describe several interesting AGT protocols along with our model such as
the binary search [2], rake-and-winnow algorithm [3], Li’s s-stage algorithm [8], and
digging algorithm [2]. This implies reasonability of our modeling of AGT protocols.
Specifically, we represent these AGT protocols by describing corresponding Group
algorithms. Due to lack of space, we only describe the binary search in the following,
where for simplicity we write only IDs to specify items.
Binary Search. The AGT protocol based on binary search (see also the bisecting algo-
rithm in [2]), which is a (log n + 1)-stage AGT protocol, is expressed by applying the
following Group algorithm.
I( j) ← Group(D( j−1) , I( j−1) ):

Step 1. If j = 1, let I( j) := I( j−1) and move to Step 3.

Step 2. Let I( j) := {0}
/ and do the following for all i ∈ [t ( j−1) ]:
( j−1) ( j−1) ( j−1) ( j−1)
• If Ii := {id1 , . . . , idk } ⊆ D( j−1) , move to the next set Ii+1 .
( j−1) ( j−1) ( j−1) ( j−1)
• I( j) ← I( j) ∪ {{id1 , . . . , idk/2 }, {idk/2+1 , . . . , idk }}.
Step 3. Output I( j) .

( j−1)
Regarding this algorithm, we note that I( j−1) = Ii , where t ( j−1) is the
i∈[t ( j−1) ]
number of sets in I( j−1) . Besides, for simplicity, we have assumed that the number of
( j)
elements in each Ii is a power of 2, though we can easily generalize it.

4 Interactive AMAC with Detecting Functionality

We introduce a model and security definition of interactive aggregate MACs with
detecting functionality (IAMD for short), and we also propose a generic construction of
IAMD starting from any aggregate MAC and any AGT protocol. In the case where an
aggregate tag is invalid, the purpose of IAMD is to trace which MAC tags are invalid by
interactive communication between an aggregator and a detector. Roughly, we consider
the following scenario by using IAMD:

• Senders: Each sender generates a MAC tag on its local message and sends this mes-
sage/tag pair to an aggregator.
1322 S. Sato and J. Shikata

• Aggregator: An aggregator receives message/tag pairs from senders, generates an

aggregates tag, and sends the messages and the aggregate tag to a detector. In the
case where there exist invalid MAC tags, the detector returns a feedback to specify
these tags. Then, the aggregator generates new aggregate tags based on the feedback
and returns them. Until the detector specifies all the invalid tags, this interactive
process is repeated.
• Detector: A detector receives messages and an aggregate tag, and verifies whether
these are valid or not. If this verification does not pass, it will detect all invalid MAC
tags by communicating with the above aggregator. When the detector can specify all
the invalid MAC tags, it outputs the list of invalid IDs.

Model and Security Definitions. IAMD consists of the following polynomial-time

algorithms (Gen, Tag, Agg, Ver, GTA), where GTA is an interactive protocol between
two Turing machines TAgg(·) and Det(·). Let n be the number of senders. Let K be a
key space, let T be a tag space, and let Tagg be an aggregate-tag space.
• Key Generation: Gen is a randomized algorithm which, on input a security parameter
λ , outputs a secret key K ∈ K.
• Tagging: Tag is a deterministic algorithm which, on input a secret key K ∈ K and a
message m ∈ {0, 1}∗ , outputs a tag τ ∈ T.
• Aggregation: Agg is a deterministic algorithm which, on input ID/tag pairs
{(idi , τi )}i∈[n] , outputs an aggregate-tag T ∈ Tagg .
• Verification: Ver is a deterministic algorithm which, on input keys {Kidi }i∈[n]
ID/message pairs M := {(idi , mi )}i∈[n] , and an aggregate tag T , outputs 1 (accep-
tance) or 0 (rejection).
• Group Testing with Aggregation: GTA takes as two sets, {(idi , mi , τi )}i∈[n] and
{(idi , Kidi , mi )}i∈[n] . Then, it runs the following Turing machines:
– TAgg takes ID/message pairs M := {(idi , mi )}i∈[n] and tags {τi }i∈[n] as input and
outputs str ∈ {complete, not-complete}.
– Det takes keys {Kidi }i∈[n] and ID/message pairs M := {(idi , mi )}i∈[n] as input
and outputs a list of IDs D = {idi | i ∈ [n] s.t. τi = Tag(Kidi , mi )}.
Then, GTA outputs (str, D).
As the correctness of IAMD, we require that an IAMD scheme meets the following
conditions:
• For all i ∈ [n], all Kidi ← Gen(1λ ), and all mi ∈ {0, 1}∗ , it holds that 1 ←
Ver({Kidi }i∈[n] , M, T ), where M := {(idi , mi )}i∈[n] , T ← Agg({(idi , τi )}i∈[n] ) and
τi ← Tag(Kidi , mi ) for all i ∈ [n].
• For all i ∈ [n], all Kidi ← Gen(1λ ), and all mi ∈ {0, 1}∗ , it holds that str = complete
and D = 0, / where (str, D) ← GTA({(idi , mi , τi )}i∈[n] , {(idi , Kidi , mi )}i∈[n] ) if TAgg
and Det follow the protocol GTA.
As the security of IAMD, we define unforgeability and identifiablity. First, we
define the following oracles: Corrupt oracle OCorr (idi ) returns the corresponding key
Kidi and sets QCorr ← QCorr ∪ {idi }, where QCorr is the list of queried IDs. Tagging ora-
cle OTag (idi , mi ) returns Tag(Kidi , mi ) and sets QTag ← QTag ∪ {(idi , mi )}, where QTag
is the list of queried ID/message pairs.
 Interactive Aggregate Message Authentication Scheme 1323

Definition 2 (Unforgeability). An IAMD scheme IAMD = (Gen, Tag, Ver, Agg, GTA)
IAMD,A (λ ) < negl(λ ) holds for any PPT adversary A against
meets unforgeability if Advuf
IAMD. AdvIAMD,A (λ ) is the advantage of A and defined as follows:
uf

⎡  ⎤
1 ← Ver({Kidi }i∈[n] , M, T )∧  ∀i ∈ [n], Kidi ← Gen(1λ );
IAMD,A (λ ) := Pr
Advuf / QCorr and  QTag ← 0;
⎣ ∃(idi , mi ), idi ∈ / QCorr ← 0; / ⎦.
(idi , mi ) ∈
/ QTag  (M, T ) ← A O Tag ,OCorr (1λ )

Regarding the identifiability of IAMD, we consider completeness and soundness:

Completeness is the security against adversaries who try to make a detector classify
valid message/tag pairs as invalid ones; Soundness is the security against adversaries
who try to make a detector classify invalid message/tag pairs as valid ones. We formal-
ize the notion of identifiability for IAMD as follows.

Definition 3 (Identifiability). An IAMD scheme IAMD = (Gen, Tag, Agg, Ver, GTA)
meets identifiability-completeness (resp. identifiability-soundness), if Advident-x
IAMD,A
(λ ) := Pr[Expident-x
IAMD,A (λ ) = 1] < negl(λ ) holds with x = c (resp. x = s) for any PPT
adversary A against IAMD. The experiment ExpIAMD,A (λ ) for x ∈ {c, s} is defined as
ident-x

follows:
Step 1: Let n := poly(λ ). For all i ∈ [n], Ki ← Gen(1λ ). QTag ← 0/ and QCorr ← 0.
/
Step 2: {(idi , mi , τi )}i∈[n] ← AOTag ,OCorr (λ ),
Step 3: Run the protocol (str, D) ← GTA({(idi , mi , τi )}i∈[n] , {(idi , Kidi , mi )}i∈[n] ).
Completeness. The experiment Expident-c IAMD,A (λ ) outputs 1 if str = complete and D ∩ {idi |
τi = Tag(Kidi , mi )} = 0/ hold. It outputs 0 otherwise.
Soundness. The experiment Expident-sIAMD,A (λ ) outputs 1 if str = complete and {idi | τi =
Tag(Kidi , mi )}\D = 0/ hold. It outputs 0 otherwise.

We claim that even if an IAMD scheme does not meet identifiability-soundness, it

may be useful in terms of the unforgeability of messages. From this viewpoint, we
define identifiability-weak-soundness which is slightly weaker than soundness. This
definition is the same as identifiability-soundness except that A is not allowed to get
a correct tag for (id, m) from Tagging oracle, and the experiment outputs 1 if there
/ D such that τi = Tag(Kidi , mi ), (idi , mi ) ∈
exists some (idi , mi ) ∈ / QTag , and idi ∈
/ QCorr .
As a relationship between unforgeability and identifiability-weak-soundness, we show
the following:
Proposition 1. If IAMD meets unforgeability, it also meets identifiability-weak-
soundness.

Proof. We construct a PPT algorithm B breaking the unforgeability of IAMD by using a

PPT algorithm A breaking identifiability-weak-soundness as follows: It is given Tagging
and Corrupt oracles and simulates the oracle access of A with these given oracles. When
A outputs {(idi , mi , τi )}i∈[n] , B runs GTA protocol and obtains (str, D). It checks whether
1324 S. Sato and J. Shikata

/ D such that τi = Tag(Kidi , mi ), idi ∈

there exists idi ∈ / QCorr , and (idi , mi ) ∈
/ QTag . If so, B
finds a set of pairs including (idi , mi , τi ) such that the aggregate-tag of the pairs is valid,
and outputs the messages and the aggregate-tag. Otherwise, it aborts the game. Then, B
breaks the unforgeability of IAMD clearly. Thus, the proof is completed. 


GIAMD: Generic Construction of IAMD. We propose a generic construction

of IAMD starting from any aggregate MAC and any AGT protocol. That is, we
use an aggregate MAC AMAC = (AMAC.Gen, AMAC.Tag, AMAC.Agg, AMAC.Ver)
and an AGT protocol AGT = (AGT.Compress, AGT.Check, AGT.Group, AGT.GT =
Query, Detect).
Our generic construction GIAMD = (Gen, Tag, Agg, Ver, GTA) is as follows:

• K ← Gen(1λ ): Output a key K ← AMAC.Gen(1λ ).

• τ ← Tag(K, m): Output a tag τ ← AMAC.Tag(K, m).
• T ← Agg({(idi , τi )}i∈[n] ): Output a tag T ← AMAC.Agg({(idi , τi )}i∈[n] ).
• 1/0 ← Ver({Kidi }i∈[n] , M, T ): Output b ← AMAC.Ver({Kidi }i∈[n] , M, T ).
• (str, D) ← GTA({(idi , mi , τi )}i∈[n] , {(idi , Kidi , mi )}i∈[n] ):
1. Let I := {(idi , datai )}i∈[n] , and A := {(idi , ansi )}i∈[n] , where let datai := τi and
ansi := (Kidi , mi ).
2. TAgg(M, (τi )i∈[n] ) and Det({Kidi }i∈[n] , M) runs Query(I) and Detect(A) respec-
tively for AGT.Compress(·) := Agg(·) and AGT.Check(·) := Ver(·).
3. Output (str, D) ← Query(I), Detect(A).

Theorem 1. If AMAC meets unforgeability, GIAMD also meets unforgeability.

Proof. Let A be a PPT adversary against GIAMD. We prove that GIAMD meets
unforgeability by constructing the following PPT algorithm B breaking the unforge-
ability of AMAC: B is given access to Corrupt and Tagging oracles of AMAC. It
runs AOTag ,OCorr (λ ). It can simulate the corrupt and tagging oracles of GIAMD by
using the given oracles of AMAC. When A outputs ({(idi , m∗i )}i∈[n] , T ∗ ), B also outputs
({(idi , m∗i )}i∈[n] , T ∗ ) if there exists some (idi , m∗i ) such that idi ∈
/ QCorr and (idi , m∗i ) ∈
/
QTag . Given Corrupt and Tagging oracle of AMAC, B can simulate the view of A com-
pletely. If A breaks the unforgeability of GIAMD, B also breaks the unforgeability of
AMAC since the condition of a forgery against GIAMD includes that of a forgery against
AMAC. Hence, the success probability of A is at most AdvuAMAC, f
B (λ ) and the proof is
completed. 


Theorem 2. If AGT meets detection-completeness and AMAC meets unforgeability,

GIAMD meets both identifiability-completeness and identifiability-weak-soundness.

Proof. GIAMD obviously meets identifiability-completeness from the detection-

completeness of AGT. In addition, GIAMD meets identifiability-weak-soundness from
Proposition 1 and Theorem 1. 

 Interactive Aggregate Message Authentication Scheme 1325

5 Concrete Constructions of IAMD

In this section, we provide several concrete constructions of IAMD and compare them
in terms of security and efficiency. As instantiation of AMACs for constructing IAMD,
we consider XOR-based and hash-based constructions as follows.
XOR-Based Construction. We construct an IAMD scheme by using a MAC func-
tion F : K × {0, 1}∗ → T and an AGT protocol. Here, the aggregation is con-
structed by taking XOR-ing for given tags in the same way as [7]. The construction
U
XIAMD = (Gen, Tag, Agg, Ver, GTA) is described as follows: Gen(1λ ) outputs K ← K
and Tag(K, m) outputs τ ← FK (m).
• T ← Agg({(idi , τi )}i∈[n] ): Output an aggregate tag T ← τ1 ⊕ · · · ⊕ τn
• 1/0 ← Ver({Kidi }i∈[n] , M, T ): Output 1 if T = FKid (m1 ) ⊕ · · · ⊕ FKidn (mn ), or output
1
0 otherwise.
We can obtain GTA = TAgg(·), Det(·) by applying Agg algorithm, Ver algorithm,
and an AGT protocol in the same way as GIAMD scheme.
Theorem 3. If F meets unforgeability, XIAMD also meets unforgeability. In addition,
if AGT meets detection-completeness and F meets unforgeability, XIAMD meets both
identifiability-completeness and identifiability-weak-soundness, however, XIAMD does
not satisfy identifiability-soundness.
Proof. From the proof of Theorem 1 in [7] and Theorem 1, XIAMD meets unforge-
ability. XIAMD also satisfies identifiability-completeness and identifiability-weak-
soundness by Theorem 2. However, this does not meet the soundness. We construct
an adversary B breaking the security as follows: It submits (id1 , m1 ), . . . , (idn , mn ) to
OTag and receive (τ1 , . . . , τn ). It computes τ1∗ ← τ1 ⊕ 1 and τ2∗ ← τ2 ⊕ 1. Then it out-
puts {(id1 , m1 , τ1∗ ), (id2 , m2 , τ2∗ ), (id3 , m3 , τ3 ), . . . , (idn , mn , τn )}. The output of B breaks
identifiability-soundness since τ1 ⊕ τ2 = τ1∗ ⊕ τ2∗ and (τ1 , τ2 ) = (τ1∗ , τ2∗ ) hold. Therefore,
the proof is completed. 


Hash-Based Construction. We construct an IAMD scheme with random oracles. We

use a MAC function F : K × {0, 1}∗ → T, an AGT protocol, and a hash function (a
random oracle) H : Tn → H, where Tn is a domain (n MAC-tag spaces) and H is a
range. The construction HIAMD = (Gen, Tag, Agg, Ver, GTA) is described as follows:
U
Gen(1λ ) outputs K ← K and Tag(K, m) outputs τ ← FK (m).
• T ← Agg({(idi , τi )}i∈[n] ): Output an aggregate tag T ← H(τ1  τ2  · · ·  τn ).
• 1/0 ← Ver({Kidi }i∈[n] , M, T ): Output 1 if T = H(FKid (m1 )  · · ·  FKidn (mn )). Output
1
0 otherwise.
The protocol GTA of HIAMD is the same as that of GIAMD except for using the
above algorithms Agg and Ver. From Theorems 1 and 2, we have the following.
Theorem 4. If F meets unforgeability, HIAMD also meets unforgeability. In addi-
tion, if AGT meets identifiability-completeness, HIAMD meets both identifiability-
completeness and identifiability-soundness in the random oracle model.
1326 S. Sato and J. Shikata

Proof. Let |τ | be the bit-length of aggregate-tags, and let qh be the number

of queries submitted to the random oracle H. In the same way as the proof
of Theorem 4 in [5], HIAMD meets the unforgeability of aggregate MACs.
q2
HIAMD,A (λ ) ≤ n · AdvF,B (λ ) + 2|τ | + 2|τ |+1 by Theorem 1. If
1
Hence, we obtain Advuf uf h

AGT meets detection-completeness, HIAMD meets identifiability-completeness by

Theorem 2. We show the identifiability-soundness. If A outputs {(idi , mi , τi )}i∈[n]
such that {idi | τi = Tag(Kidi , mi )}\D = 0, / there exists some τi such that τi =
Tag(Kidi , mi ) and H(τ1  · · · τn ) = H(Tag(Kid1 , m1 ) · · · Tag(Kidn , mn )). That is, the
pair (τ1 , . . . , τn ) and (Tag(Kid1 , m1 ) · · · Tag(Kidn , mn )) is a collision of H. Hence, we
q2h
HIAMD,A (λ ) ≤
have Advident-s τ
2 |+1
| . Therefore, the proof is completed. 


Comparison of Concrete Constructions of IAMD. We consider several concrete con-

structions of IAMD obtained by applying the following AMACs and AGT protocols:
we apply XOR-based AMAC and hash-based AMAC; and we apply the binary search
algorithm [2], rake-and-winnow algorithm [3], Li’s s-stage algorithm [8], and digging
algorithm [2]. As a result, we obtain eight IAMD constructions and these construc-
tions are summarized in Table 1. We have selected the above AGT protocols, since the
number of tests required in them is O(d log dn ) which is asymptotically optimal. More
strictly, the following result is known:

Table 1. Concrete Constructions of IAMD

The resulting IAMD Applied AMAC Applied AGT

XIAMDBIN XOR-based AMAC Binary search
XIAMDRAW XOR-based AMAC Rake-and-winnow algorithm
XIAMDLi XOR-based AMAC Li’s s-stage algorithm
XIAMDDIG XOR-based AMAC Digging algorithm
HIAMDBIN Hash-based AMAC Binary search
HIAMDRAW Hash-based AMAC Rake-and-winnow algorithm
HIAMDLi Hash-based AMAC Li’s s-stage algorithm
HIAMDDIG Hash-based AMAC Digging algorithm

Proposition 2 (Lemma 4.5.2 of [2]). For integers n, d with 0 ≤ d ≤ n, let M(d, n)

be the minimum number of tests to detect d defectives from n items. Then, we have
d log dn ≤ M(d, n) ≤ d log dn + (1 + log e)d.

Consequently, we summarize comparison of IAMD schemes in Table 2 as follows.

1. In terms of a total amount of tag-size, XIAMDDIG is the best if n > d · 2ε , where
ε = 4/ (e/ log e − 1) ≈ 4.5. This is because the difference between tag-sizes of
XIAMDLi and XIAMDDIG is ((e/ log e − 1)d log (n/d) − 4d) |τ |. XIAMDDIG is best
if (e/ log e − 1)d log (n/d) − 4d > 0, namely, n > d · 2ε . The same analysis can be
 Interactive Aggregate Message Authentication Scheme 1327

Table 2. Comparison of IAMD: d is the number of invalid tags. n is the number of senders. |τ | is
the bit-length of aggregate-tags. |h| is the bit-length of the evaluation value of hash functions. The
number of stages s in XIAMDLi and HIAMDLi is determined by the optimal parameter setting.
e is Napier’s constant. “Unknown d” means that it is possible to detect all defectives without
knowing the number of defectives. Comp. means identifiability-completeness and Sound (resp.,
wSound) means identifiability-soundness (resp., identifiability-weak-soundness).

Construction Total tag size (in the #Stage Unknown d ? Identifiability

worst-case)
 
XIAMDBIN 2d log dn + 2d − 1 |τ | log n + 1  Comp. and wSound
 
XIAMDRAW 4d log en d + 2 log n |τ | 2 Comp. and wSound
 
XIAMDLi log e d log( d ) |τ |
e n
log dn Comp. and wSound
 
XIAMDDIG d log dn + 4d |τ | d(log n + 1)  Comp. and wSound
 
HIAMDBIN 2d log dn + 2d − 1 |h| log n + 1  Comp. and Sound
 
HIAMDRAW 4d log en d + 2 log n |h| 2 Comp. and Sound

HIAMDLi log e d log d |h|
e n
log dn Comp. and Sound
 
HIAMDDIG d log dn + 4d |h| d(log n + 1)  Comp. and Sound

applied in HIAMDDIG . However, we cannot decide which is better, XIAMD construc-

tion or HIAMD construction, since MAC-tag size |τ | and hash-value size |h| are not
given to be enough to compare and (dis)advantage is depending on the choice of
underlying MACs or hash functions to be applied.
2. In terms of the number of stages, XIAMDRAW and HIAMDRAW are best. They are
most effective if we focus on time complexity because the number of communi-
cations between an aggregator and a detector is small, and it is possible to verify
message/aggregate-tags pairs in a parallel process.
3. Since we need not know the number d of invalid messages beforehand, XIAMDBIN ,
XIAMDDIG , HIAMDBIN and HIAMDDIG are very useful. This is because the under-
lying AGT protocols are based on competitive group testing.
4. In terms of security-level of identifiability-soundness, HIAMD has the advantage.
However, as we discussed in Sect. 4, we stress that XIAMD schemes are also useful
in some applications since they meet identifiability-weak-soundness.

6 Conclusion
We proposed IAMD which can aggregate multiple MAC tags and detect invalid mes-
sages by combining aggregate MACs and AGT protocols. These functionalities are use-
ful in the case where there are a lot of senders using MAC schemes. This is because we
can specify invalid messages efficiently in terms of the total size of MAC tags. Specifi-
cally, the contribution of this paper is shown as follows.
• We formalized AGT protocols which could detect specific items among a lot of
items. Specifically, we defined the model and the property (detection-completeness)
which AGT protocols need to meet. Besides, in order to show that our model
1328 S. Sato and J. Shikata

expresses AGT protocols, we gave a description of an existing AGT protocol based

on binary search in our model.
• We introduced the notion of IAMD which could compress multiple MAC tags
and detect invalid messages among lots of messages by interactive communica-
tions between aggregators and detectors. Specifically, we defined the model and
formalized the security, namely unforgeability and identifiability consisting of
identifiability-completeness and identifiability-(weak-)soundness.
• We proposed a generic construction of IAMD starting from an aggregate MAC and
an AGT protocol. We also showed that this construction met unforgeability and
identifiability-(weak-)soundness.
• We presented various instantiations of IAMD constructions by applying concrete
aggregate MACs and AGT protocols to our generic construction. We compared the
resulting constructions in terms of total tag-size, the number of stages, and secu-
rity. As a result, we achieved almost minimum number of total tag-size by applying
the digging algorithm, though there was a trade-off between total tag-size and the
number of stages in general.

Acknowledgements. This research was conducted under a contract of Research and Develop-
ment for Expansion of Radio Wave Resources funded by the Ministry of Internal Affairs and
Communications, Japan.

References
1. Dorfman, R.: The detection of defective members of large populations. Ann. Math. Stat.
14(4), 436–440 (1943)
2. Du, D.Z., Hwang, F.K.: Combinatorial Group Testing and Its Applications. Series on Applied
Mathematics, vol. 12, 2nd edn. World Scientific, Singapore (2000)
3. Eppstein, D., Goodrich, M.T., Hirschberg, D.S.: Improved combinatorial group testing algo-
rithms for real-world problem sizes. SIAM J. Comput. 36(5), 1360–1375 (2007)
4. Goodrich, M.T., Atallah, M.J., Tamassia, R.: Indexing information for data forensics. In:
Third International Conference on Applied Cryptography and Network Security, ACNS
2005, Lecture Notes in Computer Science, vol. 3531, pp. 206–221. Springer (2005)
5. Hirose, S., Shikata, J.: Non-adaptive group-testing aggregate MAC scheme. In: 14th Interna-
tional Conference on Information Security Practice and Experience (ISPEC 2018), Lecture
Notes in Computer Science, vol. 11125, pp. 357–372. Springer (2018)
6. Hwang, F.K.: A method for detecting all defective members in a population by group testing.
J. Am. Stat. Assoc. 67(339), 605–608 (1972)
7. Katz, J., Lindell, A.Y.: Aggregate message authentication codes. In: CT-RSA, Lecture Notes
in Computer Science, vol. 4964, pp. 155–169. Springer (2008)
8. Li, C.H.: A sequential method for screening experimental variables. J. Am. Stat. Assoc.
57(298), 455–477 (1962)
9. Minematsu, K.: Efficient message authentication codes with combinatorial group testing. In:
ESORICS, Part 1, Lecture Notes in Computer Science, vol. 9326, pp. 185–202. Springer
(2015)
10. Porat, E., Rothschild, A.: Explicit non-adaptive combinatorial group testing schemes. In:
ICALP, part 1, Lecture Notes in Computer Science, vol. 5125, pp. 748–759. Springer (2008)
11. Thierry-Mieg, N.: A new pooling strategy for high-throughput screening: the shifted
transversal design. BMC Bioinform. 7, 28 (2006)
 Basic Study on Targeted E-mail Attack
Method Using OSINT

Kota Uehara1, Kohei Mukaiyama1, Masahiro Fujita1,

Hiroki Nishikawa2, Takumi Yamamoto2, Kiyoto Kawauchi2,
and Masakatsu Nishigaki1(&)
1
Shizuoka University, 3-5-1 Johoku, Naka, Hamamatsu,
Shizuoka 432-8011, Japan
[email protected]
2
Mitsubishi Electric Corporation,
5-1-1, Ofuna, Kamakura, Kanagawa 247-8501, Japan

Abstract. In recent years, attackers have easily gained considerable informa-

tion on companies and individuals using open source intelligence (OSINT),
thereby increasing the threat of targeted attacks. In light of such a situation,
modeling the synergistic effect of OSINT and targeted attacks will be an
effective measure against these attacks. In this paper, we formulate a state
transition model that defines the process by which attackers gather a target’s
information by using OSINT tools. Then we categorize the targeted e-mails that
the attackers can generate in each state. The results of the analysis can be used
by the victims to estimate the extent of attacks from the contents of the targeted
e-mails, and to take appropriate measures.

1 Introduction

In recent years, the damage due to targeted e-mail attacks has increased rapidly. Tar-
geted e-mail attacks are a typical example of social engineering, causing damage (for
example, exploiting information and money, or manipulating PCs illegally) to the
targets by deceiving them. To accomplish a targeted e-mail attack, it is essential to
make the target believe that the targeted e-mail is a regular e-mail, and thus the attacker
attempts to collect information about the targeted person.
Regarding information gathering, previously attackers had to gather information on
the target through their own exertions. Typical data collection methods include human
intelligence (HUMINT), which pretends to be an administrator or officer to get close to
the target to hear information directly from them or to collect information from persons
around them, and signals intelligence (SIGINT), which eavesdrops information and
communication of the target person. However, in recent years it has become common
practice for companies and users to send information about themselves over the internet,
such as through owned media and social media. It has been reported that the Web and
social media is overflowing with information pertaining to companies and individuals,
and personal identifiable information and privacy information can be obtained by
combining public personal information [1, 2]. This information-gathering method is
called Open Source Intelligence (OSINT), and there are several tools that support OSINT
© Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1329–1341, 2020.
https://doi.org/10.1007/978-3-030-15032-7_111
1330 K. Uehara et al.

activities. By using OSINT tools, attackers can create targeted e-mails that are more
credible (the target can be more easily deceived), and hence, the threat of targeted e-mail
attacks is much greater than before. From the defender’s point of view, it is important to
consider the target e-mail attacks that use OSINT techniques beforehand [3].
In view of this situation, this study attempts to model the synergistic effect of
OSINT and targeted e-mail attacks. Specifically, assuming a targeted e-mail attack [3]
using OSINT tools, we formulate a “state transition model” that defines the process by
which attackers gather information of the target person using OSINT tools. Further-
more, in each state, we categorize targeted e-mails that the attackers can generate. The
results of this analysis provide effective knowledge for the victims to estimate the depth
of the attack from the contents of targeted e-mails, and to take measures for restoration
accordingly. Moreover, in the future, because OSINT tools will be developed and fully
supported by artificial intelligence (AI), attackers will carry out targeted e-mail attacks
by misusing AI. The findings of this study can be a foothold in analyzing the syner-
gistic effect of AI and targeted e-mail attacks.

2 Targeted E-mail Attack Using OSINT

2.1 Advancement of Social Engineering by OSINT

OSINT is an abbreviation for open source intelligence, and refers to intelligence
activities collecting necessary information from published information. The published
information (hereinafter referred to as “OSINT data”) includes a wide range of infor-
mation, for example, newspaper reports, and information written from government
officials to telephone directories. In recent years, many individuals and companies have
transmitted information related to themselves on the social media and Web, and
enormous OSINT data related to individuals and companies is now easily available.
OSINT data is useful information for purposes such as economic forecasting and
epidemic analysis. However, OSINT data is also useful in targeted e-mail attacks, more
precisely, for attackers who utilize social engineering. It is reported that personal and
privacy information can be obtained by combining OSINT data [4], and tools that
support OSINT activities are also available [5]. Attackers can now collect information
on target persons using OSINT data without performing “troublesome intelligence
activities” such as HUMINT and SIGINT. By using the information, an attacker can
create a targeted e-mail with high credibility (the target can be deceived easily).
Below, we explain these threats with concrete examples. The underlined parts
correspond to the OSINT data.
[Concrete Examples]
Assume that the attacker sends a targeted e-mail to Mr. A who is a university professor
in Japan.
1. The attacker uses the name of Mr. A to search the e-mail address on the Web, and
obtains the e-mail address whose domain is “*.ac.jp”.
2. The attacker searches related web pages using names and e-mail addresses, and
examines his researches and achievements from his website.
 Basic Study on Targeted E-mail Attack Method Using OSINT 1331

3. The attacker continues to investigate the related web pages and obtains information
that Mr. A previously gave a lecture at company X.
4. The attacker searches the Web about company X to examine the related web pages,
and obtains information such as the e-mail naming convention that the employee
use or the business that they are focusing on currently.
5. The attacker pretends to be an employee of company X by e-mail spoofing. The e-
mail body includes something like the following; “your lecture was great,” “our
project currently focused on is related to Mr. A’s research,” “we wish you to talk
again for our company,” “we would like you to see the attached file for details.”
Then the attacker attaches a PDF file containing the exploit code and sends the e-
mail to Mr. A.
Although the attacker in this scenario initially had only information on Mr. A’s
name and profession, they were eventually able to collect considerable OSINT data
from the Web. As a result, the attacker succeeded in creating a targeted e-mail with
high credibility as shown in Step 5.

2.2 OSINT Tools

OSINT tools are software used to obtain OSINT data related to the search object
efficiently, from the enormous OSINT data available on the Web. Numerous OSINT
tools (such as Maltego and Creepy) have already been released. Below, detailed
operation of OSINT tools will be explained using Maltego and Tinfoleak, which are
representative examples of OSINT tools.
Maltego is a data collection and visualization tool developed by Paterva. When you
enter a name, domain, URL, or a combination these elements, it automatically collects
all kinds of information related to it from the Web. For instance, if you enter the name
“Masakatsu Nishigaki” for Maltego, you will receive a list of phone numbers, e-mail
addresses, and related Web sites in the format shown in Fig. 1. In this instance,
although the social media account could not be found, if there is a matching account
name on Twitter or Facebook, information on the corresponding account can also be
acquired.
In addition, by further inputting the information obtained here, it is possible to
collect information in a chain reaction. For instance, if you enter the domain name
“minamigaki.cs.inf.shizuoka.ac.jp” (Fig. 1) obtained by entering a name in Maltego,
you can obtain the PDF file, related web site, related domain, web server software
information, e-mail address, IP address, and location information obtained from the IP
address (Fig. 2).
In this way, it can be seen that the search operations defined in points 1 to 4 (in the
scenario described in the previous section) can be semi-automatically executed by
Maltego.
Tinfoleak can obtain Twitter ID input information and the following information.
While Maltego searches the whole Web on the Web, Tinfoleak collects information
limited to a specific service (Twitter).
1332 K. Uehara et al.

Tinfoleak can obtain the following information by entering a TwitterID. While

Maltego searches on the whole Web, Tinfoleak collects information limited to a
specific service (Twitter).
• User basic information (name, image, location, followers, etc.)
• Device and operating system
• Applications used, and linked social media
• Location information (coordinates) given to tweets
• Photographs submitted by users
• Hash tag used by user and time used
• Identification of other users closely related to the user
• Topics on which the user is interested (such as hobbies)
These pieces of information can be obtained by thoroughly checking and analyzing
profiles and tweets of target users and interactions with other users, and manually
searching for tweets using location information. Tinfoleak provides such information
automatically and instantaneously.
These OSINT tools are very useful for authorized users to check to what extent the
personal information of its own diffuses, and diagnose the vulnerability of the server
that is used (check the daemon and application version). However, in some cases, it has
other aspects, such as being used by attackers as a tool to support targeted e-mail
attacks (more precisely, for social engineering).

Fig. 1. Result 1 by Maltego (Input: Name) Fig. 2. Result 2 by Maltego (Input: Domain)

3 Synergistic Effect of OSINT and Targeted E-mail Attack

3.1 Threat Analysis by State Transition Model
Considering targeted e-mail attacks using the OSINT tools, this paper attempts to
model “the synergistic effect of OSINT tools and targeted e-mail attacks”. First, we
formulate the process of gathering information of attack targets using OSINT tools by
 Basic Study on Targeted E-mail Attack Method Using OSINT 1333

attackers as a state transition model. Then, in each state, we categorize targeted e-mails
that the attackers can generate.
In this paper, which is the first step of this research, we focus on the case where the
attack target of the targeted e-mail attack is a “specific individual,” and model it
accordingly. We believe that it is possible to model a targeted e-mail attack by targeting
“any member belonging to a specific department” as well, but we prefer to treat that as
the subject of our analysis from now on.

3.2 Transition of Information Held by Attackers

In a targeted e-mail attack using OSINT tools, regarding an attacker,
1. Initially, the attacker enters information about the target that he possesses into the
OSINT tools, and then obtains new information.
2. Furthermore, the attacker enters the information obtained in step 1 into the OSINT
tools to gather information in a chain reaction.
Considering the procedure of attacking, we formulate the process by which the
attackers gather information of the target person using OSINT tools as a “state tran-
sition model.”
In the initial state, “information relating to the attack target person,” which is
owned by the attacker, is written as {X0}. When an attacker inputs {X0} to the OSINT
tool and new information X1 related to the attack target can be obtained, the state
changes to {X0, X1} (from {X0}). The attacker continues to input {X0, X1} to the
OSINT tools, then further (new) information X2 related to the attack target can be
obtained, the state changes to {X0, X1} (from {X0}). After that, this operation is
repeated.
For instance, in the state in which the attacker possesses the information {name,
phone number} of the target, when information of “address” can be obtained by
inputting “name,” “phone number” or both in the OSINT tools, the state transitions
change from {name, phone number} to {name, phone number, address}.
The attacker freely uses arbitrary OSINT tools and repeats the OSINT activities of
the above steps 1 and 2. Here, the OSINT tools group (Recon-NG, Maltego, Creepy,
Metagoofil, Tinfoleak, EmailHarvester, the Harvester, SpiderFoot, and ExifTool),
which are included in Buscador [6], which is a Linux virtual machine for OSINT
collection, is assumed.1

3.3 Formulation of State Transition Model

When the authors took the attacker’s position and tried OSINT activities by actually
using the nine kinds of OSINT tools named in the previous section, it was possible for

1
Each OSINT tool has its own characteristics, and “input information” for the OSINT tool and
“collectable information” obtained as output from the input information are partially different for
each tool. Due to space limitations, the list of “input information” and “collectable information” of
each OSINT tool is omitted.
1334 K. Uehara et al.

“Process of using the OSINT tool by an attacker to collect information on targets one
after another” as a state transition model (Fig. 3).
In the state transition model to be formulated this time, the following assumptions
are provided for simplifying the model.
• The type of OSINT data to be considered is confined to name, phone number, e-
mail address, social media account, hobby, location information, friendship, affili-
ation organization name, managerial name in the organization, and organization’s
public information.
• Build a state transition model with the assumption that the attacker holds only the
attack target {name} as the initial state.
• Do not consider information that can be inferred from the held information. (For
example, it can be easily guessed that the subject belongs to the educational
institution at the time when it is found that the domain of the mail address to be
attacked is “ac.jp,” but this time, do not perform such deductions.)
• If you enter the social media account into the OSINT tools, you can always acquire
information on hobbies, location information, and friendships of people in that
account (*a in Fig. 3). (Actually, some people include location information in
tweets on Twitter, whereas some people do not include them at all. However, for
simplicity this time, we will assume that these three pieces of information are
known from the social media in general.)
• Several organizations currently publish IR information and the like in web pages,
therefore if you enter an organization’s name in the OSINT tools, you will be able
to acquire information related to the names of the executives, business contents, and
group companies (affiliates) (*b in Fig. 3).
Presently we will focus on targeted e-mail attack (targeted attack via e-mail),
therefore we exclude “address,” which is information related to postal mail from the
OSINT data handled.2

3.4 Targeted E-mail that Attackers Can Generate in Each State

Typical examples of targeted e-mails that an attacker can create in the four states (states
(3), (4), (7), (10)) in which e-mails can be sent in the state transition model shown in
Fig. 3 are shown in Figs. 4, 5, 6, 7 and 8.

2
When we actually tried OSINT activities, it was rare (only when the customer management database
is in an open state etc., due to misconfiguration etc.,); the case where the address could be acquired
by the OSINT tool. Therefore, excluding addresses in this analysis is reasonable, also from the
meaning that the address is not “information that can be easily acquired by the OSINT tool.”
 Basic Study on Targeted E-mail Attack Method Using OSINT 1335

Fig. 3. State transition model of the process by which attackers gather information

• State (3) (Fig. 4):

In state (3), because the information on the attack target held by the attacker is only
“mail address” and “name,” the content of the mail is not dependent on the attack target
person. Although it can be said to be targeted mail because the name is written in the
text of the mail, it is close to the content of general phishing mail.
• State (4) (Fig. 5):
In state (4), an attacker can include the “phone number” of the attack target in the
text of the mail. Even if you have nothing to do with the e-mail, the information
contained in the body of the e-mail is certainly yours, so the possibility that target is
deceived (they may access the link in the e-mail) is thought to be increased. Moreover,
the fear that his/her phone number is known to the sender of the e-mail may cause
anxiety in the target person, and calm judgment will be lost.
In this case, the “address” of the target person was excluded from consideration, but
if the attacker could obtain the address (instead of the telephone number) of the attack
target person, targeted e-mail can be created as shown in Fig. 6.
• State (7) (Fig. 7):
In state (7), the attacker has acquired information on his/her hobby, position
information, and friendship from the social media account name of the target person,
such that it is able to create a targeted e-mail impersonating a friend of the target. In this
state, it is possible to include the private information of the attack target in the mail, and
1336 K. Uehara et al.

it is possible to create a targeted e-mail with high credibility. Furthermore, it is known

that mail sent by a sender impersonating an intimate person is effective to make the
recipient blind [7], hence, the degree of the threat is considered to be high.
• State (10) (Fig. 8):
In state (10), the attacker can create a targeted e-mail like that was a trigger of the
massive information leakage incident [8] that the Japan Pension Service caused in May
2015. In case of this incident, the subject and body of the e-mail is included in the
contents of the business and the executive’s name in which the Japan Pension Service
published on the web page, thus a highly reliable targeted e-mail was created and used.

Fig. 4. Targeted e-mail in state (3) Fig. 5. Targeted e-mail in state (4)

Fig. 6. Targeted e-mail in state in which an Fig. 7. Targeted e-mail in state (7)
attacker has {name, email address, address}

4 Discussion
4.1 Type of Targeted E-mail in Each State
As mentioned in Sect. 3.1, in this paper, we narrowed down the target of targeted e-
mail attack to “specific individuals,” but here we classify “specific individuals” into
 Basic Study on Targeted E-mail Attack Method Using OSINT 1337

Fig. 8. Targeted e-mail in state (10) [8]

three more categories (Table 1). First, the “specific individual” is divided into two
depending on whether or not the attack target is “an individual belonging to a specific
organization.” Then, depending on whether or not the attack target is uniquely specified
(whether the number k of persons who can be an attack target is 1 or 2 or more), the
two are further divided into two. As shown in Table 1, the three types among these are
referred to as “specific individual type,” “carpet bombing type,” and “specific member
type.”
For instance, even if an attacker in state (1) could acquire “Gmail address related to
the name of the target person” by the OSINT activities, it is also possible that it is the
mail address of another person with the same name. In other words, in state (3) it
cannot be said that the attack target person can be uniquely identified (Fig. 9). Such a
situation can arise, for instance, when the attacker’s purpose is “to deceive one of
multiple persons of the same name.” That is, the target type mail in state (3) is sent to
the target of the same name in the same sentence (carpet bombing type).
On the other hand, it can be said that phone numbers and affiliated organizations are
unique information for identifying specific individuals (Fig. 10).3 Therefore, the target
type e-mail in the state (4) and the state (8) aims only at the attack target for the purpose
of the attacker “deceiving a specific person” or “deceiving a specific person belonging
to a specific organization” (Specific personal type, specific member type).
The tendency of the targeted e-mail type in each state is shown in Fig. 11.

3
In the case of a large-scale organization, there may be people with the same first and last name, but
here we consider an organization of moderate scale.
1338 K. Uehara et al.

Table 1. Type of targeted mail

Attack target Number of attack target indicated by k
k=1 k  2
Specific individual Specific individual type Carpet bombing type
Individual belonging to organization Specific member type

Fig. 9. State in which an individual cannot Fig. 10. State in which an individual can be
be uniquely identified uniquely identified

Fig. 11. Type of targeted e-mail in each state

 Basic Study on Targeted E-mail Attack Method Using OSINT 1339

4.2 Use of This Model and Template

It is expected that state transitions in the process of attackers collecting information on
target persons and typing of targeted e-mails created in each state can be utilized as the
following three findings.
First, it is to grasp the depth of the targeted e-mail attack. When a targeted e-mail is
sent to you, the amount of information the attackers are collecting about you can be
estimated. Moreover, in the future, it is possible to predict what type of targeted e-mail
will be sent when attackers’ information gathering progresses. This allows you to pay
attention to the expected target type e-mails, and you can know the possibility that
another attacker will send similar targeted e-mails. Namely, it is considered useful for
risk analysis and advance measures.
Second, it is to grasp what “information about yourself” is disclosed and should be
stopped to prevent attackers from creating highly reliable targeted e-mails. From the
state transition model shown in Fig. 3, for example, it can be seen that it is effective to
refrain from disclosing information in the form of co-occurrence of your name (real
name) and social media account so as not to bring the attacker to the state (7).
Third, it is possible to introduce countermeasures for targeted e-mail attacks
according to the degree of information disclosure of individuals and organizations.
Considering the possibility that targeted e-mail with high credibility may be sent using
more information obtained by OSINT, it is necessary to raise countermeasures against
targeted e-mail attacks.

5 Related Work

Regarding the effectiveness of OSINT in social engineering, studies with respect to

references [3, 9, 10] are being conducted.
Ball et al. mention that there is the possibility that OSINT will be used for social
engineering and criminal activity because the amount of data accumulated all over the
world is exponentially increasing, and further discuss ways to launch spear-phishing
mail attacks on organization employees using OSINT [3]. In the paper, they state that
attackers introduce the OSINT tools for information gathering and describe a method to
use a dedicated tool for spear phishing. Edwards et al. categorized the OSINT data into
two categories, Bootstrap (data used as a trigger for attack) and Accentuator (data
supplementary used to increase the effectiveness of attacks), and they showed how the
data is used in social engineering [9]. They also investigated the extent to which
information such as name of employee, phone number, or e-mail address can be
gathered from OSINT about enterprises responsible for utilities such as water supply,
gas, and electricity. Further, they revealed the threat of social engineering caused by
them.
Silic et al. examined the effectiveness of social engineering using social media for
Fortune 500 companies [10]. Specifically, they utilized OSINT, and created a “fake
social media account” to impersonate an employee of the targeted company, and
became a member of the social media private group composed of regular employees,
then revealed whether it is possible to obtain information from the group. As a result, it
1340 K. Uehara et al.

is clear that employees are easily deceived, vulnerable to social engineering, and that
organizations currently do not have any methods to control over security threats from
social media.
In this paper, we survey multiple OSINT tools, and formulate the information
gathering process by attackers as a “state transition diagram,” focus on targeted e-mail
attacks and concretely explain how to utilize information in each state (to create tar-
geted e-mail). Therefore, this paper complements or reinforces the existing research
above-mentioned.
Research on machine learning and social engineering is also being conducted [11].
Singh et al. propose a learning model in which target of targeted attack is CFO
(Chief Financial Officer) of a company and machine prediction predicts whether person
is “able to be targeted (easy to be cheated)” by machine learning [11]. The learning data
is the age and gender of CFO, the number of followers and Tweets on Twitter, whether
phishing is successful via Twitter or LinkedIn, and so on. By machine learning, it is
possible to identify a person who is targeted (deceived) with 80% accuracy, and it is
clarified that the evolution of social engineering using it.
Regarding automatic generation of targeted e-mail, Iwata et al. propose analysis
methods of received mails for automatic generation of training e-mails in targeted e-
mail attack training [12]. After analyzing the user’s received e-mail BOX, they confirm
what kind of e-mail you trusted and opened, and give a comment such as “It is similar
to an everyday e-mail, but its unnaturalness makes you notice that it is a targeted e-
mail” to the e-mail, then a training e-mail is created. Research conducted by Singh et al.
and Iwata et al. shows that attackers can perform sophisticated social engineering by
utilizing big data technology and AI technology. From now on, it is important to
prepare for sophisticated social engineering and targeted e-mail.

6 Conclusion

In this paper, we formulate the process by which attackers gather information of targets
using the OSINT tools as a state transition model, and categorize targeted e-mails that
attackers can generate in each state.
The obtained findings are used for risk analysis such as grasping the depth of
attack, assuming what type of targeted e-mail may be received by disclosing infor-
mation, and selection of required security measures. Briefly, the pre/post-action mea-
sures mentioned above are expected to be possible.
Here, we mainly analyzed “targeted e-mail attacks by targeting a specific indi-
vidual.” However, we shall further investigate “targeted e-mail attack targeting orga-
nizations” in the future. In addition, although presently only the information that can be
mechanically collected by the OSINT tools was taken into consideration, information
that can be inferred from the information and information that the user unintentionally
made public (for example, such as metadata of the public file) should also be con-
sidered. In the future, we will improve these state transition models by taking this
information into consideration.
 Basic Study on Targeted E-mail Attack Method Using OSINT 1341

References
1. Acquisti, A., Gross, R., Stutzman, F.: Face recognition and privacy in the age of augmented
reality. J. Priv. Confidentiality 6, 1–20 (2014)
2. Rainie, L., Kiesler, S., Kang, R., Madden, M., Duggan, M., Brown, S., Dabbish, L.:
Anonymity, privacy, and security online. Pew Research Center (2013)
3. Ball, L.D., Ewan, G., Coull, N.J.: Undermining-social engineering using open source
intelligence gathering. In: Proceedings of 4th International Conference on Knowledge
Discovery and Information Retrieval (KDIR), pp. 275–280. SciTePress-Science and
Technology Publications (2012)
4. Best, C.: OSINT, the internet and privacy. In: EISIC, p. 4 (2012)
5. INFOSEC: Top Five Open Source Intelligence (OSINT) Tools. https://resources.
infosecinstitute.com/top-five-open-source-intelligence-osint-tools/. Accessed 30 July 2018
6. IntelTechniques.com: Buscador OSINT VM. https://inteltechniques.com/buscador/index.
html. Accessed 3 Aug 2018
7. Chen, S., Fitzsimons, G.M., Andersen, S.M.: Automaticity in close relationships. In: Social
Psychology and the Unconscious: The Automaticity of Higher Mental Processes, pp. 133–
172 (2007)
8. Japan Pension Service: Investigation Result Report on Information Leakage Cases due to
Unauthorized Access (in Japanese). https://www.nenkin.go.jp/files/kuUK4cuR6MEN2.pdf.
Accessed 7 Aug 2018
9. Edwards, M., Larson, R., Green, B., Rashid, A., Baron, A.: Panning for gold: automatically
analysing online social engineering attack surfaces. Comput. Secur. 69, 18–34 (2017)
10. Silic, M., Back, A.: The dark side of social networking sites: understanding phishing risks.
Comput. Hum. Behav. 60, 35–43 (2016)
11. Singh, A., Thaware, V.: Wire Me Through Machine Learning. Black Hat, USA (2017)
12. Iwata, K., Nakamura, Y., Inamura, H., Takahashi, O.: An automatic training system against
advanced persistent threat. In: 2017 Tenth International Conference on Mobile Computing
and Ubiquitous Network (ICMU), pp. 1–2. IEEE (2017)
 Network Location Privacy Protection
with Multi-access Edge Computing

Ping Zhang1 , Mimoza Durresi2(B) , and Arjan Durresi1

1
Department of Computer and Information Science,
Indiana University Purdue University Indianapolis, Indianapolis, IN, USA
{zhang58,adurresi}@iupui.edu
2
Department of Informatics, Mathematics and Statistics,
European University of Tirana, Tirana, Albania
[email protected]

Abstract. Real world incidents show leaking of communication and

location privacy can lead to serious consequence. Many research efforts
have tried to anonymize individual user from aggregated location data.
However, just the communication between the mobile user and their
peers could reveal important privacy of the mobile user, such as location
history. In this paper, we proposed a scalable system built with Multi-
access Edge (MEC) that can hide the mobile user’s network location
and traffic from communication peers. This system creates a dynamic
distributed proxy network for each mobile user to minimize performance
overhead and operation cost.

1 Introduction

Internet access paradigm is rapidly shifting to full mobility. The upcoming 5G

cellular network takes step further that it targets at replacing traditional wired
Internet last hops and thus connecting every device [2,8,12,23]. This trend not
only converts existing Internet end-hosts to mobile hosts, it also extends mobile
Internet audience and enables new business and paradigms However, the adverse
side effect of the full mobile Internet also becomes more prominent. Particularly
network location and identity privacy are facing more challenges since every
Internet end host essentially becomes mobile end host. Thus there are more
data and characteristics exposed to each end host, but protection mechanism
does not improve equally or being addressed by the new access technology. For
example, when a mobile end host connects to another end host, connections are
set up on its exposed public IP address, which is either its actual public Internet
attach point, or a gateway close to its physical location. That means using its
IP address all its peers can identify the approximate geolocation of it.
These collected network location history can be analyzed to profile and iden-
tify individual users and project its future location statistically [14,20]. The
precision can be further improved when combined with other contexts such as
content user access.
c Springer Nature Switzerland AG 2020
L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1342–1352, 2020.
https://doi.org/10.1007/978-3-030-15032-7_112
 Network Location Privacy Protection with Multi-access Edge Computing 1343

The tighter bound of people’s identity to their mobile devices and applica-
tions started to raise lots of privacy concerns. Research communities and industry
have responded: on the service side a number of technologies were proposed to
anonymize identity information when aggregating statistic or providing location-
based service [11]; on the client side mobile users are promoted to give explicit
permission to applications of using the device’s geographic location, contact list,
storage, etc. in order to protect users’ privacy [4]. However, network access is
usually granted without explicit approval since most applications need Internet
access to function. Through either private VPN service or multiple relay net-
works like Tor [7], Internet users can hide where they are when communicate
with peer hosts or websites and hide whom they talk to from ISP. However,
today’s VPN service does not provide extra support for mobility and adds per-
formance overhead as traffic always goes through a static relay end host.
Fortunately, the new ubiquitous computation capacity that is provided by
Multi-access Edge Computing (MEC) [1,3] offers the opportunity to protect
network location privacy with low performance overhead and operation cost.
Compared to public Cloud Computing which aggregates computation and data
at centralized data centers, MEC utilizes the computation and storage capabil-
ity of ubiquitous mobile edge networks. An application running in MEC is much
closer to end customers and thus have less latency and higher availability. Mobil-
ity Support System (MSS) [24–27] is a system proposed to protect mobile users’
network privacy while enhancing mobility support without sacrificing perfor-
mance and costs. It overcomes VPN’s inherent overhead and management issue,
by automatically picking optimal proxy for each connection and also providing
a bridge for better generic mobility support. MSS achieves that by leveraging
distributed proxies simultaneously while “pushing” these proxies closer to com-
munication destination to minimize performance penalty and improve privacy.
MSS leveraging MEC can elevate privacy protection strength and efficiency
to another level. It can protect mobile Internet end host from revealing their
actual network address and history with the MEC services IoT devices already
benefit from. In this paper, we will present our design of incorporating MEC
into MSS.

2 Related Works
Privacy issue caused by mobility, especially location, has been well studied. De
Montjoye et al. [6] found that from a large set of anonymous movement data,
using four data points of hourly data can identify 95% unique users. Given the
identified movement pattern of this identified user, they can even construct a
history of this user’s locations from the anonymized data set. Ma et al. reached
the same conclusion [14].
The TOR network [7] is an overlay network designed to allow an Internet
end-host to secretly communicate with peer end-hosts without exposing either
content, source, or destination to traffic carrier.
1344 P. Zhang et al.

One popular research area is to protect against user profiling from Location
Based Service (LBS) while can still use it [5]. Wernke et al. [21] surveyed dif-
ferent identify protection types and common mechanisms to protect and attack
privacy. Researches also show obfuscated location data while can improve loca-
tion privacy but cannot stop adversary to infer relative precise Point Of Interests
(POI) [16].
Another focused area is to increase anonymity of collected user location
data [11], limiting shared location information, or evaluating privacy exposure
level before sharing location data [4]. On the other hand, there are researches
pointed out that because human mobility trace is very unique [6], even com-
pleted anatomized data can still be used to extract patterns and identify indi-
viduals [14,20].
Standards of MEC are actively developed by cellular industry and standard-
ization group [9]. It is considered as one building block of 5G network [10,15,18],
and its capability of offloading computation from core network is especially
emphasized [19]. Roman et al. [17] studied and summarized common security
threats and challenges of multiple paradigms used in mobile edge computing
and mobile cloud computing, and pointed out needed improvement of security
design.

3 Design of MSS with MEC

The original MSS design [24,25] is designed to protect network privacy with
very low latency penalty. Compared to traditional VPN, MSS not only has per-
formance advantage but also overcomes single server’s inherent limitation such
as single point of failure and bandwidth bottleneck. It achieves those through
a novel paradigm of allocating distributed proxies close to every peer node. On
the other hand, MSS accomplish these benefits with comparable or less opera-
tion cost by sharing resources among users and connections, and dynamically
adjusting fleet size and distribution.
The incorporation of MEC provides MSS capability to expand geolocation
presence greatly and thus reduce performance overhead. It also provides oppor-
tunities for MSS to reduce system operation cost. With MEC MSS elevates its
network paradigm from the edge of the network to network core.

3.1 MSS Basic Design

MSS consists of three different parties: Mobile Node, Peer Node, and Mobility
Service Provider (MSP).
• Mobile Node is the mobile device hosting mobile user’s identity and applica-
tions. Each Mobile Node has an MSS agent deployed, which handles incoming
and outgoing traffics. All direction connections managed by MSS agent are
identity-based, either to proxy or another Mobile Node.
• Peer Nodes are Internet hosts on the other end of traffic. They can be a
web server, ordinary host, or another Mobile Node equipped with MSS agent.
 Network Location Privacy Protection with Multi-access Edge Computing 1345

• MSP manages a fleet of servers, called Virtual Routers, that are dynami-
cally allocated from Cloud service provider’s data centers. Proxies are a ten-
ant of Virtual Router: each Virtual Router hosts multiple Proxies relaying
different Mobile Nodes’ traffic, up to Virtual Router’s resource limit.
When a Mobile Node wants to connect to a Peer Node, MSS agent on the host
will request a Proxy from MSP control plane. This Proxy will be allocated at a
network location as close as possible to the Peer Node, and it connects directly to
the Peer Node. The traffic between the Mobile Node and Peer Node is tunneled
through Proxy using identity-based connection [13] between Mobile Node and
Proxy. The proxy will be the exposed network address and won’t change no
matter where Mobile Node moves to. Therefore Mobile Node’s real network
address and movement are entirely hidden from the Peer Node. When Peer
Node is an ordinary Internet host and connection is bound on IP address, MSS
provides additional mobility support that traditional network protocols can work
transparently without interruption even if Mobile node is temporarily offline
or changes network addresses. All of these benefits are achieved without much
network latency overhead because Mobile Node’s movement will never deviate
its route from optimal one much, including scenarios Mobile Node performing
vertical handover which could change network location dramatically.
To receive MSS’s privacy protection and mobility enhancement, Mobile
Node’s user only needs to sign a single contract with an MSP, and this MSP
becomes the only one knowing Mobile Node’s network whereabouts. MSPs are
independent of infrastructure provider (such as ISP), and any MSP can have a
global presence, though their capability of where and how much they can allocate
Virtual Router will determine their services’ performance.
There are two levels of fleet management operated by MSP: Proxy manage-
ment and Virtual Router management. Each Mobile Node’s Proxy fleet forms an
overlay control plane to monitor and aggregate metrics, and delegate signaling
between the Mobile Node and MSP control plane. In addition to on-demand
Proxy allocation, MSP also periodically adjusts Proxy distribution to comply to
user’s Service Level Agreement (SLA). On a higher and broader level, MSP con-
tinuously monitors resource utilization of its Virtual Routers, and dynamically
allocate or remove capacity to maintain a healthy availability. This is the key to
drive down operating cost.

3.1.1 Privacy
Distance, λ, is defined as how different the exposed network address is in term
of relative geographical distance, which is derived from the mapping of a network
address to registered geographical locations. In general the larger the distance the
better privacy. We use function dis(x, y) to represent the approximate location
between network attach point x and y. Then for a given combination of IoT
device (m), remote server (n), and Proxy Server (p), location privacy λm, n, p
can be evaluated as:
dis(m, p)
λm, n, p =
dis(m, n)
1346 P. Zhang et al.

3.1.2 Cost
The Cost, χ, is total operation cost of running all proxies and control plane
controllers, plus traffic/bandwidth cost if applies.

C= costproxy + costcontroller + costtraf f ic

Generally, the controller is relatively fixed fleet, and much smaller compared
to the proxy fleet in all three setups. When implemented distributed it can be
hosted on existing Proxies. Therefore we generally consider it as a constant.
Traffic can be either absorbed into Proxy cost in the form of flat-rate pricing.
Single proxy cost is determined by its location, cloud service provider and cloud
host type, and bandwidth Proxy node provides. As result for a Mobile Node its
overall operation cost is sum of all hour charge of Proxy Servers and bandwidth
cost, which MSS optimize the former.

χM N = χproxy ∗ hourproxy + χtraf f ic

3.1.3 Performance
Communication latency and bandwidth are two most perceptible measurement
of mobile device’s experience. Since generally, the proxy can always provide
enough bandwidth for a mobile device, MSS does not consider bandwidth as a
variance for performance optimization, but rather an SLA requirement for MSS
control plane to choose Proxy. Round Trip Time (RTT) of traffic between the
mobile device and its remote server directly and via Proxy are the significant
performance difference MSS optimizes for. It is defined as proxy overhead, the
difference between direct connect RTT and via proxy RTT, that smaller means
better performance.

μ = RT T (m, p) + RT T (p, n) − RT T (m, n)

3.1.4 Score
Score, S, is one aggregated metric unifying performance and privacy metric,
based on five inputs: mobile device location, remote server location, Proxy can-
didate location, acceptable minimum performance, and acceptable minimum pri-
vacy. The value range of Score is between 100 and −100, while any value below
or equal 0 means the candidate does not satisfy minimum requirement. k is
weighing constant that the sum of all k equals to 100%.
maxμ − μ λm, n, p − minλ
S= ∗ kμ + ∗ kλ − χproxy ∗ kχ
maxμ + μ λm, n, p + minλ

3.2 Incorporating MEC

With MEC MSS can even find “next-door” proxy for every Peer Node. These
RANs are also generally owned by the same provider and connected with globally
 Network Location Privacy Protection with Multi-access Edge Computing 1347

managed private links, which can provide better QoS than the public Internet.
Additionally, those network controllers and routers which implement MEC are
originally specialized of handling network traffic. MSS can leverage MEC to
augment performance and reduce operation cost. To incorporate MEC, MSS
designs are changed in the following areas:

1. Proxy application. Proxy is changed that can directly run as MEC applica-
tion where VM is abstracted out from platform interface. Otherwise, Virtual
Router will still be allocated as ordinary VM on mobile edge host. Both their
functionality and role remains the same.
2. Specialized Proxy. When MEC allows access to RAN especially managing
customize routing rules, Proxy is replaced by routing table entries and packet
repeater. In this case, Virtual Router will reduce to an application listening
to MSP signals and accordingly maintaining routing rules and policies of the
network device it runs on. This mode reduces operation cost, and can also
improve performance due to less traffic processing.
3. Network representation. In the simple mode where MSP does not have deep
integration with the cellular network, MSP just treats MEC Edge Clouds as
the equivalent of public Cloud data centers with less capacity. When MSP
can have deeper integration, such as routing traffic within the cellular net-
work, MSP’s network graph distinguishes cellular network and the public
Internet where two network connects through multiple Internet gateways. In
this mode, any connection cross two network boundary must go through one
selected Internet gateway, and this gateway implements Proxy if privacy met-
ric meets the requirement. Otherwise proxy will have to allocated in a public
Cloud data centers and gateway will be chosen at the place with minimum
cost only.
4. Proxy allocation algorithm. MSP gives priority to MEC when allocating
Proxy, but will retreat to public Cloud when the region’s MEC is not suffi-
cient either in availability or performance. So a Mobile Node’s Proxies could
consist of MEC routing rules, MEC application, and public Cloud hosted
ones.

When MSS can have deeper integration with MEC (or cellular provider imple-
ments MSS), it gains more benefits besides the operation cost saving. For exam-
ple, when MSP use MEC Edge Cloud as ordinary public Cloud, traffic from
Mobile Node will first exit RAN into public Internet, go into another RAN to
reach Proxy, exit second RAN, and finally reach Peer Node. As a comparison,
a deeper integrated MSP will directly route traffic within cellular data network
and exit at an Internet gateway very close to Peer Node. On the other hand,
the cellular provider has inherent advantages of becoming an MSP. For exam-
ple, mobility Management is already one core functionality of a cellular network.
MSS’s functionality such as tracking Mobile Node’s location, looking up Mobile
Node, replicating mobility information across the network, etc. can be easily
adapt from cellular network’s similar functions. It also benefits MSS users since
they do not even need to share their real network location to anyone else.
1348 P. Zhang et al.

4 Simulation and Result Analysis

To validate our design and algorithm we setup simulation trying to mimic real-
world topology and traffic pattern. Simulations were conducted on a topology
generated by Inet3 [22] topology generation tool. With it, we created 5000 nodes
on a 2D plain with dimension 10000 of 10000 to emulate a simplified view of the
Internet. Each node represents a public Internet subnet that is publicly routable
and reachable. Their coordinates represent their geographic location, while hops
between two nodes represent the network distance. Due to the mobility nature,
every event in the simulation has a timestamp, and all events are merged onto the
same timeline during one simulation iteration. The Maximum event timestamp
is 2000.
During this period, Mobile Node performs a random walk in a 2D plane and
connects to the geographic nearest node as its Internet attaching point. This
simulates an Internet mobile user roams across different networks and uses the
nearby gateway as its exposed Internet address. Each Mobile Node randomly
moves 20 times at random timestamp. For each Mobile Node, we create a pool
of 20 Peer Nodes, and their locations are randomly selected from the 5000 nodes.
These are Peer Nodes either initiating connections to Mobile Node or receiving
connection coming from Mobile Node.
Each Peer Node can have 1 to 10 connections during the simulation period,
and timestamp is also uniformly random generated. On the other hand, Mobile
Nodes randomly nominate 3 locations for hosting listening proxies to receive
an incoming connection. This design is trying to simulate one ordinary Internet
mobile use’s average day access pattern. We generate traces of 100 Mobile Nodes
and compare the average of Mobile Node’s metrics, to counter the potential skew
caused by accidental non-uniform randomness.
We simulate to compare MSS versus other models. The comparison is based
on the view of individual customer’s privacy and performance metrics when one
customer moves while communicates with its peers. Models include:

1. Typical VPN user. We simulate by selecting one static Proxy Server selected
around the middle of the optimal router between the Mobile Node and its
first Peer Node. It does give favor to first Peer Node, but statistically, it does
not make a difference since we have 19 other Peer nodes for simulation.
2. Typical cell data mobile user. We simulate by keeping a proxy server that
follows Mobile Node move (randomly picked within two hops of Mobile Node’s
attaching point, and re-selects when Mobile Node’s access point is moving
away from attaching point). This proxy server simulates the Internet Gateway
which relays a cellular data network user’s IP packets.
3. MSS based on public Cloud only. Among the 5000 nodes, we randomly select
10 “seed” nodes that are not too close to each other to represent public Cloud
service provider’s data centers. We randomly “grow” the data center footprint
by tainting nodes directly connect to it in 2 to 3 hoops, to simulate public
cloud data center covering large fleets of servers and owning a large amount
of IP addresses.
 Network Location Privacy Protection with Multi-access Edge Computing 1349

4. MSS based on MEC. Among the 5000 nodes, we exclude leaf nodes (those
have only one connection to neighbors) and core nodes (those have the highest
number of connections to neighbors), and all the rest are the candidate of
MEC nodes. This simulates the ubiquitous presence of cellular data network
and potential coverage of MEC.

Performance overhead is measured as the extra hops compared to optimal

route, applying our Performance Equation. Figure 1a shows Cumulative Distri-
bution Function (CDF) of performance overhead of four competitors. “VPN”
apparently has largest performance penalty as all traffics go through a static
point no matter when Mobile Node moves to. On the other hand “Cellular Data
Network” has the lowest performance overhead since the data path is almost
always near optimal as the Internet Gateway is always nearby. However, note
that this low overhead accompanied with multiple dysconnectivity of existing
connections as each change of Internet Gateway will disturb ongoing traffic and
connection. “MSS” in two configurations show great performance overhead that
is very close to “Cellular Data Network” without sacrificing neither connectiv-
ity nor privacy. Figure 1b presents a point-in-time performance overhead among
four. It is clear that VPN always has high performance overhead cost. MEC based
MSS generally has lower performance overhead comparing to Public Cloud based
MSS. “Cellular Data Network” performance overhead decreased in last one third
time range mostly due to Mobile Node not moving much during that period.

9000 300
VPN
Cellular Data Network
MSS with Public Cloud
8000 VPN MSS with MEC
Cellular Data Network 250
MSS with Public Cloud
7000 MSS with MEC
Network Performance Overhead (CDF)

Network Performance Overhead

6000 200

5000
150
4000

3000 100

2000
50
1000

0 0
0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 1600 1800 2000
Time Time

(a) Overall Comparison (b) Per Sampling Point Comparison

Fig. 1. Performance overhead comparison

Location privacy is measured based on our location privacy evaluation equa-

tion. Figure 2a displays the average location privacy metrics. VPN unsurprising
dominate as it only exposed the VPN address no matter where the Mobile Node
or Peer Node, and in general Mobile Node would select a VPN server far from its
actual location. MSS in two configurations perform good, and MEC based one
beat public cloud-based one. Cellular Data Network generally doesn’t protect
Mobile Node’s location privacy much, unless in rural, large cell areas where an
Internet Gateway covers a large area.
1350 P. Zhang et al.

1.2 1.4
VPN VPN
Cellular Data Network Cellular Data Network
MSS with Public Cloud MSS with Public Cloud
MSS with MEC MSS with MEC
1.2
1

1
Location Privacy (Average)

0.8

Location Privacy (Min)

0.8

0.6

0.6

0.4
0.4

0.2
0.2

0 0
0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 1600 1800 2000
Time Time

(a) Average (b) Min

Fig. 2. Location privacy comparison

The overall signaling cost, including outgoing (setting up a connection to a

Peer Node), lookup (incoming connection from Peer Node), and update (update
all Proxies when Mobile Node changing attaching point) are illustrated in Fig. 3a.

Fig. 3.

4500 200
MSS with Public Cloud 1
Cellular Data Network MSS with Public Cloud 4
MSS with Public Cloud 1 MSS with Public Cloud 16
4000 MSS with Public Cloud 4 MSS with Public Cloud 32
MSS with Public Cloud 16 MSS with MEC 1
MSS with Public Cloud 32 MSS with MEC 4
3500 MSS with MEC 1 MSS with MEC 16
Network Performance Overhead (CDF)

MSS with MEC 4 150 MSS with MEC 32

MSS with MEC 16
Network Performance Overhead

MSS with MEC 32

3000

2500
100
2000

1500

50
1000

500

0 0
0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 1600 1800 2000
Time Time

(a) Overall Comparison (b) Per Sampling Point Comparison

Fig. 4. Scaling performance overhead comparison

 Network Location Privacy Protection with Multi-access Edge Computing 1351

We further experimented with the different maximum allowed Proxies, i.e.,

maximum 1, 4, 16, and 32 Proxies for each Mobile Node in the simulation.
Figure 4a shows that when allowing more Proxies privacy metric improved. On
the other hand, the percentage of improvement starts to become less for our
simulation setup when allowed maximum Proxies is more than 4, and became
less significant when allowed Proxies increased to 16. Our further analysis shows
the sweet spot is around 8, and the reason is that 8 Proxies are about large
enough to cover the 20 Peer Nodes we selected for each Mobile Node.

5 Conclusions

In this paper, we discussed privacy issues a communication connection of Inter-

net mobile user could expose and corresponding attack model. Based on that
we proposed a design to protect it with a dynamic distributed Proxy network,
built upon our previously designed Mobility Support System. An Internet user
can receive ubiquitous Internet mobility support while network location privacy
is well protected, with a negligible performance overhead and minimized oper-
ation cost. Our preliminary simulation results show that our design meets our
expectation, and there is still headroom to optimize for performance and cost in
the future.

Acknowledgements. This work was partially supported by the National Science

Foundation under Grant No. 1547411.

References
1. Ahmed, A., Ahmed, E.: A survey on mobile edge computing. In: 2016 10th Inter-
national Conference on Intelligent Systems and Control (ISCO), pp. 1–8, January
2016
2. Barolli, L., Honma, Y., Koyama, A., Durresi, A., Arai, J.: A selective border-
casting zone routing protocol for ad-hoc networks. In: 2004 Proceedings of the
15th International Workshop on Database and Expert Systems Applications, pp.
326–330 (2004)
3. Beck, M.T., Werner, M., Feld, S., Schimper, T.: Mobile edge computing: a taxon-
omy (2015)
4. Boutsis, I., Kalogeraki, V.: Location privacy for crowdsourcing applications. In:
Proceedings of the 2016 ACM International Joint Conference on Pervasive and
Ubiquitous Computing, UbiComp 2016, pp. 694–705. ACM, New York (2016)
5. Damiani, M.L.: Privacy enhancing techniques for the protection of mobility pat-
terns in LBS: research issues and trends. In: European Data Protection: Coming
of Age, pp. 223–239. Springer, Dordrecht (2013)
6. de Montjoye, Y.-A., Hidalgo, C.A., Verleysen, M., Blondel, V.D.: Unique in the
crowd: the privacy bounds of human mobility. Sci. Rep. 3 (2013). Article number
1376
7. Dingledine, R., Mathewson, N., Syverson, P.: Tor: the second-generation onion
router. In: Proceedings of the 13th USENIX Security Symposium (2004)
1352 P. Zhang et al.

8. Durresi, M., Durresi, A., Barolli, L.: Sensor inter-vehicle communication for safer
highways. In: 19th International Conference on Advanced Information Networking
and Applications, AINA 2005, vol. 2, pp. 599–604 (2005)
9. ETSI: Multi-access edge computing
10. Garvelink, B.: Mobile edge computing: a building block for 5G (2015)
11. Gedik, B., Liu, L.: Protecting location privacy with personalized k-anonymity:
architecture and algorithms. IEEE Trans. Mob. Comput. 7(1), 1–18 (2008)
12. Harold Owens, I.I., Durresi, A.: Video over software-defined networking (VSDN).
Comput. Netw. 92, 341–356 (2015)
13. Laganier, J., Koponen, T., Eggert, L.: Host identity protocol (HIP) registration
extension. RFC 5203 (experimental), April 2008
14. Ma, C.Y.T., Yau, D.K.Y., Yip, N.K., Rao, N.S.V.: Privacy vulnerability of pub-
lished anonymous mobility traces. IEEE/ACM Trans. Netw. 21(3), 720–733 (2013)
15. Nunna, S., Kousaridas, A., Ibrahim, M., Dillinger, M., Thuemmler, C., Feussner,
H., Schneider, A.: Enabling real-time context-aware collaboration through 5G and
mobile edge computing. In: 2015 12th International Conference on Information
Technology - New Generations, pp. 601–605, April 2015
16. Primault, V., Mokhtar, S.B., Lauradoux, C., Brunie, L.: Differentially private loca-
tion privacy in practice. CoRR, abs/1410.7744 (2014)
17. Roman, R., Lopez, J., Mambo, M.: Mobile edge computing, fog et al.: a survey and
analysis of security threats and challenges. Future Gener. Comput. Syst. 78(Part
2), 680–698 (2018)
18. Tran, T.X., Hajisami, A., Pandey, P., Pompili, D.: Collaborative mobile edge com-
puting in 5G networks: new paradigms, scenarios, and challenges. IEEE Commun.
Mag. 55(4), 54–61 (2017)
19. Tran, T.X., Pompili, D.: Joint task offloading and resource allocation for multi-
server mobile-edge computing networks. CoRR, abs/1705.00704 (2017)
20. Tsoukaneri, G., Theodorakopoulos, G., Leather, H., Marina, M.K.: On the infer-
ence of user paths from anonymized mobility data. In: 2016 IEEE European Sym-
posium on Security and Privacy (EuroS&P), pp. 199–213, March 2016
21. Wernke, M., Skvortsov, P., Dürr, F., Rothermel, K.: A classification of location
privacy attacks and approaches. Pers. Ubiquit. Comput. 18(1), 163–175 (2014)
22. Winick, J., Jamin, S.: Inet-3.0: internet topology generator. Technical report,
Department of Electrical Engineering and Computer Science, University of Michi-
gan (2002)
23. Xie, W., Goyal, M., Hosseini, H., Martocci, J., Bashir, Y., Baccelli, E., Durresi, A.:
Routing loops in DAG-based low power and lossy networks. In: 2010 24th IEEE
International Conference on Advanced Information Networking and Applications,
pp. 888–895 (2010)
24. Zhang, P., Durresi, A.: Cloud aided internet mobility for privacy protection. In:
IEEE International Conference on Communications, ICC 2017, Paris, France, 21–
25 May 2017, pp. 1–6 (2017)
25. Zhang, P., Durresi, A., Jain, R.: Cloud aided internet mobility. In: Proceedings
of IEEE International Conference on Communications, ICC 2013, Budapest, Hun-
gary, 9–13 June 2013, pp. 3688–3693 (2013)
26. Zhang, P., Durresi, M., Durresi, A.: Mobile privacy protection enhanced with multi-
access edge computing. In: 2018 IEEE 32nd International Conference on Advanced
Information Networking and Applications (AINA), pp. 724–731 (2018)
27. Zhang, P., Durresi, M., Durresi, A.: Multi-access edge computing aided mobility
for privacy protection in internet of things. Computing 1–14 (2018)
Author Index

A Balzano, Walter, 1217

Aalsalem, Mohammed Y., 977 Barolli, Admir, 366
Abbasi, Raza Abid, 26, 39, 53, 82 Barolli, Leonard, 108, 156, 238, 251, 366, 397
Abdullah, Muhammad, 95 Barreto, Priscila Solis, 655
Abid, Mohamed, 214 Ben Chaabene, Seif, 619
Adekotujo, Akinlolu Solomon, 1133 Ben Yahia, Sadok, 619
Ahmad, Noman, 95 Benjemmaa, Afef, 963
Ahmad, Waleed, 26, 53 Berradia, Tahar, 341
Akagi, Hideyuki, 775 Besrour, Amine, 1228
Akhtar, Muhammad, 39 Bhatia, Ashutosh, 325, 1254, 1279
Albuquerque, Natália Porfírio, 559 Bistarelli, Stefano, 546
Alcarria, Ramón, 1059 Bondan, Lucas, 376
Alchieri, Eduardo, 655, 859 Bordel, Borja, 1059
Almiani, Muder, 846 Borges, Felipe, 859
Alves Junior, Joilson, 690 Bouallegue, Ridha, 751
Amanulla, 82 Boujnah, Noureddine, 751
Amato, Alessandra, 1217 Boukhalfa, Fouzi, 286
Amersho, Betsegaw Lemma, 1047 Butt, Ayesha Anjum, 14, 67
Amir, Muhammad, 53 Bylykbashi, Kevin, 156
Amraoui, Noureddine, 1228
Arahori, Yoshitaka, 470 C
Aritsugi, Masayoshi, 1291 Caetano, Marcos F., 859
Aseeri, Mohammed, 214 Cai, Yun-Zhan, 725
Awan, Tayyab, 53 Calsavara, Alcides, 523
Ayadi, Aya, 214 Chan, Hau, 1106
Ayadi, Rami, 214 Chand, Annas, 14, 95
Ayed, Mounir Ben, 963 Chauhan, Ankit, 1240
Ayub, Nasir, 1 Chen, Chin Kang, 571
Azzouna, Nadia Ben, 341, 1157 Chen, Huiping, 1106
Chen, Lo-An, 725
B Cheng, Feng, 1240
Baba, Kensuke, 265 Cheng, Kai, 1147
Baba, Takahiro, 265 Chi, Hsuan, 895
Babbar-Sebens, Meghna, 1169 Chiang, Meng-Shu, 737
Bahrami, Mehdi, 419 Choudhary, Shridhar, 419

© Springer Nature Switzerland AG 2020

L. Barolli et al. (Eds.): AINA 2019, AISC 926, pp. 1353–1357, 2020.
https://doi.org/10.1007/978-3-030-15032-7
1354 Author Index

Choukair, Zièd, 431 Gomes, Demis, 1011

Christodoulopoulos, Christos, 988 Goncalves, Glauco, 1011
Cozzolino, Giovanni, 1217 Gondow, Katsuhiko, 470
Cui, Baojiang, 166 Goto, Kenta, 584
Cuka, Miralda, 238, 251 Granville, Lisandro Zambenedetti, 376, 485
Graupner, Hendrik, 1240
D Grüner, Andreas, 200
da Rocha, Helbert, 690 Gudino, Lucy J., 677
da Silva, Marcus Vinicius Brito, 485 Guegan, Loic, 1047
Dao, Duy-Tuan, 737
Dao, Thanh Chung, 952 H
Date, Susumu, 1024 Hadded, Mohamed, 286
de Araújo, Joseana Macêdo Fechine Régis, 559 Hagen, Svein Thore, 120
de Carvalho Ferreira, Gabriel, 655 Hanif, Murtaza, 67
de Oliveira Silva, Flávio, 643 Hansdah, R. C., 1254
de Souza Pereira, João Henrique, 643 Haribabu, K., 325
Delporte-Gallet, Carole, 272 Hartanto, Rudy, 458
Dhurandher, Sanjay Kumar, 702, 713 Hayashibara, Naohiro, 535
Dinesh Ram Kumar, T., 325, 1254 Hegglid, Gunne John, 120
Do, Ba Lam, 952 Heimfarth, Tales, 446
Dong, Fan, 313 Hernandes, Sediane C. L., 523
dos Santos, Carlos R. P., 376 Hirooka, Takumu, 535
Duarte Jr., Elias P., 376 Huang, Chung-Ming, 737
Duolikun, Dilawaer, 132, 144, 225 Hussain, Farookh Kadeer, 313
Durr-e-Adan, 977 Hussain, Farookh Khadeer, 1000
Durresi, Arjan, 1169, 1342
Durresi, Mimoza, 1342 I
Dutta, Kamlesh, 188 Iijima, Yusuke, 512
Dutta, Sabyasachi, 1266 Ikeda, Daisuke, 265
Ikeda, Makoto, 156, 238, 251
E Ikram, Mohammed A., 1000
El Ayeb, Walid, 431 Islam, Md. Saiful, 927
Elmazi, Donald, 156, 238, 251 Islam, Muhammad Talha, 14
Endo, Patricia Takako, 1011 Ito, Fumi, 1193
Enokido, Tomoya, 108, 132, 144, 225 Izumi, Kiyoshi, 301, 940

F J
Faisal, Hafiz Muhammad, 26, 39, 82 Jacobs, Arthur Selle, 485
Fan, Jiashi, 1106 Javaid, Nadeem, 1, 14, 26, 39, 53, 67, 82, 95,
Fatima, Aisha, 14, 977 977
Fauconnier, Hugues, 272 Jitsunari, Yuma, 470
Feng, Yaokai, 458
Ferreira, Leylane, 1011 K
Fiorese, Adriano, 834 Kanzaki, Akimitsu, 775
Frunza, Marius-Cristian, 810 Karmakar, Kallol, 1303
Fujita, Masahiro, 1329 Kasliwal, Mehul, 677
Kato, Yuka, 1193
G Kaur, Davinder, 1169
Garcia, Vinícius F., 376 Kawaguchi, Hiryu, 388
Gayvoronskaya, Tatiana, 200 Kawakami, Tomoya, 787
Ge, Mouzhi, 1094 Kawauchi, Kiyoto, 1329
Ghodhbane, Jihed, 751 Kayes, A. S. M., 498
Ghorbel, Oussama, 214 Kejriwal, Rajas, 677
Giacomin, João Carlos, 446 Kelner, Judith, 1011
Author Index 1355

Khan, Inam Ullah, 95 Mujeeb, Sana, 1

Khan, Raja Farhat Makhdoom, 67 Mukaiyama, Kohei, 1329
Khan, Sajjad, 26, 53, 82 Muramatsu, Akinori, 1291
Khan, Wazir Zada, 1 Mussadaq, Fahad, 39
Khan, Zahoor Ali, 14, 26, 39, 95 Mustafin, Ilgiz, 810
Khattak, Muhammad Umar, 1 Myrizakis, George, 1181
Kobayashi, Akio, 301, 512
Kobayashi, Kou, 512 N
Kohana, Masaki, 301, 512 Nagaraju, Shamanth, 677
Kortas, Nawel, 1034 Nakamura, Shigenari, 108, 132
Kosuke, Kaneko, 883 Nandi, Sukumar, 821
Kouichi, Sakurai, 883 Nandi, Sunit Kumar, 821
Koyama, Akio, 1071 Ngaffo, Armielle Noulapeu, 431
Ksantini, Riadh, 1228 Nguyen, Binh Minh, 952
Kunugi, Yuya, 1071 Nguyen, Kinh, 498
Nishigaki, Masakatsu, 1329
L Nishikawa, Hiroki, 1329
Lee, JooYoung, 810, 1133 Nishino, Hiroaki, 584
Leu, Fang-Yie, 571 Nissato, Masahiro, 407
Liang, Xiaobing, 166 Nobre, Jéferson C., 376
Lin, Chu-Hsing, 895, 904
Lin, Sian-Jhe, 725 O
Liu, Yi, 156 Ogiela, Lidia, 176, 181
Longo, Luca, 1133 Ogiela, Marek R., 176
Loo, Jonathan, 762 Ogiela, Urszula, 181
Loukides, Grigorios, 1106 Ohkawa, Takeshi, 407
Ltifi, Hela, 963 Ohmori, Motoyuki, 667
Lu, Hsiao-Wen, 904 Okamoto, Shusuke, 397
Oma, Ryuji, 132
M Ootsu, Kanemitsu, 407
Ma, Qian, 166 Orgerie, Anne-Cécile, 1047
Magableh, Basel, 846 Osawa, Kyohei, 940
Mahato, Dharmendra Prasad, 188 Ouechtati, Hamdi, 1157
Mapp, Glenford, 762 Ozawa, Eri, 1193
Marcuzzo, Leonardo C., 376
Matsui, Kanae, 1205 P
Matsui, Yuki, 1024 Pacheco, Luis, 859
Matsumoto, Satoru, 799 Panja, Somnath, 1266
Matsuo, Keita, 156, 251 Payne, Christian, 914
Matsushima, Hiroyasu, 940 Pekar, Adrian, 353
Mazzante, Gianmarco, 546 Pellenz, Marcelo Eduardo, 523, 690
Mazzara, Manuel, 1133 Penna, Manoel C., 523
Mbarek, Bacem, 1094 Penna, Manuel C., 690
Meinel, Christoph, 200, 1240 Pereira Júnior, João Eurípedes, 643
Melcher, Elmar Uwe Kurt, 559 Perkusich, Ângelo, 559
Micheletti, Matteo, 546 Petrakis, Euripides G. M., 988, 1082, 1181
Miyata, Naoki, 667 Pfitscher, Ricardo José, 485
Monshizadeh, Shohreh, 120 Pinigin, Arseny, 1133
Monteiro, Tania L., 690 Pitner, Tomas, 1094
Moreira, Andre, 1011
Moscato, Francesco, 1217
Mostarda, Leonardo, 546 Q
Mühle, Alexander, 200 Qasim, Umar, 67
Muhlethaler, Paul, 286 Quinson, Martin, 1047
1356 Author Index

R Souissi, Ilhem, 341

Rafique, Muhammad Zeeshan, 67 Souza, Giovanni V., 376
Rahayu, Wenny, 498 Spohn, Marco Aurélio, 559
Rathee, Sandhya, 325 Sreejith, V., 677
Raynal, Michel, 272 Srinivasan, Sridhar, 595
Raza, Muhammad, 1000 Studiawan, Hudan, 914
Riha, Zdenek, 1279 Subramaniyan, Archana, 631
Rivera, Samuel J., 1169 Sugiyama, Hiroki, 407
Rosa, Pedro Frosi, 643 Sukmana, Muhammad I. H., 1240
Rosendo, Daniel, 1011 Sultana, Tanzeela, 977
Rosiyadi, Didi, 571 Sumioka, Motoshi, 419
Ruan, Mona B. H., 353 Sun, Bang, 166
Susanto, Heru, 571
S Suzuki, Hiroyuki, 1071
Sadok, Djamel, 1011 Suzuta, Ichiroh, 667
Said, Lamjed Ben, 341, 1157
Sakai, Keiya, 1205
T
Sakaji, Hiroki, 301, 512, 940
Takano, Yasunao, 301, 388, 512
Sakamoto, Shinji, 366, 397
Takizawa, Makoto, 108, 132, 144, 181, 225,
Sakuraba, Akira, 1123
238, 251, 366, 702
Sakurai, Kouichi, 458, 1266
Taniar, David, 927
Sakuta, Hiroshi, 388, 512
Teranishi, Yuuichi, 787
Sánchez, Álvaro, 1059
Thiruvasakan, Lakshmi Priya, 762
Sánchez-de-Rivera, Diego, 1059
Thomas, Ian, 419
Sandhu, Jasminder Kaur, 188
Tiezzi, Francesco, 546
Santos, Guto Leoni, 1011
Torkura, Kennedy A., 1240
Santosa, Paulus Insap, 458
Tsai, Meng-Hsun, 725
Sato, Fumiai, 607
Tsakos, Konstantinos, 1082
Sato, Goshi, 1123
Tu, Doan Quang, 498
Sato, Shingo, 1316
Tupakula, Uday, 1303
Schaeffer-Filho, Alberto E., 376
Seah, Winston K. G., 353
U
Shabbir, Shaista, 977
Ubaid, Ayesha, 313
Shagdar, Oyunchimeg, 286
Uchida, Noriki, 1123
Sharma, Nabin, 1000
Uehara, Kota, 1329
Sharma, Subodh, 883
Uehara, Minoru, 872
Shen, Bojie, 927
Ur Rehman Khan, Sajawal, 82
Shibata, Yoshitaka, 1123
ur Rehman, Shujat, 82
Shikata, Junji, 1316
Uslu, Suleyman, 1169
Shimada, Takashi, 940
Shimojo, Shinji, 787, 1024
Shota, Johjima, 883 V
Shuja, Sahibzada Muhammad, 67 Valadares, Dalton Cézane Gomes, 559
Siddavaatam, Richa, 713 Varadharajan, Vijay, 1303
Singh, Jagdeep, 702 Vieira Jr., Paulo, 834
Singh, Nagendra Pratap, 188 Vien, Quoc-Tuan, 762
Singh, Pranav Kumar, 821 Vishvakarma, Deepak Kumar, 1279
Soares, Piercarlo Fermino, 446
Soe, Yan Naung, 458 W
Sohel, Ferdous, 914 Wang, Junhu, 927
Solis, Priscila, 859 Wang, Yu-Ting, 725
Sood, Nipun, 677 Watashiba, Yasuhiro, 1024
Author Index 1357

Wilderman, Mattias, 1011 Youssef, Habib, 1034

Woungang, Isaac, 702, 713 Yuan, Chen, 872
Yukonhiatou, Chaxiong, 787
Y
Yamamoto, Hirofumi, 940 Z
Yamamoto, Takumi, 1329 Zahid, Maheen, 1
Yamashita, Yuki, 940 Zhang, Ping, 1342
Yashkina, Ekaterina, 1133 Zhao, Bing, 166
Yeferny, Taoufik, 619 Zheng, Yong, 595, 631
Yokota, Takashi, 407 Zouari, Belhassen, 1228
Yoshihisa, Tomoki, 787, 799 Zubair, Adam, 1133
Yoshikawa, Takashi, 1024 Zuraiz, Maria, 53