Technology

SASDL and RBATQ: Sparse Autoencoder

With Swarm Based Deep Learning and
Reinforcement Based Q-Learning for
EEG Classification
Sunil Kumar Prabhakar , Member, IEEE, and Seong-Whan Lee , Fellow, IEEE

Abstract—The most vital information about the electrical Impact Statement—Efficient deep learning techniques for
activities of the brain can be obtained with the help EEG signal classification
of Electroencephalography (EEG) signals. It is quite
a powerful tool to analyze the neural activities of the
brain and various neurological disorders like epilepsy,
I. INTRODUCTION
schizophrenia, sleep related disorders, parkinson disease HE physical activities of the nervous system can be com-
etc. can be investigated well with the help of EEG signals.
Goal: In this paper, two versatile deep learning methods
are proposed for the efficient classification of epilepsy
T prehensively reflected by the EEG signals [1]. If there is
any change in the brain function caused due to neurological
and schizophrenia from EEG datasets. Methods: The disorders, then it can be detected by EEG signals. In the field of
main advantage of using deep learning when compared medicine, an objective basis for diagnosing certain disorders is
to other machine learning algorithms is that it has the provided by the information processing of EEG signals, thereby
capability to accomplish feature engineering on its own. enabling the clinicians to provide effective treatment for the par-
Swarm intelligence is also a highly useful technique to
solve a wide range of real-world, complex, and non-linear ticular brain disorder. Earlier, a manual detection and analysis of
problems. Therefore, taking advantage of these factors, the EEG waveforms was done and due to its intensive labor and
the first method proposed is a Sparse Autoencoder (SAE) long-time consumption, automated classification of EEG signals
with swarm based deep learning method and it is named to diagnose the neurological disorder came into existence [2].
as (SASDL) using Particle Swarm Optimization (PSO) Therefore, classification of EEG signals is quite a vital task with
technique, Cuckoo Search Optimization (CSO) technique
and Bat Algorithm (BA) technique; and the second tech- respect to the identification, diagnosis and even prevention of
nique proposed is the Reinforcement Learning based on brain related disease. In this paper, the classification of epilepsy
Bidirectional Long-Short Term Memory (BiLSTM), Attention EEG signals and schizophrenia EEG signals are dealt in much
Mechanism, Tree LSTM and Q learning, and it is named detail. Epilepsy is a chronic disease characterized by sudden
as (RBATQ) technique. Results and Conclusions: Both and repeated seizures [3]. Due to various initiating locations
these two novel deep learning techniques are tested on
epilepsy and schizophrenia EEG datasets and the results and transmission modes of the abnormal electrical activities
are analyzed comprehensively, and a good classification in brain, different clinical manifestation occurs such as loss of
accuracy of more than 93% is obtained for all the datasets. consciousness, limb convulsions, behavioral problems etc [4].
Index Terms—Deep learning, EEG, PSO, Q-learning, rein-
The most prevalent technique to examine the brain activities in
forcement learning. epileptic patients is with the help of EEG. For epileptic patients,
the EEG signals of their brain activity are split into interictal,
pre-ictal and ictal states [5]. An unusual pattern is exhibited
Manuscript received November 2, 2021; revised February 20, 2022 in the EEG signals where the seizure occurs. A distributed
and March 16, 2022; accepted March 17, 2022. Date of publication pattern is also sometimes exhibited in the EEG signal where
March 23, 2022; date of current version May 27, 2022. This work
was supported in part by Institute for Information & Communications the seizure occurs. A distinctive pattern is also shown by the
Technology Promotion grant funded by Korea government under Grant EEG signals of interictal state and preictal state. Therefore, to
2017-0-00451, in part by the Development of BCI based Brain and Cog- differentiate these epileptic states, these patterns in the EEG
nitive Computing Technology for Recognizing User’s Intentions using
Deep Learning under Grant 2015-0-00185, in part by the Development signals are highly useful so that the occurrence of a seizure
of Intelligent Pattern Recognition Softwares for Ambulatory Brain Com- can be known thereby reducing the deadly effects it has on
puter Interface under Grant 2019-0-00079, and in part by the Artificial the patients [6]. Seizure detection and classification has been
Intelligence Graduate School Program, Korea University. The review of
this paper was arranged by Editor Laura Astolfi. (Corresponding author: studied for the past two decades with the help of machine
Seong-Whan Lee.) learning and deep learning techniques, and a good survey about
The authors are with the Department of Artificial Intelligence, it can be found in [7], [8] enabling the authors not to repeat
Korea University, Seoul 02841, South Korea (e-mail: sunilprab-
hakar22@gmail.com; sw.lee@korea.ac.kr). the past works again and again. However, the most important
Digital Object Identifier 10.1109/OJEMB.2022.3161837 ideas incorporating machine learning and deep learning since
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

58 VOLUME 3, 2022
PRABHAKAR AND LEE: SASDL AND RBATQ: SPARSE AUTOENCODER WITH SWARM BASED DEEP LEARNING AND REINFORCEMENT 59

the past four years is discussed here for the better understanding EEG series splitting reported an accuracy of 92.91% [33], deep
of the readers. A transfer learning along with semi-supervised convolutional neural networks reported 98.07% for non-subject
learning for seizure classification from EEG signals was based testing and 81.26% for subject based testing [34], spectral
proposed by Jiang et al., where the average accuracy was shown based analysis reporting 96.77% [35], swarm computing tech-
to be higher than 95% in most cases [9]. A tunable Q wavelet niques with classifiers reporting 92.l7% [31], Short Time Fourier
transform dependent on multiscale entropy was proposed for Transform (STFT) with CNN reporting 97.00% [36], Partial
automated classification of epileptic EEG signals where the Least Squares technique reporting 98.77% [37], multivariate
highest accuracy of even 100% was achieved in few cases [10]. Empirical Mode Decomposition (EMD) reporting 93.00% [38],
A local mean decomposition (LMD)-based feature analysis with continuous wavelet transform (CWT) with CNN reporting an
Support Vector Machine (SVM) was utilized by Zhang and accuracy of 98.60% [39], a simple CNN reporting 98.96% [40],
Chen where the classification accuracy reached an accuracy sparse depiction with nature inclined classification and deep cum
of 98.10% [11]. In the year 2018, for automated classification transfer learning reporting 98.72% [30] and Collatz pattern re-
of epilepsy from EEG signals, deep learning approaches was porting an accuracy of 99.47% [41] are some of the most famous
proposed in [12], [13] and a morphological component analysis works proposed recently. In this work, the key contributions are
based SVM classification was proposed in [14], and these three as follows and no previous works have been reported in literature
approaches produced a high classification accuracy of more than using the two developed novel deep learning models.
95% as per the consideration of their problem requirement. A i) Initially a sparse autoencoder with swarm based deep
scalogram based convolution network from EEG signals was neural network using PSO was developed for classifying
proposed in [15], a matrix determinant-based approach was epilepsy and schizophrenia datasets.
utilized in [16], cross-bispectrum analysis for seizure detection ii) Secondly, reinforcement learning based on Q-learning
in [17], a novel random forest model with grid search optimiza- was implemented successfully to classify epilepsy and
tion in [18] are some of the famous works in 2019 and almost schizophrenia datasets.
all the works have achieved a classification accuracy of 90% The organization of the work is as follows. Section II explains
to 100% depending on type of case study. In the year 2020, a the development of the SASDL, and Section III explains the
novel convolutional based neural network model [19], improved RBATQ model. Section IV explains the results and discussion
Radial Basis Function (RBF) analysis [20], Power Spectral and Section V gives the conclusion.
Density (PSD) based deep CNN [21], imagined EEG signal
analysis through fully convolutional networks [22], empirical
mode decomposition analysis along with its derivative [23], a II. DEVELOPMENT OF SASDL MODEL
bat algorithm based SVM [24] are some of the famous works for An autoencoder model is developed to mitigate the dimen-
seizure classification from EEG signals with almost all the works sionality of the input [42]. A feedforward neural network is
reporting a good classification accuracy of more than 90%. In utilized by this form of unsupervised learning and the autoen-
2021, a Jacobi polynomial transform based technique with Least coder has both encoding and decoding plan An input x is usually
Square SVM (LS-SVM) [25], an adaptive synthetic sampling trained and x is reconstructed to be quite similar to the input x
approach [26], a fractal-based seizure detection technique [27], as much as possible. Many kinds of autoencoders are available
Principal Component Analysis (PCA) based Genetic Algorithm in literature such as sparse autoencoder, denoising autoencoder,
(GA) [28], significance of channel selection techniques [29] are stacked autoencoder etc [42]. When there is a huge data space,
utilized for seizure classification from EEG signals where it the reconstruction of the raw data by the autoencoder can fail
reported a classification accuracy of more than 90% for most as it might fall into replication of the tasks. The sparse autoen-
of the classification cases. In 2022, sparse analysis with deep coder has usually lower output dimensions and it persuades the
and transfer learning models were developed with an ensemble autoencoder in reconstructing the raw data from the most useful
cum nature inclined classification for epilepsy classification features instead of replicating it once again. In this study, a sparse
reporting classification accuracies of more than 90% for epilepsy autoencoder is chosen which helps to extract the highly useful
classification [30]. patterns which would have a very low dimensionality. These
As the paper discusses schizophrenia classification also from feature vectors are once again selected by the PSO/CSO/BA and
EEG signals, recent literature about it is also discussed in the pa- finally fed into a simple deep neural network which comprises
per as follows. Schizophrenia is a serious mental disorder where of two hidden layers along with a Softmax output layer. The
people interpret reality in an abnormal manner Schizophrenia re- input vector to the PSO/CSO/BA is given from the bottleneck
sults in a combination of delusion, hallucination, and disordered of the sparse autoencoder. The bias units are the neurons termed
thinking thereby the daily functions are severely impaired [31]. as (+1) and these are added to the feed forward neural network
Therefore, schizophrenia involves a range of problems with cog- with the help of cost function. In order to get a most preferable
nition, emotion, and behaviour. An exact cause of schizophrenia reconstruction of the input x, this step is highly useful, and it
is not known, but a combination of brain chemistry, genetics can be achieved without overfitting. The cost function of the
and environmental factors may contribute to the development autoencoder comprises of three steps. Assuming a dataset with a
of this disease [32]. EEG signals are a great boon to analyze total of N training samples (x1 , x2 , . . . , xn ), where the ith input
this disorder and some of the famous works are utilized in is indicated by xi . The reconstruction of the input xi is trained
this field are as follows. For schizophrenia EEG analysis, the by the developed SAE with the help of function hW,b (xi ) so that
60 IEEE OPEN JOURNAL OF ENGINEERING IN MEDICINE AND BIOLOGY, VOL. 3, 2022

its proximity to xi is very close. The squared error, sparsity term nest, the host birds discover the egg laid by the cuckoo with a
and the weight decay are the three important sections of the cost probability pa ∈ [0, 1]. The host bird can then decide to either
function. The weight decay aids to avoid overfitting. For all N eliminate the egg or even abandon the nest completely. The
training samples, the mean square error along with the weight algorithm of the cuckoo search is developed using these three
decay and sparsity term is expressed as: rules and is shown in Algorithm 2. A levy flight is generally
implemented when new solutions x(t+1) are generating for a
1 1 
N
Jsparse (W, b) = hW,b (xi ) − xi  cuckoo ‘c’ and is expressed as:
N i=1 2
(t+1) (t)
xi = xi + α ⊕ Levy(λ) (5)
λ 
nl−1 sl sl+1

sl
l 2
+ (Wji ) +β KL(pp̂j ) where α > 0 denotes the step size. Generally, α = O(L/10) is
2
l=1 i=1 j=1 j=1
utilized in most cases, where L denotes the characteristic scale
(1) of the problem of interest. For a random walk, the (5) is projected
where the sparse penalty term is represented by β, KL indicates as a stochastic equation. Depending on the current location and
the Kullback–Leibler divergence. The value of λ should be the transition probability, the random walk in a Markov Chain
carefully chosen because a low value of it leads to overfitting is modeled. The entry wise multiplications are expressed by the
and a high value of λ leads to underfitting. Here ReLU is chosen product ⊕. To explore the search space here, the random walk
as the activation function represented as a, which expresses the through Levy flight process is more efficient as it has a longer
average activated value of the hidden layer, and it is represented step length. With the help of a Levy distribution, a random walk
as: where random step length is obtained is provided by the Levy
flight and is projected as:
1  2 i
N
p̂j = (a (x )) (2)
N i=1 j Levy ∼ u = t−λ , (1 < λ ≤ 3) (6)

where p represents the sparsity parameter. The calculation of the

C. BA
sparsity term is usually done to make p̂j look identical and close
to p as much as possible. Activation and deactivation of neurons In this process, for a typical bat algorithm, the following
on the hidden layer is done by this parameter. idealized rules are utilized [45]. In order to sense the distance,
echo location is used by the bats and the difference between the
A. PSO prey and the different background barriers are known by the bats.
In a random manner the bats can fly with a particular velocity
A famous population-based algorithm utilized to solve opti-
vi at a position xi . The wavelength of their emitted pulses is
mization problems is PSO [43]. The number of particles consti-
adjusted quickly. Based on the proximity of the target, the rate
tute the total population, and each particle indicates a candidate.
of pulse emission r ∈ [0, 1] can be adjusted. The loudness varies
The best solution is searched for by means of updating the
from a large positive value A0 to a minimum value Amin , though
velocity and particle vectors as per the equation:
in many ways the loudness can vary. For simplicity purposes,
vid (t + 1) = w ∗ vid (t) + c1 ∗ r1 ∗ (Pid − xid (t)) + c2 ∗ r2 the following approximations can be utilized. Generally, the
frequency factor f in a specific range [fmin , fmax ] correlates to a
∗ (Pgd − xid (t)) (3) specific wavelength range [λmin , λmax ]. For an easy implemen-
xid (t + 1) = xid (t) + vid (t + 1) (4) tation, any wavelength can be used depending on the specific
problem. By means of adjusting the frequencies the range of
The velocity of the particle i in the dth dimension is represented the wavelength can be adjusted. While fixing the wavelength λ,
as vid . The position of the particle i in the dth dimension is the frequency too can be varied as λ and f are closely related.
represented as xid . In the dth dimension, Pid represents the local For simplicity reasons, it is assumed that f ∈ [0, fmax ]. It is well
best and Pgd represents the global best. The random numbers known that higher frequencies possess short wavelength and can
between 0 and 1 is represented by r1 , r2 . The w represents the travel only a shorter distance. The typical range is only a few
inertia weight and c1 , c2 represents the acceleration coefficient meters for bats and the rate of pulse is in the range of [0, 1],
for both exploitation and exploration purposes. Due to its ver- where 1 implies the highest pulse emission rate and 0 implies
satility as proven in literature, PSO is chosen here in this work no pulses and the procedure of it is shown in Algorithm 3.
and is shown in Algorithm 1.
D. Overall Framework of the Work
B. CSO
The overall framework with testing and training using the
When dealing with CSO, the following tree main rules are
SASDL is depicted from Figs. 1 and 2. Initially, the dataset is
used in expressing the cuckoo search process [44]. Firstly, one
split into a training set and test set. The training set is passed
egg is laid by a cuckoo at a particular time and its egg is dropped
to SAE and the bottleneck output of the SAE is fed to the
in a randomly chosen nest. Secondly, only the best nests which
PSO/CSO/BA and then the respective output of it is fed to the
possess high quality eggs is progressed and carried on to the
DNN. The PSO/CSO/BA is used to select the best particles with
next generation. Thirdly, with a fixed number of available host
the help of Algorithms 1, 2 and 3 respectively.
PRABHAKAR AND LEE: SASDL AND RBATQ: SPARSE AUTOENCODER WITH SWARM BASED DEEP LEARNING AND REINFORCEMENT 61

Fig. 1. Training of the proposed SASDL method.

Algorithm 1: PSO Implementation to the DNN.

Input: Population Size P opulsize , generation gen
popul ← Initialize the particles randomly until the total
number of particles reach P opulsize ;
gbest,i ← Empty, 0
while i < gen do
for particle p in popul do
p ← Position updation of p using standard PSO
operation.
fitness ← Compute the fitness for p using the standard
fitness evaluation critic
Fitness updation of p by fitness
if fitness > fitness of the personal best then
Update the personal best of p with the p;
end if
end for
Fig. 2. Testing of the proposed SASDL method.
gbest ← Best particle updation among the current gbest
and pop
As far as the PSO is concerned, the inertia weight is considered i ← i + 1;
as 0.64. The acceleration coefficient c1 and c2 is considered as end while
1.524 and the population size is set as 30 and the total number of Return gbest
generations is assigned as 30. All the values were finally chosen Post process it by sending it to the DNN.
after several trial and error-based experimentation efforts.
As far as the CSO is concerned, to fasten up the local search,
the generations of the new solutions by using Levy walk is implying that the bats are progressing towards reaching the
utilized. The parameters used in our experiments are as follows, optimal solution. In our experiment, the value of n is chosen as 40
nests n = 20, α = 1.5 and pα = 0.45. virtual bats.
As far as BA is concerned, the choosing of the parameters
requires some trial-and-error experimentation in this process. III. DEVELOPMENT OF RBATQ MODEL
By means of randomization, every bat should possess different To analyze the decision process of reinforcement learning,
values of both pulse emission rate and loudness. In this exper- in this paper, three deep learning techniques are utilized such
iment, the initial loudness A0i is considered as 1 and the initial as Bidirectional LSTM, attention mechanism along with Tree
emission rate ri0 is chosen as 0.5 as the values can be between LSTM. To get the control policy, Q-learning algorithm is utilized
ri0 ∈ [0, 1]. If there is an improvement in the new solution, there here in this work.
will naturally be an updation of the loudness and emission rates,
62 IEEE OPEN JOURNAL OF ENGINEERING IN MEDICINE AND BIOLOGY, VOL. 3, 2022

s2 = T ree(X; θ2 ). X indicates the features of the input signal

Algorithm 2: CSO Implementation to the DNN.
and the state parameters are expressed by θ1 and θ2 respectively.
Input : Objective function f (x), x = (x1 , . . . , xd )T and the Action: In the environment, there are quite a collection of pre-
initial population generation with host nests xc defined actions denoted by A, such as action a1 , action a2 , action
while (t < MaxGeneration) a3 respectively. The initial decision decides to consider a1 or
Random generation of a solution by Levy flight a2 , and the next decision decides to consider a3 or a4 . For every
Evaluation of the fitness Fc by the cuckoo action, the reward obtained is represented by R = r1 , r2 , r3 , r4 .
Random choosing of the nest among n, say d In a state S, an action ‘a’ is usually considered by the agent and
if (Fc > Fd ) a reward ‘r’ is received from the environment. The transition
New solution replaces j adaption of the decision procedure is chosen accordingly.
end if Transition and Reward function: The agent considering a1 at
Abandon a fraction (pα ) of worse nests s1 is then transmitted to se by means of effectively utilizing a
Generate new nests and its respective solutions state transition tuple (s1 , a1 , r1 , se ). An agent usually receives
Project only the best or quality solutions a reward r1 if the judgement of a1 is correct. If the judgement
Analyze the current best by ranking the solutions of a1 is incorrect, then in order to push the utilizing judgement
end while of the initial decision, r1 can be set accordingly. The rest of the
Post processes it by sending it to the DNN. state transition tuples and its respective reward function can be
assigned in a similar manner.
Algorithm 3: BA Implementation to the DNN.
B. BiLSTM Layer
Input : Bat population initialization xi and vi
(i = 1, 2, . . . , n) Every LSTM component in the BiLSTM layer comprises of
Frequency initialization fi , pulse rate initialization ri , three multiplicative gates such as input gate it , forget gate ft and
loudness initialization Ai output gate ot . The proportion of information can be controlled
while (t < Max number of iterations) by these gates and helps it to progress on to the next time step.
Generation of new solutions by frequency adjustment In each LSTM unit, a memory cell ct is also kept which helps
Velocity updation is done to analyze the preceding state thereby the features of the current
The location/solutions updation is also done input signal can be well memorized. For every LSTM unit, the
if (rand > ri ) data sources are as follows: the feature vector xt at time t, hidden
Best solution is selected among a set of solutions state vector ht−1 and ht+1 (before and after time t, along with
Local solution is generated among the best solutions the cell vector ct−1 ). The implementation of forward passes are
end if as follows:
New solution generation by random flying it = σ(Wxi xt + Whi ht−1 + Wci ct−1 + bi ) (7)
if (rand < Ai and f (xi ) < f (x0 ))
Acquire the new solutions ft = σ(Wxf xt + Whf ht−1 + Wcf ct−1 + bf ) (8)
Enhance ri and mitigate Ai gt = tanh(Wxc xt + Whc ht−1 + Wcc ct−1 + bc ) (9)
end if
Ranking of bats to find the current best x∗ ct = it gt + ft ct−1 (10)
end while ot = σ(Wxo xt + Who ht−1 + Wco ct + bo ) (11)
Post process it by sending it to the DNN
ht = ot tanh(ct ) (12)
where the weight matrices are represented by W , bias vectors
A. Reinforcement Learning (RL)
are represented as b. The subscripts indicate the meaning as per
In order to learn the control policies of the agent in an efficient the name suggestion as it is commonly represented in BiLSTM
manner, the most commonly used framework is RL, and it is concept [47]. The logistic function is indicated by σ. The exe-
done by means of active interaction with its environment [46]. cution of the backward passes with respect to time are carried
State: Three states such as the initial state s1 , transition out in a same fashion as the forward passes. At a time t, the
state s2 along with the end state se is present in the internal hidden state vectors of two directions ht and ht are computed
state S of the environment. Representing the state directly from simultaneously in the BiLSTM layer, therefore past features and
the signal is quite a difficult task as there are no appropriate future features can be efficiently utilized in a specific time frame.
measures to assess it and therefore to extract the features of The hidden state vectors of two directions ht and ht is passed
signal, deep learning techniques are used which helps to indicate to a Softmax layer at a particular time t and it is represented as:
the circumstances in the decision process. Initially, to realize
the feature extraction, a bidirectional LSTM [47] is used and to yt = soft max(Why ht + Wh y ht + by ) (13)
generate the initial state, attention-based methods [48] are used Here the weight matrices are expressed by W and the bias
and it is represented as s1 = Att(X; θ1 ). To create the transition vector is represented by b. Attention mechanism is applied to
state s2 , Tree-LSTM [49] is utilized here, and it is represented as the BiLSTM.
PRABHAKAR AND LEE: SASDL AND RBATQ: SPARSE AUTOENCODER WITH SWARM BASED DEEP LEARNING AND REINFORCEMENT 63

C. Tree LSTM
Algorithm 4: Q-Learning Training Procedure for the Pro-
This concept was implemented in the field of natural language posed RBATQ Method.
processing (NLP) however, the idea has been tried to biosignal Start BiLSTM, Attention mechanism and Tree-LSTM with
processing for the first time in this paper. The development random parameters
of the tree LSTM starts from its leaf node, and it is done in η=0
a recursive manner up to the root [49]. On the hidden state Pre-training of BiLSTM, Attention mechanism and
vector of the antecedent element, the non-linear transformation Tree-LSTM process
is carried out so that s2 is generated. It serves as an important For every epoch = 1,2 do
transition predicament in the decision process and therefore For every input signal X do
s2 is denoted as: s2 = T ree(X; θ2 ) where θ2 indicates all the Utilize deep learning model for automated feature
essential criterion in the Tree-LSTM. Once the transition state extraction of X and produce S1 and S2
s2 is generated, it is progressed to a Softmax output layer so that For t = 1,2 do
yr is obtained which indicates the probability of various kinds r, s = reward and state after considering the action
for a relation mention. A category with the highest probability π(s)
is chosen, so that a3 or a4 can be determined easily.
a = π(s)
yr = sof t max(Wsy s2 + by ) (14)
Implement Gradient descent step:
Here the weight metric is represented by W and the bias vector  
∂Eη ∂Qη (s, a)
is specified by b. A softmax layer is utilized at every dependency − = E 2(Qπ (s, a) − Qη (s, a))
tree so that the category for the root node is predicted when the ∂η ∂η
given inputs X are discovered at its respective children nodes. 1 t−1 π  
Qπ (s, a) = r+ Q (s, a )
t t
D. Q-Learning Updation Process:
 
An approved form of reinforcement learning technique is 1 t−1   ∂Qη (s, a)
Q-learning algorithm [50]. For the agent, an optimal state-action η =η+α r+ Qn (s, a ) − Qη (s, a)
t t ∂η
value function Q(s, a) can be easily used to learn it. By means
of consultation of Q(s, a), the agent considers an action a in α = update step, r = reward function
state s, which is nothing but the simple estimation of the action’s (s, a ) = state action pair
anticipated long-term reward. By means of analyzing a sequence π(s) = arg maxa Qη (s, a )
of actions, some cumulative rewards can be maximized. For s = s, a = a
every state-action pairs, it is quite difficult to obtain Q(s, a) End for
as the state space is infinite in the decision process. There- End for
fore, using a novel network, Q(s, a) is approximated which End for
can specify Q(s, a) as a parameterized outcome represented as
Qη (s, a) = M LP (ϕ(X; θ), a, η). s1 = Att(X; θ1 ) is referred
by s1 = ϕ(X; θ1 ) and s2 = T ree(X; θ2 ), where θ is calculated pre-trained initially. All the parameters in BiLSTM are indicated
by means of pretraining the deep learning models. The parameter as θ0 , all the parameters in attention layer are specified as θ1 and
in the neural network is represented by η and it is learnt by all the parameters in Tree-LSTM is indicated as θ2 are these
implementing the famous stochastic gradient descent step with are the main training parameters used. Deep learning is used to
the help of RMSprop. The degree of approximation is measured represent the features and RL is used to combine these three tasks
with respect to the least squares error in order to estimate the in the final decision process. The standard conventional pipeline
real value function Qπ as follows: architectures fail to enable the information to flow in a sequential
manner, but this RL method combines all the tasks in a sequential
Eη = E[(Qπ (s, a) − Qη (s, a))2 ] (15)
manner and allows to make decisions too. The decisions may
where E represents the least square value. Instead of the real have problems initially but after several epochs, a good stability
value function Qπ (s, a), the estimated value function Qη (s, a) is can be obtained. A global updation of the parameters s is done
used by the Q-learning. In the middle of the estimation Qη (s, a) in this architecture and therefore an eventual convergence is
and the expectation Qπ (s, a), the discrepancy is reduced when achieved later. Hence the feedback from decision-making can
the parameters are updated during every epoch. There is a be obtained easily by the RL method thereby enabling the data
continuous updation of values when the agent progresses from to progress easily in the global architecture. The Q-learning
a random Qη (s, a) by means of utilizing the decisions and training procedure is expressed in Algorithm 4.
obtains the suitable reward. By carefully selecting the actions
with the highest Qη (s, a ), the agent can expand its future
rewards accordingly. Ultimately, the control policy π is obtained IV. RESULTS AND DISCUSSION
by the Q-learning algorithm. When the training procedure is The proposed deep learning models has been initially eval-
carried out, BiLSTM, attention layer along with Tree-LSTM are uated on the University of Bonn dataset where it deals with
64 IEEE OPEN JOURNAL OF ENGINEERING IN MEDICINE AND BIOLOGY, VOL. 3, 2022

TABLE I
PERFORMANCE ANALYSIS OF THE PROPOSED DEEP LEARNING TECHNIQUES

epilepsy classification. Then the proposed deep learning mod- learning techniques happens, the schizophrenia EEG samples
els has been evaluated on the dataset obtained for Institute of are randomly divided into ten non-overlapping folds due to
Psychiatry and Neurology, Poland. As far as the Bonn dataset is the adoption of a 10-fold cross validation technique here. The
concerned, the epilepsy datasets are categorized into A, B, C, D dimensionality representation of the input is reduced by SAE
and E sets. The normal category dataset is present in set A and set where the size of the input is about (4097 × 100) for epileptic
B, the interictal category dataset is present in set C and set D and dataset and (5000 × 45) for schizophrenia dataset. It is reduced
the ictal category dataset is present in set E. The classification to about (2500 × 50) for epileptic dataset and (5000 × 15) for
problems discussed here are A-E, B-E, C-E, D-E, AB-E, AC-E, schizophrenia dataset. These useful features are provided by the
CD-E, ACD-E, ABCD-E. As far as the schizophrenia dataset bottleneck of the SAE that is fed to the PSO/CSO/BA. The size
is considered, it is just normal case versus schizophrenia case. of the bottleneck comprises of 9000 hidden units. After it is
All the explicit datasets for it are given in the reference [51], passed to PSO/CSO/BA, a total of 4500 features are obtained.
[52]. In the epilepsy dataset, 100 single channel EEG recordings The classifier comprises of two hidden layers and an output
are present which has a sampling rate of 173.61 Hz along with layer where the sizes of units are expressed as 2250, 500 and 2
a time duration of 23.6 seconds. The sampling of these time respectively. To specify the probability of each class, Softmax
series is done into 4097 data points and then all these 4097 regression is utilized in the output layer. In between the fully
data points are further split into 23 chunks, here about 2300 connected neural networks, dropout is utilized to prevent the
samples are present in each category. For the deep learning overfitting. In between the two classes, the maximum probability
techniques, the 2300 EEG signals are randomly divided into is chosen as the final decision of the classifier. To compute
ten non-overlapping folds due to the adoption of a 10-fold cross the cost function of the classifier, cross entropy is utilized and
validation technique utilized here for evaluation. When dealing then a weight decay term was added to it subsequently. The
with schizophrenia datasets, each channel has about 225,000 cost function is minimized by the SAE and the PSO/CSO/BA
samples and therefore the data is specified into a matrix format selects the most important features as it is given as input to the
of [5000 × 45]. As it has about 19 channels, it is specified DNN. The completion of the training process is done in about
exactly as [5000 × 45 × 19]. When the implementation of deep 50 iterations and the batch size was set as 10. The value of
PRABHAKAR AND LEE: SASDL AND RBATQ: SPARSE AUTOENCODER WITH SWARM BASED DEEP LEARNING AND REINFORCEMENT 65

the sparsity parameter p is chosen as 0.08, the weight decay

λ is set as 0.01 and the sparse penalty term β was chosen as 4
respectively. To adjust the classifier parameters, fine tuning of the
deep neural network classifier was done on the last 20 iterations
so that the cost function of the Softmax was minimized. For
parameter updation, Adam optimizer is used. The evaluation of
the model was done using a 10-fold cross validation technique.
As for the RBATQ deep learning model, the hyperparameters
implemented are as follows. The state size for all the LSTM
units is set as 250 and the dimension of the hidden layer is fixed
as 100. The non-linear function utilized is tanh. The dropout
rate is set as 0.75, initial learning rate is 0.002. Mini batch size
is set as 25 and the constraint of maximum norm regularization is
set as 3. The performance metrics analyzed here are sensitivity, Fig. 3. Performance analysis of Good Detection Rate (GDR) %.
specificity, and accuracy and is tabulated in Table I.
On analyzing Table I, it is inferred that for the epilep-
tic dataset (A-E), a good classification accuracy of 97.75%
is obtained when utilizing RBATQ model, 98.55% accuracy
with SASDL-PSO model, 98.50% accuracy with SASDL-CSO
model, 98.26% accuracy with SASDL-BA model and 96.52%
for SAE with DNN model. For the epileptic dataset (B-E),
a good classification accuracy of 97.42% is obtained when
utilizing RBATQ model, 97.85% accuracy with SASDL-PSO
model, 97.64% accuracy with SASDL-CSO model, 97.55%
accuracy with SASDL-BA model and 98.03% for SAE with
DNN model. For the epileptic dataset (C-E), a good classification
accuracy of 94.57% is obtained when utilizing RBATQ model,
98.11% accuracy with SASDL-PSO model, 96.62% accuracy
with SASDL-CSO model, 96.18% accuracy with SASDL-BA Fig. 4. Performance analysis of error rate (%).
model and 95.8% for SAE with DNN model. For the epilep-
tic dataset (D-E), a good classification accuracy of 94.74% dataset (BCD-E), a good classification accuracy of 93.68%
is obtained when utilizing RBATQ model, 97.52% accuracy is obtained when utilizing RBATQ model, 98.07% accuracy
with SASDL-PSO model, 96.56% accuracy with SASDL-CSO with SASDL-PSO model, 97.46% accuracy with SASDL-CSO
model, 96.74% accuracy with SASDL-BA model and 95.91% model, 97.21% accuracy with SASDL-BA model and 97.6% for
for SAE with DNN model. For the epileptic dataset (AB-E), SAE with DNN model. For the schizophrenia dataset, a good
a good classification accuracy of 94.4% is obtained when classification accuracy of 94.97% is obtained when utilizing
utilizing RBATQ model, 98.4% accuracy with SASDL-PSO RBATQ model, 97.95% accuracy with SASDL-PSO model,
model, 98.04% accuracy with SASDL-CSO model, 97.96% 96.22% accuracy with SASDL-CSO model, 96% accuracy with
accuracy with SASDL-BA model and 98% for SAE with DNN SASDL-BA model and 96% for SAE with DNN model.
model. For the epileptic dataset (CD-E), a good classification The Good Detection Rate (GDR) and Error Rate Analysis for
accuracy of 94.29% is obtained when utilizing RBATQ model, the Deep learning models is plotted in Figs. 3 and 4 respectively.
98.55% accuracy with SASDL-PSO model, 96.57% accuracy As inferred from Fig. 3 for the proposed SASDL-PSO model
with SASDL-CSO model, 95.46% accuracy with SASDL-BA produces a high GDR and then it is followed by the proposed
model and 95.22% for SAE with DNN model. For the epilep- SASDL-CSO model and the SASDL-BA model. The proposed
tic dataset (AC-E), a good classification accuracy of 94.4% RBATQ model and the ordinary SAE-DNN model produce a
is obtained when utilizing RBATQ model, 97.25% accuracy comparatively low GDR when compared to the other classifiers.
with SASDL-PSO model, 95.90% accuracy with SASDL-CSO As inferred from Fig. 4, a low error rate is obtained for the
model, 95.51% accuracy with SASDL-BA model and 95.49% proposed SASDL-PSO model. A high error rate is obtained for
for SAE with DNN model. For the epileptic dataset (ACD-E), the SAE-DNN model, and the remaining three models too have a
a good classification accuracy of 93.89% is obtained when slightly higher error rate than the proposed SASDL-PSO model.
utilizing RBATQ model, 97.89% accuracy with SASDL-PSO
model, 96.63% accuracy with SASDL-CSO model, 95.87% ac-
A. Comparison With Previous Works for Epilepsy
curacy with SASDL-BA model and 95.45% for SAE with DNN
Bonn Dataset and Schizophrenia Dataset
model. For the epileptic dataset (ABCD-E), a good classification
accuracy of 94.04% is obtained when utilizing RBATQ model, Though thousands of papers are published online every year in
98.48% accuracy with SASDL-PSO model, 96.99% accuracy epilepsy and schizophrenia classification, a few selected impor-
with SASDL-CSO model, 95.95% accuracy with SASDL-BA tant and recent works which have analyzed many combinations
model and 95.64 % for SAE with DNN model. For the epileptic of the epilepsy problem has been considered and the results have
66 IEEE OPEN JOURNAL OF ENGINEERING IN MEDICINE AND BIOLOGY, VOL. 3, 2022

TABLE II
COMPARISON WITH PREVIOUS RESULTS FOR THE EPILEPSY BONN DATASET

TABLE III sometimes giving less classification accuracy than the previous
COMPARISON WITH PREVIOUS RESULTS FOR THE SCHIZOPHRENIA DATASET
results by a minor margin. The main intention of this work is
to analyze a swarm based deep neural networks along with a
Reinforcement based Q-learning for epilepsy and schizophrenia
datasets and the results are projected.

V. CONCLUSION
To study and analyze the neuronal dynamics within the human
brain, the most standard tool utilized by the researchers and
clinicians is EEG. For the EEG dependent analysis of various
neurological disorders, visual inspection of these huge datasets
is very difficult. Therefore, feature extraction techniques and
automated classification schemes have been developed in the
past. With the advent of deep learning, manual feature extraction
is not necessary as it is aided by the deep learning process itself.
In this paper, two novel deep learning techniques one with the
help of swarm intelligence and another with the help of Rein-
forcement learning such as SASDL and RBATQ are proposed
in this paper and tested for two EEG datasets such as epilepsy
dataset and schizophrenia dataset. The highest classification ac-
been compared with them and reported in Tables II and III. The curacy of 98.55% was obtained with the proposed SASDL-PSO
best result of 98.55% has been obtained for the A-E problem with method and 97.75% was obtained with the proposed RBATQ
the proposed SASDL-PSO model, 98.03% for B-E problem with method for epilepsy dataset. The highest classification accuracy
the SAE-DNN model, 98.11% for the C-E problem with the of 97.95% was obtained with the proposed SASDL-PSO method
proposed SASDL-PSO model, 97.52% for D-E problem with and 94.97% was obtained with the proposed RBATQ method
the proposed SASDL-PSO model, 98.4% for AB-E problem for schizophrenia dataset. Future works aim to develop more
with the proposed SASDL-PSO model, 98.55% for CD-E prob- interesting deep learning models to classify the EEG datasets
lem with the proposed SASDL-PSO model, 97.25% for AC-E with a high classification accuracy. Moreover, these developed
problem with the proposed SASDL-PSO model, 97.89% for deep learning models are planned to be implemented for other
ACD-E problem with the proposed SASDL-PSO model, 98.48% biosignal datasets such as Electrocardiogram (ECG), Photo-
for ABCD-E problem with the proposed SASDL-PSO model, plethysmogram (PPG), Electrooculogram (EOG) etc for the
and 98.07% for BCD-E problem with the proposed SASDL- diagnosis of various medical disorders.
PSO model. For schizophrenia classification, the best result of
97.95% is obtained with the proposed SASDL-PSO model. The REFERENCES
proposed results have more or less reached the similar results [1] H.-I. Suk and S.-W. Lee, “Subject and class specific frequency bands
when compared to the previous state of the art results, sometimes selection for multi-class motor imagery classification,” Int. J. Imag. Syst.
giving more classification accuracy than the previous results and Technol., vol. 21, no 2, pp. 123–130, 2011.
PRABHAKAR AND LEE: SASDL AND RBATQ: SPARSE AUTOENCODER WITH SWARM BASED DEEP LEARNING AND REINFORCEMENT 67

[2] M.-H. Lee et al., “EEG dataset and OpenBMI toolbox for three BCI [24] A. Naser, M. Tantawi, H. A. Shedeed, and M. F. Tolba, “Automated EEG-
paradigms: An investigation into BCI illiteracy,” Giga Sci., vol. 8, no 5, based epilepsy detection using BA-SVM classifiers,” Int. J. Med. Eng.
pp. 1–16, 2019. Informat., vol. 12, no. 6, pp. 620–625, 2020.
[3] R. Hussein, H. Palangi, R. K. Ward, and Z. J. Wang, “Optimized deep [25] L. C. D. Nkengfack, D. Tchiotsop, R. Atangana, V. L. Door, and D. Wolf,
neural network architecture for robust detection of epileptic seizures using “Classification of EEG signals for epileptic seizures detection and eye
EEG signals,” Clin. Neuriophysiol., vol. 130, no. 1, pp. 25–37, 2019. states identification using jacobi polynomial transforms-based measures
[4] A. Subasi and M. Ismail Gursoy, “EEG signal classification using PCA, of complexity and least-squares support vector machines,” Informat. Med.
ICA, LDA and support vector machines,” Expert Syst. Appl., vol. 37, no. 12, Unlocked, vol. 23, 2021, Art. no. 100536.
pp. 8659–8666, 2010. [26] A. Alhudhaif, “A novel multi-class imbalanced EEG signals classification
[5] N. Zahra, N. Kanwal, N. S. Rehman, S. Ehsan, and K. D. McDonald-Maier, based on the adaptive synthetic sampling (ADASYN) approach,” Peer J
“Seizure detection from EEG signals using multivariate empirical mode Comput. Sci., vol. 7, 2021, Art. no. e523.
decomposition,” Comput. Biol. Med., vol. 88, pp. 132–141, 2017. [27] A. Humairani, B. S. Atmojo, I. Wijayanto, and S. Hadiyoso, “Fractal based
[6] H. Ocak, “Automatic detection of epileptic seizures in EEG using discrete feature extraction method for epileptic seizure detection in long-term EEG
wavelet transform and approximate entropy,” Expert Syst. Appl., vol. 36, recording,” in Proc. 2nd Int. Conf. Sci. Technol., 2021, pp. 1–10.
no. 2, pp. 2027–2036, 2009. [28] M. K. M. Rabby, A. K. M. K. Islam, S. Belkasim, and M. U. Bikdash,
[7] S. Saminu et al., “A recent investigation on detection and classification of “Epileptic seizures classification in EEG using PCA based genetic algo-
epileptic seizure techniques using EEG signal,” Brain Sci., vol. 11, no. 5, rithm through machine learning,” in Proc. ACM Southeast Conf., 2021,
2021, Art. no. 668. pp. 17–24.
[8] A. Shoeibi et al., “Application of deep learning techniques for automated [29] A. A. Ein Shoka et al., “Automated seizure diagnosis system based on fea-
detection of epileptic seizures: A review,” Int. J. Environ. Res. Public ture extraction and channel selection using EEG signals,” Brain Inf., vol. 8,
Health, vol. 18, no. 11, 2021, Art. no. 5780. 2021, Art. no. 1. [Online]. Available: https://doi.org/10.1186/s40708-021-
[9] Y. Jiang et al., “Seizure classification from EEG signals using transfer 00123-7
learning, semi-supervised learning and TSK fuzzy system,” IEEE Trans. [30] S. K. Prabhakar and S.-W. Lee, “ENIC: Ensemble and nature in-
Neural Syst. Rehabil. Eng., vol. 25, no. 12, pp. 2270–2284, Dec. 2017, clined classification with sparse depiction based deep and transfer learn-
doi: 10.1109/TNSRE.2017.2748388. ing for biosignal classification,” Appl. Soft Comput., vol. 117, 2022,
[10] A. Bhattacharyya et al., “Tunable-q wavelet transform based multiscale Art. no. 108416.
entropy measure for automated classification of epileptic EEG signals,” [31] S. K. Prabhakar, H. Rajaguru, and S.-H. Kim, “Schizophrenia EEG signal
Appl. Sci., vol. 7, no. 4, 2017, Art. no. 385. classification based on swarm intelligence computing,” Comput. Intell.
[11] T. Zhang and W. Chen, “LMD based features for the automatic Neurosci., vol. 2020, 2020, Art. no. 8853835. [Online]. Available: https:
seizure detection of EEG signals using SVM,” IEEE Trans. Neural //doi.org/10.1155/2020/8853835
Syst. Rehabil. Eng., vol. 25, no. 8, pp. 1100–1108, Aug. 2017, [32] L. Cardoso et al., “Abstract computation in schizophrenia detection
doi: 10.1109/TNSRE.2016.2611601. through artificial neural network based systems,” Sci. World J., vol. 2015,
[12] M. Ullah Hussain, E.-U.-H. Qazi, and H. Aboalsamh, “An automated 2015, Art. no. 467178. [Online]. Available: https://doi.org/10.1155/2015/
system for epilepsy detection using EEG brain signals based on deep 467178
learning approach,” Expert Syst. Appl., vol. 107, pp. 61–71, 2018. [33] V. Jahmunah et al., “Automated detection of schizophrenia using non-
[13] X. Wei et al., “Automatic seizure detection using three-dimensional CNN linear signal processing methods,” Artif. Intell. Med., vol. 100, 2019,
based on multi-channel EEG,” BMC Med. Informat. Decis. Mak., vol. 18, Art. no. 101698.
2018, Art. no. 111. [Online]. Available: https://doi.org/10.1186/s12911- [34] S. L. Oh, J. Vicnesh, E. J. Ciaccio, R. Yuvaraj, and U. R. Acharya,
018-0693-8 “Deep convolutional neural network model for automated diagnosis of
[14] A. G. Mahapatra et al., “Epilepsy EEG classification using morpholog- schizophrenia using EEG signals,” Appl. Sci., vol. 9, 2019, Art. no. 2870.
ical component analysis,” EURASIP J. Adv. Signal Process., vol. 2018, [35] R. Buettner, D. Beil, S. Scholtz, and A. Djemai, “Development of a
2018, Art. no. 52. [Online]. Available: https://doi.org/10.1186/s13634- machine learning based algorithm to accurately detect schizophrenia based
018-0568-2 on one-minute EEG recordings,” in Proc. 53rd Hawaii Int. Conf. Syst. Sci.,
[15] Ö. Turk and M. S. Ozerdem, “Epilepsy detection by using scalogram based 2020.
convolutional neural network from EEG signals,” Brain Sci., vol. 9, no. 5, [36] Z. Aslan and M. Akin, “Automatic detection of schizophrenia by applying
2019, Art. no. 115. deep learning over spectrogram images of EEG signals,” Traitement du
[16] S. Raghu, N. Sriraam, A. S. Hegde, and P. L. Kubben, “A novel approach Signal, vol. 37, no. 2, pp. 235–244, 2020.
for classification of epileptic seizures using matrix determinant,” Expert [37] S. K. Prabhakar, H. Rajaguru, and S.-W. Lee, “A framework for
Syst. Appl., vol. 127, pp. 323–341, Aug. 2019. schizophrenia EEG signal classification with nature inspired optimization
[17] N. Mahmoodian, A. Boese, M. Friebe, and J. Haddadnia, “Epileptic algorithms,” IEEE Access, vol. 8, pp. 39875–39897, 2020.
seizure detection using cross-bispectrum of electroencephalogram signal,” [38] P. T. Krishnan, A. N. Joseph Raj, P. Balasubramanian, and Y.
Seiuzre: Eur. J. Epilepsy, vol. 66, pp. 4–11, 2019. Chen, “Schizophrenia detection using multivariate empirical mode de-
[18] X. Wang, G. Gong, N. Li, and S. Qiu, “Detection analysis of epileptic EEG composition and entropy measures from multichannel EEG signal,”
using a novel random forest model combined with grid search optimiza- Biocybernet. Biomed. Eng., vol. 40, no. 3, pp. 1124–1139, 2020.
tion,” Front. Hum. Neurosci., vol. 13, 2019, Art. no. 52. doi: 10.3389/fn- [39] A. Shalbaf, S. Bagherzadeh, and A. Maghsoudi, “Transfer learning with
hum.2019.00052. deep convolutional neural network for automated detection of schizophre-
[19] W. Zhao et al., “A novel deep neural network for robust detection of nia from EEG signals,” Phys. Eng. Sci. Med., vol. 43, pp. 1229–1239,
seizures using EEG signals,” Comput. Math. Models Med., vol. 2020, 2020, 2020.
Art. no. 9689821. [40] K. Singh, S. Singh, and J. Malhotra, “Spectral features based convolutional
[20] D. Zhou and X. Li, “Epilepsy EEG signal classification algorithm neural network for accurate and prompt identification of schizophrenic
based on improved RBF,” Front. Neurosci. vol. 14, 2020, Art. no. 606. patients,” Proc. Inst. Mech. Eng., vol. 235, no. 2, pp. 167–184, 2021.
doi: 10.3389/fnins.2020.00606. [41] M. Baygin, O. Yaman, T. Tuncer, S. Dogan, P. D. Barua, and U. R. Acharya,
[21] Y. Gao, B. Gao, Q. Chen, J. Liu, and Y. Zhang, “Deep convo- “Automated accurate schizophrenia detection system using collatz pattern
lutional neural network-based epileptic electroencephalogram (EEG) technique with EEG signals,” Biomed. Signal Process. Control, vol. 70,
signal classification,” Front. Neurol., vol. 11, 2020, Art. no. 375. Sep. 2021, Art. no. 102936.
doi: 10.3389/fneur.2020.00375. [42] M. Al-Qatf, Y. Lasheng, M. Al-Habib, and K. Al-Sabahi, “Deep learning
[22] C. Gómez et al., “Automatic seizure detection based on imaged-EEG approach combining sparse autoencoder with SVM for network intrusion
signals through fully convolutional networks,” Sci. Rep., vol. 10, 2020, detection,” IEEE Access, vol. 6, pp. 52843–52856, 2018.
Art. no. 21833. [Online]. Available: https://doi.org/10.1038/s41598-020- [43] J. Guan and L. Jia, “A multi-objective particle swarm optimization algo-
78784-3 rithm for solving human resource allocation problem,” IPPTA: Quart. J.
[23] O. K. Cura et al., “Epileptic seizure classifications using empirical mode Indian Pulp Paper Tech. Assoc., vol. 30, no. 8, pp. 144–149, 2018.
decomposition and its derivative,” BioMed. Eng. OnLine, vol. 19, 2020, [44] J. Zhao, S. Liu, M. Zhou, X. Guo, and L. Qi, “An improved binary cuckoo
Art. no. 10. [Online]. Available: https://doi.org/10.1186/s12938-020- search algorithm for solving unit commitment problems: Methodological
0754-y description,” IEEE Access, vol. 6, pp. 43535–43545, 2018.
68 IEEE OPEN JOURNAL OF ENGINEERING IN MEDICINE AND BIOLOGY, VOL. 3, 2022

[45] L. F. Zhu, J. S. Wang, H. Y. Wang, S. S. Guo, M. W. Guo, and W. Xie, “Data [50] D. Pandey and P. Pandey, “Approximate q-learning: An introduction,”
clustering method based on improved bat algorithm with six convergence in Proc. 2nd Int. Conf. Mach. Learn. Comput., Bangalore, India, 2010,
factors and local search operators,” IEEE Access, vol. 8, pp. 80536–80560, pp. 317–320.
2020. [51] R. G. Andrzejak, K. C. Lehnertz, F. Rieke, P. Mormann, and C. E. Elger,
[46] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, “Indications of nonlinear deterministic and finite dimensional structures
Cambridge, MA, USA: MIT Press, 1998. in time series of brain electrical activity: Dependence on recording region
[47] W. J. Lu et al., “A CNN-BiLSTM-AM method for stock price prediction,” and brain state,” Phys. Rev. E., Stat. Nonlinear Soft Matter Phys., vol. 64,
Neural Comput. Appl., vol. 33, pp. 4741–4753, 2020. 2001, Art. no. 061907.
[48] H. Ge, Z. Yan, W. Yu, and L. Sun, “An attention mechanism based [52] E. Olejarczyk and W. Jernajczyk, “Graph-based analysis of brain connec-
convolutional LSTM network for video action recognition,” Multimedia tivity in schizophrenia,” PLoS One, vol. 12, 2017, Art. no. e018862.
Tools Appl., vol. 78, no. 14, pp. 20533–20556, 2019.
[49] K. S. Tai, R. Socher, and C. D. Manning, “Improved semantic representa-
tions from tree-structured long short-term memory networks,” May 2015,
arXiv:1503.00075v3.