Layer recurrent neural network based intelligent user activity classification model using smartphone

Abstract

User activity classification is one of the most popular research topic in the domain of health care and social care, since this automated technology can provide monitoring and understanding of activities of patients. Smartphone inbuilt sensors based User Activity Classifier (UAC) recognizes user activities using features extracted from sensors like accelerometer and gyroscope in build in smartphones. In this research paper, we are proposing a new user activity classifier system using Layer Recurrent Neural Network (LRNN) which is Artificial Neural Network (ANN). We utilize synthesized data, containing features of user activity classification system, extracted from the raw data recorded in smartphones. With these derived features, we train and test Layer Recurrent Neural Network classifier for user activity classifier. In order to evaluate this system, we have compared the performance of this Layer Recurrent Neural Network based user activity classifier against the convention Multilayer Perceptron (MLP) and Naive Bayes based user activity classifier. Test results show that the proposed Layer Recurrent Neural Network -based user activity classifier is able to recognize user activities reliably and outperforms the Multilayer Perceptron based user activity classifier. We have achieved the classification accuracy of 98.56% for the activities. The results are much more accurate than Multilayer Perceptron based classifier and Naive Bayes classifier.

Keywords

Human activity classification artificial neural network naive bayes classifier smartphone

1 Introduction

Ambient intelligence is one of the emerging discipline that brings intellect to our everyday ambience and makes those surrounding sensitive to human beings and User Activity Classification is an important task of it. User activity classification has emerged one of the impelling field of research over the last decade. It is a very necessary and daring domain which can support many important omnipresent applications [1]. User activity classification is a method to determine and recognize different user activities via external sensors like accelerometer and gyroscope.

The use of ingrained and pre-installed sensors in the design of smartphones and other smart gadgets has opened the pathway to a large number of uses and operations that were difficult to exercise in later years without using and modern sensors. Acceleration-metre, digital compass, gyroscopic metre, Global Positioning System, mic-recorders and video camera are examples of some of these sensors. These sensing smart-equipment’s assists us to gather unaltered raw information, to be utilised in the utilization and usage of different function and employment, from these smart equipment. The Global Positioning System, for e.g., is utilised in tracking based software and on the flip side luminous sensing devices are utilised in estimating proximity among various clients and the smart devices. An acceleration meter is gravitational force sensitive, thus any shift in the smartphone’s orientation will generate different accelerometer readings [2]. These readings act as inputs to many installed software applications in these smart-devices, majorly for detecting motion, sensing user’s posture and various user interaction. Many applications depend hugely on the raw information retrieved by linking smartphone’s acceleration meter. The client’s synergy with the smart technology, while playing and using entertainment application like games, requires that the user constantly change the angular orientation of the smart device against the gravity. Another famous use of acceleration meters in smart devices is using the raw information acquisition to change the screen display view. When the client angularly rotates the smart gadget from horizontal to vertical (or vice versa) the display of screen will rotate accordingly. Gyroscope also does the similar thing but unlike an accelerometer which measures linear acceleration of movement, a gyrometer measures the angular rotational velocity.

In last decade, user activity classification has evoked significant interest among researchers in the areas of healthcare, social care, and life care services, since it allows automated data monitoring and analysing of any person’s activities [3]. The process of User Activity Classification involves feature extraction from sensors, modelling and conversion of raw data into useable form and applying classification techniques. User activity classification via smart phone sensor is a challenging task because of the frequency and amount of data received, both accelerometer and gyroscope in all direction, in addition to the noise in the data recorded. Though many researchers have been exploring sensor based User Activity Classification systems due to their practical applications, accurate classification of user activities still remains as a major challenge.

2 Related work

First significant breakthrough in UAR came in 1999, which as that now one can distinguish between two-dimensional and three-dimensional methods. In that same year a new taxonomy was shown and explained with main focus subjects motion analytics. In 2003 a hierarchical action categorization was proposed to find hierarchy. In is also important to know that action and activity are not same asproposed. In 2011 tree structured taxonomy became popular for classification purposes. In 2013 three-dimensional and motion capture technology started being exploited mainly using three dimensional death analysis. In last couple of years image processing based user activity classification has become quite popular. User activity classifier can also be used in for speck and voice recognition. Many security systems nowadays uses user voice recognition technique to identify a person. User Activity Classification is now being seen as a three dimensional GPS.

Research gap related with this research domain is: Most of the researches in the field of User activity classification is confined with respect to the devices used to extract various features from the patient. None the research uses feature extraction quality of sensors we come across very easily in this smart environment. Unlike other research work which uses high quality sensors and cameras to obtain user activity data we will be using data from the sensor values of accelerometer and gyroscope build in our smart phones. The challenge faced is this approach is that data that is recorded is mixed with noise and other external factor from the environment. Also data obtained for some activities are very scarce. This approach provide a cheap and user feasible method for User activity classification.

3 Data set

The attributes chosen for this dataset come from the acceleration-meter and gyro-scope three-axial un-synthesised signals t_Acc-XYZ and t_Gyro-XYZ. These time domain signals (time is donated by prefix’t’) were recorded at a steady rate of fifty hertz. Then they are distilled with the help of a median filter and a third order low pass Butterworth filter with a corner frequency of twenty hertz to clear away noise. Similarly, the signals from acceleration meter were then disassociated into user’s body and gravitational acceleration signals (tBdAcc-XYZ and tGrvityAcc-XYZ) by employing one more low pass Butterworth filter with a corner frequency of 0.3 Hz [4, 5]. Similarity afterwards, the user’s body linear acceleration and angular velocity were retrieved and synthesised in time domain to retriece Jerk signals (tBdAccJerk-XYZ and tBdGyroJerk-XYZ). Also the magnitude/amplitude of these three-dimensional signals were computed by utilising the Euclidean norm (tBdAccMag, tGrvityAccMag, tBdAccJerkMag, tByGyroMag, tBodyGyroJerkMag). Finally a Fast Fourier Transform (FFT) was enforced to some of these signals which resulted in producting fBdAcc-XYZ, fBdAccJerk-XYZ, fBdGyro-XYZ, fBdAccJerkMag, fBdGyroMag, fBdGyroJerkMag. (prefix ‘f’ indicates signals with frequency domain). These signals were utilised for the assessment of variables of the feature vector for each and every activity pattern:

’-XYZ’ is used to denote 3-axial signals in the X, Y and Z directions.

tBdAcc-XYZ tGrvityAcc-XYZ tBdAccJerk-XYZ

tBoGyro-XYZ tBdGyroJerk-XYZ tBdAccMag

tGrvityAccMg tBdAccJerkMag tBdGyroMag

tBdGyroJerkMag fBdAccMag fBdAcc-XYZ fBdGyro-XYZ fBdAccJerkMag

fBdGyroMag fBdGyroJerkMag gravi-tyMean

tBdAccMean tBdAccJerkMean tBdGy-roMean

tBdGyroJerkMean

Many other vectors retrieved by calculating the mean of the signals in a signal window sample which are utilised by the angle () function variable are grav-ity Mean, tBdAccMean, tBdAccJerkMean, tBdGy-roMean, tBdGyroJerkMean [6, 7].

Mathematical functions used for synthesising the raw input data are mean, correlation, standard deviation, max index, median, kurtosis, skewness, max, bands Energy, min, angle, Simple moving average., mean Frequency, energy, interquartile range, entropy, arCoeff.

They all carried out a set of activities comprising of 6 elemental activities: three static condition (standing still, sitting, lying down) and three dynamic activities (walking, walking upstairs and walking downstairs). The study also consisted of postural changeover that took place between the static postures. These are: stand->sit, sit->stand, sit->lie, lie->sit, stand->lie, and lie->stand (see Table 1). All the subjects were carrying a smartphone (Samsung Edge S8) in their pockets during the study’s implementation. We seizes three-axial linear acceleration and three-axial angular velocity at a consistent rate of fifty hertz using the pre-installed accelerometer and gyro-scope in the smartphone. The captured dataset was randomly divided into two batches, in which seventy percent of the subjects were chosen for providing the training dataset and thirty percent the testing dataset. The sensor signals (accelerometer and gyroscope) were pre-synthesized by the application of noise filters and then sampled in fixed-width sliding windows of 2.56 seconds and fifty percent overlap (128 readings per window). The accelerometer signal data, which has acceleration because of gravitational force (9.8 m/s^2) and body movement components, was separated using a Butterworth low-pass filter into user’s body acceleration and gravity (9.8 m/s^2) [8]. The gravitational force is pretended of having very few frequency components, therefore it was concluded to use a filter with 0.3 Hz cut-off frequency. From each monitor, a vector of five sixty-one features was retrieved by calculating different variables from frequency domain and time domain. The dataset is then separated in two groups and can be utilised separately.

Data set of all twelve activities consists of 10929 activity cases in which 7767 are reserved for training and the remaining 3162 for testing.If we don’t consider state transition activities there are 7415 train cases for first 6 classes and 2996 test cases.

For activity 1, there are 1226 train and 496 test cases. Activity 2 consists of 1073 train and 471 test cases. Activity 3 consists of 987 train and 420 est cases. Activity 4 consists of 1293 train and 508 test cases. Activity 5 consists of 1423 train and 556 test cases. Activity 6 consists of 1413 train and 545 test cases. Activity 7 consists of 47 train and 23 test cases. Activity 8 consists of 23 train and 10 test cases. Activity 9 consists of 75 train and 32 test cases. Activity 10 consists of 620 train and 25 test cases. Activity 11 consists of 90 train and 49 test cases. Activity 12 consists of 57 train and 27test cases.

4 Methodology

The Layer Recurrent Neural Network based design of classifier enables us to make and design a wide variety of neural network configurations. It is feasible to build, train and use neural networks.

Various steps for implementing User Activity Classification using ANN:-

4.1 Feed forward neural networks

ANN enables us to form and implement Multilayer Perceptron (MLP) neural networks. A Multilayer Perceptron is a feed forward artificial neural network can be illustrated by:

one input layer with R number of layers of inputs.

M–1(numbers) hidden layers, where each and every layer m has an random number of neurons S^m.

and one output layer with S^M number of neurons, which correspond to the number of outputs of the neural network

The trailing notation enables n short description of a MLP which produces the number of inputs R, M number of layers and S^m number of neurons in each and every layer m: $nn = [R S^{1} S^{2} \dots \dots S^{M}]$ (1)

In a Multilayer Perceptron each and every layer has all connection to the coming following layer, which implies that each and every neuron output generated in the layer m acts like an input to each and every neuron in layer m + 1 (and inputs are connected to all neurons in the first layer). Figure 1 shows an example of a design of a Multilayer Perceptron with two neurons, two inputs with two hidden layers each and an output layer with only one neuron (and therefore only one output $\hat{y}$ ). As you can see these is no feedback path in the design of Multilayer Perceptron. The Multilayer Perceptron can be described with $nn = [2 2 2 1]$ (2)

Fig. 1

MLP Architecture with [2 2 2 1] configuration.

The Multilayer Perceptron shown in the example in Fig. 1 consists of five mathematical unit called neurons. The structure of these mathematical unit is shown in Fig. 2.

Fig. 2

Structure of a neuron.

Neurons are the mathematical summative units in an artificial neural network. They can have many inputs p which are multiplied by the weights w corresponding to each connection and added up together with the bias weight b for each corresponding connection to the summation output n. Then the neuron output of the ANN “a” is derived by the transfer function (also known as activation function) f (n). $n = \sum^{i} (wipi) + b$ (3) $a = f (n)$ (4)

Generally various distinct transfer function can be utilised in an artificial neural network. In ANN the transfer function can be described as:

a = tanh (n) (Hyperbolic tangent) for each and every neurons in hidden layers

y = a = n(Linear function) for each and every neurons in the output layer.

The array-matrix definition and design allows a feasible and simple to understand description of a Multilayer Perceptron. Therefore the input p, the ANN outputs $\hat{y}$ , the added outputs n, the layer outputs a, the activation functions f (n) and the bias weights b for each and every connection of one layer m are represented by the arrays $\underline{p}, \hat{\underline{y}}, {\underline{n}}^{m}, {\underline{a}}^{m}, {\underline{f}}^{m}, {\underline{b}}^{m}$ (the superscript represents the layer m and the subscript the number of the neuron or input). $\underline{p} = [\begin{matrix} p \\ . . . \\ pR \end{matrix}]$ (5) $\underline{\hat{y}} = [\begin{matrix} \hat{y} \\ . . . \\ \hat{y} s^{M} \end{matrix}]$ (6) ${\underline{n}}^{m} = [\begin{array}{l} n_{1}^{m} \\ ... \\ n_{S^{m}}^{m} \end{array}]$ (7) ${\underline{a}}^{m} = [\begin{array}{l} a_{1}^{m} \\ ... \\ a_{S^{m}}^{m} \end{array}]$ (8) ${\underline{f}}^{m} = [\begin{array}{l} f_{1}^{m} \\ ... \\ f_{S^{m}}^{m} \end{array}]$ (9) ${\underline{b}}^{m} = [\begin{array}{l} b_{1}^{m} \\ ... \\ b_{S^{m}}^{m} \end{array}]$ (10)

The connection bias weights w are represented by the matrix ${\tilde{IW}}^{1, 1}$ which contains the connection weights of the first layer and the matrices ${\tilde{IW}}^{m, 1}$ which contain the weights that connect the outputs of layer l with layer m. For the example in Fig. 1 the connection matrices are: ${\tilde{IW}}^{1, 1} = [\begin{matrix} w_{1, 1}^{1} & w_{1, 2}^{1} \\ w_{2, 1}^{1} & w_{2, 2}^{1} \end{matrix}]$ (11) ${\tilde{LW}}^{2, 1} = [\begin{matrix} w_{1, 1}^{2} & w_{1, 2}^{2} \\ w_{2, 1}^{2} & w_{2, 2}^{2} \end{matrix}]$ (12) ${\tilde{LW}}^{3, 2} = [\begin{matrix} w_{1, 1}^{3} & w_{1, 2}^{3} \\ w_{2, 1}^{3} & w_{2, 2}^{3} \end{matrix}]$ (13)

Figure 3 shows the array-matrix illustration of the MLP of Fig. 1.

Fig. 3

Multilayer Perceptron with two hidden layers.

4.2 Recurrent neural networks

Layer Recurrent Neural Network based artificial neural network also enables us to define various different topologies of artificial neural networks, which are networks, where connections among various units form a directed cycle. In ANN this is exercised and utilised by joining the all the ANN outputs of a given layer m with the input of previous layers <m or with its own layer input. Tapped Delay Lines (TDL) are delays that can be introduced by a delay operators z^-d where d is the unit by which time discrete signal is delayed. To introduce the delay the Sets DI^l,m and DL^l,m are used to define TDL. TDL consists of each and every delays d_i among a connections from the output of layer l of ANN to the input of layer m of ANN. Consequently for every d_i ∈ DI^l,m or d_i ∈ DL^l,m there is supposed to be a connection matrix ${\tilde{IW}}^{m, l} [d_{i}]$ or ${\tilde{LW}}^{m, l} [d_{i}]$ . This is shown in Fig. 4. This Tapped delay lines integrated ANN design makes it possible to include three different types of TDL. The influence of these setting on the neural network structure is explained with the help of Fig. 5.

Fig. 4

Taped Delay Lines in elaborated (a) and simplified (b) illustration.

Fig. 5

Layer Recurrent Neural Network (LRNN) with two hidden layer.

Input delays dIn ∈ [0, 1, 2, …] (blue; default for MLP dIn = [0]):

This allows to delay the inputs $\underline{p}$ of the neural network by any real-valued time-step d ≥ 0. dIn cannot be empty because inputs need to be connected with the neural network. If the current input has be used in the ANN, dIn has to contain 0. Since this only delays the inputs, this will not lead to a recurrent network. ${DI}^{1, 1} = dIn$ (14)

Output delays dOut ∈ [1, 2, …] (green; default for MLP dOut ∈ []

This enables to add a recurrent connection of the outputs $\hat{\underline{y}}$ of the neural network to its first layer Since this adds a recurrent connection if dIn is non-empty, the delays must be greater than zero i.e. d > 0. A neural network with this a connection is a Layer Recurrent Neural Network. ${DL}^{(1, M)} = dIn$ (15)

Internal delays dIntern ∈ [1, 2, …] (red; default for MLP dOut = []):

It allows to introduce internal delays inside the network. Since this adds recurrent connections if dIntern is non-empty, the delays must to be greater than zero d > 0. A neural network with such a connection will be a Layer Recurrent Neural Network. ${DL}^{m, l} = dIntern \forall (m ⩽ l | {DL}^{m, l} \neq {DL}^{1, M}$ (16)

In Layer Recurrent Neural Network based user activity classification model all only have no delays direct connections. This is because none of the activity occurrence depends of the occurrence of the previous activity, than is why TDL is set asnull.

4.3 Training the neural network

Once a neural network is created, it can be trained. To train a neural network, training data is required. If you have a system, that produces the output. $\underline{y}$ , when given the input $\underline{p}$ , (see Fig. 6), ( $\underline{p}, \underline{y}$ ) represents one sample of training data.

Fig. 6

Generated training data set $\tilde{P}$ and $\tilde{Y}$ of a system.

Fig. 7

A system with input $\underline{p}$ and $\underline{y}$ output.

Fig. 8

Training a neural network.

Training data is defined by an input matrix $\tilde{P}$ and an output (or target) matrix $\tilde{Y}$ containing Q samples of training data (see Fig. 7). For static systems (Multilayer Perceptron based neural networks) it is only essential that element q of the input matrix corresponds to element q of the output matrix (in any given order). For dynamic systems (recurrent neural networks) the samples have to be in the correct time order. For both systems, the training data should represent the system as good as possible. $\tilde{P} = [\underline{p} [1] \underline{p} [2] \dots \dots . . \underline{p} [q] \dots \dots \dots . \underline{p} [Q]]$ (17) $\tilde{Y} = [\underline{y} [1] \underline{y} [2] \dots \dots . . \underline{y} [q] \dots \dots \dots . \underline{y} [Q]]$ (18)

With the input and output training data, neural network can be trained. Training a neural network means, that all weights in the weight vector $\underline{w}$ , which contains all connection weights $\tilde{IW}$ and $\tilde{LW}$ and all bias weights $\underline{b}$ , are updated step by step, such that the neural network output $\hat{\tilde{Y}}$ matches the training data output (target) $\tilde{Y}$ . The objective of this optimization is to minimize the error E (cost function) between neural network and systemoutputs.

Generally there are different methods to calculate the error E (cost function) for neural network training like Quadratic cost function, Quadratic cost function, Exponentional cost function, Hellinger distance function, Kullback—Leibler divergence function, Generalized Kullback—Leibler divergence function, Itakura—Saito distance, etc.

In ANN the mean squared error is used, which is necessary to apply the Levenberg-Marquardt algorithm.The training repeats adapting the weights of the weight vector $\underline{w}$ until one of the two termination conditions becomes active (see Fig. 8):

the maximal number of iterations (epochs) k_max is reached

the Error is minimized to the goal E ≤ E_stop

Fig. 9

Getting output from trained Artificial Neural Network.

The Levenberg—Marquardt algorithm, a second order optimization method that uses the Jacobian matrix $\tilde{J}$ to approximate the Hessian matrix $\tilde{H}$ , is used a weights optimisation algorithm to modify the values of neural networks weight in each connection in order to make Layer Recurrent Neural Network based classifier learn with respect to the cost function E. In ANN the Jacobian matrix is calculated using the Real-Time Recurrent Learning (RTRL) algorithm [9]. Real-Time Recurrent Learning (RTRL) algorithm is much faster as compared to the conventional Back Proportion Through Time learning algorithm because unlike Back Proportion Through Time learning algorithm, which is a static learning algorithm (iterating process stops for every single epoch), Real-Time Recurrent Learning (RTRL) algorithm is a dynamic learning algorithm.

4.4 Using the trained neural network for future purpose

Once a neural network is trained successfully, it can be used to calculate the neural network outputs for new (different from the training data) input data. The input data $\tilde{P}$ for using the neural network has the same structure than for training. The neural network calculates the output data $\hat{\tilde{Y}}$ which has the same structure than the training output data Q. Any arbitrary number of data samples Q can be used, resulting in the same amount of output samples (see Fig. 9). $\tilde{P} = [\underline{p} [1] \underline{p} [2] \dots \dots . . \underline{p} [q] \dots \dots \dots . \underline{p} [Q]]$ (19) $\hat{\tilde{Y}} = [\hat{\underline{y}} [1] \underline{\hat{y}} [2] \dots \dots . . \hat{\underline{y}} [q] \dots \dots \dots . \hat{\underline{y}} [Q]]$ (20)

4.5 Naive bayes classifier

Naive Bayes classifier is a simple probabilistic classifier based on applying Bayes theorem with strong (naïve) independence assumption between the features [10]. We use Naïve Bayes Classifier in this research paper to classify the user activities and compare it with Multilayer Perceptron (MLP) Neural Networks and Layer Recurrent Neural Networks (LRNN).

Naive Bayes classifier works on the concept of Bayes theorem:- $P (\frac{c}{x}) = \frac{P (\frac{x}{c}) P (c)}{P (x)}$ (21)

5 Results and discussions

To evaluate User Activity Classification system, we have compared the process on Multilayer Perceptron (MLP) Neural Networks, Naive Bayes classifier and Layer Recurrent Neural Networks (LRNN) respectively using the same 10929X561 dataset. Table 2 shows the accuracies of the proposed User Activity Classification system via Layer Recurrent Neural Networks and the comparison results of using Multilayer Perceptron for all listed twelve activity classes and Table 3 shows accuracies of first sixactivities.

Table 1
Activity Labels

Activity	Activity Code for LRNN	Activity Code for MLP
WALKING	1	100000000000
WALKING_UPSTAIRS	2	010000000000
WALKING_DOWNSTAIRS	3	001000000000
SITTING	4	000100000000
STANDING	5	000010000000
LAYING	6	000001000000
STAND TO SIT	7	000000100000
SIT->STAND	8	000000010000
SIT ->LIE	9	000000001000
LIE ->SIT	10	000000000100
STAND->LIE	11	000000000010
LIE ->STAND	12	000000000001

Table 2

Comparison of accuracies of UAC based on MLP and LRNN for all twelve classes

ACTIVITY NAME	CLASSIFICATION ACCURACY (%)
	MLP	NBSK	LRNN
WALKING	83.8	88.1	98.8
WALKING UPSTAIRS	95.3	87.1	98.7
WALKING DOWNSTAIRS	94.2	65.7	97.6
SITTING	93.3	72.2	96.9
STANDING	86.7	92.1	97.8
LAYING	99.6	98.7	98.9
STAND TO SIT	31.4	73.9	39.1
SIT TO STAND	0.0	44.4	70.0
SIT TO LIE	44.4	43.8	43.8
LIE TO SIT	47.5	36	52.0
STAND TO LIE	45.8	61.2	44.9
LIE TO STAND	10.0	18.5	48.1
MEAN(SD)	61.0(34.0)	65.2(24)	73.89(25.23)

Table 3

Comparison of accuracies of UAC based on MLP and LRNN for first six classes

ACTIVITY NAME	CLASSIFICATION ACCURACY (%)
	MLP	NBSK	LRNN
WALKING	83.8	88.1	98.8
WALKING UPSTAIRS	95.3	87.1	98.7
WALKING DOWNSTAIRS	94.2	65.7	97.6
SITTING	93.3	72.2	96.9
STANDING	86.7	92.1	97.8
LAYING	99.6	98.7	98.9
MEAN(SD)	92.15(5.3)	83.9(11.40)	98.12(0.75)

The results for Multilayer Perceptron and Naive Bayes classifier show the lower performance compared to Layer Recurrent Neural Network. The results show that the average accuracy of Multilayer Perceptron is 61% with standard deviation (SD) of 34.0 and for Naive Bayes classifier it’s slightly higher with 65.2% and a lower standard deviation (SD) of 24.0.

The results by Layer Recurrent Neural Network show the better classification accuracy of 73.89% with standard deviation of 25.23. The LRNN based user activity classifier is 12.89% more accurate than Multilayer Perceptron based User Activity Classifier and 8.74% more accurate that Naive Bayes classifier. Table 4 shows a confusion matrix for twelve activities using Multilayer Perceptron, Table 5 shows a confusion matrix for twelve activities using Naive Bayes classifier and Table 6 shows a confusion matrix for twelve activities using Layer Recurrent Neural Network.As one can notice the mean accuracy of UAC based system for all twelve activity is quite low. Table 3 shows the accuracies of the proposed User Activity Classification system via LRNN and the comparison results of using Multilayer Perceptron and Naive Bayes classifier for only first six listed activity classes. The results for Multilayer Perceptron and Naive Bayes classifier show the lower performance compared to Layer Recurrent Neural Network. The results show that the average accuracy of Multilayer Perceptron is 61% with standard deviation (SD) of 34.0 and for Naive Bayes classifier it’s slightly higher with 65.15% and a lower standard deviation (SD) of 24.0. The results by Layer Recurrent Neural Network show the better classification accuracy of 73.89% with standard deviation of 25.23. The Layer Recurrent Neural Network based user activity classifier is 12.89% more accurate than Multilayer Perceptron based User Activity Classifier and 8.74% more accurate that Naive Bayes classifier. Table 4 shows a confusion matrix for twelve activities using Multilayer Perceptron, Table 5 shows a confusion matrix for twelve activities using Naive Bayes classifier and Table 6 shows a confusion matrix for twelve activities using LRNN. As one can notice the mean accuracy of UAC based system for all twelve activity is quite low. Table 3 shows the accuracies of the proposed User Activity Classification system via RNN and the comparison results of using Multilayer Perceptron and Naive Bayes classifier for only first six listed activity classes. The results show that the average accuracy of Multilayer Perceptron is 92.15% with standard deviation of 5.3 and for Naive Bayes classifier it’s 83.9% with standard deviation of 11.4.The results by Layer Recurrent Neural Network show the better classification accuracy of 98.12% with standard deviation of 0.75. The Layer Recurrent Neural Network based User Activity Classifier is 5.97% more accurate than Multilayer Perceptron based User Activity Classifier and 14.22% more accurate then Naive Bayes classifier. This significant increase in mean accuracy is due the abandon dataset availability of first six activity classes as compares to last six activity classes. Tables 4, 5 and 6 shows a confusion matrix for 12 activities using Multilayer Perceptron and Layer Recurrent Neural Network.

Table 4

Confusion matrix for MLP based UAC (all classes)

TARGET CLASS
OUTPUT CLASS	Class code	1	2	3	4	5	6	7	8	9	10	11	12	% accuracy
	1	485	53	13	0	4	1	0	0	0	0	3	0	83.8
	2	3	405	15	1	0	0	0	0	1	0	0	0	95.3
	3	8	11	392	0	0	2	2	0	0	0	0	1	94.2
	4	0	0	0	446	24	0	3	0	2	0	3	0	93.3
	5	0	1	0	58	526	20	2	0	0	0	0	0	86.7
	6	0	0	0	2	0	522	0	0	0	0	0	0	99.6
	7	0	0	0	1	2	0	11	9	3	1	7	1	31.4
	8	0	0	0	0	0	0	0	0	0	0	0	0	0.0
	9	0	0	0	0	0	0	0	0	4	0	5	0	44.4
	10	0	0	0	0	0	0	0	0	1	19	1	19	47.5
	11	0	1	0	0	0	0	5	0	19	2	27	5	45.8
	12	0	0	0	0	0	0	0	1	2	3	3	1	10.0
	% accuracy	97.8	86.0	93.3	87.8	94.6	95.8	47.8	0.0	12.5	76.0	55.1	3.7	89.8

Table 5

Confusion matrix for NBS based UAC (all classes)

TARGET CLASS
OUTPUT CLASS	Class code	1	2	3	4	5	6	7	8	9	10	11	12	% accuracy
	1	437	16	43	0	0	0	0	0	0	0	0	0	88.10
	2	30	410	20	0	0	0	11	0	0	0	0	0	87.0
	3	78	66	276	0	0	0	0	0	0	0	0	0	65.7
	4	0	0	0	367	133	0	6	2	0	0	0	0	72.2
	5	0	0	0	33	512	0	10	0	0	1	0	0	92.0
	6	0	0	0	0	0	538	0	0	0	5	1	1	98.7
	7	0	2	0	0	0	0	17	2	1	0	1	0	73.9
	8	0	0	0	0	0	0	1	4	3	0	0	1	44.4
	9	0	0	0	0	0	0	0	1	14	8	3	0	43.75
	10	0	0	0	0	0	0	0	0	1	9	10	5	36
	11	0	0	0	0	1	0	2	0	15	1	30	0	61.2
	12	0	0	0	0	0	0	0	0	10	8	4	5	18.5
	% accuracy	80.2	83.0	81.4	91.8	79.3	100	36.2	44.4	31.8	29.0	53.6	41.7	82.9

Table 6

Confusion matrix for LRNN based UAC (all classes)

TARGET CLASS
OUTPUT CLASS	Class code	1	2	3	4	5	6	7	8	9	10	11	12	% accuracy
	1	490	6	0	0	0	0	0	0	0	0	0	0	98.8
	2	5	465	1	0	0	0	0	0	0	0	0	0	98.7
	3	4	6	410	0	0	0	0	0	0	0	0	0	97.6
	4	0	0	1	492	12	1	0	1	1	0	0	0	96.9
	5	0	1	0	8	544	0	0	2	1	0	0	0	97.8
	6	0	0	1	0	3	539	0	2	0	0	0	0	98.9
	7	1	0	2	0	4	1	9	1	4	1	0	0	39.1
	8	0	0	0	1	0	1	0	7	0	1	0	0	70.0
	9	1	0	0	0	0	0	0	0	14	4	5	8	43.8
	10	0	0	0	0	0	1	1	1	0	13	2	7	52.0
	11	0	0	0	2	1	1	1	3	12	3	22	4	44.9
	12	0	0	0	0	1	0	3	1	3	2	4	13	48.1
	% accuracy	97.8	97.3	98.8	97.8	96.3	99.1	64.3	38.9	40.0	54.2	66.7	40.6	95.4

Table 7

Compiled overall result

Number of classes considered Classifier used	All 12 classes			First 6 classes
	MLP classifier	NBS classifier	LRNN classifier	MLP classifier	NBS classifier	LRNN classifier
Mean accuracy (per class)	61.0 %	65.15 %	73.89 %	92.15 %	83.99 %	98.12 %
Standard deviation	34.0%	24.0 %	25.23 %	5.33 %	11.41 %	0.75 %
Overall accuracy	89.8 %	82.85 %	95.4 %	92.78 %	85.84 %	98.36 %
Error	10.2 %	17.15 %	4.6 %	7.22 %	14.16 %	1.63%

In case of activity number 1 that is walking which is a dynamic activity, both Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier predicts with an accuracy of 83.8% and 88.1% which is much lower to the accuracy of Layer Recurrent Neural Network based classifier. This is because they are not able to distinguish between walking, walking upward and walkingdownward.

In case of activity number 2 that is waking upstairs which is a dynamic activity, Multilayer Perceptron Neural Network based classifier predicts with the accuracy of 95.3% and Naive Bayes classifier prediction accuracy is 87.0% which are significantly less than Layer Recurrent Neural Network based classifier which is 98.7%.This is because both Multilayer Perceptron based classifier and Naive Bayes classifier wrongfully classifies walking upstairs activity, majorly, as walking normally and walking downstairs. This happens because accelerometer data only vary in only one axis to all three different dynamic activity which makes a static neural network like Multilayer Perceptron and Naive Bayes classifier to get wrongfully trained. But Layer Recurrent Neural Network based classifier easily distinguishes between these activities because of recurrent network designing, therefore giving significantly better results compared to Multilayer Perceptron Neural Network based classifier.

Similarly in case of activity number 3 that is waking upstairs which is a dynamic activity, Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier predicts with the accuracy of 94.2% and 65.7 which is less than Layer Recurrent Neural Network based classifier which is 98.8%. This is because Multilayer Perceptron based classifier and Naive Bayes classifier wrongfully classifies walking downstairs activity, majorly, as walking normally and walking upstairs. This happens because accelerometer data only vary in only one axis to all three different dynamic activity which makes a static neural network like Multilayer Perceptron and Naive Bayes classifier, to get wrongfully trained. But Layer Recurrent Neural Network based classifier easily distinguishes between these activities because of recurrent network designing, therefore giving significantly better results compared to Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier.

In case of activity number 4 that is sitting which is a static activity, Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier predicts with the accuracy of 93.3% and 72.2% which is significantly less than Layer Recurrent Neural Network based classifier which is 97.8%. This is because Multilayer Perceptron based classifier and Naive Bayes classifier wrongfully classifies sitting activity, majorly, as standing. This happens because accelerometer data and gyroscope readings only vary a little in all three different static activity which makes a static neural network like Multilayer Perceptron and Naive Bayes classifier to get wrongfully trained. But Layer Recurrent Neural Network based classifier easily distinguishes between these activities because of recurrent network designing, therefore giving significantly better results compared to Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier.

In case of activity number 5 that is standing which is a static activity, Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier predicts with the accuracy of 86.7% and 92.0% which is less than Layer Recurrent Neural Network based classifier which is 97.8%. This is because Multilayer Perceptron based classifier and Naive Bayes classifier wrongfully classifies standing activity, majorly, as sitting. This happens because accelerometer data and gyroscope readings only vary a little in all three different static activity which makes a static neural network like Multilayer Perceptron and Naive Bayes classifier to get wrongfully trained. But unlike previous cases Layer Recurrent Neural Network based classifier also wrongfully classifies these activities. Though the accuracy of Layer Recurrent Neural Network based classifier is better than Multilayer Perceptron based classifier and Naive Bayes classifier but Multilayer Perceptron based classifier also does a respectable job in classifying this activity.

In case of activity number 6 that is laying which is a static activity, Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier predicts with the accuracy of 99.6% and 98.7% which is slightly more or less than Layer Recurrent Neural Network based classifier which is 98.9%.

In case of activity number 7 that is stand to sit which is a state transition activity, Multilayer Perceptron Neural Network based classifier predicts with the accuracy of 31.4% which is slightly less than Layer Recurrent Neural Network based classifier which is 39.1%. This is because Multilayer Perceptron based classifier wrongfully classifies stand to sit activity. This happens because of the lack of training data cases. Layer Recurrent Neural Network based classifier also does not provides good results but is slightly better than Multilayer Perceptron based classifier because of recurrent network designing. But Naive Bayes classifier performs better with 73.9%. This is because Naive Bayes classifier is based on Bayes theorem which sometime can perform better with there is less data in training set.

In case of activity number 8 that is sit to stand which is a state transition activity, Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier predicts with the accuracy of 0.0% and 44.4% which is significantly less than Layer Recurrent Neural Network based classifier which is 70.0%. Multilayer Perceptron based classifier is not able to classify a single case of this activity correctly. This happens because of the lack of training data cases. Layer Recurrent Neural Network based classifier also does provides good results and is significantly better than Multilayer Perceptron based classifier and Naive Bayes classifier because of recurrent network designing.

In case of activity number 9 that is sit to lie which is a state transition activity, Multilayer Perceptron Neural Network based classifier, Naive Bayes classifier and LRNN based classifier, all predicts with the accuracy of around 44%.

In case of activity number 10 that is lie to sit which is a state transition activity, Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier predicts with the accuracy of 47.5% and 36.0% which is slightly less than Layer Recurrent Neural Network based classifier which is 52.0%. This happens because of the lack of training data cases. Layer Recurrent Neural Network based classifier does not provides good results and but still are significantly better than Multilayer Perceptron based classifier and Naive Bayes classifier.

In case of activity number 11 that is stand to lie which is a state transition activity, Multilayer Perceptron Neural Network based classifier and Layer Recurrent Neural Network based classifier predicts with the accuracy around 45% which is significantly less than Naive Bayes classifier which is 61.2%.

In case of activity number 12 that is lie to stand which is a state transition activity, Multilayer Perceptron Neural Network based classifier and Naive Bayes classifier predicts with the accuracy of 10.0% and 18.5% which is significantly less than Layer Recurrent Neural Network based classifier which is 48.1%. This is because Multilayer Perceptron based classifier and Naive Bayes classifier wrongfully classifies lie to stand activity. This happens because of the lack of training data cases. Layer Recurrent Neural Network based classifier also does not provides good results but is significantly better than Multilayer Perceptron based classifier and Naive Bayes classifier because of recurrent network designing.

The Multilayer Perceptron based User Activity Classification system was tested for 3162 test samples. In this test (shown in Table 4) the system classified 2839 activities correctly and 323 activates incorrectly making the accuracy of the classifier system to be 89.8%. Confusion matrix table (Table 4) shows target classes (desired result) and output classes (calculated result).This is to show how the accuracy of User Activity Classification system decreases from seventh class onwards.Accuracy of first six classes of activity can also be calculated from the confusion matrix (Table 4, 5, 6). In 2996 test cases Multilayer Perceptron classified 2806 samples correctly, therefore making the accuracy of Multilayer Perceptron for first six classes 92.78%.

Similarly, Layer Recurrent Neural Network based User Activity Classification system was tested for 3162 test samples. In this test (shown in Table 6) the system classified 3017 activities correctly and 323 activates incorrectly and 144 activities incorrectly making the accuracy of the classifier system to be 95.4%, which is better than Multilayer Perceptron based user activity classifier system by 5.6%. Accuracy of first six classes of activity can also be calculated from the confusion matrix (Table 6). In 2996 test cases Multilayer Perceptron classified 2940 samples correctly, therefore making the accuracy of Layer Recurrent Neural Network for first six classes 98.36%, which is better than Multilayer Perceptron based user activity classifier by 5.58%. Layer Recurrent Neural Network based classifier is much more accurate in classifying last six activities than Multilayer Perceptron based classifier. Overall result has been compiled and presented in Table 7.

6 Conclusion

Layer Recurrent Neural Network based User Activity Classifier achieved mush higher accuracy than the conventional Multilayer Perceptron based user activity classifier and Naive Bayes classifier. This is because LRNN based classifier uses its internal memory and its recurrent network design to process arbitrary sequences of inputs. It uses a finite sequence to predict or label each element of the sequence based on both the past and the future context of the element in different layers. We also hypothesized that L based classifier produces better result than Multilayer Perceptron based classifier in cases of scarce information. Activities like Walking, Walking Upstairs, Walking, Downstairs, Sitting, Standing, Laying have larger dataset for both testing and training as compared to activities like Stand To Sit, Sit To Stand, Sit To Lie, Lie To Sit, Stand To Lie, Lie To Stand making it harder to predict and classify the right activity. Layer Recurrent Neural Network based User Activity Classifier tends to perform better as compared to Naive Bayes classifier and Multilayer Perceptron based classifier. Layer Recurrent Neural Network based User Activity Classifier also is more accurate in distinguishing between all three dynamic activities as well as the three static activities unlike Multilayer Perceptron based classifier and Naive Bayes classifier which are less accurate than Layer Recurrent Neural Network based classifier.

However, rooms for improvements exist: while dynamic and non-dynamic (static) activities can be efficiently classified with classifier but activates with scarce dataset still present misclassification overlaps.

References

, Tong

and Ji

, Activity Classification with Smartphone Sensors, it Tsinghua science and technology issn 1007-0214 02/11, 19(3) (2014), 235–249.

Ayu

M.A.

, Mantoro

, Matin

A.F.A.

and Basamh

S.S.O.

, Intelligent Environment Research Group (INTEG). Recognizing User Activity Based on Accelerometer Data from a Mobile Phone, IEEE Symposium on Computers & Informatics, 2011.

Parka

S.U.

, Parka

J.H.

, Al-masnia

M.A.

, Al-antaria

M.A.

, Uddinb

Z.Md.

and Kima

T.S.

, A Depth Camera-based User Activity Classification via Deep Learning Re-current Neural Network for Health and Social Care Ser-vices, Conference on ENTERprise Information Systems / International Conference on Project MANagement / Con-ference on Health and Social Care Information Systems and Technologies, CENTERIS / ProjMAN / HCist (2016), 5–7.

Esfahani

and Malazi

H.T.

, PAMS: A new position-aware multi-sensor dataset for human activity recognition using smartphones, IEEE Conferences (2017), 1–7.

Käse

and Babaee

, Gerhard Rigoll Multi-view human activity recognition using motion frequency, IEEE Conferences (2017), 3963–3967.

Paniagua

, Flores

, Srirama

S.N.

, Mobile Sensor Data Classification for Human Activity Recognition using MapReduce on Cloud, The 9th Inter-national Conference on Mobile Web Information Sys-tems, Procedia Computer Science10 (2012), 585–592.

Liu

and Liu

, Recognizing Human Activi-ties Based on Multi-Sensors Fusion, IEEE Conferences (2010), 1–4.

Reyes-Ortiz

J.L.

, Anguita

, Oneto

and Parra

, Smartphone-Based Classification of User Activities and Postural Transitions Data Set, https://archive.ics.uci.edu/ml/datasets/Smartphone-Based+Classification+of+User+Activities+and+Postural+Transitions.

Williams

J. Ronald

and Zipser

David

, Alearning algorithm for continually running fully recurrent neural networks, Neural computation1(2) (1989), 270–280.

10.

Castro

, Coral

, Rodriguez

, Cabra

and Colorado

, Wearable-based human activity recognition using an IoT approach, Journal of Sensor and Actuator Networks6(4) (2017) 28.

Layer recurrent neural network based intelligent user activity classification model using smartphone

Abstract

Keywords

1 Introduction

2 Related work

3 Data set

4 Methodology

4.1 Feed forward neural networks

Table 1 Activity Labels

References

Table 1
Activity Labels