Gesture recognition and information recommendation based on machine learning and virtual reality in distance education

Abstract

The dynamic and static gesture recognition in the distance education application scenario is not mature enough in theory at present, and still has a large space for development, and the application of gesture recognition in education is relatively insufficient. The purpose of this article is to combine gesture recognition with teacher classroom education and introduce a dynamic gesture recognition method. Moreover, this study introduces the data collection and preprocessing in detail and converts the data of the gesture action area into gray value images, and then uses the improved algorithm to perform classification. In addition, this study designs a control experiment to analyze the performance of the algorithm in this study and compares the accuracy of algorithm recognition from the perspective of simple background and complex background. The research results show that teaching gesture recognition in distance education can effectively improve education efficiency, with high accuracy, and can be directly applied to the system.

Keywords

Machine learning virtual reality distance education gesture features feature recognition

1 Introduction

Because of the cultural differences and language barriers between teachers and students, gestures are extremely important for the smooth progress of foreign Chinese education.

In classroom education, in addition to expressing instruction in language, teachers often supplement body language with gestures, actions, expressions, etc. to break the language barrier and ensure the smooth progress of classroom communication. Gestures make classroom education more vivid and vivid, and make classroom management more direct and convenient, thus saving more education time and ensuring the effectiveness of education. In a practical foreign language Chinese classroom, many problems are worthy of attention and discussion, such as how do teachers use gestures to assist education, what attitudes and expectations do students have for teachers’ gestures, and what characteristics and problems exist in the gestures used by teachers in education, etc.

With the vigorous development of computer-related technologies, human-computer interaction (HCI [1]), artificial intelligence and other technologies have developed rapidly in recent years. Virtual reality devices such as smart glasses and artificial intelligence robots gradually enter people’s daily production and life from scientific laboratories, which brings convenience to our lives and also allows us to enjoy the impact of technology on our senses. Technology is changing our lives in a subtle way, and the interaction between human and machines is also changing in a huge wave of technology. The original human-computer interaction is based on the Command Line Interface (CLI), and professional programmers interact with the computer through command line statements and keyboards [2]. In 1973, Xerox developed the graphic display technology on the Alto computer known as the “first personal computer’’. Subsequently, Appleof the United States developed a Macintosh operating system based on a mouse and a graphical interface, which marked HCI’s entry into the era of graphical user interfaces (GUI). Until 2008, Microsoft first proposed the Natural User Interface (NUI), which is an interactive interface designed according to the psychological pattern of people’s daily behavior. The nature of the natural user interface is simple and easy to learn, which greatly reduces the gap between humans and machines, and gives the user a sense of reality in terms of sensory feeling and behavior. NUI generally incorporates a variety of natural human-computer interaction methods, such as voice, touch, gestures, etc. [3]. Gesture recognition, as the most natural and intuitive way in modern natural human-computer interaction, is gradually being studied by more and more individual. Gesture recognition technology has matured in the wave of machine learning and artificial intelligence, and has been widely used in somatosensory games, smart homes, classroom education, and virtual reality (VR). The emergence of gesture recognition helps users get rid of the shackles brought by mechanical interaction devices such as mouse and keyboard and build a richer bridge between human and machine than CLI and GUI.

As an emerging human-computer interaction method, the application prospects of gesture recognition are obvious to all. However, the dynamic and static gesture recognition in real application scenarios is currently not mature enough in theory and still has a large room for development [4]. On the other hand, the application of gesture recognition in education is relatively inadequate. The purpose of this article is to combine gesture recognition with teacher classroom education, so that teachers can easily control multimedia education tools and make the education atmosphere more active to achieve efficient education without using keyboard and mouse.

2 Related work

The rapid development in the fields of machine learning, artificial intelligence and pattern recognition has greatly promoted the transformation and continuous improvement of gesture data collection devices. The first is “data gloves” with sensors as the core. It is a wearable device that collects joint information of the hands from sensors in different parts, and then calculates position and shape information during gesture movement. The literature [5] increased the classification characteristics and reduced the number of model parameters through the combination of artificial intelligence network (ANN) and hidden Markov model (HMM), thereby reducing the iterative complexity of the algorithm. The accuracy of tracking and recognizing isolated words and simple sentences in sign language is over 90%. The literature [6] used the DGMM dynamic Gaussian mixture model to identify 274 entries in the Chinese sign language dictionary on the basis of data gloves. Moreover, the accuracy of this method is as high as 98.2%. In addition, data gloves are also used in the development of smart medical and interactive somatosensory fields. The literature [7] made a great breakthrough in the matching accuracy of real hands and virtual hands through the combination of data gloves and medical simulation in the medical field. The advantage of data gloves is that the collected data has less redundancy, which can save a lot of calculation costs. The disadvantage is that the price is relatively high, which results in higher costs, and the data gloves need to be worn continuously during the data collection stage, which reduces the user’s comfort. Subsequently, with the rapid development of computer vision technology, the “camera” began to appear. Due to the relatively low price of ordinary cameras, many scholars have begun to use camera-based methods to study gesture interaction technology. The literature [8] used the camera to collect data on gesture movement to track and recognize gestures. Due to the influence of environment and background interference, the conditions of experimental simulation are restricted, which in turn affects the robustness of the algorithm in real scenarios. The advantage of the camera is that the data collection cost is lower, and the disadvantage is that the target is easily interfered and affected by the environmental background. In recent years, more and more remarkable achievements have been made in the field of new natural human-computer interaction, and new somatosensory interactive devices such as Leap Motion and Kinect have remarkable functions. Among them, Kinect2.0 is deeply loved by research scholars, and it can simultaneously acquire color images and depth images of the target. In addition, it can also obtain skeleton data of the human body, which greatly promotes the development of gestures in the field of human-computer interaction and dynamic sign language recognition [9]. Gesture interaction technology based on new somatosensory interaction devices is currently lagging behind in domestic development. Therefore, there is an urgent need to develop dynamic gesture detection and tracking algorithms based on new somatosensory interaction devices. With the continuous improvement and perfection of gesture data collection equipment, the effectiveness of gesture detection and tracking algorithms in gesture interaction technology is also increasing [10]. The research on gesture detection methods are mainly divided into static and dynamic. Among them, the research of static gesture detection method [11] is an earlier method, and the representative one is the detection method based on regional features. Regional features mainly include shape and texture features of target gestures, Haar features and HOG features. Static gestures can achieve a better detection effect due to its stable invariance. The dynamic gesture detection method [12] is mainly based on the detection of motion, such as the use of geometric descriptors to describe the context and the location of the palm by fingertip detection. Due to the arbitrary deformation and maneuverability of dynamic gestures, the robustness of the detection algorithm is relatively poor. The literature [13] solves the problem caused by the movement and deformation of gestures in the process of gesture detection and tracking by using compressed sensing. The research on gesture tracking methods is also divided into micro-deformation and arbitrary deformation. Among them, the tracking algorithm for micro-deformed gestures is relatively simple, mainly including tracking based on features such as templates, outlines and shapes, which can usually achieve better tracking effects. For the opening and closing gestures, the literature [14] proposed a gesture tracking method based on region growth and contour detection.

The research on gesture interaction technology in new human-computer interaction and dynamic sign language recognition started abroad early. The literature [15] set out to solve the communication problem of deaf-mute people and used some iconic symbols to express the semantic information of gestures, which laid a good foundation for the development of sign language. So far, for the representation method of gesture semantic information, people still use the trajectory, shape, position, size and other information of gestures during the movement process. Since the design of the American Sign Language computer program completed in [16], countries have begun to combine the characteristics of their own national languages and conducted research on the sign language recognition technology used by the hearing-impaired people in their daily communication. For example, the United States, Canada and Germany have made sign language recognition technology develop rapidly during the same period. Since then, gesture interaction technology has also made relatively breakthrough progress. The research on human motion tracking and recognition was earlier, and the literature [17] proposed the concept of “human motion perception’’. Subsequently, British scholar David Man “successively proposed the” computer vision “theory, which provided a solid theoretical basis for vision-based target detection and tracking technology. Moreover, as the most flexible part of the human body, the hand has attracted the attention of researchers. At present, the first two of the gesture interaction technology, the detection and tracking of target gestures, have developed into a multidisciplinary interdisciplinary research involving image processing, computer vision and artificial intelligence [18]. The literature [23] addresses the various problems in the field of vehicle communication with the suggestion of a mutual unified and dispersed spectrum sensing model. The application of the mutual cognitive paradigm minimizes conflict and multiple unknown problems. The literature [24] discusses the problem of vast volumes of big data and introduces the SmartBuddy idea of an adaptive and smart world incorporating human activity and human dynamics. The literature [25] talks about the development in parallel reconfigurable computing systems of a directed acyclic graph for video coding algorithms for motion estimation. Partitioning algorithm also plays a major role in speeding up the production of images. The article [26] deals with leveraging IoT and BigData Analytics in real-time applications using the Hadoop platform. The above-mentioned processes enable the deployment of an IoT-based Smart City. The article [27] centers on IoT and its major part in sophisticating the human practices and endeavors. This paper moreover managed with the collection of different information from different assets that are associated to the web [28, 29].

3 Data collection

The channel state information, CSI, can express the amplitude and phase information of each subcarrier corresponding to the frequency domain. It makes full use of the idea of orthogonal frequency division multiplex, $(\begin{matrix} Orthogonal Frequency Division Multiplex \\ OFDM \end{matrix})$ so that the wireless signal fully utilizes multiple sub-channels between the transmitter and the receiver to achieve efficient communication. Even if the target only makes a relatively small movement, CSI can perceive the movement very sensitively and show it on each subcarrier. Therefore, some researchers use CSI to do research on human behavior recognition and indoor intrusion detection [19].

This article uses the Intel5300 network card as the receiving end AP and the TL - WR845N dual-antenna router as the transmitting end DP, which can collect channel information of 30 subcarriers. Each set of CSI data is given in the form of a complex matrix. The mathematical description of CSI is: $y = Hx + n$ (1)

Among them, y represents the received data, x represents the transmitted data, H is the CSI matrix, and n is the Gaussian white noise. Then, the CSI matrix can be estimated as: $\tilde{H} = \frac{y}{x}$ (2)

The CSI of a single subcarrier can be described as: $H (i) = | H (i) | e^{j sin ∠ H (i)}$ (3)

In the formula, H (i) represents CSI on the i-th subcarrier channel, |H (i) | represents the signal amplitude of the i-th subcarrier channel, and ∠H (i) represents the signal phase of the i-th subcarrier channel. Each set of CSI data contains the amplitude and phase information of the signals on all 30 subcarrier channels, so the CSI matrix H can be expressed as [20]: $H = {[H_{1}, H_{2}, \dots, H_{30}]}^{T}$ (4)

When actually collecting CSI data, in order to save bandwidth, a multi-input and output model (Multi - inputMulti - outputMIMO) is used. If it is assumed that there are M antennas at the transmitting end and N antennas at the receiving end, the channel state information of all CSI data streams can be expressed as [21]: $H = [\begin{matrix} H_{11} & H_{12} & \dots & H_{1 M} \\ H_{21} & H_{22} & \dots & H_{2 M} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ H_{N 1} & H_{N 2} & \dots & H_{NM} \end{matrix}]$ (5)

In the formula, H_NM represents the channel state information between the Mth antenna at the transmitting end and the Nth antenna at the receiving end.

In order to collect more and richer gesture CSI data, the AP-end uses 3 antennas to send signals, the DP-end uses 2 antennas to receive signals, and an 2 × 3 MIMO array is formed. The data collector performs four custom gestures between DP and AP, namely, waving left, waving right, waving forward, waving backward. The experimental scene is located in the laboratory of Science and Technology Building of Feicui Lake Campus of Hefei University of Technology. The schematic diagram of the data collection scene is shown in Fig. 1.

Fig. 1

Schematic diagram of gesture movement collection.

We set the sampling rate of CSI to 800 Hz, and the DP-end uses the Ubuntu16.04 system to connect to the AP-end (TL - WR845N dual-antenna router). Then, the terminal is opened, and the CSI data packet is obtained by way of PingAP. The original CSI data obtained is a matrix of 2 × 3 ×30.

Each original CSI data packet is divided into 6 data streams, or it is called 6 links, and each link has 30 subcarriers. Therefore, the obtained CSI data form can be expressed as: $CSI = [\begin{matrix} H_{1, 1} & H_{1, 2} & \dots & H_{1, 30} \\ H_{2, 1} & H_{2, 2} & \dots & H_{2, 30} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ H_{6, 1} & H_{6, 2} & \dots & H_{6, 30} \end{matrix}]$ (6)

Among them, H_i,j is the CSI value of the signal status information of each subcarrier, i represents the i-th link, and j represents the j-th subcarrier.

4 CSI data processing

The obtained CSI data packets include 2 × 3 ×30 = 180-dimensional data in total. Too high dimensions mean that the complexity is too large, so the data needs to be reduced in dimension. The signals of 30 subcarriers on each link are averaged, and the average value obtained represents the data value of the link. Then, the adaptive weighted fusion method is applied to these 6 links, and finally a set of values is obtained. The fused CSI data is subjected to Kalman filtering. After removing the noise, threshold segmentation is performed to obtain the CSI data of the gesture action area.

We assume that there are n links in total, and the corresponding n sets of CSI data are X₁, X₂, ⋯ , X_n respectively. The adaptive weighted fusion is to use an adaptive method to obtain the optimal weighting factor that matches the n sets of data, so that the total variance of the n sets of data is minimum. $\hat{X}$ represents the weighted fusion value of the n groups of data. The adaptive weighted fusion algorithm model diagram is shown in Fig. 2.

Fig. 2

Model diagram of adaptive weighted fusion algorithm.

We assume that the variances of these n sets of data respectivelyare $σ_{1}^{2}, σ_{2}^{2}, \dots, σ_{n}^{2}$ and the actual value to be estimated is X. These n sets of data are independent of each other and are unbiased estimates of x. The weighting factors of these n sets of data are W₁, W₂, ⋯ , W_n respectively, and the value $\hat{X}$ after fusion can be expressed by the following formula:

${\begin{matrix} \hat{X} = \sum_{i = 1}^{n} W_{i} X_{i} \\ \sum_{i = 1}^{n} W_{i} = 1 \end{matrix}$ (7)

The total variance σ² is:

$\begin{matrix} σ^{2} = E [{(X - \hat{X})}^{2}] \\ = E [{(\sum_{i = 1}^{n} W_{i} X - \sum_{i = 1}^{n} W_{i} X_{i})}^{2}] \end{matrix}$ (8)

After finishing the formula, the obtained σ² is: $σ^{2} = E [\sum_{i = 1}^{n} W_{i}^{2} {(X - \hat{X})}^{2}] = \sum_{i = 1}^{n} W_{i}^{2} σ_{i}^{2}$ (9)

Among them, $σ_{i}^{2}$ represents the variance of the ith group of data, the total variance is a multivariate quadratic function of the weighting factor. The minimum value $σ_{min}^{2}$ of σ² is solved: ${\begin{matrix} σ_{min}^{2} = min (\sum_{i = 1}^{n} W_{i}^{2} X_{i}^{2}) \\ \sum_{i = 1}^{n} W_{i} = 1 \end{matrix}$ (10)

min(·) is the operation to find the minimum value. According to the theory of multivariate function seeking extreme value, the optimal weighting factor is obtained when the total variance is minimum: $W_{i}^{*} = \frac{1}{σ_{i}^{2} \sum_{i = 1}^{n} \frac{1}{σ_{i}^{2}}} (i = 1, 2, \dots, n)$ (11)

At this time, the corresponding minimum value $σ_{min}^{2}$ of the total variance n is: $σ_{min}^{2} = \frac{1}{\sum_{i = 1}^{n} \frac{1}{σ_{i}^{2}}}$ (12)

It can be seen from formula (12) that in order to find the optimal weighting factor, the variance $σ_{i}^{2} (i = 1, 2, \dots, n)$ of each group of CSI data must first be found. We assume that two mutually independent links are α, β, their measurement values are X_α, X_β, the corresponding measurement errors are e_α, e_β, and the true value to be estimated is X. Then, there is the following relationship: ${\begin{matrix} X_{α} = X + e_{α} \\ X_{β} = X + e_{β} \end{matrix}$ (13)

The variance of link α is: $σ_{α}^{2} = E [e_{α}^{2}]$ (14)

X_α, X_β is not related to each other, the mean is zero, and X_α, X_β is not related to X. Therefore, the cross-covariance function R_αβ of X_α, X_β is: $R_{α β} = E [X_{α} X_{β}] = E [X^{2}]$ (15)

The self-covariance function R_αα of X_α is: $R_{α α} = E [X_{α}^{2}] = E [X^{2}] + E [e_{α}^{2}]$ (16)

According to formulas (14), (15), (16), we can obtain: $σ_{α}^{2} = E [e_{α}^{2}] = R_{α α} - R_{α β}$ (17)

Among them, the value of R_αα, R_αβ can be obtained from the estimated value of the time domain. If it is assumed that the number of acquisitions of CSI is k and the time domain estimate of R_αα, R_αβ is R_αα (k) , R_αβ (k) respectively, then there is the following formula: ${\begin{matrix} \begin{matrix} R_{α α} (k) = \frac{1}{k} \sum_{m = 1}^{k} X_{α} (m) X_{α} (m) = \frac{k - 1}{k} R_{α α} (k - 1) \\ + \frac{1}{k} X_{α} (k) X_{α} (k) \end{matrix} \\ \begin{matrix} R_{α β} (k) = \frac{1}{k} \sum_{m = 1}^{k} X_{α} (m) X_{β} (m) = \frac{k - 1}{k} R_{α β} (k - 1) \\ + \frac{1}{k} X_{α} (k) X_{β} (k) \end{matrix} \end{matrix}$ (18)

If relevant calculations are taken for link α (α ≠ β ; α = 1, 2, ⋯ , n) and link β (β ≠ α ; β = 1, 2, ⋯ , n), and the average value ${\bar{R}}_{α β} (k)$ of R_αβ (k) is used as the estimated value of R_αβ, then: $R_{α β} = {\bar{R}}_{α β} (k) = \frac{1}{n - 1} \sum_{β = 1 β \neq α}^{n} R_{α β} (k)$ (19)

From formula (19), the time domain estimates of R_αα and R_αβ can be obtained, and the variance $σ_{i}^{2}$ of each data link can be further obtained. Then, the optimal weighting factor of each group of data is obtained according to formula (11). Finally, the fusion value $\hat{X}$ is calculated according to formula (7). Figure 3 shows the fusion value $\hat{X}$ obtained after adaptively weighting the CSI data of 6 links.

Fig. 3

Waveform diagram of adaptive weighted fusion CSI data.

Next, the fused CSI data is denoised. Although a general low-pass filter can remove noisy signals, it will also filter out some useful signals. Moreover, when the gesture is slow, the filtering effect is not good. Kalman filter is a state equation of a linear system. Its basic idea is to use the input and output of the system to observe data, and then estimate the state of the entire linear system. The calculation principle is as follows: $Z_{t} = F_{t} Z_{t - 1} + B_{t} U_{t} + Q_{t}$ (20)

Among them, Z_t represents the state vector at time t, F_t represents the state transition matrix, B_t represents the control matrix, U_t is the control vector, and Q_t is the n₁-dimensional noise matrix. If S_t is used to denote the observation vector of the system at time t, then the observation equation of the system can be expressed as: $S_{t} = P_{t} Z_{t} + V_{t}$ (21)

observation noise. Both V_t and Q_t in formula (20) are Gaussian white noise, which has the following properties: $E [Q_{t}] = 0, E [V_{t}] = 0$ (22) $E [Q_{t} Q_{t}^{T}] = C, E [V_{t} V_{t}^{T}] = D$ (23)

Among them, C represents the prediction noise covariance matrix, D represents the measurement noise covariance matrix. Kalman is used to estimate the state quantity of the system at time t. At time t, the state prediction value $Z_{t}^{'}$ can be expressed as: $Z_{t}^{'} = F_{t} Z_{t - 1} + B_{t} U_{t}$ (24)

The predicted value $G_{t}^{'}$ of the covariance matrix at time t of the system can be expressed as: $G_{t}^{'} = F_{t} G_{t - 1} F_{t}^{T} + C_{t}$ (25)

In Equation (25), G_t-1 is the calculated value of the covariance matrix of the system at time t - 1. The Kalman gain matrix K_t can be expressed as: $K_{t} = \frac{G_{t}^{'} P_{t}^{T}}{P_{t} G_{t}^{'} P_{t}^{T} + D_{t}}$ (26)

The optimal estimate of the state quantity at time t of the system is: ${\hat{Z}}_{t} = Z_{t}^{'} + K_{t} (S_{t} - P_{t} Z_{t}^{'})$ (27)

The covariance matrix is updated: $G_{t} = (I - K_{t} P_{t}) G_{t}^{'}$ (28)

Among them, I is the identity matrix.

The result of Kalman filtering of the CSI data obtained by adaptive weighted fusion is shown in Fig. 4.

Fig. 4

Waveform diagram of CSI data after Kalman filtering.

The CSI data obtained after Kalman filter denoising is a left-hand wave from 0 to 4 seconds. Although the noise is filtered, there are many meaningless data. Therefore, it is necessary to perform threshold division on the CSI data including the gesture action, and divide the CSI data in the gesture action occurrence area. During the CSI data analysis, it was found that the state change of the phase difference variance between the CSI antennas is closely related to the detection of gestures. Therefore, this paper uses the sliding window method based on the threshold to detect the phase difference of CSI. The specific steps are as follows:

(1) The CSI data is collected in a quiet environment without any action, and the average value μ and normalized standard deviation σ of the signal in the sliding window are calculated.

(2) The threshold δ is determined according to μ and σ calculated in the previous step, and the relationship between μσ and δ satisfies: $μ + 6 σ ⩽ δ$ (29)

(3) The signal flow in the sliding window is collected to compare the average value of the signal in the sliding window with the threshold value. If the average value is greater than δ, it is determined that the signal is a signal in the gesture action area. In the actual experimental process, since the dynamic gesture is a continuous process, the signal in the gesture action area should be continuous, and it has a starting point and an ending point. After determining the starting point and the ending point of the gesture action, the piece of CSI data is segmented out.

After the Kalman filtered CSI data is divided by the threshold by the above method, the CSI data of the gesture action area is shown in Fig. 5. In the figure, the red point indicates the starting point, and the green point indicates the end point.

Fig. 5

Schematic diagram of CSI data after threshold segmentation.

5 Algorithm performance analysis

The purpose of image stylized migration is to get the content from the content picture, and then extract the style from the style image, and finally reconstruct a stylized image. Since most of the style conversion algorithms target unpaired images, that is, generating an image with similar content and different styles, the loss function usually used such as the mean square error does not guide the generation of style images well. The image style conversion can be completed by constraining the image style and content through the perceptual loss of the feature domain, but neither the algorithm speed nor the quality of the generated image has shown too good performance. In this paper, the GAN network structure is used to transform the gesture image. The anti-loss in the adversarial training can help the generator to generate a higher-quality and more stylized image. The cycle consistency test is used to constrain the image space and maintain the content of the generated image. A lot of experiments are carried out in this paper to verify the effectiveness of the stylized image migration algorithm proposed in this paper. First, this article introduces the construction of the gesture depth image data set, and then shows the results of the style conversion of the images of different real gesture data sets. Subsequently, this paper conducts some self-contrast experiments to verify the effectiveness of the image stylized migration algorithm proposed in this paper. Finally, this study compares the image stylization migration algorithm proposed in this paper with the previous image style conversion algorithms.

This paper collects 100,000 hand depth image data sets by collecting remote education pictures. In order to make the gesture gestures rich and diverse and complete, this paper imitates various gesture gestures in the BigHand data set to generate model hand images. In addition, 1000 gesture images were generated for testing. A part of depth imagesofgesture are shown in Fig. 6:

Fig. 6

Model gesture data set.

It can be seen from the figure that a large part of the image generated by the model software has a black background with a depth value of 0. This part of the background has no effect on the style conversion of the image but will affect the effect of style conversion. In this paper, we first process the model data set to find the centroid of the hand image, use an appropriate size image frame to crop out the hand image, and move the hand in the image to the center of the image. As the network training, the input is an image with a size of 256 * 256, the labeling of the joint points of the hand will be biased because the image is scaled according to an abnormal ratio. In this paper, a crop frame with a size of 320 * 320 is used to crop the image at the center of the image, and then scale scaling and normalization are performed. In this paper, the hand gesture image generated by the model is compared with the actual captured gesture image. The boundary of the hand in the real image has frizzy edges when imaging due to light reflection and refraction, and the image quality is average, and the resolution is lower. The conversion of the model hand image into the real hand image is mainly to learn the depth change of the finger due to refraction and reflection. The style conversion of the real hand image is mainly to process the frizz of the finger edge into a smooth depth change.

This article shows the results of the style conversion of gesture images. For the training set, the X domain uses 100,000 depth image data generated by the model in this paper, the test uses 1000 images, and the Y domain uses the real hand image ICVL data set and BigHand data set for experiments. Among them, the ICVL data set uses 17,000 images as the training set and 600 images as the test set, and the BigHand data set uses 80,000 images as the training set and 2000 images as the test set. Before the network training, all the images of the training set were scaled, otherwise the images of different scales could not calculate the loss function during training. Because this article uses a fully convolutional network that can process images of any size, there is no need to scale and preprocess the images during the test. The partial result of the style conversion of the gesture image of the BigHand dataset and the model gesture image is shown in Fig. 7.

Fig. 7

The result of gesture conversion model data.

Figure 7 shows the conversion of BigHand real data into model data. The two images are a group, the left image is the real image, and the image on the right is the style converted image. It can be seen from the figure that the real hand image has frizzy edges, and the edges are very smooth after being converted into the model hand. The part of the shadow that is occluded in the real image after conversion is expressed as a difference in depth, as shown by the yellow box in Fig. 7. It can be seen from the four sets of images in the third and fourth columns that the real data will have noise points or redundant parts. For example, in the second row of the third column and the third row of the image, there is an extra finger part, and there is a noise point in the wrist part, as shown by the green box in Fig. 7. After conversion, the redundant part and the noise part are removed, which shows that the network structure in this paper also has a certain ability to remove noise. The model gesture image has a neat wrist. It can be seen from the figure that after the style conversion of the real hand image, the wrist becomes very neat, and the overall thickness of the finger is more in line with the law of the model data.

Gesture tracking plays an important role in the traditional dynamic gesture recognition method. The extraction of dynamic gesture features is based on the fact that gestures can be tracked correctly, otherwise the extracted features do not have a high degree of discriminability. Moreover, its recognition of gestures has not improved, and even reduced the accuracy of recognition. Gesture tracking is a sub-problem of target tracking, so we can apply a general target tracking algorithm to gesture tracking. In distance education, teaching gesture recognition can effectively improve education efficiency.

Finally, the performance of the model is analyzed to explore the accuracy of gesture recognition in distance education. After describing the experimental data set and experimental details in detail, we compared our method with many other methods on the data set. We first compare the average accuracy of each method, and then give the experimental results of this method of 10-fold cross-validation. Finally, the data of the experimental results are analyzed. The results of the algorithm in this study in distance teaching gesture recognition are shown in Tables 1 and 2. Table 1 is the gesture recognition results of the simple background, and Table 2 is the gesture recognition results of the complex background. The statistical diagrams corresponding to the two tables are shown in Figs. 8 and 9.

Table 1

Statistical table of recognition accuracy of distance education gestures in simple background

Serial number	Accuracy (%)	Serial number	Accuracy (%)
1	97	26	97
2	99	27	99
3	96	28	99
4	99	29	97
5	99	30	96
6	96	31	99
7	97	32	97
8	95	33	99
9	96	34	99
10	95	35	95
11	96	36	99
12	97	37	99
13	98	38	96
14	96	39	98
15	98	40	99
16	95	41	97
17	100	42	97
18	99	43	98
19	98	44	98
20	98	45	100
21	98	46	96
22	97	47	97
23	96	48	98
24	95	49	98
25	97	50	99

Table 2

Statistical table of recognition accuracy of distance education gestures in complex background

Serial number	Accuracy (%)	Serial number	Accuracy (%)
1	86	26	87
2	86	27	93
3	89	28	90
4	94	29	89
5	92	30	91
6	87	31	93
7	86	32	93
8	89	33	93
9	89	34	95
10	94	35	95
11	90	36	92
12	92	37	91
13	93	38	87
14	91	39	94
15	91	40	87
16	91	41	85
17	88	42	90
18	94	43	92
19	91	44	90
20	85	45	88
21	92	46	93
22	90	47	86
23	87	48	87
24	87	49	87
25	89	50	87

Fig. 8

Statistical diagram of recognition accuracy of distance education gestures in simple background.

Fig. 9

Statistical diagram of recognition accuracy of distance education gestures in complex background.

Tables 1 and 2 are the results of distance teaching gesture recognition in simple background and complex background, respectively. In actual teaching, distance teaching gesture recognition will be disturbed by various background factors, so it needs to be analyzed in different situations. A it can be seen from Figs. 8 and 9 that the gesture recognition rate of simple background is basically distributed above 95%, indicating that the algorithm of this study has a better effect on gesture recognition under simple background. Moreover, under a complex background, the recognition rate of this research algorithm is between 85% -95%, which basically meets the actual teaching needs. This shows that the algorithm of this study can meet the needs of gesture recognition in distance education, and it can be applied to the distance education system.

7 Conclusion

In the development of artificial interaction and artificial intelligence, computer vision hardware and machine learning algorithms are developing rapidly, and the prospects in this field are becoming more and more extensive. Based on real sense technology, this paper conducts research on the combination of teacher multimedia education and gesture recognition, and proposes a continuous dynamic gesture recognition method based on CSI and YOLOv3. For the topic of real-time recognition of static gestures, this study decided to use CNN-based target detection algorithm. The algorithm uses CNN to train the static target image to be tested and can extract target features autonomously without complex image preprocessing. Moreover, the detection and recognition process of the algorithm is integrated, and the detection and recognition speed arefast, and the accuracy is high, so it is the current mainstream target detection algorithm. Moreover, this study introduces the process of CSI data collection and preprocessing in detail and converts the CSI data of the gesture action area to generate a gray value image and uses the YOLOv3 algorithm to train and recognize the gray value image. In addition, this study verifies the four customized dynamic gestures by recognizing the confusion matrix and obtains a better recognition effect. This method overcomes the defect that the YOLOv3 algorithm is not suitable for processing dynamic target recognition and uses the advantages of CSI to collect human dynamic behavior to provide a new research direction for the topic of dynamic gesture recognition.

References

Ichikawa

and Masakura

, Motion capture depends upon the common fate factor among elements, Perception 46(12) (2017), 301006617720123.

Zhang

, Wang

, Chu

, et al., Application of optical motion capture technology in power safety entitative simulation training system, Optics & Photonics Journal 6(8) (2016), 155–163.

Wang

S.K.

, Xie

and Hu

, Research on protection property of running sportswear fabrics based on 3-D motion capture system, Textiles and Light Industrial Science and Technology 3(1) (2014), 57–62.

Miura

, Kaiga

, Shibata

, et al., Low-dimensional feature vector extraction from motion capture data by phase plane analysis, Journal of Information Processing 25(2) (2017), 884–887.

Frost

D.M.

, Beach

T.A.C.

, Campbell

T.L.

, et al., Can the Functional Movement Screentrademark be used to capture changes in spine and knee motion control following 12 weeks of training? Physical Therapy in Sport 23(2) (2016), 50–57.

Zhang

, Yan

X.K.

and Liu

A.G.

, An introduction to a newly-developed “acupuncture needle manipulation training-evaluation system” based on optical motion capture technique, Acupuncture Research 41(6) (2016), 556–559.

Joo

, Simon

, Li

, et al., Panoptic studio: A massively multiview system for social interaction capture, IEEE Transactions on Pattern Analysis & Machine Intelligence 107(99) (2016), 1–1.

Sourioux

, Bestaven

, Guillaud

, et al., 3-D motion capture for long-term tracking of spontaneous locomotor behaviors and circadian sleep/wake rhythms in mouse, Journal of Neuroence Methods 295(2) (2017), 51–60.

Ancillao

, Savastano

, Galli

, et al., Three dimensional motion capture applied to violin playing: A study on feasibility and characterization of the motor strategy, Computer Methods & Programs in Biomedicine 149(5) (2017), 19–27.

10.

Komisar

, Novak

A.C.

and Haycock

, A novel method for synchronizing motion capture with other data sources for millisecond-level precision, Gait & Posture 51(2) (2016), 125–131.

11.

Sá

Fátima

, Marques

António

, Rocha

N B F

, et al., Kinematic parameters of throwing performance in patients with schizophrenia using a markerless motion capture system, Somatosensory Research 32(2) (2015), 77–86.

12.

Prim

G.D.S.

, Berenice Santos Gonçalves and M.L.H. Vieira, A representação do corpo e do movimento: uma análise da interatividade do motion capture, Design & Tecnologia 5(9) (2015), 23.

13.

ngel-López

Juan Pablo

and de la Peńa

Nelson Arzola

, Kinematic hand analysis using motion capture technology, IFMBE Proceedings 49(11) (2015), 257–260.

14.

Kim

M.K.

, Kim

T.Y.

and Lyou

, Performance improvement of an AHRS for motion capture, Journal of Institute of Control 21(12) (2015), 1167–1172.

15.

Zhou

Z.M.

and Chen

Z.W.

, A survey of motion capture data earning as high dimensional time series, International Journal of Multimedia and Ubiquitous Engineering 10(9) (2015), 17–30.

16.

Khan

M.A.

, Multiresolution coding of motion capture data for real-time multimedia applications, Multimedia Tools & Applications 76(15) (2016), 1–16.

17.

Puupponen

, Wainio

, Burger

, et al., Head movements in finnish sign language on the basis of motion capture data: A study of the form and function of nods, nodding, head thrusts, and head pulls, Sign Language & Linguistics 18(1) (2015), 41–89.

18.

Lee

and Yoo

, Low-cost 3D motion capture system using passive optical markers and monocular vision, Optik International Journal for Light & Electron Optics 130 (2016), 1397–1407.

19.

Fry

A.C.

, Herda

T.J.

, Sterczala

A.J.

, et al., Validation of a motion capture system for deriving accurate ground reaction forces without a force plate, Big Data Analytics 1(1) (2016), 11–23.

20.

Rahman

M.M.

, Analysis of finger movements of a pianist using magnetic motion capture system with six dimensional position sensors, Transactions of the Virtual Reality Society of Japan 15(4) (2017), 243–250.

21.

Park

S.W.

, Park

H.S.

, Kim

J.H.

, et al., 3D displacement measurement model for health monitoring of structures using a motion capture system, Measurement 59(3) (2015), 352–362.

22.

Giannetti

, Petrella

and Bach

, et al., bone position measurement using high-frequency ultrasound validated with 3-D optical motion capture systems: a feasibility study, Journal of Medical & Biological Engineering 37(7) (2017), 1–8.

23.

Paul

, Daniel

, Ahmad

and Rho

, Cooperative cognitive intelligence for internet of vehicles, IEEE Systems Journal 11(3) (2017), 1249–1258.

24.

Paul

Anand

, Ahmad

Awais

, Mazhar Rathore

and Jabbar

Sohail

, Smartbuddy: defining human behaviors using big data analytics in social internet of things, IEEE Wireless Communications 23(5) (2016), 68–74.

25.

Paul

, Jiang

Y.C.

, Wang

J.F.

and Yang

J.F.

, Parallel reconfigurable computing-based mapping algorithm for motion estimation in advanced video coding, ACM Transactions on Embedded Computing Systems (TECS) 11(S2) (2012), 1–18.

26.

Rathore

M.M.

, Paul

, Hong

W.H.

, Seo

H.C.

, Awan

and Saeed

, Exploiting IoT and big data analytics: Defining smart digital city using real-time urban data, Sustainable Cities and Society 40 (2018), 600–610.

27.

Antonov

Andrey

, Nenov

Svetoslav

and Tsvetkov

Tsvetelin

, Impulsive controlability of tumor growth, Dynamic Systems and Applications 28 (2019), 93–109.

28.

Wang

Lili

and Cheng

, Application of panel data regression function model to financial risk management of bank resource allocation, Dynamic Systems and Applications 29(3) (2020), 683–694.

29.

Paul

, “Internet of Things: A primer’, R Jeyaraj Human Behavior and Emerging Technologies 1(1) (2019), 37–47.