Abstract
Online classroom teaching is difficult to identify students’ learning status in real time. Therefore, we need to combine intelligent image recognition technology to analyze student status through eye movement features. This study solves the problem of inaccurate positioning of the initial position of the shape model in the process of eyelid matching through machine learning. Moreover, this study improves the algorithm and uses the AK-EYE model based on the combination of ASM algorithm and Kalman filtering to establish a local feature model for each feature point. According to the gray information in the normal direction of the feature point, the local gray information is modeled. After training through the sample set to obtain the state model, the target eye can be searched, and the pose parameters can be determined. Finally, this study designs a control experiment to analyze the performance of the model proposed in this study. The research shows that the algorithm proposed in this paper has a high recognition accuracy and has a practical basis, which can be used as one of the subsequent classroom teaching system algorithms.
Introduction
In the course of widespread use of online education technology, it has gradually exposed some problems: the flow of information between teachers and learners under the current framework is basically one-way, and it is difficult for publishers of online courses to obtain timely feedback from students on the content or difficulty of the courses. At present, most online learning systems need to track learners’ learning status through means such as classroom tests and after-school exercises. These methods are generally not real-time and require students to actively provide feedback. Moreover, the amount of information contained in the feedback data is limited, so it is difficult to fully describe the students’ listening status. Therefore, today’s online learning systems generally require an effective learning state feedback mechanism as an aid. As far as the current needs are concerned, it must meet four points: 1. Effective and accurate: It needs to provide a reliable and valuable learning status to ensure the validity of the assessment. 2. Tightly integrate the classroom: The student status we obtain must be related to the content of the classroom, so that we can evaluate the learning status more reasonably. 3. Real-time: The real-time nature of the learning status feedback is especially important for online classrooms, especially for online classrooms with small-class live broadcast teaching. Teaching teachers can use this as a basis to adjust the teaching methods on the spot, thereby improving teaching efficiency. 4. Convenient statistical analysis: Unlike the dozens of students in traditional classrooms, the audience of a class in online education is often thousands and more diverse. In this situation, teachers need to pay more attention to the classroom status of the student community. Therefore, only by counting the large-scale student feedback data can we effectively respond to the teaching quality evaluation under the new conditions [1].
Facial features have always been an important source of information reflecting people’s emotional states. In addition, the cameras on modern terminal devices have gradually become the standard configuration, so that these devices can easily obtain the learners’ faces.
Today, people have tried to extract various facial features to help the analysis of learning states, including sight line and head orientation estimation, facial texture, facial organ displacement and deformation. Among them, the change of students’ sight is considered by us to be the most active among the learner’s many facial features and can best reflect the learning status, and our work is also based on this understanding. The system we designed detects and records the changes in people’s sight during learning and uses the records as the basis to estimate what learning behavior students are doing at this time. These recorded learning behaviors have great reference value for later teaching quality assessment, classroom management, curriculum and classroom planning and design. People often say “eyes are the windows of the soul”, so most of the learning content of our learning depends on vision. Moreover, inferring a person’s cognitive state by monitoring eye movements is an intuitively feasible method [3]. In fact, we observe that students’ faces are generally in a relatively calm state when reading literature, observing pictures, and listening to lectures. If there is a local feature that has been in dynamic for a long time, then the answer is undoubtedly the eye. When people perform almost all learning behaviors, their visual focus will constantly switch on learning materials, and different learning states and learning materials will also cause this switching to exhibit different characteristics [4].
Based on the above analysis, this study recognizes the eye movement characteristics of English online classroom students based on the recognition of the eye movement characteristics of students to obtain online classroom information of students, and controls the classroom based on the obtained information.
Related work
Visual search refers to the process of directing visual attention to clues related to the environment to make individuals clearly understand how to prepare and manipulate skills in specific situations [5]. Visual search is a way to direct visual attention to the area of most interest. It can effectively reflect the use of athletes’ visual information, including the search strategy for the use of related information and the process of providing useful information for technical actions. Sports psychologists gradually pay attention to the visual search of athletes in sports skills. For this visual search, appropriate extraction modes and strategies can improve the athlete’s ability to judge in the game and the ability to make appropriate decisions in advance, and then enable the athlete to score and win the game. In sports scenes, there are two aspects of visual search: one is that information search is based on the information needed to complete a certain action, and the other is to select the appropriate action based on the information obtained by visual search.
The processing of information by humans depends largely on vision. About 80% –90% of the external information is obtained through the human eye, so the eyes are known as “the window of the soul” [6]. With the development and progress of science and technology, more and more researchers began to study visual characteristics. Studies have shown that the pattern of eye movement extraction is closely related to changes in human psychology. There are three basic ways of eye movement: fixation, saccades, and pursuit movement [7]. (1) Fixation: In normal visual observation, eye movement appears as a rapid jump between a series of stagnation points on the observed object, and the eyeball usually stays for at least 100 s or longer, which is called fixation [8]. The eyes are not absolutely still when fixing. In order to observe the object, the eyeball always has a slight jitter, and its amplitude is generally less than 1 degree. The vast majority of information can only be obtained and processed during Fixation [9]. (2) Saccade: It is a sudden change of fixation point or fixation position, which is usually unconscious. The spatiotemporal information of stimulation can be obtained during the saccade, but it is difficult to form a clear stimulation image. Thus, a fast search of the visual field and selection of stimulus information are realized. (3) Pursuit movement motion: When the observed object is in relative motion with the eye, the eye will follow the object in order to ensure that the eye is always watching the object [10].
With the continuous development and progress of technology, more and more human eye detection methods appear. Researchers at home and abroad have classified human eye detection methods from the following perspectives: The literature [11] divided it into two types according to the change of illumination: general light source and special light source. The literature [12] divided it into a method based on template matching, a method based on gray projection, a method based on knowledge and a method based on statistics. The literature [13] modeled from different image features and divided human eye detection methods into shape methods based on image corners and contour features, apparent methods based on image overall texture features, and hybrid methods of the two. The literature [14] divides human eye detection methods into knowledge-based methods and data-based methods according to pattern recognition principles. The knowledge-based method refers to that people define the relationship between certain description features and categories from the prior knowledge of known objects, so as to formulate relevant criteria, and allow the computer to identify the labels of unknown samples through these criteria; Data-based methods mean that people learn certain features in a certain number of samples and certain laws that exist between statistical categories, and establish a classification model. The statistics-based human eye detection method is a very active research hotspot in recent years and is often used in the research of target detection. Compared with other methods, it has the advantages of high detection accuracy, strong robustness, and fast speed [15]. At present, the statistics-based human eye detection method mainly involves hidden Markov (HMM), subspace methods, support vector machines (SVM), Adaboost (Adaptive Boosting), artificial neural networks (ANNs) and other machine learning methods [16]. The future development trend of human eye detection technology includes: (1) Fusion of two or more features [17]; (2) Combination of two or more classification algorithms [18]; (3) Use certain special methods in a specific application [19]; (4) Adopt generally applicable methods under large-scale data [20]; (5) Deep learning method based on convolutional neural network [21]. The research and development of human eye detection based on computer vision and pattern recognition has been receiving much attention. Through the use of algorithms in the field of pattern recognition and machine learning to study human eye detection, and the use of human eye banks for testing, the results greatly promote the research and development of pattern recognition theory. In recent years, applications based on human eye detection and eye tracking have emerged in many fields, but the problems of its high cost, complex system, and large noise interference have also become increasingly prominent. Therefore, in-depth research on human eye detection is not only challenging, but also of great significance.
Eyelid matching based on AK - EYE model
We know that the ASM algorithm is divided into two parts: training and matching: first, the feature points of the training set are calibrated by manual calibration, and then the feature points are trained to obtain the shape model, and then the target object is matched by matching the key points.In order to obtain the point distribution model of our target object, we selected face images of different ages and different genders for model training. However, considering the actual running speed of the program, the selected photos should not be too many.
Before training the model, we need to mark the feature points. This paper chooses to manually calibrate the feature points to calibrate the feature points of the target object on each image to present the shape information of the target object. Then, the shape model is established, and a local texture model is created to match the model to the target object. Finally, Kalman filtering is used to predict and update the initial position of the entire model’s motion matching process.
When marking, we need to pay attention to the following issues:
(1) The number of marked points should not be too much. In principle, the more marked points, the more accurate the description of the shape of the model. However, in fact, too many mark points will cause the matching time to become longer, seriously affecting the real-time performance of the entire algorithm. This paper counts the distance between the manually marked feature points and the feature points of the shape obtained by the actual algorithm search and records the mean square deviation of each point and the actual shape point, and the mean square error of the total points. After many tests and comprehensive considerations of accuracy and performance, this study selects 68 calibration points.
(2) The position of the mark should be selected at key features, such as: corner points, high curvature points, T-joints, etc. At the same time, among the boundary points with contour features, other intermediate points (equal points) are selected by equal-space sampling to form a set of boundary points of the target contour, which is called a set of landmark points. Moreover, the selection of these points corresponds to the characteristics of the target. The specific situation is shown in Fig. 1:
Schematic diagram of the location of the landmark point.
According to the traditional algorithm requirements, when marking the eyes, the marking is only taken at the edge of the eyelid, which is easy to cause the problem of contour convergence when matching. In order to solve this problem, we also mark the borders of the eyebrows, and marks the contours of the face and the feature points around the nose and mouth, which greatly improved the accuracy of matching and also extended the follow-up experiments. As shown in Fig. 2(a), it is the defined feature point marker, and as shown in Fig. 2(b), it is the calibrated 68 facial feature points.
(a) Calibration of facial feature points (b) Calibration of 68 facial feature points.
By manual annotation, the calibration points of each face image in the training set form a vector:
Among them, (x ik , y ik ) represents the k-th feature on the i-th training sample, and n training samples constitute n shape vectors. The entire training set forms a vector group: X = (x1, x2, ⋯ , x i , ⋯ , x N ), and N is the number of pictures in the training set.
In the above steps, we have calibrated the key feature points and established the training sample set. However, in order to keep the training sample data in the same comparable coordinate system to eliminate the non-shape interference caused by different angles, distances, and posture changes in the picture, and to make the corresponding points in different samples comparable, it is necessary to normalize or align the shape of the training samples.
This paper uses the Procustes method for normalization, that is, this paper selects the appropriate translation, rotation and scaling operations for each sample to ensure that they are as similar as possible and the difference between these shapes and the average shape is minimal. Among them, the average shape is calculated from the entire sample set.
In the process of using the Procustes method to align the training set, the parameters of each sample need to be calculated: The scale θ
i
, the rotation angle s
i
and the translation vector t are adjusted. If only two shapes X1, X2 are considered, the shape description is as follows:
The shape X1 is scaled, translated, and rotated to align with shape X2, that is, to minimize the value of E. The formula of E is as follows:
Among them, W is a diagonal matrix, and increasing the diagonal matrix can make the feature points more stable. The more stable the feature point, the smaller the position it moves. The diagonal matrix w can be obtained by the following calculation method. We set R
KI
to represent the distance between the kth calibration point and the first calibration point in an image, and set R
KI
to represent the variance between different images in the entire training set. By calculating w
k
, the diagonal matrix W is obtained. The expressions of the diagonal elements of the transformation M, the vector t and the diagonal matrix W are as follows:
The above is the case of two samples. In order to align all M samples in the training sample set, we use Procustes analysis. The basic idea of this method is to minimize the distance between all shapes and the average shape. The minimization formula is as follows:
The steps of using Procustes to align the point distribution model are as follows: All face models in the training set are aligned to the first face model. The average model of the face is calculated. All face models are aligned to the average face model. The step (2) and the step (3) are repeated until the algorithm converges.
If we put all the shape point sets before and after alignment into one picture, we can find that some corner points are relatively concentrated, while others are more scattered. The point distribution model is to find the coordinate changes of these scattered points. As shown in Fig. 3, it intuitively reflects the distribution of feature points of training samples before and after shape alignment.
Distribution of feature points before and after face sample alignment.
However, these scattered points are not irregularly distributed, and they are related to other points. Each pair of samples in the training set can be represented by 2n-dimensional points. The training set is a 2n-dimensional space set. Each point in this space is set with a set of coordinates, and the average of these point sets is the origin of the coordinates. The shape of these marked points is similar to other shapes in the training set. By moving these marked points in this space, a new shape is created. After the alignment of the shape vector is completed, the change trend of the model should be counted next. This process should be realized by the method of principal component analysis (PCA). The specific statistical process is as follows:
(1) The shape information of the training sample set is calculated, and the average shape
Among them, N represents the number of samples in the training set and X i is the i-th training sample.
(2) The formula for solving the covariance matrix S is as follows:
(3) The eigenvalue λ i and the eigenvector p i m are calculated, and then the eigenvalue λ i is arranged in order from large to small.
Among them, P i is the eigenvector and corresponds to the i-th eigenvalue λ i of the covariance matrix S.
(4) The statistical shape model of training samples is constructed.
Among them, P is the transformation matrix, which is composed of the first m feature vectors p
i
of the covariance matrix S extracted in the previous step. b is an m-dimensional vector, and is the weight of the corresponding feature vector, which is used to control the change of the shape of the feature point. When b is 0, X is the origin of coordinates (average shape). Moreover, b cannot be too large. If it is too large, the entire shape changes will beyond the range of the face. In this paper, the following constraints are imposed on b:
Through such constraints, the value of b is limited to 3 times the standard deviation of the average value. According to known feature vectors and feature values, different training set shapes can be generated.
In each iteration, in order to make the feature points find a better new position, we also need to build a local feature model for each feature point.
For a fixed feature point of each sample in the training sample set, the distribution of gray values around it should be similar.Any shape area around the feature point can be considered,However, considering the real-time requirements of the system, this paper mainly models the local gray information through the gray information in the normal direction of the feature points.
Each corresponding grayscale template is actually a statistical model about 2n + 1 grayscale values. The bright spots adjacent to each feature point are connected, and the normal direction passing through this connection line is the normal direction of each feature point. Along the normal direction, the gray values of n pixels are taken on both sides of the feature point. In order to reduce the influence of light, this paper uses the gray gradient of the two points before and after to represent,The creation of local features is shown in Fig. 4.
Creation of local features.
A vector with length 2n + 1 is constructed. That is, on both sides of the j-th feature point on the i-th training image, we can select n pixels along the normal direction of the point to differentiate the gray values of the pixels contained in the vector to obtain a local texture. Similarly, n local textures g of the j-th feature point can be obtained. After that, the average and variance of these feature points are obtained. The local features of other feature points are also obtained through the same operation. In this way, the distance between the new feature g of a feature point and the local feature can be expressed by the Mahalanobis distance. The specific process is as follows:
(1) The local gray vector g
ij
of the j-th feature point on the i-th sample in the training sample set is calculated.
The local gray vector is composed of the gray information of 2n + 1 pixels centered on the point in the normal direction of the feature point.
(2) The gray gradient vector (g
ij
) ′ is calculated:
(3)
(4) The average gray vector
The normalized gray derivative vector satisfies the Gaussian distribution, so the Mahalanobis distance between the normalized gray derivative vector
The Mahalanobis distance can calculate the probability that the candidate feature point comes from this feature point. In the process of searching for feature points, it can determine the corresponding probabilities of different points and then determine the best candidate point.
After obtaining the shape statistical model reflecting the two-dimensional shape change rule of the target object image and the local area gray model reflecting the calibration point characteristics, the target of the unknown image can be segmented. At this point, the shape model has been established. Next, we will use this model to search the eyes in the target image.
After the ASM shape model trained by the sample set is established, the target eyes can be searched, and the pose parameters can be determined. First, the average shape is affine transformed, we set the shape parameter to b = 0. Then, the statistical shape model
The above formula represents that the average shape is rotated counterclockwise by θ and scaled by s at its center, and then it is translated by t to obtain the initial model X. Among them,
(1) Calculate new location
The initial shape X is usually inconsistent with the current target shape, so the shape X must be adjusted to make it best match the real target shape, that is, to find the best position of the face calibration point. When searching, the active shape model establishes a local texture model for each calibration point and uses this as a search criterion to update the position of each key feature point. As shown in Fig. 5, the local texture model is searched for the best match in the normal direction of feature points.
Image search matching.
For the i-th feature point in the model, along the normal direction of the point, the algorithm uses the point as the center origin to select 1 (1 > m) pixels on the left and right sides of the point. Then, the algorithm solves the derivative of the gray value of the 1 pixel and normalizes the obtained result to obtain a local feature. It contains 2 (1 - m) + 1 sub-local features. Then, the distance is calculated using the above formula to obtain the new coordinates of the feature point at this moment, thereby generating a displacement. Then the distance between G
j
(d) and the local features of the feature point is:
The point corresponding to the smallest distance (Dist) is the best position of the feature point. All feature points are continued to take the above operation to obtain the calculated shape vector dX.
(2) Calculate shape and update parameters.
In the above step, we provide all the feature points with their new positions and combine their displacements into a vector.
Next, we need to use affine transformation and adjust its parameters to make the position x of the current feature point close to the corresponding new position X + dX. After the affine transformation, the parameter variable ds, dθ, dt of the affine transformation can be obtained, and the new shape parameter is s (ds + 1) , dθ + θ, dt + t. After that, the position is calculated and the parameter dX is adjusted. The specific announcement is as follows:
The formula for permission dX is as follows:
After finding dX according to the above formula, we have to find a new db to make the following formula true:
In order to make the shape change in a suitable range, we also need to limit the value range of the parameter b. Therefore, after calculating the changes in shape and related parameters, we need to update the parameters immediately:
Moreover, we need to calculate the new shape according to the formula
Schematic diagram of best point matching.
In the active shape model algorithm, if the distance between the best matching point of the feature point search and the current point is not greater than the sampling interval, the point is considered to be convergent. When the number of converged feature points reaches 90% of the total, the entire algorithm is considered to converge, and the search matching is ended.
The flow chart of human eye detector training is shown in Fig. 7. The training process is: (1) First, the human eye image as the training sample is preprocessed with size normalization; (2) Next, the MSLB-LBPH and Co-HOG features of the image are extracted and fused to obtain a high-dimensional feature vector; (3) Finally, based on high-dimensional feature vectors, a continuous AdaBoost algorithm based on a lookup table weak classifier is used to train a cascade classifier, thereby obtaining a human eye detector. The trained human eye detection can detect the human eye area in the normalized human face image.
Human eye detector training flowchart.
After the necessary decoding and transcoding, the input image we obtained will be loaded into memory and stored in the form of a gray matrix. Since it is impossible for us to determine the environment of the learner in advance, the image obtained at the beginning is likely to include many other objects besides the learner. We need to distinguish the face area of the learner we are interested from other background objects on the picture at this time. This can greatly improve the accuracy of subsequent steps such as eye point positioning and iris positioning, and can reduce the processing range of subsequent steps, eliminate interference items, and improve calculation efficiency.
Template matching refers to the process of finding a suitable target in the global image. It is assumed that the features of the pupil to be found and the pupil template are very similar, and the goal is to find the central area of the pupil through algorithm design. We search from the eye area, set the starting point in the area, and then constantly move the position of the point. After moving to the new position, the template is compared again, and the similarity between the new area and the template is recorded. Finally, we can obtain a maximum similarity value to determine the pupil area.
The continuous AdaBoost training cascade classifier is used, and each classifier uses a training process of 180,000 positive examples and 120,000 negative examples. Among them, some parameter settings of the cascade classification model are shown in Table 1 and Fig. 8.
Statistical table of weak classifiers of each layer of cascade classifier
Statistical diagram of weak classifiers of each layer of cascade classifier.
After constructing the above model process, the performance of the model is analyzed. The students in the English online teaching classroom were subjected to eyeball feature recognition to judge their learning status. Each student was recognized for a total of 40 minutes, and a total of 90 students are recognized. The obtained state is compared with the actual state to investigate whether the state is in line with reality. The results are shown in Table 2 and Fig. 9.
Statistical table of the accuracy rate of feature recognition of classroom students’ eye movements
Statistical diagram of the accuracy rate of feature recognition of classroom students’ eye movements.
It can be seen from Fig. 7 and Table 1 that the algorithm model proposed in this study has an accuracy rate of more than 85% for the recognition of the students’ eye movements, which shows that the algorithm proposed in this study has certain practical effects.
Classroom students ‘eye movement feature recognition can dynamically analyze students’ status, which is convenient for improving classroom teaching strategies in real time. In this paper, the algorithm of eye movement features in the visual system is studied in detail. In order to solve the problem of inaccurate positioning in the tracking process, this study proposes to combine the ASM algorithm and Kalman filtering and use the obtained AK-EYE model to locate the eyelid in real time. The results show that the model has obtained good results. In view of the fact that there are very few acquisition devices for obtaining eye information alone, and the price is expensive, in this study, binocular stereo vision data collection equipment designed by our laboratory is used to collect experimental data sets, and template matching was used to accurately locate pupils. The results show that the cost of this study is low, and the effect is significant. Moreover, this study designed a set of intuitive and effective spontaneous eye movement data collection methods combined with actual needs, which are proved to be in line with the expected results and subjective feelings in the subsequent system real test. The research results show that the algorithm proposed in this paper has a certain practical effect.