Research on ResNet50 forage image classification algorithm integrated with SE attention

Introduction

As an important feed resource in animal husbandry, forage grass is important for the sustainable development of animal husbandry. Forage image classification is a challenging task due to the characteristics of diverse growth status. The traditional forage classification method mainly relies on artificial observation and empirical judgment, and it has problems such as strong subjectivity and low efficiency. Traditional image classification methods require manually extracted features with limited accuracy and efficiency. Therefore, it is feasible to introduce deep learning technology to solve the problem of forage image classification. With the continuous development of computer vision and machine learning, forage image classification has gradually attracted the attention of researchers. By constructing forage grass image dataset and using deep learning algorithm, intelligent forage classification can be realized. It can not only improve the accuracy and efficiency of classification but also provide real-time forage quality assessment and management suggestions for growers.

In the study of image classification, feature extraction refers to the extraction of representative features from images. Common methods include traditional feature extraction based on texture, shape, and color, as well as convolutional neural network (CNN) in deep learning. The classifier design is to make classification decisions based on textracted features. Common methods include support vector machine (SVM), random forest, and deep neural network. COVID-19 Excess Mortality Collaborators applied logistic regression, random forest, and support vector machine to classify 3616 patients with COVID-19 and 10,192 healthy people. ¹ Esraa et al. used the QCNN model and a modified ResNet (50) pre-trained model, for enhancing the biomedical image classification in the MNIST medical dataset. During the training phase, the weights are updated using the Adam optimizer, while ResNet (50) is used to reduce the computational cost. The classification accuracy achieved 99.6% accuracy, 99.7% precision, 99.6% recall, and 99.7% F1 score. This method can effectively improve the accuracy of remote sensing image scene classification, which is better than the mainstream self-supervised image classification methods SimSiam, SwAV, MoCov2, and Deepcluster, and has good application value. ² Lingxiao Wang et al. proposed an attention mechanism feature fusion deep learning model with only 14 million (M) parameters. The accuracy, sensitivity, and specificity of low-grade intraepithelial neoplasia were 94.5%, 93.0%, and 96.5%, respectively, achieving state-of-the-art classification performance. ³ Xiaohua Zhang Jiang et al. proposed an improved ResNet-50 architecture, which incorporated the coordinate attention (CA) module and weight-adaptive multi-scale feature fusion (WAMFF) to enhance the ResNet-50’s image feature extraction capabilities. Furthermore, compared with mainstream deep networks, such as AlexNet, VGG16, DenseNet, MNASNet, and GoogLeNet on the same dataset, the top-1 accuracy rate increased by 7.3%, 3.19%, 4.98%, 6.04%, and 3.87%, respectively. The experimental results demonstrate that the improved model is effective in improving the identification accuracy of apple leaf diseases and insect pests and enhancing the model’s effective feature extraction capabilities. ⁴ In view of information loss, inadequate feature extraction, network overfitting, and training difficulty in deep learning, Namrye Son et al. proposed a medium- and long-term power forecasting. Deep neural networks (DNNs) and long short-term memory (LSTM) were employed for the time series prediction. ⁵ In the model, in order to achieve effective classification performance, the last layer of ResNet 150 is replaced by long and short-term memory (LSTM) and deep neural network (DNN). To improve the performance of classification stage, a hybrid optimization algorithm called Hybrid Barnacle Mating Bird Swimming Optimization (HBM-BSO) was used to perform parameter optimization in LSTM and DNN. The analysis of experimental results shows that this method is significantly superior to the commonly used multi-plane classification models for plant leaves with different performance indicators. ⁶ Qin Haihua et al. proposed a method for diagnosing a rolling bearing, based on CBAM_ResNet and ACON activation function. At first, the collected bearing working vibration signals are processed into input samples to preserve the original features, maximizing the preservation of original features. Then the CBM-ResNet fault diagnosis model is built. Utilizing the advantages of Convolutional Neural Networks (CNNs) in classification tasks and key feature extraction, Convolutional Block Attention Module Networks (CBAMs) are embedded into residual blocks. In order to avoid model degradation, the convolutional block attention module network (CBAM) is embedded in the residual block. By enhancing the interaction between channel and spatial information, the capability to extract key features of the model can be improved. At last, in order to improve the feature representation and generalization ability of the model, shallow features are adaptive activated by an activated or unactivated (ACON) activation function. ⁷ Yuhua et al. proposed a neural network model based on residual network and self-attention mechanism based on multimodal theory. The performance of this model is significantly better than GRU (Gating Cycle Unit), ResNet (Residual Network), CNN (Convolutional Neural Network), and LSTM (Long and Short-term Memory). The accuracy of the model is 94.38%, the precision is 94.92%, the recall is 98.94%, and the F1 scores is 96.89%. ⁸ It can be seen that the model is indeed effective in studying intelligent ventilation decision-making.

Forage grass image classification technology still faces challenges, including image quality, changes in lighting conditions, and similarity between categories. In the future, research can be conducted from multiple aspects, such as improving image preprocessing algorithms, optimizing feature extraction methods, and designing more effective classifier models, in order to improve the accuracy and robustness of forage grass image classification. In this paper, ResNet 50 model is used to construct image recognition model for 13 forages. Images in RGB format are taken with random angles of mobile phone. Accelerate training use GPU in Linux environment. Based on the deep neural network, the image recognition and classification model is built. Parameter fine-tuning and freezing weight parameters are used to realize automatic recognition and classification. It can improve the speed and accuracy of forage grass image recognition and meet practical needs in the future.

The ResNet50 in deep learning in materials and methods

Residual neural network

Neural Network is a computational model that simulates the neural system of the human brain to solve various machine learning and deep learning tasks. The system is composed of a large number of neurons, which transmit and process information through connections between them. The neural network consists of multiple layers, each consisting of multiple neurons. The input layer of the neural network receives the external input data, which is calculated and processed through multiple intermediate layers, and the output result is in the output layer. The core component of the neural network is the neurons. Each neuron receives a set of inputs, calculated by weighted and activation functions to transfer the results to the neurons in the next layer. During the training process, by learning and adjusting the weights of connections between neurons, the network can adapt to specific tasks and data.

Traditional deep neural networks face the problem of gradient disappearance and gradient explosion, which makes the model difficult to train. In the deep network, the input information is gradually lost or becomes unstable, which makes it difficult to optimize. To solve this problem, ResNet takes the idea of residual learning. Residual network is a deep learning model that introduces residual connectivity to solve the training difficulties of deep neural networks. By introducing the skip connection in the network, the information can directly skip some levels for transmission, alleviating the problem of information loss and gradient disappearance.

The core idea of residual learning is not directly learning nonlinear transformation. By using the residual blocks, the input features are added directly to the output to form a skipped connection. Networks can learn identity maps without the need for learning transformations. They can simply transfer inputs to outputs while preserving important features.^9–11

Residual connection allows information to skip several layers in the network, avoiding information loss and gradient disappearance problems. It enables the network to stack deeper, learn more complex and deeper feature representation, and improve the model performance and accuracy. The residual principle is shown in Figure 1, and the core idea can be summarized as formula (1).

F ( x ) = H ( x ) − x

(1)

OPEN IN VIEWER

Figure 1.

Residual schematic.

Residual network (ResNet) is the key residual unit in the structure. The residual unit contains cross-layer connections, which transmit input information across layers through curves. Then it adds up to the result of the convolution operation. Assuming that the input image is x, the output after convolution operation is F(x), and the final output is H(x) = F(x) + x. The residual is F(x), and the objective of the network is shown in formula (1).

In ResNet, the input image x undergoes two residual block convolution operations before being fused with x and is output after being activated by the ReLU activation function. Before merging x and F(x), path fusion is achieved through shortcut connection. To ensure the fusion of feature maps, the size of the feature map output through residual block convolution is the same as the size of the input x. This design offers two benefits. Residual units allow for direct transfer of information in the network, helping to capture finer feature changes. The residual function is easier to optimize, so that the network can be deeper and improve the network performance and convergence speed.

In conclusion, the residual network introduces cross-layer connections and path fusion through the residual units, so that the network can be deeper and easier to optimize. This structural design improves the training problem of the deep network and achieves good results.

Residual neural network construction

The SE attention mechanism module

The attention mechanism used in convolutional neural networks can enhance the model’s focus on important input features The SE attention mechanism is realized through two stages: squeeze and excitation. In the “squeeze” stage, the global information of channel features is obtained by global average pooling, and in the “excitation” stage, the neural network layer is used to learn and rearrange the weights of these channel features and enhance the response of useful features.^12–14 The SE attention schematic is shown in Figure 2. The figure shows the insertion of attention mechanism module in the network structure. In the residual block, global average pooling is performed to compress the two-dimensional features of each channel to 1, and the feature graph [h, w, c] = => [1,1, c]. Weight values are generated for each channel through excitation, and the correlation between channels is constructed through the fully connected layer, so that the number of weight values is consistent with the number of channels in the input feature map. The function of scale is to weight the previously normalized weights onto the channel features.

OPEN IN VIEWER

Figure 2.

SE attention schematic.

CBAM attention mechanism

CBAM is an attention mechanism based on spatial and channel attention. The CBAM attention mechanism adds a CBATM module to each convolutional block of the convolutional neural network for learning the characteristic channel and spatial attention. The CBATM module includes two sub-modules. They are, respectively, referred to as channel attention module and spatial attention module. The channel attention module generates channel attention vectors by learning the relationship between the channels. The spatial attention module does so by learning the relationship between each spatial position. The output of these two sub-modules is multiplied to obtain the final attention vector.

Depth-separable convolution

Deep separable convolution is a basic operation in the convolutional neural network, which can reduce the number of network parameters and computation and improve the operation speed and efficiency of the model. Traditional convolution operations include two parts: kernel weights and bias terms. It needs to perform convolution calculations on each channel of the input feature map and produce the same number of channel output as the convolution kernel.

In deep separable convolution, the convolution operation is divided into two parts: deep convolution and point-by-point convolution. The deep convolution only performs convolutions for each channel of the input feature map and produce channel outputs equal to the number of convolution kernels. In point convolution, each channel output is combined by 1x1 convolution check to generate the final output feature map.

Experimental results and analysis of the 2-ResNet50 forage recognition model

Data source

In this paper, the dataset is collected at random angle. Experimental platform is Linux server. GPU RTX is 3090. Video memory is 24G. The framework of deep learning is adopted, namely, PyTorch 1.11.0. The collection location is the Shalqin Base of the Chinese Academy of Agricultural Sciences, and the collection time is mid-July. Shooting phone is Mi 10S, aperture is 1.69, and ISO is 50. The focal length of the camera is 6.72 mm, with an equivalent focal length of 24 mm and a shutter speed of 1/1448 seconds. During shooting, no flash was used and ultra-wide angle was not turned on. Use the default parameters of the Mi phone camera to take photos, from around 9 am to 12 am, full of light. The photo size is 5792 × 4344 px, and the average photo size is 15 MB. Randomly collect a large number of grass images from three different perspectives to form a sample dataset, with the image type being RGB. The dataset contains 13 different types of grass, including ice grass, licorice, Suaeda salsa, reed, horse Tang, nettle, alfalfa, smooth bromegrass, sheep grass, oat, needle grass, and pighair grass. Each contains 145 pictures; some of the forage grass pictures are shown in Figure 3.

OPEN IN VIEWER

Figure 3.

Images of the forage dataset.

Residual neural network construction

The batchsize was fixed to 20, and the epoch was fixed to 30. The partition ratio between the training set and the test set in the experiment was 8:2. The training set contained 1480 images, and the validation set contained 356 images. ResNet50 is used to complete training. During the training process, some parameters of the fully connected layer were modified to achieve classification.

Dynamic adjustment of the learning rate

The adaptive method Reduce LR On Plateau determines the adjustment strategy to reduce the learning rate according to the indicators on the validation set and controls the learning rate by setting some parameters.

The fusion of attention mechanisms and depthwise separable convolution module

This experiment adopts SE attention mechanisms and CBAM attention mechanism. The CBAM module is added in the last stage. The CBAM attention mechanism is added between the third convolutional layer of the fourth residual block and the fourth convolutional layer of the fourth residual block, as shown in Figure 4. The accuracy of the improved network has increased to 96.09%, while the accuracy of the original network is 93.8%. The total training and validation time is 2420.81 seconds.

OPEN IN VIEWER

Figure 4.

Fusion of attention mechanisms and depthwise separable convolution module.

In ResNet50, the SE attention mechanism module is applied after the second convolutional layer and a batch normalization layer in each residue block structure. The SE module is added after the second convolutional layer of BasicBlock. In the forward method of BasicBlock, after passing through the first convolutional layer, BatchNorm layer, and ReLU activation function, the output passes through the second convolutional layer and BatchNorm layer. Then, the output is processed through SE Layer. Applying the SE attention module, the accuracy has increased from 93.8% to 98.04%. The total training and validation time is 2536.76 seconds. As shown in Figure 5, the left is the original residual network module, and the right is the addition of the SE attention mechanism module.

OPEN IN VIEWER

Figure 5.

Apply SE attention mechanism module in ResNet50. (a) The original network residual module. (b) The SE attention mechanism added.

Using the original SE module, the depthwise separable convolutional module is added in the first convolutional layer, as shown in Figure 6. The accuracy is 99.16%, and a total training and validation time is 2292.38 seconds.

OPEN IN VIEWER

Figure 6.

Apply SE attention mechanism module and add the depth-separable convolution in ResNet50. (a) The original network residual module. (b) Depthwise separable convolutions added.

As shown in Figure 7, the depthwise separable convolutions are added in both the first and second convolutional layer, simultaneously. The accuracy is 87.71%, and the training and validation time is 2475.70 seconds.

OPEN IN VIEWER

Figure 7.

Apply SE attention mechanism module and add two depthwise separable convolutions in ResNet50. (a) The original network residual module. (b) Adaptive average pooling layer with new activation tensor.

The model optimizes the SE attention mechanism by replacing the global average pooling layer with an adaptive average pooling layer. Meanwhile, a new activation tensor is created at the ReLU activation function instead of the original input tensor, which can improve the stability of the neural network, as shown in Figure 8. The accuracy is 96.09%, and the training and validation time is 285.28 seconds.

OPEN IN VIEWER

Figure 8.

Apply adaptive average pooling layer with new activation tensor in ResNet50.

In summary, using the optimized SE module and adding a depthwise separable convolution module to self.conv1, the accuracy can be improved to 99.44%, and a total training and validation time is 2333.30 seconds. This is the best solution. Using different modules in the optimization process of SE model, comparison of experimental results is shown in Table 1.

OPEN IN VIEWER

Table 1.

Comparison of results of different modules in the SE model optimization process.

Join the module	F1 (%)	Recall (%)	Time/s	Loss	Precision (%)
CABM	95.70	96.13	2420.81 s	0.1869	96.09
SE	98.05	98.08	2539.76 s	0.1310	98.04
SE + Single-layer depth separable convolution	99.20	99.27	2292.38 s	0.0997	99.16
SE + Double-layer depth separable convolution	86.66	86.13	2475.70	0.7016	87.71
Optimized SE	95.70	96.13	2603.84 s	0.1869	96.09
Optimized SE + Double-layer depth separable convolution	86.66	86.13	2475.70 s	0.7016	87.71
Optimized SE + Single-layer depth separable convolution	99.52	99.54	2333.30 s	0.0435	99.44

Experimental results and discussion

To verify the performance of the proposed network, it is compared with three models: ResNet18, ResNet34, and Efficient Net. The experimental results are shown in Table 2.

OPEN IN VIEWER

Table 2.

Accuracy, recall, and F1 values for the four models.

Model	Precision (%)	Recall (%)	F1 (%)
ResNet18	89.30	88.40	87.00
ResNet34	91.60	90.40	89.60
EficientNet	96.30	96.80	95.10
This article model	99.44	99.54	99.52

The experimental results show that the accuracy can achieve 99.44% and recall is 99.54%, using the proposed scheme to train the ResNet50 deep neural network. It is significantly higher than the other three networks. The method described in this article has good recognition performance and robustness in the field of forage grass images classification and recognition. It can meet practical application needs.

Conclusion

This article focuses on the collection, identification, and classification of 13 types of forage, including alfalfa, smooth bromegrass, ice grass, and so on. The classification model adopts a deep learning network based on ResNet50. By conducting comparative experiments on the dataset, it is found that the best accuracy is achieved when the epoch is 30 and the batchsize is 20. The experimental results indicate that it is feasible to apply the ResNet50 network with SE attention mechanism and depthwise separable convolution to this dataset for classification inference. The accuracy of this model can reach 99.06%, which has strong feature extraction and classification capabilities. It can accurately classify and recognize forage grass. This study has certain guiding significance for the classification and identification of forage in the planting bases of the Academy of Agricultural Sciences and provides reference and foundation for further research.

Statements and declarations