A brief review and challenges of object detection in optical remote sensing imagery

2.1.1 Limitations of remote sensing images

The optical remote sensing images have several drawbacks with respect to shapes, sizes, occlusions, resolutions, pixels and so on, of objects. The detectors generally utilize the region of interests (ROIs) which are not appropriate frequently. There is a need to improve the selection of required training objects. The exact number of pixels that hold the characteristics of an object should be selected for training. By considering the resolution, if a detector is trained on high-resolution images then it is difficult to get optimal accuracy on low-resolution images and it will generate more false alarms. Nowadays, the researchers are delivering the solutions for this problem in various ways such as fine-tuning, wise selection of layers of the neural net, and scaling of images. The key problems in point form are given below:

(1)

The targeted object frequently appears at diverse scales and scale changes due to different resolution of images. This limitation becomes the reason for the poor performance of object detection.

(2)

The objects in high-resolution images are comparatively very small which reduces the accuracy when CNN features are pooled by the uppermost layer at a lower resolution. After several down-scaling of the sample, the feature map becomes 1/16 of the input image which causes the loss of key information of small objects and creates false alarms.

(3)

In remote sensing imagery, the labeled data is very less and accessible annotated data is not enough for optimal performance of object detection. This consumes much time to collect new labeled data and it is very expensive as well.

(4)

The remote sensing data is generated by distinctive tools (e.g. satellite, airborne, airplane, UAV and SAR, etc.) with varying resolution. It increases the plea for emerging a distinctive object detection method which could be applicable for all kinds of remote sensing imageries.

2.1.2 Image enhancement approaches for object detection

There are several different sized objects in various datasets and this reduces the performance of different classifiers/detectors due to irregular change in size and shape. There is a dire need to improve the optical remote sensing training images for object detection using geometric transformations. The training images have a great impact on object detection performance. Several well-intentioned contributions for image enhancement have been proposed in past decades [57, 58, 59, 60] still there is a gap to make more improvements in training images. Another wise way is to compress training images to reduce the required storage space for high-resolution remote sensing images. While lossless compressed images take similar training as compared to original images but it is advantageous for storage space. On the other hand, lossy images reduce the accuracy of object detection due to a lack of information. The optimal image enhancement approaches for training images are mentioned as under:

a)

Image denoising

b)

Image filtering

c)

Edge sharpening

d)

Image rotation

e)

Image scaling

f)

Image compression etc.

The compression of the object detection model will improve detection efficiency. Because the deep network model will occupy a large amount of storage space and its application in a mobile terminal is limited, compression of the object detection model will also be an important research idea in the future. The research based on analog network aims to design special supervised and unsupervised tasks to make object detection promising. The main idea is to train smaller object detection models by using the trained deep network object detection framework. The training of the small framework becomes more efficient through the training of additional tasks. Therefore, the overall framework of object detection can be compressed on the premise of ensuring the performance of object detection. Specifically, the object detection with deep training can be achieved. In addition, unsupervised training tasks can be added to strengthen network training. Additional tasks can be done to the shallow network by setting up coders and decoders. The training of service will further improve the efficiency of network training.

2.1.3 Limitations of current object detection methods

The search ability for large-scale remote sensing images is weak. At present, most of the algorithms focus on object detection for a small size image containing the object to be interpreted, and the ability to interpret the object from a large-scale remote sensing image is relatively weak. The algorithm has a high time complexity, and cannot fully meet the actual needs of automatic interpretation of the surge data, fast detection and recognition of objects. On the other hand, due to the difference of detector angle and distance from the object, the features presented by the object change greatly, which makes the database of the object characteristic view larger and consumes a lot of storage space. At the same time, the computational complexity of related algorithms increases in the process of object search. Existing interpretation algorithms have strong pertinence. When interpreting different kinds of objects in a remote sensing image, different algorithms need to be designed to make the same remote sensing image that needs to be processed many times, this increases the processing time. Therefore, different types of object interpretation are worthy of attention and research.

Objects in remote sensing images are affected by a variety of random factors, such as the number of unknown objects, the probability of complete or partial occlusion of objects, the probability of unknown objects, etc., which makes the object information incomplete. The research work of knowledge base remote sensing is not comprehensive, and the existing algorithms do not fully integrate the important prior knowledge such as object environment, so it is necessary to further mind the characteristics of the object environment.

3.2.1 Region-based convolutional neural network (R-CNN)

In 2014, Girshick et al. [67] introduced CNN model into the object detection field and proposed the R-CNN model. In SPP-Net proposed by He et al. [41] in 2015 and Fast R-CNN proposed by Girshick [42], respectively, several problems are improved by them such as 1) For each candidate region, to extract feature expression of fixed length, it needs to be adjusted to a uniform size (i.e. resized patches as shown in Fig. 2), which may lead to loss of image information to some extent; 2) Candidate region recommendation, depth feature extraction, step by step SVM training for large-scale data so the process is usually costly. 3) The consideration of the quantity of training time and storage space; each candidate region covers a large proportion of each other which results in repeated feature extraction.

OPEN IN VIEWER

According to the intuitive ideas, object detection refers to the location of the object which consists x, y, width and height of bounding box, which covers the object, whereas, classification tells us that which object is located. In other words, classification/recognition gives us the class label for the object. The whole detection process of R-CNN is shown in Fig. 2. R-CNN algorithm can be divided into four steps: 1) Candidate region generation: input a natural image and extract about 2000 candidate regions by Selective Search method; 2) feature extraction: normalize the image of each candidate region, and input the deep convolution network CNN for convolution, pooling and other operations to extract features; 3) Classification judgment: extracted features; Linear SVM classifier is fed to judge the classification; 4) position refinement: the candidate box position is refined by boundary regression to get accurate object region.

To overcome the time-consuming problem of R-CNN feature extraction, SPP-Net [41] proposed that the whole image features should be extracted first, instead of the cumbersome convolution calculation for each candidate region, and then the feature of candidate region should be intercepted on the feature map before classification, and this method needs 0.5 seconds to process an image.

In SPP-Net, He et al. [41] extract feature expression for each candidate region by using spatial pyramid pooling method, which effectively avoids losing information in the process of adjusting the size of the candidate region, and makes the detection accuracy of SPP-Net surpass R-CNN model. Girshick proposed a ROI Pooling layer in Fast-RCNN. Using this layer, only one depth feature is extracted. Then, according to the location of each candidate region mapped to the ROI Pooling layer, the feature expression of uniform length is extracted by combining the idea of SPP-Net. In addition, because of the introduction of the ROI Pooling layer, feature extraction and classification model training are completed in the same network. Therefore, compared with R-CNN and SPP-Net, the performance of Fast-RCNN in training, testing speed and detection accuracy has been improved.

OPEN IN VIEWER

The detection flow of SPP-Net is shown in Fig. 3. Although the overall detection process remains unchanged, the following improvements have been made on the basis of R-CNN: 1) it is no longer necessary to unify the images into fixed-size by intercepting and normalizing operations in advance, so as to solve the problem of image distortion and information loss caused by normalization operations; 2) using spatial pyramid structure to replace the pooling layer of the last convolution layer.

Therefore, SPP-Net has the following advantages: 1) By defining a scalable pooling layer, SPP-Net can process images of arbitrary aspect ratio and arbitrary scale, produce fixed size output, and improve the robustness of extracted features through multi-scale; 2) Because the features of all candidate regions are extracted directly from the overall feature mapping, the repeated counting of convolution layer is effectively solved. The efficiency is improved by calculating problems. But at the same time, there are still some problems in SPP-Net: 1) the isolated training process makes it impossible to train the network parameters as a whole, and still needs to preserve a large number of intermediate results; 2) the convolution layer and the connection layer located on both sides of the spatial pyramid pooling layer still need to be adjusted separately, which limits the effect of deep convolution network; 3) the extraction of candidate areas is still very important in the whole detection process.

Fast-RCNN [42] is mainly improved by accelerating R-CNN. The detection flow of Fast-RCNN is shown in Fig. 4. In Fast-RCNN, firstly, a series of candidate regions are generated by one candidate region method; secondly, convolution neural network feature maps of input images are generated by multiple convolution layers; secondly, candidate region pooling layer maps candidate regions to convolution neural network feature maps, and extracts feature maps of the same size for candidate regions of different sizes; finally, pooling is located in candidate regions. After the layer, the final feature is extracted, and the classification score and the location of the detection box are predicted and refined based on the feature.

OPEN IN VIEWER

The improvement of Fast-RCNN can be summarized as follows: 1) Compared with R-CNN, the detection quality is improved; 2) Multi-task loss layer is introduced by combining classification and border regression loss function, and single-level training process is exploited which improves the accuracy of the algorithm; 3) end-to-end training is utilized except for candidate region extraction and all features are stored in display memory without additional disk space. 4) Referring to the idea of SPP-Net, a single fixed-scale ROI layer is proposed and the full-connection layer is speeded up by SVD decomposition.

However, Fast-RCNN still has the following shortcomings: 1) SS method is still used to extract candidate regions, and the relative feature extraction is still time-consuming; 2) real end-to-end training and testing are not achieved, which cannot meet the requirements of real-time detection; 3) GPU acceleration is used, but the candidate region extraction method is still CPU implementation.

Faster-RCNN is an advanced improvement of Fast-RCNN [68]. The region proposal network (RPN) is proposed in Faster-RCNN. It takes only 10 ms to generate regions by applying the RPN on GPU with the convolutional neural network. The region proposal generation and Fast-RCNN have been merged into a new network model to form a real-time object detection framework which enhances the detection speed and accuracy. The object detection structure of Faster-RCNN has shown in Fig. 5.

Ren et al. proposed a Regional Proposal Network (RPN) based on CNN in 2017 [43], which is known as RPN. The experiment results show that compared to the conventional region proposal algorithm such as Selective Search (SS), RPN achieves higher accuracy by recommending fewer regions (RPN: 300, Selective Search: 2000). The RPN has been combined with Fast-RCNN and trained them alternately, and acquired the unified detection model which is known as Faster-RCNN. In addition, Redmon et al. also proposed a network model without region proposals, known as YOLO (You Only Look Once) [45]. This proposed model predicts the conceivable locations of the objects in the images using regression. The model has achieved optimal real-time performance on GPU and the detection speed is about 45 FPS (frame per second). Nevertheless, Faster-RCNN has maintained its definite advantages in terms of detection accuracy. However, Faster-RCNN has some shortcomings such as it has not reached to the real-time detection. The way of getting region boxes before classification needs a huge amount of computations. This limitation has led to another advance approach, is called as Mask-RCNN [69]. The important features which make the difference between region-based CNN object detectors are given in Table 1.

OPEN IN VIEWER

In present years, many scholars have presented CNN models for vehicle detection in aerial images because CNN has excellent feature representation ability. Ammour et al. has proposed the detection model based on deep learning for vehicle detection in unmanned aerial vehicle (UAV) images [70]. This proposed model is analogous to the R-CNN framework as mentioned above. The key difference is that they use segmentation rather than the region proposal algorithm. The method of image segmentation describes two advantages: the processing speed of the segmentation is better than selective search, and the extracted regions do not overlap with each other. However, the extracted region by the segmentation method needs post-processing before using it on CNN. Qu et al. has proposed a detection method for UAV images which is based on SPP-Net and region proposal algorithm [71]. However, in these two detection methods, the CNN framework has been used for feature extraction, and there are still analogous problems with R-CNN models. The regions are very important in order to enhance accuracy; therefore, RCNN is utilized to produce accurate bounding boxes [72]. These boxes are horizontal at the first stage and rotated at the second stage which was adopted from cascade RCNN [73] and it verifies that the regression of targets by this approach can produce more precise results. The detection accuracy is also concerned with the dataset’s characteristics. More specifically, the performance of Faster-RCNN has been improved for fixed input size in terms of average precision [74]. In region-based methods, SS is prominently involved to enhance the accuracy. A systematic region proposal method was proposed in comparison with SS to reduce the processing speed with optimal accuracy [75].

3.3.1 You only look once (YOLO)

YOLO algorithm proposed by Redmon et al. in 2016 is a convolutional neural network that can predict multiple box locations and classes at once. The network design strategy of the YOLO algorithm continues the core idea of Google Net [76] which reveals end-to-end object detection in a real-time scenario. It gives a full role to the advantages of fast speed, but its essence towards accuracy has declined. However, the YOLO 9000 [77] algorithm proposed by Redmon et al. in 2016 improves its accuracy on the speed of the original YOLO algorithm. There are two main improvements: 1) a series of improvements have been made on the original YOLO detection framework to make up for the lack of detection accuracy; 2) a method of combining object detection and training is proposed. The training network of the YOLOv2 algorithm [78] can adjust dynamically under certain circumstances by the down-sampling method. This mechanism can make the network predict different size pictures and balance the speed and accuracy of detection. The YOLO framework carries out the detection likewise regression problem. It transforms an image into S × S grid then each cell of grid predicts bounding boxes B with their confidence and class probabilities can be described by C. The final tensor for predictions can be expressed as S × S × (B × 5 + C).

3.3.2 Single shot multibox detector (SSD)

In 2016, Liu et al. proposed the SSD algorithm [46], which combines YOLO’s regression idea and Faster-RCNN anchor mechanism to achieve the coexistence of speed and accuracy. The original YOLO algorithm recognizes objects by dividing image into S × S grid. When small objects are detected with this framework, the accuracy will decrease. In the SSD algorithm, the full connection layer of the YOLO algorithm is removed, so the object of any size can be detected, and the performance is basically not good. The biggest difference between training and training SSD test sets using candidate regions and standard detectors for pooling is that ground truth needs to be assigned a set of fixed sets to detect a particular output in the output. When this assignment is determined, the loss function and backward propagation can achieve end-to-end applications. In a word, SSD combines the regression idea in YOLO and anchor mechanism in Faster-RCNN and uses the multi-scale regional features in all positions of the whole graph to carry out regression. It not only keeps the fast speed of YOLO but also ensures that the region prediction is as accurate as Faster-RCNN. The shallow layers in SSD might not yield sufficient high-level features to execute predictions for small size objects. Furthermore, SSD shows optimal results for a complex environment but it requires a large amount of training data. The training process of SSD is acquired from a multi-scale MultiBox approach which exhibits that the model could deal with a variety of instances for each class [79, 80] and it is modified in order to handle multi-class object categories. The weighted sum of localization loss ( L 𝑙𝑜𝑐 ) and confidence loss ( L 𝑐𝑜𝑛𝑓 ) provides the total objective loss. It the number of matching of default boxes will be zero then the overall loss function will become zero.

OPEN IN VIEWER

Finally, based on the previous related work, we investigate that the sliding windows approach has the lowermost difficulty in execution and there is essentially a recurrence of executing image classification while it might be very slow during the process of training and testing. Though the accuracy of image classification relies on the effectiveness of the network it needs the expertise to build such a mature network. The region proposal approach has reduced the training and testing time compared with the conventional sliding window approaches and helps to enhance the detection accuracy. Indeed, Faster-RCNN attains the maximum accuracy as compared to other methods [81]. By means of testing and training time, the well-known SSD is considerably faster than the state of the art methods whereas it eliminates the region proposal approach, but it costs the reduced accuracy as compared to those techniques with region proposals.